Intercoder reliability in content analysis
Content analysis is an important method in both the social sciences and the humanities. In many content analyses, scholars include that variables that are of a subjective nature (e.g., codings of moral behavior in media content, interpretative language variables like metaphor or irony, coherence relations etc.). For such interpretative variables, it is sometimes very difficult to achieve satisfactory levels of reliability as expressed through e.g., Cohen’s Kappa or Krippendorff’s Alpha. What are the consequences of studying such subjective variables in content analysis?
In this symposium, which will take place on Friday July 3rd, 2015, speakers from different methodological backgrounds (social sciences, humanities) will reflect on intercoder reliability in their content-analytic work. Questions to be addressed include: How should intercoder reliability be approached? Should every analysis always achieve sufficient levels of reliability, or can exceptions be found? What are some of the do’s and don’ts when making reliable coding sheets for content analyses? Does the choice of reliability statistic (e.g., Cohen’s Kappa, Krippendorff’s Alpha, etc.) impact the analysis? What is the relation between reliability and validity?
The symposium is open for a maximum of 30 participants. For questions and registration, please contact Dr. Christian Burgers at c.f.burgers@vu.nl.
Program
| Time | Activity | Speaker |
| 9.00 – 9.30 | Walk-in with coffee | |
| 9.30 – 9.45 | Opening | Christian Burgers (VU) |
| 9.45 – 10.15 | Taming the messiness of our data: On intercoder reliability in discourse research | Wilbert Spooren and Renske van Enschot (Radboud University Nijmegen) |
| 10.15 – 10.45 | The one-coder reliability | Christine Liebrecht (University of Amsterdam) |
| 10.45 – 11.15 | MIP, VIP and HIP: Modeling and measuring figuration in a 3D framework for language use | Gerard J. Steen (University of Amsterdam) |
| 11.15-11.30 | Coffee break | |
| 11.30-12.00 | Non-trivial agreement: Rethinking corrections of reliability coefficients for agreement by chance | Jan Kleinnijenhuis (VU University Amsterdam) |
| 12.00-12.30 | The (im)possibilities of machine learning as replacement of human annotation | Florian Kunneman and Antal van den Bosch (Radboud University Nijmegen) |
| 12.30-13.15 | Lunch | |
| 13.15-13.45 | Was this shooting justified? The struggle with interpretative categories when coding video games | Tilo Hartmann (VU University Amsterdam) |
| 13.45-14.15 | Constructing taxonomies for classifying respondents’ answers: An example from experimental research on the perceived reasonableness of argumentation | Roosmaryn Pilgram (University of Amsterdam) |
| 14.15-14.45 | Unpacking CA intercoder reliability: Manifest vs. latent content | Lisa Sparks (Chapman University / University of California, Irvine) |
| 14.45-15.00 | Coffee break | |
| 15.30-17.00 | General discussion | |
| 17.00-… | Borrel at the Basket |
The symposium will be held in Room Z-009 of the Metropolitan Building (Buitenveldertselaan 3, Amsterdam).