Codebooks and Decision Models

Content analysis and the codebook

Researchers have coded text by hand for most of a century. Communication scholars counted themes in wartime propaganda and newspapers in the 1940s, and the method they developed, called content analysis, is now standard in political science, sociology, public health and environmental policy (Krippendorff 2019; Neuendorf 2017). Qualitative researchers use a closely related practice to code interviews and field notes (Saldaña 2025).

The central document in content analysis is the codebook. A codebook lists each question, or code, that a reader will apply to every unit of text, such as a paragraph, a speech or a news article. A good codebook entry has a short name, a precise definition, the allowed values, and examples of text that does and does not qualify. The codes come in a few types that match the levels of measurement in statistics: yes/no codes, categories with no order, and ordered scales.

Large research programs publish their codebooks so that others can apply them the same way. The Comparative Agendas Project codes bills, speeches and news stories in many countries into about 20 major policy topics, and the Manifesto Project codes every sentence of party election manifestos into 56 policy categories. Both codebooks run to dozens of pages, and both have been revised many times as coders found cases the definitions did not cover.

Because a codebook is applied by people, researchers measure whether different people apply it the same way. Two or more coders label the same sample independently, and the researcher reports how often they agree. Raw percentage agreement overstates reliability when most units are easy, because coders agree on the obvious cases by chance alone. Cohen's kappa (Cohen 1960) and Krippendorff's alpha (Hayes and Krippendorff 2007) correct for this. When coders disagree, the team discusses the cases, rewrites the definitions, and codes again. A codebook is finished only after several rounds of this.

Hand coding is slow and expensive. A trained coder might read a few hundred paragraphs a day, so most studies coded a sample instead of the whole corpus. Our 37 reports contain about 12,000 paragraphs, which would take one person several weeks to code once.

Supervised text classification

From the 2000s, researchers began to train statistical classifiers to extend hand coding to larger corpora. People code a training sample with the codebook, a model learns to reproduce their labels, and the model labels the rest. Grimmer and Stewart (2013) review these methods for political text, and their main advice still holds: automated methods amplify human judgment and do not replace it, so every application has to be validated against careful reading.

Climate research has used this approach directly. ClimateBERT, a language model trained on climate text, has been used to classify corporate climate risk disclosures (Bingler et al. 2022) and to detect environmental claims in company reports (Stammbach et al. 2023). The main cost of supervised classification is the labelled training data, often thousands of examples for each code. Laurer et al. (2023) show that models pre-trained on general reasoning tasks can reach good accuracy with far fewer labels.

Coding with a language model

General language models such as GPT, Claude, Gemini and DeepSeek can apply a codebook with no training examples at all. The researcher writes the codebook into a prompt, adds one paragraph, and asks the model to reply with a label, usually in a fixed format such as JSON. Code then reads the label out of the reply. Gilardi et al. (2023) found that ChatGPT matched or beat crowd workers on several annotation tasks at a small fraction of the cost, and Ziems et al. (2024) tested many models on a broad set of social science coding tasks. These papers started a rapid shift toward model-assisted coding.

The same papers, and the guidance that followed, describe the problems a researcher has to manage (Törnberg 2024; Chae and Davidson 2025).

• A language model produces text, so its reply can be malformed, wordy, or a label that is not in the codebook. Structured output modes reduce this but do not always enforce it.
• Answers change with small changes in prompt wording and from one run to the next.
• The model does not report how sure it is. Asking it for a confidence number gives another piece of generated text, which is not a calibrated probability.
• Models follow long, detailed codebooks unevenly. Halterman and Keith (2025) found that models given a codebook with no further training often did not follow its definitions.
• Commercial models change or disappear, which makes results hard to reproduce. Palmer et al. (2023) argue that research using proprietary models should explain why an open model would not do.

Language models are at their best on tasks that need generated text, such as filling in a detailed record for each goal a company has set, with a quote as evidence. We use one for exactly this in Session 10.

Decision models

A decision model is built for the narrow question at the heart of coding. Jev, released by TypeSafe AI in September 2026, is the first model of this kind to be widely available. TypeSafe calls it a System One model, after Daniel Kahneman's distinction between fast, intuitive judgment and slow, deliberate reasoning. Jev never generates text. The caller sends a piece of text (the state) and a set of typed questions, and Jev returns a probability for every possible answer.

Jev's three question types correspond to the three types of code in a classic codebook.

A choice question takes a description of each label, and a score question takes a description of each level. These descriptions are the codebook's definitions, so writing a Jev request is close to writing a codebook entry. Because the answer is always one of the allowed values, there is nothing to parse and no invalid label to handle.

The probabilities are the main difference from a language model. TypeSafe says it trains Jev so that its probabilities are calibrated: of all the answers given probability 0.8, about 80% should be right. Calibration is a property of many answers together and does not guarantee that any single answer is right. The probabilities let a researcher choose a cut-off, add probabilities instead of counting labels, or send the uncertain cases to a person or a larger model. TypeSafe's documentation recommends setting cut-offs by testing on your own data, and different models need different cut-offs for the same question.

Decision models are also fast and cheap. Jev charges about 4 cents per million tokens of input and nothing for output, and an answer usually comes back in under a second. Coding all 12,000 paragraphs of our corpus with one question costs about 20 cents.

Most early uses of Jev are inside software rather than research. Companies use it to route customer messages, screen content for safety, sort email, and decide when an automated step should stop and ask a person (InfoQ 2026). An independent test on three public benchmarks found Jev less accurate than the largest language models at sorting text into 77 categories, but competitive at content moderation and far cheaper. Its confidence was a good guide on the classification tasks: answers given 0.9 or more were right 94–96% of the time (AI/ML API 2026). That test was run by a company that resells these models, and on 300 examples per task, so treat its numbers as a rough guide.

Decision models have limits that matter for research.

• Jev gives no reasons for its answers. To understand a code you have to read the paragraph yourself.
• TypeSafe reports that Jev is unreliable at counting, arithmetic and comparing dates, and recommends doing these in code.
• Jev is closed and hosted only by TypeSafe, and the company has not published a paper, training data or model size.
• Answers vary slightly between runs. In our tests, two runs of the same yes/no question over the whole corpus selected 2,337 and 2,330 paragraphs.
• Different decision models can disagree a great deal. When we tested Laya, an open-weights decision model that accepts the same requests, its scale answers clustered in the middle of every scale and it missed several quiet retreats from goals that Jev caught.

Comparison of methods

Codebooks in Module 3

Module 3 follows the content analysis workflow, with models doing the repetitive part. You write your codebook in plain English before your coding agent turns it into code. Jev applies it to every paragraph, which is the step that used to take weeks. You then label a sample of paragraphs yourself, before looking at Jev's answers, and compare the two. This is the reliability check from hand coding, with Jev as the second coder. The same caution applies: most paragraphs are easy "no" cases, so overall agreement looks high, and the paragraphs that you or Jev coded "yes" are the ones to study.

When you and Jev disagree, you do what a coding team does. You read the paragraph again, decide who was right, and rewrite the code if its wording was the problem. Keeping each version of the codebook, with a note on what changed and why, is part of the assignment because it is part of the method.

In Session 10 we use Jev to choose the paragraphs that mention goals and a language model to extract a record of each goal from them. This filter-then-extract design uses each kind of model for the task it does best. We also check every quote the language model returns against the source text, and compare our results with two independent assessments.