Calibration

Beta.

A raw probability depends on the content and on how the question and its answers are worded. spinf returns extra numbers with every answer so the scores can be made comparable.

The numbers

• p: the probability of an answer among the answers you listed (they sum to 1), read from the model's next-word distribution at the end of your template.
• logprob: the model's log probability of the answer itself, before that normalisation: the exact string (its tokens' log probabilities summed), with its case variants (rise, Rise, RISE) added when case_insensitive is on.
• floor_p and floor_residual: the same, scored with empty content. This is the question's built-in bias. The lift p - floor_p is what the content itself says. The floor of a question is computed once per call, however many contents share it.
• residual (per question and combination): 1 − Σ P(answers), the probability that went to anything other than your answers. See below.

Reading the residual

The residual is mostly set by the template and the words, not by the content:

• After a template like "…\nAnswer:{?}" the model also expects many other words, so even a good one-word question can have a residual of 0.8–0.9. In a live call, "Will oil prices rise or fall next month?" on a news headline gave p(" rise") = 0.64 against a floor of 0.56, with a residual of 0.92.
• For answers of several tokens (" energy sector"), the probability of the exact string is small, and the residual is close to 1 however well the answer fits. p still separates the answers: 0.98 for " energy sector" on an Exxon headline.

So judge a question on p and the lift, not on the residual's size. The residual is useful within one question: a content whose residual is much higher than usual for that question is one the question does not fit well, and deserves less weight. spinf adds a low_option_mass warning when a question's one-word answers take less than 10% of the probability; take it as a hint to check the words and the template.

Choosing a template

The template is part of the question. Rewording it changes what the model is asked, and so p and the lift, not only the residual. Naming the answers in the template ("… Answer rise or fall.") lowers the residual, but in our tests it also moved some lifts from positive to negative. Before relying on a template:

• score a few contents you know the answer to, with two or three wordings, and keep the one that separates them best;
• then keep it fixed: scores are comparable over time only under the same template, answers and system_fingerprint.

Calibrating against a baseline

A cell is one score: one question (a template with one combination of its placeholders) on one piece of content. A raw cell carries the bias of its wording, so calibrate it before you compare or aggregate it:

1. Score a baseline set of documents of the same kind: for example a random sample of articles from the same sources and period.
2. For each question and answer, keep the distribution of its lift over the baseline.
3. Express each new score as a percentile or z-score of that distribution. This gives an absolute, comparable probability that is stable across questions, sources and time.

Macro cells: many cells, one better-calibrated read

Don't rely on one cell for something you want to measure. Ask it through several cells, and combine them:

• Several wordings of the same question (3 to 5), each calibrated on its own.
• Related questions that point the same way, such as "Does the article report a supply cut?" and "Is supply getting tighter?" (flip the sign of those that point the other way).
• Related placeholder values, such as {commodity} = crude oil, Brent, WTI.

Average their calibrated scores into a macro cell. The wording bias and the noise of each cell partly cancel, so the macro cell is better calibrated and more stable than any single cell. The extra cells cost little: the content is read once per call, and each question adds only its own tokens.

Then aggregate the macro cells over many documents (for example per day, per company or per segment), and standardise each aggregate on its own trailing window.

For decisions on single items, calibrate on labelled items instead: choose the threshold on p, or on a macro cell's score, that gives the precision and recall you need.

The floor depends only on the question, so for a fixed set of questions you can keep the floors of a first call and send "empty_floor": false afterwards. Floors from different calls typically agree to within a few hundredths; recompute them when the system_fingerprint changes.

The oil index use case shows this at scale: thousands of questions asked of 15M articles, aggregated into a daily read that correlates with the crude price level at 0.8 or more in three of the last four years.