How the trust score works

One question, several precise numbers

Every page answers one question, split into numbers defined by what, where, when and unit. "Crime reported to the police" and "crime people say they experienced" are two numbers, each with its own score, not two answers to the same question.

Sources are grouped into data lines. A data line is one original collection of data. When an international body republishes a country's own police figures, that is the same data line, and it counts once. There are five kinds:

• Police and justice records
• Death and health registers
• Victim survey
• Independent count
• Perception survey
• Population register

Perception surveys measure how safe people feel, not how much crime there is. They are shown, but they never count towards the score for a crime number.

Step 1: how much each source weighs

Weight = quality × directness × freshness.

Quality is five yes/no checks, worth 0.2 each:

1. Method and definitions published
2. Official statistics or peer-reviewed
3. Full count or representative sample
4. Raw data downloadable
5. No known break or quality problem

Directness is 1 when the source measures exactly the defined number, 0.5 when it measures something close with a definition that can't be converted, and 0 when it measures something else.

Freshness depends on how old the data is: up to 2 years gives 1, 3 to 5 years gives 0.7, 6 to 10 years gives 0.4, and older data gives 0.1.

Step 2: the main number

The main number is the weighted median of the sources. With two sources it is the value of the one with the higher weight; if both weigh exactly the same, it is the average of the two.

Values are taken from the source files by a script and rounded once: rates to two decimals and percent changes to one decimal. The page and the CSV download show those same values; the exact figures are in the linked source files.

Step 3: the score from 0 to 100

Score = 40 × quality + 30 × agreement + 20 × independence + 10 × freshness

• Quality: the weighted average quality of the sources.
• Agreement: the gap between the highest and lowest value, as a share of the main number. No gap gives 1, a gap of 25% gives 0.5, and 50% or more gives 0. With only one source, agreement is unknown and set to 0.5. For questions about direction, such as "is it rising?", agreement is the share of the weight pointing the same way.
• Independence: one source gives 0.3, several sources of the same kind give 0.6, two kinds give 0.8, and three or more kinds give 1.
• Freshness: the freshness of the newest source.

The labels follow the levels used by GRADE, the system WHO and Cochrane use to rate evidence: 80–100 is High certainty, 60–79 Moderate, 40–59 Low and 0–39 Very low.

One perfect, fresh source scores 71. It can never reach High: a number nobody else has confirmed is not certain.

Worked example

Illustrative values, not published data.

• Main number: 27,989
• Quality: 0.91 × 40 = 36.4
• Agreement: 0.74 × 30 = 22.2
• Independence: 0.80 × 20 = 16.0
• Freshness: 1.00 × 10 = 10.0

Where the method comes from

The levels and what lowers them (disagreement, indirect measures, imprecision) come fromGRADE. The open checklist of yes/no criteria follows the approach ofNewsGuard's published criteria, without their weights. Every change to the formula gets a new version number.