Fact-checking and quality gates

Before a batch of Insporta questions is published it passes a fixed sequence of gates, cheapest first: an author’s self-check, mechanical checks on structure, length signal, distractor tells and duplicates, a reading pass by a reviewer with the neighbouring levels in front of them, and a fact check by a person against the primary source that verifies the key and all three distractors of every question. Facts confirmed along the way are recorded in a per-sport register that the next author reads first.

The author’s self-check

The author checks their own batch against the rules before anyone else sees it:

• every test holds exactly ten questions, every question four options;
• no question refers to a source document;
• stems inside the length corridor, and key length not signalling the answer across the batch;
• every distractor checked as false, not merely as not-the-key;
• no topic tag repeated;
• no two questions in the batch teaching the same fact;
• nothing from the “already covered” list re-asked;
• contested claims marked as contested, and safety answers naming the channel.

The list mirrors the rules described in How Insporta writes a test. It is necessary and it is not sufficient, which is why the gates follow.

The mechanical gates

Structure. Exactly ten questions per test, four options, one key, every required field present, and stems inside the corridor described in Reading load and length.

Length signal. Across the batch, the key must not be the longest option, or the shortest, much more often than the quarter that chance gives. This is a distribution over the batch, not a pass or fail on each question, because a per-question fix moves the tell rather than removing it.

Distractor tells. Invented rules and restrictions, and absolutes that the key does not carry, are counted across the batch, see How distractors are built. Both are mechanical, and both survived an author reading every line.

Tag uniqueness. Every topic tag must be unique across the whole sport. This was added as hygiene, and it found what nothing else had: run over triathlon’s 1500 questions it surfaced two genuine duplicates in level 1 that the overlap check had missed and no reader had hunted for. A dedicated audit then found 24 repeats inside level 1, against 14 in level 2 and 10 in level 3, the count falling as each batch was written under more of the rules. Cheap invariants catch things they were not designed to catch.

Word-overlap duplicates. Every question is compared against the batch and against everything already published. It works at roughly 65 percent shared words and misses the same question asked in different words; lowering the threshold starts matching legitimately distinct questions instead.

These gates check form, not truth. A clean structural report means the batch is well formed.

The reading pass

A reviewer reads the batch with the neighbouring levels open, looking for what the overlap check cannot see: two questions that teach the same fact in different words, a test built on a perspective rather than a subject, a mechanism asked from the side another question already covered. The limits are known. An author’s own re-reading found five of fourteen duplicates in one batch, all inside a single sub-level and none of the cross-sub-level pairs; “I read it all” is not a gate. The dedicated pass, with both levels in front of the reader, is. A reviewer can also correct the author on the subject itself, and in the first football pilot did, twice.

The fact check

A person checks every question against the primary source: the Laws, the playing conditions, the competition rules, the technical report or the coaching reference cited in the question’s source field. The check verifies all four options. Most findings are not wrong keys but wrong options that turn out to be true: in the first triathlon batch, eleven problems in a hundred questions, most of them true distractors. A true distractor is also the fault most likely to show up later, when a question’s statistics are examined to see whether it separates learners who know the sport from those who do not, see How to validate a sports knowledge test.

The fact check reads in both directions, for claims stated more firmly than the evidence allows and for claims hedged that the evidence supports, as described in Evidence, claims and safety. Any repair pass that rewrites questions is re-checked against what it wrote, not what it meant: two of the fourteen duplicates in one triathlon batch were introduced by the previous batch’s own fix.

The verified-facts register

A fact that a reviewer confirmed or corrected has to outlive the batch, or it will come back. A triathlon rule corrected in level 1, that athletes may share nutrition and only assistance from non-participants is barred, returned as an error in level 3 because the new author had no way to know. Each sport now keeps a register of every position checked against a primary source, with the corrected errors, the genuinely contested claims and the things that should not be over-hedged. Authors read it before writing, reviewers check content against it, and anyone who contradicts an entry needs a primary source, not an opinion.

The registers are public: football, cricket, futsal, triathlon.

What the gates found the first time

The first pilot batch of one hundred football questions was run through all of these gates, and independent reviewers found something in 69 questions: the key longest in 93, an invented rule in 36, an absolute the key never carries in 31, eight duplicate pairs, two whole tests built on the wrong idea, and two factual blockers. Every finding became a rule or a check. The gates are not there because authors are careless; they are there because length, tells and repeats are invisible when you read for meaning.