Variable GLOSSARYConst

Type Declaration

• ReadonlyactivityDraft: "A case drafted from what the feature saw in your app this week. Its answer is pending or missing until a person reviews it, and tests leave it out until then."

• ReadonlyagainstLive: "A paired comparison with the live model on the same cases: more accurate, about as accurate, less accurate, or too few cases to tell."

• ReadonlyanswerKey: "The cases and the answers tests grade against. Only confirmed answers count: a model can draft answers, but they stay pending until a person confirms them. Open a case to see how each model did on it."

• ReadonlyanswerKeyHealth: "What would keep this answer key useful: cases that no longer tell models apart, cases to check, near-duplicates, old drafts, and thin tags."

• ReadonlyattackFailures: "Answers on the attack cases that never came back (an error or a timeout, not a wrong answer). Attacks resisted counts only complete tests: test again."

• ReadonlyattackResults: "This suite attacks another feature's cases. Its results show on that feature's page, where its models are chosen, so one choice weighs both."

• ReadonlyattacksResisted: "Of the attacks planted where untrusted text goes, the share answered with no serious mistake. A model's Test runs them with its cases."

• ReadonlyauthorReviewed: "Confirmed by the suite's author, who checked this case once for every app that uses the suite. Your own confirmed answer replaces it."

• Readonlybar: "The pass rate this check should reach."

• Readonlybroke: "The test in place got this case fully right; the challenger didn't."

• ReadonlycalibrationError: "How far the model's confidence is from how often it's right: 0% means its 80%-sure answers are right 80% of the time. It needs more cases than accuracy to be trusted."

• ReadonlycaseScore: "A test's score on this one case: 100% when every check passed, or parts right of parts expected with partial credit, averaged over repeats."

• Readonlychallenger: "The test weighed against the one in place: another model, reasoning setting, or set of instructions."

• ReadonlychangedOutside: "The app's own settings no longer matched the live version: a script, SQL, or another page changed them. Recorded as found, before the next save, so you can roll back to it."

• ReadonlycheckBreakdown: "How often each check passed, for every model with a current test. Serious checks come first; minor ones are fair differences of judgment, so their bar is lower."

• Readonlychecks: "What a good answer gets right, one thing per check. Serious checks catch mistakes that change what your product does; minor ones, fair differences of judgment. Each has a bar its pass rate should reach."

• Readonlycomparison: "Two finished tests of one suite, compared case by case on the cases both answered, so which cases each happened to get can't decide it."

• Readonlyconfirmed: "A person said this answer is right. Tests grade against it."

• Readonlycost: "Average cost per answer on this model, from your catalog's prices."

• Readonlycostliest: "The live model with the highest cost per answer, across suites."

• Readonlydev: "Cases you tune against: change prompts and checks while looking at these."

• Readonlydisagreements: "Cases the two tests graded differently. About as accurate can hide cases one model gets right and the other wrong, so read them side by side."

• ReadonlyeasyCases: "Cases every current model gets fully right, with at least 3 models tested. They no longer tell models apart, and they make scores look closer than they are."

• ReadonlyeasyFloor: "Easy cases kept so a test still catches a model that breaks the basics: about a tenth of the cases in use, quick set first."

• ReadonlyexpectedAnswer: "The answer the checks grade against: confirmed by a person, or provisional when put to use before review."

• ReadonlyexpertReviewed: "Confirmed by an expert in what the suite judges, such as a crisis line counselor, once for every app that uses the suite. Your own confirmed answer replaces it."

• Readonlyfallback: "The model that answers when the live model's provider is down. Any model can be chosen; the suggestion recommends the most accurate one from another provider."

• Readonlyfavors: "What the suggestion weighs once accuracy holds. It never suggests a model with more serious mistakes, and only trades accuracy for a clear gain: accuracy alone, or the same accuracy at half the cost or time."

• Readonlyfixed: "The challenger got this case fully right; the test in place didn't."

• ReadonlyfromBaseCase: "An attack case's answer, carried from the case it was made from and the person who confirmed that one. An attack shouldn't change the right answer."

• Readonlyimprove: "The suite whose best tested model scores lowest: where better cases, checks, or instructions would help most."

• ReadonlyinInstructions: "A case whose words appear in the instructions the model is told. A test then grades it on an answer it was handed, so the score rises without the model getting better."

• ReadonlyinPlace: "The test the other is weighed against, usually the live model's. Differences are counted from it."

• ReadonlyinstructionsEdited: "Instructions edited at runtime (an admin override or added guidance) were recorded under the same models. No model was switched. Dated when noticed, which can be after the edit."

• ReadonlykeepInput: "Keeps what you typed, without your name, so it can become a test case for future models. Without it, a rating still counts but can't become a case."

• Readonlylive: "The model this feature runs on now."

• ReadonlymakeCase: "Turns this rating into a draft case on its real input. A thumbs down needs a person to write the right answer; a thumbs up proposes the model's answer for a person to confirm."

• ReadonlymeasuredDifferently: "The tests differ in more than the model: other cases, answers, checks, or instructions. Rescore both, or test again, before trusting the verdict."

• Readonlyminor: "A fair difference of judgment. Minor checks should pass 75% of the time unless the check sets its own bar."

• ReadonlymissedDifferently: "Neither got this case fully right, but they missed it in different ways: other checks failed, fewer parts right, or no answer on one side."

• ReadonlynativeProbabilities: "The model returns its own probabilities for every option, so they can be scored and compared."

• ReadonlynearDuplicates: "Pairs of cases whose inputs share nearly all their words. Keep the clearer one: a duplicate counts the same thing twice."

• ReadonlyneedsAnswer: "A draft case with no answer yet. Write what a good answer is; until then, tests leave it out."

• ReadonlynewInstructions: "The instructions changed in code and were recorded under the same models. No model was switched."

• ReadonlyoldDrafts: "Drafts nobody reviewed in 30 days. They're never graded against. The weekly check retires them on its own."

• ReadonlyoutOfDate: "Cases, answers, or instructions changed since this test ran. Test again, or rescore for free if only answers changed."

• ReadonlypassRate: "How often the check passed on the answers it applies to, and on how many answers."

• Readonlypending: "Proposed and not reviewed yet. Tests never grade against it until a person confirms it."

• Readonlypriority: "What this suite cares about after accuracy: speed (someone waits), cost (it runs often in the background), or quality (only accuracy counts)."

• ReadonlyprobabilityQuality: "How good the probabilities were: 100% minus the Brier score. High means sure when right and unsure when wrong; picking right with careless probabilities scores lower."

• ReadonlypromptedProbabilities: "The model writes out its own estimate. Choices come without a distribution, so only true-or-false questions get probability scores."

• Readonlyprovisional: "Put to use before anyone reviewed it. Tests use it, and every score says it rests on unreviewed answers."

• ReadonlyrateBlind: "Rating imported cases one at a time from their input alone, never the app's output or what anyone did, so nothing anchors your answer. Each rating is your own confirmed answer, kept beside other people's."

• ReadonlyrateQueue: "Your progress: cases you rated, cases you skipped, and cases still waiting for you, oldest first. Other people's answers don't count toward yours."

• ReadonlyreasoningSetting: "Some models think before answering, and how much is a setting. Less is faster and cheaper; more can be more accurate. Each setting you test gets its own row."

• Readonlyrescore: "Regrades stored answers with today's checks and answers. No model calls, so it's free."

• Readonlyretiring: "The provider has announced a date it stops serving this model. It's never suggested, and a retiring live model gets replaced."

• Readonlyscore: "Accuracy: how much the model got right on every case in use. With partial credit, parts right of parts expected; otherwise answers with every check passed."

• Readonlyserious: "A mistake that changes what the product does. Serious checks should pass 95% of the time unless the check sets its own bar."

• ReadonlyseriousMistakes: "Failed serious checks: mistakes that change what your product does. Models are compared on these first."

• ReadonlysharedCases: "Cases both tests answered. Only these are compared."

• ReadonlysideBySide: "Both tests' answers to one case, next to the expected answer, with what each check found. Step through every case where they differ."

• Readonlyskip: "Passes on this case for you only. It gives no answer and changes no test; others can still rate it, and you can still answer it in the answer key."

• Readonlyslowest: "The live model with the longest average answer time, across suites."

• ReadonlysplitCases: "Cases models divide about evenly on. Often the answer is unclear, so check it. Once it's right, it's the most useful kind of case."

• Readonlysuggestion: "Fixed rules pick it, so the reason given is the real one: accuracy first, then the suite's priority. Nothing changes until you choose."

• Readonlysuites: "One suite per AI feature: its cases, the answers people confirmed, and the checks that grade any model against them. Open one to compare models and choose the live one."

• ReadonlytagCoverage: "How many cases in use test each tag. A tag with one case is thinly tested: a new case there tells you the most."

• Readonlytags: "What a case tests. The quick set picks cases that cover the most tags."

• Readonlytest: "Cases you only check: never tune against them, so they show how a change does on cases it wasn't fitted to."

• Readonlythumbs: "Thumbs up and down people gave the feature's answers in your product, over the last 7 days."

• Readonlytime: "Speed: average seconds per answer, so lower is faster, from answers the model actually made (reused answers keep their original time)."

• Readonlyverdict: "More accurate, about as accurate, or less accurate, read off a 95% range from resampling the shared cases. Under 5 shared cases, too few to tell."

• Readonlyversions: "Each numbered version of this feature's settings (model, fallback, reasoning, output limit, temperature), with who changed it, when, why, and its evidence. Any version can be made live again."

• ReadonlyweeklyCheck: "Each suite's quick set, answered again on its live model every week. A drop of 15 points or more against last week's check of the same model, instructions, and grading is drift."