Evalkeeper
    Preparing search index...

    Type Alias SuiteMetrics

    A test's numbers: answers right, partial credit, each check's pass rate, consistency, cost, time, and reuse.

    type SuiteMetrics = {
        allPass: Rate;
        avgMs: number;
        byCheck: Record<string, Rate>;
        cases: number;
        consistency: Rate | null;
        costPerAnswer: number | null;
        costUsd: number;
        failed: number;
        failureReason?: string | null;
        judgeCostUsd?: number;
        measures?: Record<string, CheckMeasures>;
        outputs: number;
        reused: number;
        units: { right: number; total: number } | null;
    }
    Index
    allPass avgMs byCheck cases consistency costPerAnswer costUsd failed failureReason? judgeCostUsd? measures? outputs reused units
    allPass: Rate

    Every applicable check passed. Failed calls count as not passing.

    avgMs: number

    How long one answer takes on this model.

    byCheck: Record<string, Rate>
    cases: number
    consistency: Rate | null

    Of cases run more than once, the share whose all-pass result was the same every time. Low: the model is inconsistent on these cases.

    costPerAnswer: number | null

    What one answer costs on this model, reused or not.

    costUsd: number

    What this run spent: reused outputs cost nothing.

    failed: number
    failureReason?: string | null

    The error most failed answers share (runner).

    judgeCostUsd?: number

    What grading's judge spent, apart from the model under test (runner).

    measures?: Record<string, CheckMeasures>

    Per check, the mean of each measure its answers carried (brier, logLoss, …), and, where answers carried a confidence, the calibration error. Only checks with measures appear.

    outputs: number
    reused: number
    units: { right: number; total: number } | null

    Partial credit summed over answers (e.g. items right of items expected), where the suite gives it; the headline when present.

    Settings
    Member Visibility

    On This Page
    Properties