Evalkeeper
    Preparing search index...

    Type Alias Suite<Input, Output, Label>

    An eval suite: how to call one AI feature on a model and how to grade what comes back. Code says how to evaluate; the store holds the cases and answers.

    type Suite<Input, Output, Label> = {
        answerGuide?: AnswerGuide;
        answerTimeoutMs?: number;
        caseKind?: "typed" | "imported" | "fixture";
        checks: CheckDef[];
        concurrency?: number;
        decisions?: boolean;
        description: string;
        fields?: { input: FormField[]; label: FormField[] };
        graderVersion: string;
        input: z.ZodType<Input>;
        key: string;
        label: string;
        labelHelp?: string;
        labelValue: z.ZodType<Label>;
        operation: string;
        priority: SuitePriority;
        rateBlind?: boolean;
        reviews?: readonly ShippedReview[];
        starter?: {
            input: Input;
            label: Label;
            note: string;
            smoke?: boolean;
            split: "DEV" | "TEST";
            tags: string[];
        }[];
        unitNoun?: string;
        describe?(input: Input): string;
        draftFromActivity?(
            args: {
                existing: { input: Input; tags: string[] }[];
                known: ReadonlySet<string>;
                limit: number;
                since: Date;
            },
        ): Promise<ActivityDraft<Input, Label>[]>;
        draftMore?(
            args: {
                count: number;
                existing: { input: Input; tags: string[] }[];
                focus: string | null;
            },
        ): Promise<{ input: Input; label: Label; note: string; tags: string[] }[]>;
        grade(
            input: Input,
            output: Output,
            label: Label | null,
            ctx?: { onCost: (usd: number) => void },
        ): Promise<CheckResult[]>;
        gradeTogether?(
            answers: { input: Input; label: Label | null; output: Output }[],
        ): CheckResult[][];
        inputFromForm(form: FormData): Input;
        labelFromForm?(form: FormData): Label;
        labelFromOutput?(output: Output): Label;
        run(
            input: Input,
            ctx: {
                maxOutputTokens?: number | null;
                model: string;
                onCost: (usd: number) => void;
                reasoningEffort?: ReasoningEffort | null;
                temperature?: number | null;
            },
        ): Promise<Output>;
        title(input: Input): string;
        withoutLeaks?(
            input: Input,
            source: { sourceRef: string | null },
        ): { input: Input; removed: string[] };
    }

    Type Parameters

    • Input
    • Output
    • Label
    Index
    answerGuide? answerTimeoutMs? caseKind? checks concurrency? decisions? description fields? graderVersion input key label labelHelp? labelValue operation priority rateBlind? reviews? starter? unitNoun?
    describe? draftFromActivity? draftMore? grade gradeTogether? inputFromForm labelFromForm? labelFromOutput? run title withoutLeaks?
    answerGuide?: AnswerGuide

    How a person should judge an answer, where it takes more than each field's help: opened from the info button beside the answer key's heading, as "How to judge an answer". Without it, labelHelp shows there. For people only: it never reaches the model, so it doesn't change the instructions hash.

    answerTimeoutMs?: number

    How long one answer may take before it counts as failed. Default 60s; multi-step agent turns need more. Keep it under the run budget.

    caseKind?: "typed" | "imported" | "fixture"

    "typed": cases are text someone types (added by hand, or drafted). "imported": cases are frozen from the app's own data, then rated by a person. "fixture": cases are structured inputs, shown with describe() and edited as JSON.

    checks: CheckDef[]
    concurrency?: number

    How many answers a test asks for at once. Default 6; slow operations need more to finish before the platform's time limit.

    decisions?: boolean

    Graded as decisions (typed answers with probabilities, as from the AI SDK's experimental_decide), so pages show probability quality and calibration next to accuracy, and which models' probabilities are native or prompted.

    description: string
    fields?: { input: FormField[]; label: FormField[] }

    The form's fields, so pages can render one for any suite. Every field has a plain label and help: people confirming answers weeks later won't remember what a field means.

    graderVersion: string

    Bump when checks change, so runs say which grading they used.

    input: z.ZodType<Input>
    key: string
    label: string
    labelHelp?: string

    Fixture suites: what the expected answer's fields mean.

    labelValue: z.ZodType<Label>

    The EXPECTED label's value.

    operation: string

    The AI feature it tests, as your app names it ("summarize_ticket").

    priority: SuitePriority
    rateBlind?: boolean

    False when blind rating doesn't fit the suite's cases: each answer is one person's own (a verdict on their own data), or your app has no rate page. The answer key then offers no Rate blind.

    reviews?: readonly ShippedReview[]

    Reviews of starter cases that ship with the suite: its author or an expert confirmed a case's answer once, for every app, so importStarters adds it confirmed instead of pending. Each names who, what qualifies them, and when, and counts only while the case's input and answer are what they saw; the suite contract fails on one that doesn't.

    starter?: {
        input: Input;
        label: Label;
        note: string;
        smoke?: boolean;
        split: "DEV" | "TEST";
        tags: string[];
    }[]

    Drafted cases to start from, added as drafts with PENDING labels.

    unitNoun?: string

    What the headline counts: "items", "tickets", … Default "answers".

    • Drafts cases from what the feature saw in your app since a date (inputs people rated or admins flagged), so the answer key keeps up with real use. The weekly check calls it through draftFromActivity; drafts arrive with a PENDING answer or none, never confirmed. Strip names, contact details, and anything identifying from each input. Return at most limit, and skip anything whose sourceRef is in known.

      Parameters

      • args: {
            existing: { input: Input; tags: string[] }[];
            known: ReadonlySet<string>;
            limit: number;
            since: Date;
        }
        • existing: { input: Input; tags: string[] }[]

          The suite's cases, retired ones too, to avoid near-repeats.

        • known: ReadonlySet<string>

          The sourceRef of every draft made before: skip these.

        • limit: number

          The most drafts to return: what's left of this week's allowance.

        • since: Date

          Look at activity from this date on.

      Returns Promise<ActivityDraft<Input, Label>[]>

    • Drafts new cases aimed at what the answer key is missing, for a person to review (they arrive with PENDING answers).

      Parameters

      • args: {
            count: number;
            existing: { input: Input; tags: string[] }[];
            focus: string | null;
        }

      Returns Promise<{ input: Input; label: Label; note: string; tags: string[] }[]>

    • Grades one output. label is the confirmed one, or null. A check that asks a judge model reports its cost through ctx.onCost.

      Parameters

      • input: Input
      • output: Output
      • label: Label | null
      • Optionalctx: { onCost: (usd: number) => void }

      Returns Promise<CheckResult[]>

    • Checks that compare a test's answers with each other, such as ranking. Gets every graded answer of one repeat; returns extra checks for each, in the same order. Pure; runs after a test and on every regrade.

      Parameters

      Returns CheckResult[][]

    • The operation's output as an answer, so a thumbs-up can propose it (pending until a person confirms it). Omit when outputs and answers differ too much to propose one.

      Parameters

      Returns Label

    • Calls the operation on exactly this model, recorded as an eval.

      Parameters

      • input: Input
      • ctx: {
            maxOutputTokens?: number | null;
            model: string;
            onCost: (usd: number) => void;
            reasoningEffort?: ReasoningEffort | null;
            temperature?: number | null;
        }
        • OptionalmaxOutputTokens?: number | null

          The output-token cap the test runs at, like temperature. Null: send none (the call's own default).

        • model: string
        • onCost: (usd: number) => void
        • OptionalreasoningEffort?: ReasoningEffort | null

          A reasoning setting to test; undefined: the operation's own.

        • Optionaltemperature?: number | null

          The temperature the test runs at: the operation's live one when the test was started. Null: send none. Undefined from runners before temperature was versioned.

      Returns Promise<Output>

    • Removes anything the input learned from its own case before it's frozen, so a case can't carry its answer: facts the app learned from the person's reaction to that same item, for example. importCases, importStarters, and adding a case on the pages run it, and report what it removed per case so a person can check it. Match on something specific to the case (the company's name), not something general (a job title), or it removes what other cases rightly share.

      Parameters

      • input: Input

        The case's input, already checked against input.

      • source: { sourceRef: string | null }

        Where the case came from: its sourceRef, or null for a case added by hand.

      Returns { input: Input; removed: string[] }

      The input without what leaked, and a plain line for each item removed (empty when nothing was).

    Settings
    Member Visibility

    On This Page
    Properties
    Methods