Evalkeeper
    Preparing search index...

    Type Alias RunnerOptions

    What the runner needs from your app: storage, suites, models, instruction hashes, and, for the board and drift checks, routing.

    type RunnerOptions = {
        candidates?: (
            operation: string,
        ) => { fallbackModels?: string[]; models: string[] };
        continueRun?: (
            runId: string,
            opts: Required<Omit<RunEvalOptions, "requestStartedAt">>,
        ) => Promise<boolean>;
        estimateCallCost?: (
            operation: string,
            model: string,
        ) => Promise<number | null>;
        instructionTexts?: (operation: string) => string[] | Promise<string[]>;
        liveEffort?: (
            operation: string,
        ) => ReasoningEffort | null | Promise<ReasoningEffort | null>;
        models: ModelCatalog;
        promptVersion: (operation: string) => string | Promise<string>;
        requestLimitMs?: number;
        retiringOnBoard?: "always" | "inUse";
        route?: (operation: string) => Route | Promise<Route>;
        store: EvalStore;
        suites: AnySuite[];
    }
    Index
    candidates? continueRun? estimateCallCost? instructionTexts? liveEffort? models promptVersion requestLimitMs? retiringOnBoard? route? store suites
    candidates?: (
        operation: string,
    ) => { fallbackModels?: string[]; models: string[] }

    The models an operation may use: models can be live; fallbackModels (default: the same) can stand in during an outage.

    continueRun?: (
        runId: string,
        opts: Required<Omit<RunEvalOptions, "requestStartedAt">>,
    ) => Promise<boolean>

    Starts a fresh request that calls runEval(runId, opts). True when one took it; false (or absent) lets the test fail as out of time.

    estimateCallCost?: (operation: string, model: string) => Promise<number | null>

    What one answer costs on a model, from your usage records; null: unknown.

    instructionTexts?: (operation: string) => string[] | Promise<string[]>

    Everything the operation's model is told, as text: system text, prompt templates, schema and tool descriptions (instructionParts in @crowandraven/evalkeeper/ai-sdk, from the parts you hash). Lets casesInInstructions warn when a case was copied into them.

    liveEffort?: (
        operation: string,
    ) => ReasoningEffort | null | Promise<ReasoningEffort | null>

    The live reasoning setting for an operation; null: the model's own. Defaults to the route's, when route is given.

    models: ModelCatalog

    Your models, for providers, prices, and retirements.

    promptVersion: (operation: string) => string | Promise<string>

    The current instructions' hash for an operation (see instructionsHash in @crowandraven/evalkeeper/ai-sdk). A test records it, so editing the instructions marks older tests out of date.

    requestLimitMs?: number

    How long one request may run, in ms. A test answers what fits, then hands the rest to continueRun (a fresh request). Default: no limit.

    retiringOnBoard?: "always" | "inUse"

    Which retiring models (ModelInfo.retiring) the board lists. "always" (the default): every candidate, though a retiring one is never suggested. "inUse": only one a feature still runs on, as its live or fallback model, so nobody spends a test on a model that's going away.

    route?: (operation: string) => Route | Promise<Route>

    Which model each operation runs on now, its fallback, and its reasoning setting. Your app owns routing; the board reads it.

    store: EvalStore
    suites: AnySuite[]

    Settings
    Member Visibility

    On This Page
    Properties