Evalkeeper
    Preparing search index...

    Interface EvalStore

    Where Evalkeeper keeps cases, answers, tests, and outputs. Implement it for another database, and run describeStoreContract against it.

    interface EvalStore {
        addLabel(caseId: string, label: NewLabel): Promise<LabelRecord>;
        createCase(data: NewCase): Promise<CaseRecord>;
        createOutput(data: NewOutput): Promise<OutputRecord>;
        createRun(data: NewRun): Promise<RunRecord>;
        findReusableHashes?(
            requestHashes: string[],
            since: Date,
        ): Promise<Set<string>>;
        findReusableOutput(
            requestHash: string,
            since: Date,
        ): Promise<OutputRecord | null>;
        getRun(id: string): Promise<RunRecord | null>;
        listCases(
            suite: string,
            opts?: { includeRetired?: boolean },
        ): Promise<CaseRecord[]>;
        listOutputs(runId: string): Promise<OutputRecord[]>;
        listOutputsFor?(runIds: string[]): Promise<OutputRecord[]>;
        listRuns(
            suite: string,
            opts?: { limit?: number; status?: RunStatus },
        ): Promise<RunRecord[]>;
        setCaseStatus(
            caseId: string,
            status: "DRAFT" | "ACTIVE" | "RETIRED",
        ): Promise<void>;
        setLabelStatus(labelId: string, status: LabelStatus): Promise<void>;
        setOutputScores(id: string, scores: CheckResult[]): Promise<void>;
        updateRun(
            id: string,
            patch: Partial<Omit<RunRecord, "id">>,
            onlyIf?: { status: RunStatus },
        ): Promise<void>;
    }
    Index
    addLabel createCase createOutput createRun findReusableHashes? findReusableOutput getRun listCases listOutputs listOutputsFor? listRuns setCaseStatus setLabelStatus setOutputScores updateRun

    • Adds an answer to a case (answers are never overwritten). A SKIPPED mark (value null) is kept the same way, and leaves the case's updatedAt alone, so a skip doesn't make tests out of date.

      Parameters

      Returns Promise<LabelRecord>

    • Which of these request hashes have an output findReusableOutput would return, in one call, for estimating a test's cost. Optional: without it the runner looks each hash up.

      Parameters

      • requestHashes: string[]
      • since: Date

      Returns Promise<Set<string>>

    • The newest original (not reused, no error) output with this hash.

      Parameters

      • requestHash: string
      • since: Date

      Returns Promise<OutputRecord | null>

    • A suite's cases that aren't retired, with their answers, oldest first. With includeRetired, retired ones too (an import checks them, so a case someone retired isn't added again).

      Parameters

      • suite: string
      • Optionalopts: { includeRetired?: boolean }

      Returns Promise<CaseRecord[]>

    • The outputs of several tests in one call, for the board on a remote database. Optional: without it the runner calls listOutputs per test.

      Parameters

      • runIds: string[]

      Returns Promise<OutputRecord[]>

    • Changes a case's status, e.g. retiring it.

      Parameters

      • caseId: string
      • status: "DRAFT" | "ACTIVE" | "RETIRED"

      Returns Promise<void>

    • Parameters

      • id: string
      • patch: Partial<Omit<RunRecord, "id">>
      • OptionalonlyIf: { status: RunStatus }

        Only if the run still has this status (a guard against races).

      Returns Promise<void>

    Settings
    Member Visibility

    On This Page
    Methods