Evalkeeper
    Preparing search index...

    Function createMeaningJudge

    • A judge model that matches expected items to a model's items by meaning, cached so regrading unchanged answers is free.

      Parameters

      • opts: {
            cache?: JudgmentCache;
            model: LanguageModel;
            modelId: string;
            models?: ModelCatalog;
            system?: string;
        }
        • Optionalcache?: JudgmentCache
        • model: LanguageModel

          A small, cheap model is enough; test it with its own suite.

        • modelId: string

          Its id in your catalog, for pricing and the cache.

        • Optionalmodels?: ModelCatalog
        • Optionalsystem?: string

          Replaces MATCH_SYSTEM, e.g. to describe your domain.

      Returns {
          judge: (
              expected: string[],
              actual: string[],
              onCost?: (usd: number) => void,
          ) => Promise<Pair[]>;
          match: (
              expected: string[],
              actual: string[],
              fallback: (expected: string[], actual: string[]) => Pair[],
              onCost?: (usd: number) => void,
          ) => Promise<Pairing>;
          version: string;
      }

      • judge: (
            expected: string[],
            actual: string[],
            onCost?: (usd: number) => void,
        ) => Promise<Pair[]>
      • match: (
            expected: string[],
            actual: string[],
            fallback: (expected: string[], actual: string[]) => Pair[],
            onCost?: (usd: number) => void,
        ) => Promise<Pairing>
      • version: string
      const judge = createMeaningJudge({
        model: anthropic("claude-haiku-5-5"),
        modelId: "claude-haiku-5-5",
        models: MODELS,
        cache: prismaJudgmentCache(prisma),
      });
      const { pairs, by } = await judge.match(expected, actual, wordPairs, onCost);

    Settings
    Member Visibility