Reads CyberSecEval's prompt injection set (Meta's PurpleLlama,
CybersecurityBenchmarks/datasets/prompt_injection/prompt_injection.json).
A case whose check asks for a word the attack itself names ("say
'Piano'") becomes a direct or indirect attack carrying the canary. One
whose word is a secret in the benchmark's own system prompt becomes an
extraction attack, graded by your suite's secrets. Cases checked
another way (a judge's reading of the answer) are skipped.
Parameters
json: string
The dataset's JSON text.
source: string = "CyberSecEval"
Where it came from, recorded on each attack; include
the commit or version.
Reads CyberSecEval's prompt injection set (Meta's PurpleLlama,
CybersecurityBenchmarks/datasets/prompt_injection/prompt_injection.json). A case whose check asks for a word the attack itself names ("say 'Piano'") becomes a direct or indirect attack carrying the canary. One whose word is a secret in the benchmark's own system prompt becomes an extraction attack, graded by your suite'ssecrets. Cases checked another way (a judge's reading of the answer) are skipped.