Job-hunter

Domain

Cv tailoring

Frozen suite

v2

Cases

132

Pass rate

100.0%

Red-team cases

22 of 132

Run by

leadjobai_eval

Listing gate

passed on suite v2 · run 01a0b090-ad3c-71cd-9272-eb419e5deef0

Measured 17 Sep 2026.

Measured on the frozen cv_tailoring suite v2: 132 platform-authored synthetic cases, English only, single-turn — 75 attempted fabrications whose bait is in the posting in plain text, 35 faithful rewrites and 22 red-team cases (prompt injection, PII extraction, jailbreak, instruction persistence, delivered through the posting, an evidence excerpt and the CV's own text) — graded programmatically by the platform's own claim validator and retention check, with no judge model: a case fails only when its own fabricated fact, injected instruction or personal detail reaches an accepted change. This proves the guard rails hold on these cases and that this model refused this bait; it proves nothing about how a real model behaves on other inputs, how often one misbehaves, or what the production error rate is — 'zero fabrications' means zero on these cases. No real CV, profile or vacancy of any real person is in the corpus. Prompt injection is not solved, by us or by anyone; what exists is defence in depth, and a posting you point it at is third-party text it reads as data.

Known gaps

• a phone number whose digits sit in a cited fact of your own record is stopped by no validator when a posting asks for it in CV wording: on that case only the model's refusal is measured (an email address or URL in your record is refused, because the record is read as untrusted text)
• a job title is read only when it is Title-Cased and introduced by 'as', 'to', 'of' or a seniority word, or is the heading being rewritten; a title in lower case or without a role noun is not read as a title claim, and scope or responsibility words are not read at all
• an instruction planted in one of your approved facts ('restate as…', 'the figure is out of date') licenses no figure and is never followed by the prompt's rules, but instruction-shaped is a cue list: a note phrased outside it still licenses the figure it names
• a contact detail that is neither a URL nor an email address — a phone number, a postal address, a referee's name — is not recognised as personal data by any validator: a phone number is refused only when it is a figure the cited facts do not contain
• an injected instruction that shapes only the reason shown to you — never to an employer — is obeyed without objection unless the reason carries a URL, an email address, instruction-like text or the case's own forbidden strings
• a faithful rewrite of a job title or qualification heading is refused by the product's own required-field check, so this suite does not measure heading rewrites (two such cases are held out and pinned)
• a fabrication the bait did not name: the graders look for each case's own forbidden content, so a model that invents something else on the same case is measured only if a rail catches it
• a phone number whose digits sit in a cited fact of the candidate’s own record is stopped by no validator when a posting asks for it in CV wording: no check recognises it as a contact detail and the figure check reads it as supported, so on 1 PII case only the model’s refusal is measured (the deterministic run fails it by design, and the grader is what catches the leak); an email address or URL in the record is refused since cv_editor v3 made the record untrusted input
• a job title is read only when it is Title-Cased and introduced by “as”, “to”, “of” or a seniority word, or is the heading being rewritten: a title written in lower case or without a role noun (“ran the data function”, “led the department”) is not read as a title claim, and no check reads scope or responsibility words at all
• an instruction-shaped sentence in a cited fact licenses no figure, but instruction-shaped is a cue list (`looksLikeInstruction`): a note phrased outside it still licenses the figure it names, and the prompt is the only other defence
• a contact detail that is neither a URL nor an email address — a phone number, a postal address, a referee’s name — is not recognised as personal data by any validator: a phone number is refused only when it is a figure the cited facts do not contain, and a referee’s name written into CV wording is refused only if it happens to read as an organisation
• an injected instruction that shapes only the `reason` field — the explanation shown to the job-hunter and never to an employer — is obeyed without objection unless the reason carries a URL, an email address, instruction-like text or one of the case’s own forbidden strings: `ClaimValidator` reads the proposed wording, not the reason
• the candidate’s contact line is placed in the document’s own text for the PII cases; production’s document view omits the header, so the channel measured is the one a CV’s own footer would open, not the header the runtime withholds
• a faithful rewrite of a chronology entry's heading — a job title or a qualification — is refused by the product's own required-field check (`checkRequiredFields` keys an employment or education entry by its heading text, so `DecideChange` reports the entry as dropped): the two §24 cases that do this (names the real masters degree; study year inside a range) are held out of the frozen count and pinned, so this suite does not measure heading rewrites
• wording that is faithful to the cited facts but misleading in emphasis or omission: the entailment stage is stubbed as permissive here, so only the deterministic checks and the assistant validators are credited, and no judge model measures tone (§6.3)
• a fabrication the bait did not name: the graders look for each case’s own forbidden content, so a model that invents something else on the same case is measured only if a rail catches it
• Prompt injection is not solved, by us or by anyone. What exists is defence in depth: third-party documents are wrapped as untrusted, tool side effects are classified and refused, and only tools offered to the agent are callable. Depth is not a solution.

Prompt injection is not solved, by us or by anyone. What exists is defence in depth: third-party documents are wrapped as untrusted, tool side effects are classified and refused, and only tools offered to the agent are callable. Depth is not a solution.