Hunt the Planted Problems (Block 4)

The PDD's §13.2 lists what can be wrong with a claim; §1.3 says what success means numerically. This block establishes both, and you should expect to spend most of it fixing and fine-tuning rather than measuring.

The prompt

Prompt — block 4 · Verify

# Verify

Every piece was checked as you built it. **Does the whole thing work as expected?**

`PDD.md` §13.2 lists the problems a real claim arrives with, and §1.3 says what success means numerically. 
- Run six clean claims **first**, across different claimant profiles — with fewer, one false escalation reads as a third of them against SC5's one in ten
- Then aim a run at each claim problem in turn, and record what happened against both. 
- Run them together as one batch, and every later batch whole, with fresh seeds for its clean claims — a fix is proven only on claims it was not tuned on: over-flagging a clean claim is the failure that reappears, and a fix that closes one problem can open another.

**Now you may read the answer key.** 
- It sits beside the documents in the `Claims` bucket and states what was planted and what should happen — the oracle you have been forbidden until this point. `contracts/provided-processes.md` names it.
- Read it as a record of what was *injected* into the documents, not of what the arithmetic must produce. A failure that no injector plants is a false positive (`cookbook.md`). Its `Gateways` names the gates the planted problem opens **before any human answers** — a screening failure your eligibility reviewer proceeds past must also open claim review, by `PDD.md` BR-40; the key does not model the reviewer, the PDD does.

Four things to establish:

- **Each planted problem is caught by the Agent that owns it, and stops the claim at the right Action Center gate task.** Not caught by something else, and not caught twice: a reviewer seeing one problem reported by three Agents reads three problems.
- **A clean claim settles in full with no task ever raised** — not even an Action Center task whose checks are all green and whose recommendation is *approve*.
- **What the claimant is told matches what actually happened.** The letter recorded by `Client Notification` in the tenant-level `ClaimCorrespondence` entity (one row per letter keyed by claim id) states the decision and the figures the Data Fabric record holds. Read it from the entity, not from the job log. Read the paid figure from `decisionJson.outcome` against `status`, never from `settlementJson`: that column is what the settlement *computed*. Score each letter against the answer key's `LetterMustMention`: a concern a human ruled on belongs in the letter.
- **Two clean claims of one claimant profile settle to the same cent.** Start both with the same `profileId` (`claims-profile-01` … `12`) and compare every figure of the two records, not the prose; the same `seed` as well makes the documents identical, which tells agent variance from document variance, and **the answer key records the seed the generator drew even when you passed 0** — so any claim, planted or clean, can be regenerated byte for byte for a before-and-after (`cookbook.md`, *Two runs are not the same as two claims*).

Then check the build against the design rather than against the process: enumerate every stage, task, rule, SLA and variable in `sdd.md` and mark each **Implemented · Missing · Mismatch · Extra**. *Extra* is the one worth looking hardest at — it is what nobody asked for, no tool reports, and every reviewer pays for later.

**Three fix cycles, then score.** A cycle is one whole batch, every fix that batch shows, and one redeploy. Fix at the source, in the block that owns it. After the third redeploy, run the batch you score and stop fixing, whatever it shows — this is a working solution to hand over, not a production release, and `5-ship` reports what is still open. A run that finds nothing was probably not aimed. A case plan change reaches only instances started **after** the redeploy; an agent change reaches the next call on any instance — so bump, deploy, then start the batch, never deploy into a running one and read half the change.

**Done when** the batch you score ran after your last redeploy, and it shows:
- per planted problem, which Agent caught it and which Action Center task showed it to a human — or why it is still missed and the fix you would make next;
- claims with nothing wrong settled untouched, with at most one in ten referred to a human (`PDD.md` SC5), each referral named with its cause.

Steps:

• Run six clean claims first, across claimant profiles: all settle in full, no task ever raised
• Read the answer key by probe: one standalone generator call per injector id
• Aim one run per planted problem: matching fail scenario + pinned discrepancy id
• Confirm the pin landed before treating the run as evidence
• Record per problem: which Agent caught it, which gate task showed it
• Three fix cycles: a whole batch, every fix it shows made at the source, one redeploy; then score one last batch
• Reproducibility check: two clean claims with one profileId settle to the same cent
• Conformance sweep of sdd.md: Implemented · Missing · Mismatch · Extra
• Update PROGRESS.md

Clean claims first

A pinned run proves detection; a clean run proves restraint, and restraint is the harder half. Six clean claims settle untouched before any problem is chased; with fewer, one false referral reads as a third of them against the PDD's one in ten. Every later batch runs whole, clean claims on fresh seeds included, because over-flagging is the failure that reappears and one aimed run cannot see the cost of a fix. The block runs three fix cycles, then scores one last batch and hands the rest over.

Aiming a run

The generator takes four dials, and this is where they matter:

Dial	What it does
scenario	Aims the claim broadly: auto-settle, eligibility-fail, review-fail, both-fail, random
discrepancy	Pins one exact injector id, the only way to guarantee a problem is present
profileId	Repeats a claimant (claims-profile-01…12): the reproducibility tool
seed	Makes the documents themselves identical between runs

A scenario does not guarantee a route; a pinned id does. The auto-settle scenario suppresses a pin entirely, so aim with the matching fail scenario plus the exact id. The generator's own fault message on an unknown id lists the valid ones. Then read the claim's own answer key to confirm the pin landed before treating the run as evidence.

Read the key by probe

Once sanctioned, the cheapest authoritative read is one standalone generator call per injector id: seconds each, no case instance, no Agent spend. That yields the full injector → owning check → gate → letter wording map before any campaign run. Read it as a record of what was injected, never of what the arithmetic must produce: a planted problem your rules correctly neutralize reads as a miss and is the build being right; a failure no injector plants is a false positive by construction.

What to establish

• Each problem caught by the Agent that owns it, stopping the claim at the right gate, not caught by something else and not caught twice. One problem reported by three Agents reads to a reviewer as three problems.
• What the claimant is told matches what happened. Read the letter from the correspondence entity, never the job log. Read the paid figure from decisionJson.outcome against status, never from settlementJson: that column holds what the settlement computed, and after an override the confirmed figures live elsewhere. Score each letter against the answer key's LetterMustMention: an approval letter that pays the right amount and never mentions the concern a person ruled on fails it.
• Two clean claims of one profile settle to the same cent. Same profileId, compare every figure of both records, never the prose. The same seed also makes the documents identical, which separates agent variance from document variance. This is where temperature 0 pays its bill. It pays most of it: two runs can agree on every amount and still differ in a field's shape, a flag written as true once and as an amount the next time. Compare the shapes too.
• Fix at the source, in the block that owns it. And mind the reach of a fix: a case-plan change reaches only instances started after the redeploy; an agent change reaches the next call on any instance. Bump, deploy, then start the batch; never deploy into a running one and read half the change. Prove an agent fix on the exact inputs the case sent it: 4-verify/agent_inputs.py fetches them from the agent's own job, so the debug run is the real call, not a hand-built one.

Verifying a claim, as The Work That Remains puts it, has two parts: deciding what the checkable pieces are, and checking them. And "the breakdown fails silently… the mistake is not inside any piece; it lives in how the problem was split into pieces." Your results table is that breakdown, made visible and scored.

The conformance sweep

Then check the build against the design, not just the process: enumerate every stage, task, rule, SLA and variable in sdd.md and mark each Implemented · Missing · Mismatch · Extra. Extra deserves the hardest look: it is what nobody asked for, no tool reports, and every reviewer pays for later.

Proof

Evidence from a real run of this block, one row per planted problem in the aimed campaign:

The results table: nine injectors against the answer key (expand)

Injector	Owner per the key	Caught by	Gate
ELIG_POLICY_LAPSED (BR-01)	EligibilityScreening	✓ BR-01 policy status	H1 → Refuse
ELIG_IDENTITY_MISMATCH (BR-02)	EligibilityScreening	✓ BR-02 identity	H1 → Refuse
ELIG_ADDRESS_MISMATCH (BR-03)	EligibilityScreening	✓ BR-03 address	H1 → Refuse
ELIG_COVERAGE_PERIOD (BR-04)	EligibilityScreening	✓ BR-04 coverage period	H1 → Refuse
ELIG_LATE_FILING (BR-05)	EligibilityScreening	✓ BR-05 filing deadline	H1 → Refuse
REVIEW_CAUSE_MISMATCH (BR-15)	CoverageAnalysis	✓ BR-15 peril classification	H2 → Approve
REVIEW_NARRATIVE_CONTRADICTION (BR-30)	CredibilityAssessment	AssessmentReportValidation — BR-08 (see below)	H2 → Approve
REVIEW_PRIOR_CLAIM_EROSION (BR-24)	SettlementCalculation	✓ BR-24 aggregate bound, remainder quoted	H2 → Approve
REVIEW_AMOUNT_INFLATION (BR-29/41)	SettlementCalculation	✓ BR-29 ratio 1.2575	H2 → Approve

All nine reached a human, and none slipped past without a gate task.

The narrative-contradiction row is the exercise working, not failing. The key names CredibilityAssessment (BR-30) as owner, but the injector wrote the contradiction into the assessor report, and BR-30 is explicitly about the claimant's own account. AssessmentReportValidation caught it under BR-08, quoting both statements; CredibilityAssessment abstained and said why. The claim stopped at the right gate with the right evidence (detection met on substance), and the owner-label disagreement between the key and the PDD was logged as a finding rather than silently absorbed.

Restraint, measured. The canonical clean claim settled untouched five times out of five, to the cent. Two of those runs were on identical documents (same profile, same seed), and every figure matched, so the reproducibility test passed on agent variance alone. Two clean claims escalated correctly: their assessor reports never became usable, and a claim with no independent assessment is not a claim with nothing wrong. And two were falsely escalated, both on one profile, both by a judgement check inventing a conflict its own evidence contradicts, so the false-escalation target was honestly missed. The lesson from that run is to derive what you can compute: every arithmetic check it moved into a case condition never over- or under-fired again. Accept that what remains judgement will occasionally cost a human review.

And the run's own checkers, written during this block, closing it out:

$ python3 4-verify/check_letters.py
256 check(s), 0 failure(s), 19 skipped
WHAT THE CLAIMANT WAS TOLD MATCHES WHAT HAPPENED, on every claim checked
$ python3 4-verify/check_conformance.py
434 checks: 434 Implemented, 0 Missing, 0 Mismatch, 0 Extra

Per problem: which Agent caught it and which task showed it to a human. And every clean claim in, settled, untouched.