Hindsight release audit · run 20260728T174522-89d397 · case readmission-post-discharge-followup · verdict confirmed · decision BLOCK

Healthcare · Hospital readmission

Will this patient be readmitted within 30 days?

A hospital wants to predict which patients will be readmitted within 30 days, so it can offer extra follow-up care at discharge.

Download

The record this page argues from

Every number above is read from this. Nothing is computed in the browser.

Open raw

Loading the record...

What is being audited

synthetic dataset

Model

readmission_v1_leaky

The version being considered for release

Its id in DataHub urn:li:mlModel:(urn:li:dataPlatform:mlflow,hindsight.readmission_v1_leaky,PROD)

Feature examined

followup_appointments_booked

built by the pipeline readmission_feature_pipeline — one column out of everything the model was shown

Decision made 10 Jan 2026, 09:00 UTC Nothing after this was knowable

22 days too late

Feature's data arrived 01 Feb 2026, 09:00 UTC The feature reached past the moment the decision was made.

Where the feature came from

Source table followup_appointments_after_discharge booked_at rows appear after the decision
→
Feature readmission_feature_pipeline followup_appointments_booked
→
Model readmission_v1_leaky

Table view of this path

Step	Table	Column	Status
1	`followup_appointments_after_discharge`	`booked_at`	rows appear after the decision
2	`readmission_feature_pipeline`	`followup_appointments_booked`	knowable at decision time
3	`readmission_v1_leaky`	—	knowable at decision time

Records re-tested 4,000 Audited 28 Jul 2026, 17:45 UTC Run 20260728T174522-89d397

In plain English

This model was cheating.

It scored perfect in testing. When we rebuilt it using only information that existed at decision time, it dropped to solid. About 49% of its apparent skill came from seeing the future.

BLOCK confirmed 49% of the skill was borrowed

The simplest way to think about it

Imagine a student who aces a practice exam. Impressive - until you notice the answer sheet was sitting on the desk. The score was real; the ability was not. Retake the exam without the answers and you learn what they actually know.

The exam: Will this patient be readmitted within 30 days?
The answer sheet: The model was given the patient's follow-up appointment count - but it counted appointments booked AFTER discharge, which only happen when someone is already getting sicker.
Retaking it fairly: Rebuild the data as it looked on the day, then re-test

How Hindsight knew

Nobody told Hindsight which feature was suspicious. It read DataHub — the catalog that records every table, column and pipeline in the company, and which column was built from which. Following that map backwards from the model showed one feature drawing on followup_appointments_after_discharge — a table whose rows only appear after the moment the patient is discharged.

Running on metadata recorded from a real DataHub instance. Connect DataHub to publish the finding back into the catalog.

What actually happened

Day 0 A patient is discharged. Care teams decide follow-up today. Everything the model is allowed to know stops here.
Day 22 The patient books an urgent follow-up appointment. A strong signal - and completely unavailable at discharge.
Later The examples attach every appointment, whenever it was booked. The model reads Day 22 while pretending to sit on Day 0.
Testing The model spots nearly every readmission. Because it is being told the answer.

The problem, on a calendar

One fact arrived after the decision

Everything left of the red line was knowable when the decision had to be made. Everything right of it did not exist yet. Watch the bottom row cross the line.

discharges_at_decision_time prediction_time

the decision itself

patient_history_point_in_time prior_admissions_12m

all of it existed before the decision

readmission_feature_pipeline_safe prior_admissions_12m

stops at the decision - cleared for release

readmission_feature_pipeline followup_appointments_booked

+22 days

still collecting data 22 days after the decision

Stays before the decision Feature under audit Reaches past the decision

Table view of the timeline

Asset	Column	Availability	Status
`discharges_at_decision_time`	`prediction_time`	ends at the cutoff	Source
`patient_history_point_in_time`	`prior_admissions_12m`	ends at the cutoff	Source
`readmission_feature_pipeline_safe`	`prior_admissions_12m`	ends at the cutoff	Clear
`readmission_feature_pipeline`	`followup_appointments_booked`	extends 22 days past the cutoff	Violation

How certain is this?

We read the code that built it The code that builds the feature reads from a table whose rows only appear after the decision, with nothing stopping it. That is proof, not an estimate — no retraining needed.

We rebuilt it and re-tested Performance fell when future data was removed, but not far enough to cross the configured confirmation policy. This route is supporting evidence only.

Either one is enough on its own. Hindsight reports which fired rather than blending them, so you can see exactly what settled it.

Why you can believe this

Four ways this tool could be fooling you, and what stops each

“It probably flags everything.”

A legitimate feature with a larger importance score than the leaked one is audited on every run, and must come back clear. It does.

“It can only ever say no.”

One scenario is expected to pass, and does — the same model after the repair. A gate with no “yes” is not a gate.

“The numbers were tuned to look good.”

The data generator is frozen and its seed committed. Whatever it produces is published — including a confirmation policy that was missed by 5.1 points, disclosed rather than hidden.

“An LLM decided this.”

It cannot. A language model may explain the evidence; reaching a verdict runs through deterministic code with no model in the path.

Why nobody noticed

Follow-up bookings look like ordinary patient history in the warehouse. Nothing about the column name says these were created after the patient went home.

What it costs

Extra care goes to the wrong patients. The people who actually needed follow-up are sent home unsupported.

What to do now

Do not release this model version.
Count only appointments that existed on the discharge date.
Re-run the audit; the score you get then is the real one.

Pre-production release audit

The evidence behind that conclusion

Every claim above traces to a measurement below. Nothing here is inferred by a language model.

Release decision

BLOCK Deterministic verdict: confirmed

Release decision BLOCK verdict: confirmed

Records tested 4,000 4,000 excluded post-cutoff

Advantage lost 48.8% under point-in-time rebuild

Audit runtime 0.099s readmission-post-discharge-followup

What the agent actually did

Backend activity

11 operations

Every DataHub call, SQL check and validation step, in order. Entries marked LIVE hit a real DataHub instance during this request; RECORDED entries replay responses captured from a real instance so this page works without Docker.

datahub OK Connecting to DataHub metadata graph DataHub Core v1.5.0.6 - SDK 1.6.0.16 recorded

mcp OK Discovering tools over the DataHub MCP Server mcp-server-datahub 0.6.0 - search, entity, lineage, tag, property and document tools available recorded

datahub OK Resolving the candidate model version and its features readmission-post-discharge-followup recorded

datahub OK Traversing fine-grained column lineage followup_appointments_after_discharge.booked_at -> readmission_feature_pipeline.followup_appointments_booked recorded

datahub !! Classifying the upstream source against the prediction cutoff source_kind = post_outcome - available 22 days after the decision computed

sql !! Parsing the transformation SQL with sqlglot no availability cutoff found across discharges_at_decision_time, followup_appointments_after_discharge computed

validation OK Rebuilding the feature with post-cutoff rows removed 4,000 of 4,000 records excluded by `available_at <= prediction_time` computed

validation !! Comparing honest performance against observed performance AUC 1.000000 -> 0.865901 - 48.8% of the advantage disappeared computed

validation OK Running the false-positive control legitimate feature keeps 100% of its advantage despite a larger ablation delta of 0.207554 computed

verdict !! Deterministic verdict: confirmed release decision = block - CI exit code 3 computed

writeback OK Write-back held at the human approval boundary field_tag, structured_property, audit_document, active_incident - nothing was written to the catalog computed

Directional column lineage

One feature reached across the decision

Temporal violation

The timeline above the fold shows this on a calendar. Here is the exact finding from the transformation itself:

The post-outcome source is referenced, but no directional available_at <= prediction_time cutoff was found. The transformation does not reference available_at.

Calibration regression fixture · constant across scenarios

Importance gets this exactly backwards

This controlled regression pair is deliberately held constant across audits: it proves the gate does not confuse importance with illegal availability.

Planted leaked feature post-outcome feature

0.21

Legitimate control pre-cutoff control

0.24

The longer bar is the safe one. Hindsight blocks the shorter feature and clears the longer, because importance measures whether a feature is useful — never whether the information was allowed to exist yet. A detector built on ablation flags exactly the wrong one here.

Table view of the comparison

Feature	Ablation delta	Verdict
`days_since_last_payment`	0.274773	confirmed
`prior_delinquencies`	0.207554	clear_for_release

Counterfactual test

Point-in-time reconstruction

Supporting evidence

Observed training AUC1.000000

Honest reconstructed AUC0.865901

48.8%

of the apparent feature advantage disappeared

AUC 1.000000 is expected: this synthetic planted leak is total by construction. Real leakage can be subtler, and the generator is frozen rather than tuned to look realistic.

False-positive defence

Strong signal, still safe

Clear

The legitimate prior_admissions_12m feature remains predictive because it existed before the decision. It must come back clear on every run.

Observed and reconstructed AUC0.932781

Ablation delta0.207554

Advantage retained100%

Verdictclear_for_release

Inspectable execution

Evidence trace

01 Transformation Verification violation
02 Point In Time Reconstruction failed
03 Deterministic Verdict confirmed
04 Safe Control clear for release
05 Writeback awaiting human approval

Smallest safe repair

Cut off future knowledge

SQL verified

evt.available_at <= src.prediction_time

The proposal is verified independently. Hindsight never merges or applies pipeline code automatically.

Human approval boundary

Publish evidence to DataHub

Dry-run is the default. Explicit approval writes the field tag, structured verdict, linked audit Document and active incident — then re-reads every one to prove it persisted.

Write-back is disabled on the public demo. Against a live instance this form publishes four records once a person ticks the approval box: a tag on the offending column, the verdict as a structured property, an audit document, and an open incident. Each one is read back afterwards, and a write that cannot be re-read is treated as a failure.

Clone the repository and run uv run hindsight serve against your own DataHub to exercise it. The example record it produces is in examples/audit_document.md.