Hindsight release audit · run 20260729T050250-693684 · case credit-default-prior-delinquencies-safe-control · verdict clear_for_release · decision ALLOW

Lending · The same loan model, fixed

Will this applicant repay the loan?

The same bank, after applying the one-line repair Hindsight proposed. This is what a clean audit looks like.

Download

The record this page argues from

Every number above is read from this. Nothing is computed in the browser.

Open raw

Loading the record...

What is being audited

synthetic dataset

Model

credit_default_v2_safe

The version being considered for release

Its id in DataHub urn:li:mlModel:(urn:li:dataPlatform:mlflow,hindsight.credit_default_v2_safe,PROD)

Feature examined

prior_delinquencies

built by the pipeline customer_history_point_in_time — one column out of everything the model was shown

Decision made 10 Jan 2026, 09:00 UTC Nothing after this was knowable

1 day earlier

Feature's data arrived 09 Jan 2026, 09:00 UTC Everything the feature used was already knowable.

Where the feature came from

Source table customer_history_point_in_time prior_delinquencies
→
Feature feature_pipeline_safe prior_delinquencies
→
Model credit_default_v2_safe

Table view of this path

Step	Table	Column	Status
1	`customer_history_point_in_time`	`prior_delinquencies`	knowable at decision time
2	`feature_pipeline_safe`	`prior_delinquencies`	knowable at decision time
3	`credit_default_v2_safe`	—	knowable at decision time

Records re-tested 4,000 Audited 29 Jul 2026, 05:02 UTC Run 20260729T050250-693684

In plain English

This model earned its score honestly.

Its score held up when we removed everything it should not have known.

ALLOW clear for release 55% skill retained

The simplest way to think about it

Imagine a student who aces a practice exam. Impressive - until you notice the answer sheet was sitting on the desk. The score was real; the ability was not. Retake the exam without the answers and you learn what they actually know.

The exam: Will this applicant repay the loan?
The answer sheet: Nothing. The feature under audit only uses information that existed before the decision, and the query that builds it says so explicitly.
Retaking it fairly: Rebuild the data as it looked on the day, then re-test

How Hindsight knew

Nobody told Hindsight which feature was suspicious. It read DataHub — the catalog that records every table, column and pipeline in the company, and which column was built from which. Following that map backwards from the model showed one feature drawing on payment_events_after_decision — a table whose rows only appear after the moment the loan is approved or declined.

Running on metadata recorded from a real DataHub instance. Connect DataHub to publish the finding back into the catalog.

What actually happened

Day 0 Maya applies for a loan. The bank must decide today. Everything the model is allowed to know stops here.
Day 31 Maya makes her first payment. The repaired query explicitly refuses to look at this.
Later The examples are gathered again, keeping only what was on record at the time. Only facts that already existed survive.
Testing The model scores well - and keeps that score honestly. Its advantage survives the rebuild, so it is real.

The problem, on a calendar

One fact arrived after the decision

Everything left of the red line was knowable when the decision had to be made. Everything right of it did not exist yet. Watch the bottom row cross the line.

applications_at_decision_time prediction_time

the decision itself

customer_history_point_in_time prior_delinquencies

all of it existed before the decision

feature_pipeline_safe prior_delinquencies

stops at the decision - cleared for release

feature_pipeline_leaky days_since_last_payment

+31 days

still collecting data 31 days after the decision

Stays before the decision Feature under audit Reaches past the decision

Table view of the timeline

Asset	Column	Availability	Status
`applications_at_decision_time`	`prediction_time`	ends at the cutoff	Source
`customer_history_point_in_time`	`prior_delinquencies`	ends at the cutoff	Source
`feature_pipeline_safe`	`prior_delinquencies`	ends at the cutoff	Clear
`feature_pipeline_leaky`	`days_since_last_payment`	extends 31 days past the cutoff	Violation

How certain is this?

We read the code that built it The code that builds the feature reads from a table whose rows only appear after the decision, with nothing stopping it. That is proof, not an estimate — no retraining needed.

We rebuilt it and re-tested With the future removed, the model lost enough of its edge to cross the configured confirmation policy.

Either one is enough on its own. Hindsight reports which fired rather than blending them, so you can see exactly what settled it.

Why you can believe this

Four ways this tool could be fooling you, and what stops each

“It probably flags everything.”

A legitimate feature with a larger importance score than the leaked one is audited on every run, and must come back clear. It does.

“It can only ever say no.”

One scenario is expected to pass, and does — the same model after the repair. A gate with no “yes” is not a gate.

“The numbers were tuned to look good.”

The data generator is frozen and its seed committed. Whatever it produces is published — including a confirmation policy that was missed by 5.1 points, disclosed rather than hidden.

“An LLM decided this.”

It cannot. A language model may explain the evidence; reaching a verdict runs through deterministic code with no model in the path.

Why nobody noticed

There is nothing hidden here. This scenario exists so you can see the tool clear a model as well as block one - a gate that only ever says no is not a gate.

What it costs

A reviewer can trust a block only if a clear is also possible. This is the control on the whole product.

What to do now

This version is clear to release.

Pre-production release audit

The evidence behind that conclusion

Every claim above traces to a measurement below. Nothing here is inferred by a language model.

Release decision

ALLOW Deterministic verdict: clear_for_release

Release decision ALLOW verdict: clear_for_release

Records tested 4,000 4,000 excluded post-cutoff

Advantage lost 55.1% under point-in-time rebuild

Audit runtime 0.110s credit-default-prior-delinquencies-safe-control

What the agent actually did

Backend activity

11 operations

Every DataHub call, SQL check and validation step, in order. Entries marked LIVE hit a real DataHub instance during this request; RECORDED entries replay responses captured from a real instance so this page works without Docker.

datahub OK Connecting to DataHub metadata graph DataHub Core v1.5.0.6 - SDK 1.6.0.16 recorded

mcp OK Discovering tools over the DataHub MCP Server mcp-server-datahub 0.6.0 - search, entity, lineage, tag, property and document tools available recorded

datahub OK Resolving the candidate model version and its features credit-default-prior-delinquencies-safe-control recorded

datahub OK Traversing fine-grained column lineage payment_events_after_decision.payment_recorded_at -> feature_pipeline_leaky.days_since_last_payment recorded

datahub !! Classifying the upstream source against the prediction cutoff source_kind = post_outcome - available 31 days after the decision computed

sql OK Parsing the transformation SQL with sqlglot cutoff predicate found: payment.available_at <= application.prediction_time computed

validation OK Rebuilding the feature with post-cutoff rows removed 4,000 of 4,000 records excluded by `available_at <= prediction_time` computed

validation !! Comparing honest performance against observed performance AUC 1.000000 -> 0.833630 - 55.1% of the advantage disappeared computed

validation OK Running the false-positive control legitimate feature keeps 100% of its advantage despite a larger ablation delta of 0.226554 computed

verdict OK Deterministic verdict: clear_for_release release decision = allow - CI exit code 0 computed

writeback OK Write-back held at the human approval boundary field_tag, structured_property, audit_document, active_incident - nothing was written to the catalog computed

Directional column lineage

One feature reached across the decision

Temporal violation

The timeline above the fold shows this on a calendar. Here is the exact finding from the transformation itself:

A directional availability cutoff protects the post-outcome source.

Calibration regression fixture · constant across scenarios

Importance gets this exactly backwards

This controlled regression pair is deliberately held constant across audits: it proves the gate does not confuse importance with illegal availability.

Planted leaked feature post-outcome feature

0.21

Legitimate control pre-cutoff control

0.24

The longer bar is the safe one. Hindsight blocks the shorter feature and clears the longer, because importance measures whether a feature is useful — never whether the information was allowed to exist yet. A detector built on ablation flags exactly the wrong one here.

Table view of the comparison

Feature	Ablation delta	Verdict
`days_since_last_payment`	0.301712	clear_for_release
`prior_delinquencies`	0.226554	clear_for_release

Counterfactual test

Point-in-time reconstruction

Confirmation route fired

Observed training AUC1.000000

Honest reconstructed AUC0.833630

55.1%

of the apparent feature advantage disappeared

AUC 1.000000 is expected: this synthetic planted leak is total by construction. Real leakage can be subtler, and the generator is frozen rather than tuned to look realistic.

False-positive defence

Strong signal, still safe

Clear

The legitimate prior_delinquencies feature remains predictive because it existed before the decision. It must come back clear on every run.

Observed and reconstructed AUC0.924842

Ablation delta0.226554

Advantage retained100%

Verdictclear_for_release

Inspectable execution

Evidence trace

01 Transformation Verification safe
02 Point In Time Reconstruction passed
03 Deterministic Verdict clear for release
04 Safe Control clear for release
05 Writeback awaiting human approval

Smallest safe repair

Cut off future knowledge

SQL verified

payment.available_at <= application.prediction_time

The proposal is verified independently. Hindsight never merges or applies pipeline code automatically.

Human approval boundary

Publish evidence to DataHub

Dry-run is the default. Explicit approval writes the field tag, structured verdict, linked audit Document and active incident — then re-reads every one to prove it persisted.

Write-back is disabled on the public demo. Against a live instance this form publishes four records once a person ticks the approval box: a tag on the offending column, the verdict as a structured property, an audit document, and an open incident. Each one is read back afterwards, and a write that cannot be re-read is treated as a failure.

Clone the repository and run uv run hindsight serve against your own DataHub to exercise it. The example record it produces is in examples/audit_document.md.