What Hindsight actually produces

It can watch instead of waiting. The release gate subscribes to DataHub's event stream and audits a model on creation. It may raise an incident to notify — it never publishes the evidence record itself, because a process that silently rewrites governed metadata is the thing this project argues against.

datahub actions -c docker/hindsight-action.yml

Proven end to end against a live instance: a model was emitted, DataHub published the change to Kafka, the event was consumed off the bus, and the gate audited it and returned block. Full record in evidence/integrations/.

A write that reports success but cannot be read back afterwards is treated as a failure.

This is what the next engineer or agent finds attached to the asset, instead of rediscovering the problem themselves.

# Hindsight audit: credit-default-leaked-payment-event

**Release decision:** BLOCK<br>
**Verdict:** `confirmed`<br>
**Catalog write-back:** awaiting human approval

## Evidence path

`payment_events_after_decision.payment_recorded_at`
-> `feature_pipeline_leaky.days_since_last_payment`

The transformation reads the configured post-outcome source without an
`available_at <= prediction_time` cutoff.

## The ablation trap

| Feature | Ablation delta | Verdict |
|---|---:|---|
| Planted leaked feature | 0.21 | `confirmed` |
| Legitimate pre-cutoff control | **0.24** | `clear_for_release` |

The safe feature has the larger importance delta. An ablation-only detector would get this exactly backwards.

## Point-in-time validation

| Measure | Observed | Reconstructed |
|---|---:|---:|
| AUC | 1.000000 | 0.833630 |
| Advantage over baseline | 0.301712 | 0.135342 |

The planted leak is total by construction, so observed AUC `1.000000` is expected; real leakage can be subtler. The strict-majority loss boundary is a configurable demo policy, not a universal scientific constant.

The reconstruction excluded 4,000 post-cutoff records. Only 44.858% of the
observed advantage remained, satisfying the frozen majority-loss rule.

## False-positive control

The legitimate `prior_delinquencies` feature had a large 0.226554 ablation
delta, but its AUC remained 0.924842 after point-in-time reconstruction.
Hindsight returned `clear_for_release`, demonstrating that feature importance
alone cannot confirm leakage.

## Proposed remediation

Add the following predicate to the payment join:

```sql
payment.available_at <= application.prediction_time
```

The full proposal is in `examples/remediation.sql` and independently verifies
as safe. No change has been applied automatically.

## Planned approved write-back

- field tag
- `hindsight.auditVerdict` structured property
- native DataHub audit Document
- active `CUSTOM / ML_LEAKAGE` incident

SELECT
    application.application_id,
    date_diff(
        'day',
        max(payment.payment_recorded_at),
        application.prediction_time
    ) AS days_since_last_payment
FROM applications_at_decision_time AS application
LEFT JOIN payment_events_after_decision AS payment
    ON payment.customer_id = application.customer_id
GROUP BY application.application_id, application.prediction_time;

SELECT
    application.application_id,
    date_diff(
        'day',
        max(payment.payment_recorded_at),
        application.prediction_time
    ) AS days_since_last_payment
FROM applications_at_decision_time AS application
LEFT JOIN payment_events_after_decision AS payment
    ON payment.customer_id = application.customer_id
   AND payment.available_at <= application.prediction_time
GROUP BY application.application_id, application.prediction_time;

The difference is the available_at <= prediction_time guard. Hindsight proposes it and checks it, but never merges or applies pipeline code itself.

Captured from DataHub Core v1.5.0.6 via MCP server 0.6.0 on 2026-07-27. Re-prove it against your own instance with hindsight verify-fixture-live.

Raw machine reports stay ignored as *.local.json; these are the sanitized summaries, versioned with the code that produced them.

• A high-correlation safe controli must clear on every run, despite having a larger ablation delta than the planted leak.
• One scenario is expected to pass. A gate that can only say no is indistinguishable from one that is not looking.
• The generator is frozen; measured results are published rather than tuned.
• The collapse threshold and its measured margin are disclosed, not hidden.
• An LLM may explain evidence. It can never promote a verdicti.
• This console claims anyone can read it, so CI checks that: every route, both themes, both reading levels, against WCAG 2.1 AA contrast plus accessible names, labels, heading order and landmarks.

scripts/check_accessibility.py — it found three real defects the first time it ran, including a subtle text colour that failed AA on most surfaces in both themes.

The same planted defect, swept from reaching every record down to reaching 2% of them, with a matched clean query for every case. Each bar is one band. Every bar is full — the defect was caught every time. What changes is which route caught it.

Hindsight's own scenarios are generated by code in this repository, so they cannot show whether it works on data it did not create. This is a committed dataset in a different domain, with different column names and a different leak mechanism, audited by the same deterministic routes.

Subscription churn. plan_changes_to_date is a running counter, and the training table joined in every change ever recorded - including the downgrade a customer makes while cancelling. Before the decision it is close to noise; after it, nearly the label.

Every feature, ranked by what the cutoff removes

Ranking is triage, not a verdict. A feature high in this list had most of its advantage removed by the cutoff, which is where to look first. Confirming a defect still requires the deterministic routes, one feature at a time. Note that ranking by importance would put the legitimate control near the top instead.

Take the data and run it yourself

uv run hindsight validate-point-in-time --scenario examples/adapter/scenario.json
uv run hindsight validate-point-in-time --scenario examples/adapter/scenario_wide.json
uv run hindsight sweep-features --scenario examples/adapter/scenario_wide.json

Every number above is a recorded result, not a live one — a public demo should not retrain a model because someone opened a page. A test re-runs the real audit on every build and fails if these figures drift.

Leakage detection is an active field. Everything below is real and worth using. The point is to be precise about the gap, not to dismiss any of it.

All three came out of building this rather than out of looking for something to contribute.

Full record, including what went wrong on the first attempt, in evidence/contributions/.