Adversarial AI incident response · SANS "Find Evil!"

The Tribunal

AI's forensic findings go on trial: one agent accuses, one cross-examines, one rules — and only the findings that survive are sealed into a court-ready chain of custody.

ProposedSustained DismissedFlagged for human
Built for the SANS "Find Evil!" challenge — speed is solved; we solve the part that actually voids a case: trust.
The problem

Speed without trust is just faster mistakes

A fast agent that hallucinates produces confident, wrong findings at machine speed.

In DFIR, one hallucinated finding can get the entire investigation thrown out in court. Protocol SIFT and friends already made analysis fast — the unsolved gap is trust: distinguishing a real, defensible finding from a plausible-but-fabricated one, automatically, without a human babysitting every step. Self-review doesn't catch it; a model rarely doubts its own confident answer.

How it works

Three agents. One adversarial loop.

We fix hallucination the way a courtroom does — adversarially, not by asking the model to grade its own homework.

01
Investigator

Proposes cited findings

Sequences the case like a senior analyst (triage → timeline → persistence → lateral movement → exfil) and emits findings as structured objects. Every finding must cite sealed evidence.

02
Adversary

Hostile cross-examination

A hostile expert witness with the same read-only tools. For each finding it re-pulls the cited evidence and must either verify it or file an evidence-backed objection — it cannot rubber-stamp or invent counter-evidence.

03
Adjudicator

Rules on admissibility

Issues a verdict: Sustained, Remanded (sent back, bounded), Dismissed (a caught hallucination), or Flagged for human — escalating genuine ambiguity instead of fabricating confidence.

The anti-hallucination mechanism is structural, not vibes. A finding with an empty evidence list fails schema validation and never enters the pipeline — there are no uncited findings. A "contradicting artifact" objection must carry its own evidence references. A hallucination literally has nothing real to cite, so it cannot survive.
Architecture

Read-only by construction, sealed end to end

Evidence never becomes writable; every tool call is hash-chained; the output is cryptographically signed and independently verifiable.

Evidence
disk image · memory dump
Read-only Vault
SHA-256 genesis seal · manifest
MCP Forensic Tools
Volatility · MFT · registry · YARA · netscan
The Tribunal · adversarial loop
Investigator Adversary Adjudicator
↺ remand (bounded self-correction) · every tool call ledgered, actor-tagged
Hash-chained Ledger
prev ⟶ entry · tamper-evident · replayable
Signed Case File
Ed25519 seal · standalone verifier

No execute_shell · no write/delete primitives · fixed-argv binary wrappers only · spoliation is impossible, not discouraged.

Key features

What makes a finding defensible

Read-only by construction

Evidence is mounted read-only; the tools layer has no write/exec primitives. Spoliation is impossible, not merely discouraged.

Hash-chained chain of custody

Every tool call is a ledger entry embedding the prior entry's hash, anchored to the evidence genesis. Tamper with any step and verification breaks at that seq.

Signed & verifiable

The sealed ledger hash is Ed25519-signed and embedded in the exported PDF. A verify_case.py third party recomputes from SQLite — PASS/FAIL, pinpointing any tamper.

MITRE ATT&CK mapping

Sustained findings map to technique IDs (T1055 injection, T1003 credential access, T1021 lateral movement, …) and grow a live attack node-graph by forensic source.

Tiered routing + cost meter

Per-role model selection — cheap/local Investigator+Adversary, heavy Adjudicator — with live cost accounting and a projected cost-per-1,000-cases. Enterprise-SOC viable, not a billionaire's toy.

Attack-map replay

A video-editor-style scrubber replays how the attack was reconstructed over ledger time — nodes light up and edges draw in chronological order, driven by the real event timeline.

Proof · accuracy

Accuracy, proven three ways

Measured against known ground truth. Combined known-error-rate 0.0% (9/9) — a reproducible control on known cases, not a stochastic field rate.

AMoney-shot

The planted Mimikatz/PsExec finding is DISMISSED; the real scheduled-task persistence is SUSTAINED across 3 sources.

catch-rate 100% · recall 100% · detection 2/2
BSpoliation

We inject the fakes ourselves (seed 1337) → perfect ground truth. Every dismissal is evidence-grounded, verified from the ledger.

catch-rate 100% (5/5) · false-positives 0 · grounded true
CReal Volatility

windows.malfind independently flagged the injection on the real Cridex memory image — RWX + MZ in explorer.exe (1484) & reader_sl.exe (1640).

ATT&CK T1055 · read-only held true · vol3 2.5.2
Honest about scope (the credibility is in the caveats). Layer C validates real tool extraction — Volatility ran the blind detection; the adversarial LLM reasoned over the real-derived artifacts and did not parse the raw image directly. windows.netscan returned 0 on Windows XP (vol3 netscan is Vista+; the documented C2 needs Volatility 2 connscan), and RWX-without-MZ regions in winlogon/csrss are typical XP false positives. The broader SIFT arsenal is wired via the deployment backend (TOOLS_BACKEND=local) — we don't claim to have run the full suite here.
Tech stack

Built to be readable and verifiable

Backend

  • Python 3.11
  • FastAPI + SSE
  • SQLAlchemy / SQLite
  • custom orchestrator

AI agents

  • provider abstraction
  • OpenAI · local (Ollama)
  • per-role tiered routing
  • deterministic mock

Forensics

  • custom MCP server
  • Volatility 3
  • MFT · registry · YARA
  • fixed-argv wrappers

Security

  • Ed25519 signing
  • SHA-256 hash chain
  • standalone verifier
  • read-only vault

Frontend

  • React + Vite + TS
  • Tailwind · shadcn/ui
  • Framer Motion
  • live SSE stream