From 9ae4daab5e1964a208ee2f6addce6a8dfba6be9b Mon Sep 17 00:00:00 2001 From: Codex Date: Mon, 13 Jul 2026 18:48:52 +0200 Subject: [PATCH] =?UTF-8?q?docs(spec):=20ANNEX=20A=20=E2=80=94=20the=20man?= =?UTF-8?q?ifold=20query=20(retrieve/gate/model)=20+=20DA'AT=20generalizat?= =?UTF-8?q?ion?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Operator's cosine proposal: ADOPTED as stage 1 of 3, never alone. - Stage 1 RETRIEVE: cosine on DIRECTION = matmul = the reflex (fast, exact, deterministic) - Stage 2 GATE: magnitude envelope + support + Mahalanobis -> OUT_OF_DISTRIBUTION - Stage 3 MODEL: local tangent-space/GP -> prediction + VARIANCE (the real extrapolation) THE DANGER: crises preserve direction and explode magnitude. Cosine returns similarity 1.0 for a same-shape-10x-size state — max confidence at the exact moment it is most wrong. Cosine CANNOT produce the OOD verdict; storing direction+magnitude separately is the entire crash-safety story. Same law, third coat: INDETERMINATE (venue) / stale-price (exit) / OOD (manifold). Unknown is not flat, not 'best guess'. TOPOLOGY (operator's IFF): cyclic (sin,cos) encoding = free + REQUIRED for the streak-phase grail study; component detection = cheap; persistent homology = EARN IT (offline Mode-1 diagnostic that shapes the gate, never a per-tick op). GENERALIZATION: three-stage discipline (not one metric) is a shared kernel — market fingerprint (scalar_hash is a hash reaching for this; conflict_level is latent OOD), trade-path/ADVSL, exit-decision, asset transfer (= how full-universe becomes affordable), counterparty simplex, ops/incident prefiguration, streak-wave phase. One guard defends all: confident interpolation into unexplored magnitude is the universal failure mode. Proposed name: DA'AT. Co-Authored-By: Claude Opus 4.8 --- prod/docs/SPEC_MALKHUT_ACTUALS_INTAKE.md | 368 +++++++++++++++++++++++ 1 file changed, 368 insertions(+) diff --git a/prod/docs/SPEC_MALKHUT_ACTUALS_INTAKE.md b/prod/docs/SPEC_MALKHUT_ACTUALS_INTAKE.md index 8a2de46..5a29045 100644 --- a/prod/docs/SPEC_MALKHUT_ACTUALS_INTAKE.md +++ b/prod/docs/SPEC_MALKHUT_ACTUALS_INTAKE.md @@ -459,6 +459,374 @@ venue-adapter boundary. Do not build two doorframes. --- +*[Annex A follows — the Mode-2 query mechanism, and its generalization.]* + +--- +--- + +# ANNEX A — THE MANIFOLD QUERY: three-stage search, and why it generalizes + +**Author:** Claude (Opus 4.8), on Fable's watch. **Date:** 2026-07-13. +**Ordered by:** HJ ("could something akin to cosine distance over a manifold be +used… this might have dangers BUT has possibility of extrapolating across axes in +a very convenient, exact and almost reflexive way"). +**Status:** DESIGN ANNEX to §0.5 (Mode 2 — RECOMMEND). Binding on any +implementation of manifold localization. + +--- + +## A.1 The question + +Mode 2 must take a **live market state** and find, inside the manifold Mode 1 +built, the **closest-to-absolute-best play** — by *extrapolation across axes*, not +by lookup. The operator's proposal: **cosine distance** as the vehicle. Fast, +exact, reflexive; angle over a normalized state vector. + +**Verdict: adopt it — as stage 1 of 3, never alone.** The reason is not +fastidiousness. Cosine used alone fails *specifically and catastrophically at the +moment we most need it to work*, and the failure is silent and maximally +confident. §A.3 is that argument; §A.2 is why the instinct is nonetheless right. + +--- + +## A.2 Why cosine is the right REFLEX + +| Property | Consequence for us | +|---|---| +| **Normalized cosine ≡ dot product ≡ matmul** | The entire manifold query is a BLAS/GPU operation. 10⁸ cells × 64-dim is microseconds. It fits inside the ≤25 ms Mode-2 budget with room to spare, and scales to the billions Mode 1 will produce. | +| **Exact, deterministic** | No approximate-ANN stochasticity. Seed-pin doctrine and Rule 9 (version everything, reproduce everything) survive intact. Same query → same answer, forever. | +| **Scale-invariance is sometimes SEMANTICALLY RIGHT** | A book at 2 bps / 100k depth and one at 4 bps / 200k can be *the same shape of market at different size*. Cosine captures shape, discards size. When shape is what matters, this is a feature. | +| **Degrades less badly than L2 in high dimension** | With 40+ sensors, Euclidean distances concentrate (everything equidistant). Angular similarity is the standard high-dim workaround; it is why every embedding retrieval system on earth uses it. | + +The operator's word — **reflexive** — is exact. This is the right primitive for +*reflex*. It is the wrong primitive for *judgment*. + +--- + +## A.3 THE DANGER (the one that matters): cosine is blind to magnitude, and +## magnitude is where the crisis lives + +Take a state vector pointing: *wide spread, thin depth, high toxicity, high +latency, imbalance skewed*. Under ordinary stress it points one way. + +Under a **liquidation cascade** it points **the same way** — just ten times +further out. + +**Cosine similarity between them: 1.0.** A perfect match. Maximum confidence. And +the policy it retrieves was fitted on the mild version. + +That is not a corner case; **that is the crash.** Market crises characteristically +*preserve direction and explode magnitude*. Every component moves the way it +always moves under stress — just far beyond anything in the training set. Cosine's +one trick is discarding exactly the coordinate that distinguishes "a bad Tuesday" +from "the day the book vanished." + +Three consequences, stated as law: + +> **A.3.1 — Cosine CANNOT produce the `OUT_OF_DISTRIBUTION` verdict.** It is +> structurally incapable. It returns similarity 1.0 for the single most dangerous +> input the system will ever see. Any design that derives OOD from cosine alone is +> not merely imperfect; it is inverted. + +> **A.3.2 — Discarding magnitude violates CLASS-X law.** *Extreme magnitudes are a +> FEATURE channel, never filtered.* Cosine filters magnitude by construction. It +> must therefore be paired with an explicit magnitude coordinate or gate, or it is +> in direct breach of a standing doctrine (see: 450-second latency tail; §5). + +> **A.3.3 — Confident-and-wrong beats uncertain-and-right at destroying capital.** +> A system that says "I don't know" in a crisis loses an opportunity. A system that +> says "I know this perfectly" in a crisis loses the account. + +--- + +## A.4 Three more edges (real, but survivable) + +**A.4.1 — The metric does all the work; cosine is only the last cheap step.** +On raw features the angle is meaningless: `depth_1pct_usd` ~10⁶, `imbalance` ∈ +[-1,1], `funding_rate` ~10⁻⁴. Whichever feature has the largest raw numbers *owns* +the angle. Required before any cosine: a **rank/quantile transform** (NOT z-score — +our distributions are fat-tailed and non-Gaussian; a Gaussian assumption is a lie +we would then be optimizing against), followed by PCA/whitening onto the manifold's +own coordinates. **That transform IS the model.** Version it (`metric_version`), +pin it, and treat any change to it as a change to the science — every manifold cell +built under a prior transform is invalidated. + +**A.4.2 — Cosine is a chordal distance in the ambient space, not a geodesic on the +manifold.** The Swiss-roll failure: two states near in angle can be far apart +*along the manifold*, on opposite sides of a fold. Policy performance in +toxicity/latency is exactly the sort of surface that has cliffs. Near-in-angle, +far-in-support, opposite side of a cliff = a confidently wrong recommendation. +Mitigation: stage 2's support gate, plus §A.7's topology diagnostic where the fold +structure is real. + +**A.4.3 — Heterogeneous blocks want different geometries.** Market state is +Euclidean-ish. **Counterparty mix is a simplex** (a probability distribution over +agent types — its natural metric is Hellinger or KL, *not* cosine). Inventory / +position state is something else again. One flat cosine over a concatenated vector +silently asserts they share a geometry; they do not. Use a **product metric** with +per-block treatment, and weight the blocks explicitly (the weights are +hyperparameters — fit them, do not guess them). + +--- + +## A.5 THE ARCHITECTURE: RETRIEVE → GATE → MODEL + +The operator's instinct survives intact, relocated to its correct stage. + +``` + live MarketWorldState (S1 book, S2 funding/dvol, S3 regime, + S4 latency, account, intent) + │ + ▼ + ┌───────────────────────────────────────────────────────┐ + │ TRANSFORM (metric_version) │ + │ rank/quantile → whiten → PCA to manifold coords │ + │ cyclic coords → (sin, cos) pairs [see A.7] │ + │ split: DIRECTION (unit vec) ⊕ MAGNITUDE (scalar) │ + └───────────────────────────┬───────────────────────────┘ + │ + ┌───────────────────────────▼───────────────────────────┐ + │ STAGE 1 — RETRIEVE "the reflex" ~µs │ + │ cosine (= dot product = matmul) on DIRECTION only │ + │ over the Mode-1 manifold → candidate neighbourhood │ + │ GPU/BLAS. Exact. Deterministic. Billions of cells OK. │ + │ ◀── THIS IS THE OPERATOR'S PROPOSAL, KEPT ──▶ │ + └───────────────────────────┬───────────────────────────┘ + │ k candidate cells + ┌───────────────────────────▼───────────────────────────┐ + │ STAGE 2 — GATE "do I actually know this?" │ + │ (a) MAGNITUDE: is live |v| inside the explored │ + │ magnitude interval FOR THIS DIRECTION-CELL? │ + │ (b) SUPPORT: support_count ≥ min_support? │ + │ (c) MAHALANOBIS: distance to the explored distribution│ + │ (= cosine with the covariance baked in — the │ + │ principled form of the same instinct) │ + │ FAIL ANY ⇒ RecommendationConfidence.OUT_OF_DISTRIBUTION│ + │ ⇒ NO recommendation. Doctrinal fallback. │ + │ ◀── THIS IS THE GUARD COSINE CANNOT PROVIDE ──▶ │ + └───────────────────────────┬───────────────────────────┘ + │ in-distribution neighbourhood + ┌───────────────────────────▼───────────────────────────┐ + │ STAGE 3 — MODEL "the actual extrapolation" │ + │ local linear/quadratic response surface (tangent │ + │ space) OR Gaussian process over the neighbourhood │ + │ → predicted outcome + PREDICTIVE VARIANCE │ + │ A manifold is BY DEFINITION locally Euclidean: the │ + │ tangent-space linear model is the mathematically │ + │ correct way to "extrapolate across axes" — which is │ + │ precisely what the operator described. │ + │ GP bonus: variance GROWS as you leave the data → │ + │ a second, independent OOD signal, for free. │ + └───────────────────────────┬───────────────────────────┘ + ▼ + RECOMMENDATION {policy, expected outcome, + variance, confidence, neighbours it + interpolates between, book_source, + metric_version, envelope status} + │ + ▼ + RISK GATE (hard veto) +``` + +**Why this ordering is not negotiable:** stage 1 is fast and blind; stage 2 is +cheap and sighted; stage 3 is expensive and honest. Running stage 3 on everything +is unaffordable; running stage 1 alone is how the account dies. The cost profile +and the safety profile happen to agree — which is usually the sign of a correct +decomposition. + +### Storage consequence (do this, it is cheap and it is the whole safety story) + +**Store the manifold as `direction ⊕ magnitude`, separately, per cell.** Then: +- cosine retrieval on direction = matmul (fast path preserved, exactly as the + operator wants); +- magnitude becomes a **scalar interval check** — trivially cheap, and it is the + entire crash-safety mechanism. + +95% of the speed, 100% of the guard. There is no tension here to resolve. + +--- + +## A.6 The law this creates (and it is the same law, a third time) + +| Layer | "I found something" | "But do I *know* it?" | Verdict when unknown | +|---|---|---|---| +| **Venue** (`bdc54fb`) | HTTP call returned/failed | Was the effect provably absent? | `INDETERMINATE` → **never roll back** | +| **Exit** (F5, tonight) | Price crossed a threshold | Is the price fresh? | stale → **skip, don't act on a fossil** | +| **Manifold** (this annex) | Cosine ≈ 1.0 | Is the live magnitude inside the explored envelope? | `OUT_OF_DISTRIBUTION` → **no recommendation; doctrinal fallback** | + +**Unknown is not flat. Unknown is not "best guess." Unknown is unknown.** +Three subsystems, three coats, one law. That is not a coincidence — it is what a +correct architecture looks like from three angles. + +--- + +## A.7 The topology question (the operator's "IFF IFF IFF") + +**When does topology genuinely matter, versus merely being seductive?** + +Topology earns its cost only when the manifold's **global** structure defeats +**local** metrics. Four cases where it truly does: + +| Structure | Where it bites us, concretely | Cost | Verdict | +|---|---|---|---| +| **Cyclicity** | Phase coordinates: hour-of-day, session, funding cycle, day-of-week — and **the operator's win/loss-streak wave phase vs vel_div regimes**. A circle is not a line: 23:59 and 00:01 are *adjacent*, and a flat embedding puts them maximally far apart. Cosine at the seam is catastrophically wrong. | **~zero** | **DO IT NOW.** Encode every cyclic coordinate as a `(sin θ, cos θ)` pair. This is the cheapest correct thing in the entire document and it is *required* for the streak-phase study to mean anything. | +| **Disconnected components** | Genuinely separate basins — normal market vs. halted / limit-up / liquidity-hole. Interpolating *between* components is meaningless; the straight line passes through states the market cannot occupy. | LOW | **DO IT.** Cluster the manifold (offline, Mode 1); refuse to interpolate across component boundaries. Stage 2 gate extension. | +| **Holes / voids** | Regions the market *cannot* enter (negative spread, arbitrage-forbidden configurations). A linear extrapolation across a hole predicts an impossible world — with confidence. | MEDIUM | **IFF** stage-3 variance shows structured failure. Detect offline with persistent homology / the mapper algorithm. | +| **Folds / cliffs** | The Swiss-roll (§A.4.2): chordal-near, geodesic-far. Policy performance cliffs in toxicity/latency. | MEDIUM–HIGH | **IFF** the support gate keeps admitting neighbourhoods whose stage-3 fits are bimodal or high-variance — that is the signature of a fold. | + +**The discipline for topology (this is the "IFF"):** +1. **Cyclic encoding: unconditional.** Free, and its absence is a *bug*, not a + simplification. +2. **Component detection: cheap, do it.** Clustering on the Mode-1 manifold. +3. **Persistent homology / mapper: EARN IT.** These are expensive, notoriously + noise-fooled, and seductive precisely because they produce beautiful pictures. + Run them **offline, in Mode 1 only, as a diagnostic of the manifold** — never + as a per-tick Mode-2 operation. And run them only **after** stage 3's variance + has *demonstrated* that the local model is failing in a structured way. You earn + topology by first proving the simple thing breaks. +4. What TDA is *for*, when earned: it tells you **where the folds and holes are**, + so that **stage 2's gate can be shaped correctly**. Topology's product is a + better *gate*, not a better *answer*. That is the whole of it. + +--- + +## A.8 THE GENERALIZATION (the operator's PS — and I think this is the best idea +## in the document) + +**Yes. Emphatically. The three-stage manifold query is a general primitive, and +"fingerprinting markets" is its most valuable instance.** + +### A.8.1 Market fingerprinting — the natural home + +MARAS today emits a **discrete label** (`regime` ∈ {BEARISH, CHOPPY_BEARISH, +CHOPPY, SIDEWAYS, CHOPPY_BULLISH}) plus `confidence`, `conflict_level`, and five +`tier_*` scores. That is a **classifier**: it puts a continuous world into five +boxes. + +The manifold view is strictly richer: **the market state is a point, and regimes +are regions — not boxes.** Then: +- The **fingerprint** = `direction` (the *shape* of the market) ⊕ `magnitude` (its + *intensity*) ⊕ its position on the manifold. A regime label becomes a + *coordinate*, not a category. "CHOPPY, but 3σ out along the toxicity axis, near + the boundary with CHOPPY_BEARISH" is a sentence the current system cannot say + and desperately needs to. +- Two observations already point straight at this: + - **`maras_fingerprint.scalar_hash` (UInt16) is already a fingerprint attempt — + but a HASH.** A hash has no neighbourhood: near-identical markets get unrelated + hashes. The cosine-manifold fingerprint is precisely its **continuous + generalization** — *nearest*-fingerprint instead of *exact*-hash. This looks to + me like the thing that hash was reaching for. + - **`conflict_level` (the five tiers disagreeing) is already an implicit OOD + signal.** It maps directly onto stage 2. Tier disagreement = "this market is + not cleanly inside any explored region." That is *free* OOD evidence we are + currently discarding into a float. +- And eigenscan is *already* a manifold construction: eigendecomposition of the + correlation structure **is** the coordinate system. The operator is an eigenscan + surfer; he has been navigating this manifold by hand for years. The three-stage + query is the machine that surfs with him. + +### A.8.2 The other manifolds in our system (all of them, and they are everywhere) + +| Manifold | The query it answers | Why it matters | +|---|---|---| +| **Trade-path** (`TradePathState`: mae/mfe/time_in_loss/recovery_velocity/…) | *"Have I seen a trade shaped like this before, and what happened to it?"* | This **is** the ADVSL question, and the TP_FLOOR question, and the MAX_HOLD question. Path-aware SL/TP becomes a manifold query instead of a threshold ladder. The 375-branch upside study is this, done geometrically. | +| **Exit-decision** (bars_held × pnl × cascade_count × imbalance × regime) | *"What did similar exits yield?"* | Directly answers the exit-mechanics characterization F5 is paying for right now. | +| **Asset** (the 10-dimension Asset Behavior DSL) | *"This new asset behaves like X"* → transfer its policy | **This is how full-universe picking (500 assets, the north star) becomes affordable.** You do not train 500 policies; you train the manifold and *locate* each asset on it. | +| **Counterparty mix** (the simplex) | *"Who is on the other side right now?"* | Mode 2's ecology prior. Note the different geometry (§A.4.3). | +| **Ops / incident** (disk, CH latency, scan cadence, HZ liveness, spool depth) | *"Does this look like 2026-06-22 before it broke?"* | An OOD alarm on the operational manifold. This is JIMINY/PINOCCHIO's actual job, stated properly: **fingerprint the system's own state and scream when it drifts somewhere it has never been.** Sparks, before the fire. | +| **Streak / wave** (win-loss amplitude, **phase**, vel_div regime) | *"Are our streaks in phase with the regime wave?"* | The operator's grail hypothesis (SHORT/LONG regime switch). **Requires the cyclic encoding of §A.7 to even be askable.** | + +### A.8.3 What actually generalizes (be precise — this matters) + +**What generalizes is the three-stage DISCIPLINE, not one metric.** + +Each manifold has its own geometry: Euclidean-ish (market state), simplex +(counterparty mix), cyclic (phase), categorical-mixed (asset taxonomy). A single +flat cosine over all of them is exactly the error §A.4.3 warns about. But the +*discipline* — **retrieve fast and blind → gate on magnitude and support → model +locally with variance → say OUT_OF_DISTRIBUTION when you don't know** — is +invariant across every one of them. + +So the shared artifact is a **library primitive**, not a shared metric: + +``` +ManifoldQuery( + transform: StateVector -> (direction, magnitude) # per-manifold, versioned + retrieve: cosine | product-metric | simplex-metric # per-manifold geometry + gate: magnitude-envelope + support + Mahalanobis + model: local tangent-space fit | GP -> (prediction, variance) +) -> Recommendation | OUT_OF_DISTRIBUTION +``` + +One geometry engine; many manifolds. MALKHUT queries it for policy selection. +MARAS queries it for regime fingerprint. ADVSL queries it for path outcome. The +asset store queries it for transfer. Ops queries it for incident prefiguration. + +**The single most important thing that generalizes is the failure mode.** In every +one of these manifolds, the fatal error is identical: *confident interpolation into +unexplored magnitude.* A crisis market, an unprecedented trade path, a novel asset, +a system state that has never occurred. All of them present as "familiar direction, +unfamiliar magnitude." All of them are the cosine-similarity-1.0 trap. **One guard +defends all of them**, and it is the magnitude/support gate of stage 2. + +That is why this is worth building once, properly, as a shared kernel. + +### A.8.4 A name, offered (the operator names things; I only propose) + +The house speaks Kabbalah: MALKHUT is *kingdom* — the lowest sefirah, the +**ground**, where everything finally touches the world. Fitting, for execution. + +The manifold-query engine is not a *thing that acts*; it is the faculty by which +the system **knows where it is**. In the tree that is **Da'at** — *knowledge* — the +hidden sefirah that is not counted among the ten, the one that emerges from the +others and unifies intellect with action. It is precisely the *knowing* that stands +between understanding and doing. + +**DA'AT** — the manifold kernel. It knows where we are, and — more valuable — it +knows when it does not. + +--- + +## A.9 Order of work (slots into §10) + +| # | Task | Gate | +|---|---|---| +| A1 | **Cyclic encoding** of all phase coordinates `(sin, cos)` | Free, unconditional, and a prerequisite for the streak-phase study | +| A2 | **Transform + versioning**: rank/quantile → whiten → PCA; emit `metric_version` | A metric change invalidates the manifold; test that it says so | +| A3 | **Store `direction ⊕ magnitude` separately** per manifold cell | The whole crash-safety story is this one storage decision | +| A4 | **Stage 1 cosine retrieval** (matmul, GPU) | Measured µs at 10⁸ cells; deterministic ×2 | +| A5 | **Stage 2 gate**: magnitude envelope + support + Mahalanobis → `OUT_OF_DISTRIBUTION` | **Mutation litmus: feed a same-direction-10×-magnitude state; the system MUST refuse. If it recommends, the guard is not wired.** | +| A6 | **Stage 3 local model** (tangent-space fit or GP) → prediction + variance | Variance must grow away from data (test it) | +| A7 | **Risk gate honours OOD** → doctrinal fallback policy | Test: novel regime ⇒ no confident recommendation, ever | +| A8 | Component detection (cheap clustering); refuse cross-component interpolation | | +| A9 | **IFF EARNED**: persistent homology / mapper, offline, Mode 1, as a *gate-shaping diagnostic* | Only after A6's variance shows structured failure | +| A10 | Generalize to a shared kernel (DA'AT); second consumer = MARAS fingerprint | One engine, two manifolds, before claiming generality | + +--- + +## A.10 The one-sentence summary + +> **Cosine is the reflex; the gate is the judgment; the local model is the answer — +> and when the live world is a familiar shape at an unfamiliar size, the only +> honest output is "I do not know," because that is exactly the moment the market +> is trying to kill you.** + +--- + +*Annex A written by **Claude (Opus 4.8)**, on Fable's watch, at HJ's order, +2026-07-13. The cosine proposal is the operator's; the three-stage decomposition, +the magnitude-blindness argument, the topology IFF-discipline, and the DA'AT +generalization are mine, and I stand behind them. Where I have asserted a +mathematical property (locally-Euclidean tangent spaces; GP variance growth away +from data; cosine's magnitude invariance) it is standard and checkable. Where I +have asserted something about OUR system (that `scalar_hash` is a hash reaching for +a fingerprint; that `conflict_level` is a latent OOD signal) it is an inference from +the schemas I read tonight, and it is flagged as such rather than dressed as fact.* + +— Claude + +--- + *Written by Claude (Opus 4.8) on Fable's context, at HJ's order, 2026-07-13. Every path, line number, row count, and measured value herein was verified against the live system on the night of writing. Where a number was not verified, it says