hjnormey/sentiment-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(spec): ANNEX A — the manifold query (retrieve/gate/model) + DA'AT generalization

Browse Source 
Operator's cosine proposal: ADOPTED as stage 1 of 3, never alone.
- Stage 1 RETRIEVE: cosine on DIRECTION = matmul = the reflex (fast, exact, deterministic)
- Stage 2 GATE: magnitude envelope + support + Mahalanobis -> OUT_OF_DISTRIBUTION
- Stage 3 MODEL: local tangent-space/GP -> prediction + VARIANCE (the real extrapolation)

THE DANGER: crises preserve direction and explode magnitude. Cosine returns
similarity 1.0 for a same-shape-10x-size state — max confidence at the exact moment
it is most wrong. Cosine CANNOT produce the OOD verdict; storing direction+magnitude
separately is the entire crash-safety story.

Same law, third coat: INDETERMINATE (venue) / stale-price (exit) / OOD (manifold).
Unknown is not flat, not 'best guess'.

TOPOLOGY (operator's IFF): cyclic (sin,cos) encoding = free + REQUIRED for the
streak-phase grail study; component detection = cheap; persistent homology = EARN IT
(offline Mode-1 diagnostic that shapes the gate, never a per-tick op).

GENERALIZATION: three-stage discipline (not one metric) is a shared kernel — market
fingerprint (scalar_hash is a hash reaching for this; conflict_level is latent OOD),
trade-path/ADVSL, exit-decision, asset transfer (= how full-universe becomes
affordable), counterparty simplex, ops/incident prefiguration, streak-wave phase.
One guard defends all: confident interpolation into unexplored magnitude is the
universal failure mode. Proposed name: DA'AT.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Codex
2026-07-13 18:48:52 +02:00
parent 317e9d7a13
commit 9ae4daab5e
1 changed files with 368 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

368
prod/docs/SPEC_MALKHUT_ACTUALS_INTAKE.md
View File

@@ -459,6 +459,374 @@ venue-adapter boundary. Do not build two doorframes.
---
*[Annex A follows — the Mode-2 query mechanism, and its generalization.]*
---
---
# ANNEX A — THE MANIFOLD QUERY: three-stage search, and why it generalizes
**Author:** Claude (Opus 4.8), on Fable's watch. **Date:** 2026-07-13.
**Ordered by:** HJ ("could something akin to cosine distance over a manifold be
used… this might have dangers BUT has possibility of extrapolating across axes in
a very convenient, exact and almost reflexive way").
**Status:** DESIGN ANNEX to §0.5 (Mode 2 — RECOMMEND). Binding on any
implementation of manifold localization.
---
## A.1 The question
Mode 2 must take a **live market state** and find, inside the manifold Mode 1
built, the **closest-to-absolute-best play** — by *extrapolation across axes*, not
by lookup. The operator's proposal: **cosine distance** as the vehicle. Fast,
exact, reflexive; angle over a normalized state vector.
**Verdict: adopt it — as stage 1 of 3, never alone.** The reason is not
fastidiousness. Cosine used alone fails *specifically and catastrophically at the
moment we most need it to work*, and the failure is silent and maximally
confident. §A.3 is that argument; §A.2 is why the instinct is nonetheless right.
---
## A.2 Why cosine is the right REFLEX
| Property | Consequence for us |
|---|---|
| **Normalized cosine ≡ dot product ≡ matmul** | The entire manifold query is a BLAS/GPU operation. 10⁸ cells × 64-dim is microseconds. It fits inside the ≤25 ms Mode-2 budget with room to spare, and scales to the billions Mode 1 will produce. |
| **Exact, deterministic** | No approximate-ANN stochasticity. Seed-pin doctrine and Rule 9 (version everything, reproduce everything) survive intact. Same query → same answer, forever. |
| **Scale-invariance is sometimes SEMANTICALLY RIGHT** | A book at 2 bps / 100k depth and one at 4 bps / 200k can be *the same shape of market at different size*. Cosine captures shape, discards size. When shape is what matters, this is a feature. |
| **Degrades less badly than L2 in high dimension** | With 40+ sensors, Euclidean distances concentrate (everything equidistant). Angular similarity is the standard high-dim workaround; it is why every embedding retrieval system on earth uses it. |
The operator's word — **reflexive** — is exact. This is the right primitive for
*reflex*. It is the wrong primitive for *judgment*.
---
## A.3 THE DANGER (the one that matters): cosine is blind to magnitude, and
## magnitude is where the crisis lives
Take a state vector pointing: *wide spread, thin depth, high toxicity, high
latency, imbalance skewed*. Under ordinary stress it points one way.
Under a **liquidation cascade** it points **the same way** — just ten times
further out.
**Cosine similarity between them: 1.0.** A perfect match. Maximum confidence. And
the policy it retrieves was fitted on the mild version.
That is not a corner case; **that is the crash.** Market crises characteristically
*preserve direction and explode magnitude*. Every component moves the way it
always moves under stress — just far beyond anything in the training set. Cosine's
one trick is discarding exactly the coordinate that distinguishes "a bad Tuesday"
from "the day the book vanished."
Three consequences, stated as law:
> **A.3.1 — Cosine CANNOT produce the `OUT_OF_DISTRIBUTION` verdict.** It is
> structurally incapable. It returns similarity 1.0 for the single most dangerous
> input the system will ever see. Any design that derives OOD from cosine alone is
> not merely imperfect; it is inverted.
> **A.3.2 — Discarding magnitude violates CLASS-X law.** *Extreme magnitudes are a
> FEATURE channel, never filtered.* Cosine filters magnitude by construction. It
> must therefore be paired with an explicit magnitude coordinate or gate, or it is
> in direct breach of a standing doctrine (see: 450-second latency tail; §5).
> **A.3.3 — Confident-and-wrong beats uncertain-and-right at destroying capital.**
> A system that says "I don't know" in a crisis loses an opportunity. A system that
> says "I know this perfectly" in a crisis loses the account.
---
## A.4 Three more edges (real, but survivable)
**A.4.1 — The metric does all the work; cosine is only the last cheap step.**
On raw features the angle is meaningless: `depth_1pct_usd` ~10⁶, `imbalance` ∈
[-1,1], `funding_rate` ~10⁻⁴. Whichever feature has the largest raw numbers *owns*
the angle. Required before any cosine: a **rank/quantile transform** (NOT z-score —
our distributions are fat-tailed and non-Gaussian; a Gaussian assumption is a lie
we would then be optimizing against), followed by PCA/whitening onto the manifold's
own coordinates. **That transform IS the model.** Version it (`metric_version`),
pin it, and treat any change to it as a change to the science — every manifold cell
built under a prior transform is invalidated.
**A.4.2 — Cosine is a chordal distance in the ambient space, not a geodesic on the
manifold.** The Swiss-roll failure: two states near in angle can be far apart
*along the manifold*, on opposite sides of a fold. Policy performance in
toxicity/latency is exactly the sort of surface that has cliffs. Near-in-angle,
far-in-support, opposite side of a cliff = a confidently wrong recommendation.
Mitigation: stage 2's support gate, plus §A.7's topology diagnostic where the fold
structure is real.
**A.4.3 — Heterogeneous blocks want different geometries.** Market state is
Euclidean-ish. **Counterparty mix is a simplex** (a probability distribution over
agent types — its natural metric is Hellinger or KL, *not* cosine). Inventory /
position state is something else again. One flat cosine over a concatenated vector
silently asserts they share a geometry; they do not. Use a **product metric** with
per-block treatment, and weight the blocks explicitly (the weights are
hyperparameters — fit them, do not guess them).
---
## A.5 THE ARCHITECTURE: RETRIEVE → GATE → MODEL
The operator's instinct survives intact, relocated to its correct stage.
```
live MarketWorldState (S1 book, S2 funding/dvol, S3 regime,
S4 latency, account, intent)
│
▼
┌───────────────────────────────────────────────────────┐
│ TRANSFORM (metric_version) │
│ rank/quantile → whiten → PCA to manifold coords │
│ cyclic coords → (sin, cos) pairs [see A.7] │
│ split: DIRECTION (unit vec) ⊕ MAGNITUDE (scalar) │
└───────────────────────────┬───────────────────────────┘
│
┌───────────────────────────▼───────────────────────────┐
│ STAGE 1 — RETRIEVE "the reflex" ~µs │
│ cosine (= dot product = matmul) on DIRECTION only │
│ over the Mode-1 manifold → candidate neighbourhood │
│ GPU/BLAS. Exact. Deterministic. Billions of cells OK. │
│ ◀── THIS IS THE OPERATOR'S PROPOSAL, KEPT ──▶ │
└───────────────────────────┬───────────────────────────┘
│ k candidate cells
┌───────────────────────────▼───────────────────────────┐
│ STAGE 2 — GATE "do I actually know this?" │
│ (a) MAGNITUDE: is live |v| inside the explored │
│ magnitude interval FOR THIS DIRECTION-CELL? │
│ (b) SUPPORT: support_count ≥ min_support? │
│ (c) MAHALANOBIS: distance to the explored distribution│
│ (= cosine with the covariance baked in — the │
│ principled form of the same instinct) │
│ FAIL ANY ⇒ RecommendationConfidence.OUT_OF_DISTRIBUTION│
│ ⇒ NO recommendation. Doctrinal fallback. │
│ ◀── THIS IS THE GUARD COSINE CANNOT PROVIDE ──▶ │
└───────────────────────────┬───────────────────────────┘
│ in-distribution neighbourhood
┌───────────────────────────▼───────────────────────────┐
│ STAGE 3 — MODEL "the actual extrapolation" │
│ local linear/quadratic response surface (tangent │
│ space) OR Gaussian process over the neighbourhood │
│ → predicted outcome + PREDICTIVE VARIANCE │
│ A manifold is BY DEFINITION locally Euclidean: the │
│ tangent-space linear model is the mathematically │
│ correct way to "extrapolate across axes" — which is │
│ precisely what the operator described. │
│ GP bonus: variance GROWS as you leave the data → │
│ a second, independent OOD signal, for free. │
└───────────────────────────┬───────────────────────────┘
▼
RECOMMENDATION {policy, expected outcome,
variance, confidence, neighbours it
interpolates between, book_source,
metric_version, envelope status}
│
▼
RISK GATE (hard veto)
```
**Why this ordering is not negotiable:** stage 1 is fast and blind; stage 2 is
cheap and sighted; stage 3 is expensive and honest. Running stage 3 on everything
is unaffordable; running stage 1 alone is how the account dies. The cost profile
and the safety profile happen to agree — which is usually the sign of a correct
decomposition.
### Storage consequence (do this, it is cheap and it is the whole safety story)
**Store the manifold as `direction ⊕ magnitude`, separately, per cell.** Then:
- cosine retrieval on direction = matmul (fast path preserved, exactly as the
operator wants);
- magnitude becomes a **scalar interval check** — trivially cheap, and it is the
entire crash-safety mechanism.
95% of the speed, 100% of the guard. There is no tension here to resolve.
---
## A.6 The law this creates (and it is the same law, a third time)
| Layer | "I found something" | "But do I *know* it?" | Verdict when unknown |
|---|---|---|---|
| **Venue** (`bdc54fb`) | HTTP call returned/failed | Was the effect provably absent? | `INDETERMINATE` → **never roll back** |
| **Exit** (F5, tonight) | Price crossed a threshold | Is the price fresh? | stale → **skip, don't act on a fossil** |
| **Manifold** (this annex) | Cosine ≈ 1.0 | Is the live magnitude inside the explored envelope? | `OUT_OF_DISTRIBUTION` → **no recommendation; doctrinal fallback** |
**Unknown is not flat. Unknown is not "best guess." Unknown is unknown.**
Three subsystems, three coats, one law. That is not a coincidence — it is what a
correct architecture looks like from three angles.
---
## A.7 The topology question (the operator's "IFF IFF IFF")
**When does topology genuinely matter, versus merely being seductive?**
Topology earns its cost only when the manifold's **global** structure defeats
**local** metrics. Four cases where it truly does:
| Structure | Where it bites us, concretely | Cost | Verdict |
|---|---|---|---|
| **Cyclicity** | Phase coordinates: hour-of-day, session, funding cycle, day-of-week — and **the operator's win/loss-streak wave phase vs vel_div regimes**. A circle is not a line: 23:59 and 00:01 are *adjacent*, and a flat embedding puts them maximally far apart. Cosine at the seam is catastrophically wrong. | **~zero** | **DO IT NOW.** Encode every cyclic coordinate as a `(sin θ, cos θ)` pair. This is the cheapest correct thing in the entire document and it is *required* for the streak-phase study to mean anything. |
| **Disconnected components** | Genuinely separate basins — normal market vs. halted / limit-up / liquidity-hole. Interpolating *between* components is meaningless; the straight line passes through states the market cannot occupy. | LOW | **DO IT.** Cluster the manifold (offline, Mode 1); refuse to interpolate across component boundaries. Stage 2 gate extension. |
| **Holes / voids** | Regions the market *cannot* enter (negative spread, arbitrage-forbidden configurations). A linear extrapolation across a hole predicts an impossible world — with confidence. | MEDIUM | **IFF** stage-3 variance shows structured failure. Detect offline with persistent homology / the mapper algorithm. |
| **Folds / cliffs** | The Swiss-roll (§A.4.2): chordal-near, geodesic-far. Policy performance cliffs in toxicity/latency. | MEDIUM–HIGH | **IFF** the support gate keeps admitting neighbourhoods whose stage-3 fits are bimodal or high-variance — that is the signature of a fold. |
**The discipline for topology (this is the "IFF"):**
1. **Cyclic encoding: unconditional.** Free, and its absence is a *bug*, not a
simplification.
2. **Component detection: cheap, do it.** Clustering on the Mode-1 manifold.
3. **Persistent homology / mapper: EARN IT.** These are expensive, notoriously
noise-fooled, and seductive precisely because they produce beautiful pictures.
Run them **offline, in Mode 1 only, as a diagnostic of the manifold** — never
as a per-tick Mode-2 operation. And run them only **after** stage 3's variance
has *demonstrated* that the local model is failing in a structured way. You earn
topology by first proving the simple thing breaks.
4. What TDA is *for*, when earned: it tells you **where the folds and holes are**,
so that **stage 2's gate can be shaped correctly**. Topology's product is a
better *gate*, not a better *answer*. That is the whole of it.
---
## A.8 THE GENERALIZATION (the operator's PS — and I think this is the best idea
## in the document)
**Yes. Emphatically. The three-stage manifold query is a general primitive, and
"fingerprinting markets" is its most valuable instance.**
### A.8.1 Market fingerprinting — the natural home
MARAS today emits a **discrete label** (`regime` ∈ {BEARISH, CHOPPY_BEARISH,
CHOPPY, SIDEWAYS, CHOPPY_BULLISH}) plus `confidence`, `conflict_level`, and five
`tier_*` scores. That is a **classifier**: it puts a continuous world into five
boxes.
The manifold view is strictly richer: **the market state is a point, and regimes
are regions — not boxes.** Then:
- The **fingerprint** = `direction` (the *shape* of the market) ⊕ `magnitude` (its
*intensity*) ⊕ its position on the manifold. A regime label becomes a
*coordinate*, not a category. "CHOPPY, but 3σ out along the toxicity axis, near
the boundary with CHOPPY_BEARISH" is a sentence the current system cannot say
and desperately needs to.
- Two observations already point straight at this:
- **`maras_fingerprint.scalar_hash` (UInt16) is already a fingerprint attempt —
but a HASH.** A hash has no neighbourhood: near-identical markets get unrelated
hashes. The cosine-manifold fingerprint is precisely its **continuous
generalization** — *nearest*-fingerprint instead of *exact*-hash. This looks to
me like the thing that hash was reaching for.
- **`conflict_level` (the five tiers disagreeing) is already an implicit OOD
signal.** It maps directly onto stage 2. Tier disagreement = "this market is
not cleanly inside any explored region." That is *free* OOD evidence we are
currently discarding into a float.
- And eigenscan is *already* a manifold construction: eigendecomposition of the
correlation structure **is** the coordinate system. The operator is an eigenscan
surfer; he has been navigating this manifold by hand for years. The three-stage
query is the machine that surfs with him.
### A.8.2 The other manifolds in our system (all of them, and they are everywhere)
| Manifold | The query it answers | Why it matters |
|---|---|---|
| **Trade-path** (`TradePathState`: mae/mfe/time_in_loss/recovery_velocity/…) | *"Have I seen a trade shaped like this before, and what happened to it?"* | This **is** the ADVSL question, and the TP_FLOOR question, and the MAX_HOLD question. Path-aware SL/TP becomes a manifold query instead of a threshold ladder. The 375-branch upside study is this, done geometrically. |
| **Exit-decision** (bars_held × pnl × cascade_count × imbalance × regime) | *"What did similar exits yield?"* | Directly answers the exit-mechanics characterization F5 is paying for right now. |
| **Asset** (the 10-dimension Asset Behavior DSL) | *"This new asset behaves like X"* → transfer its policy | **This is how full-universe picking (500 assets, the north star) becomes affordable.** You do not train 500 policies; you train the manifold and *locate* each asset on it. |
| **Counterparty mix** (the simplex) | *"Who is on the other side right now?"* | Mode 2's ecology prior. Note the different geometry (§A.4.3). |
| **Ops / incident** (disk, CH latency, scan cadence, HZ liveness, spool depth) | *"Does this look like 2026-06-22 before it broke?"* | An OOD alarm on the operational manifold. This is JIMINY/PINOCCHIO's actual job, stated properly: **fingerprint the system's own state and scream when it drifts somewhere it has never been.** Sparks, before the fire. |
| **Streak / wave** (win-loss amplitude, **phase**, vel_div regime) | *"Are our streaks in phase with the regime wave?"* | The operator's grail hypothesis (SHORT/LONG regime switch). **Requires the cyclic encoding of §A.7 to even be askable.** |
### A.8.3 What actually generalizes (be precise — this matters)
**What generalizes is the three-stage DISCIPLINE, not one metric.**
Each manifold has its own geometry: Euclidean-ish (market state), simplex
(counterparty mix), cyclic (phase), categorical-mixed (asset taxonomy). A single
flat cosine over all of them is exactly the error §A.4.3 warns about. But the
*discipline* — **retrieve fast and blind → gate on magnitude and support → model
locally with variance → say OUT_OF_DISTRIBUTION when you don't know** — is
invariant across every one of them.
So the shared artifact is a **library primitive**, not a shared metric:
```
ManifoldQuery(
transform: StateVector -> (direction, magnitude) # per-manifold, versioned
retrieve: cosine | product-metric | simplex-metric # per-manifold geometry
gate: magnitude-envelope + support + Mahalanobis
model: local tangent-space fit | GP -> (prediction, variance)
) -> Recommendation | OUT_OF_DISTRIBUTION
```
One geometry engine; many manifolds. MALKHUT queries it for policy selection.
MARAS queries it for regime fingerprint. ADVSL queries it for path outcome. The
asset store queries it for transfer. Ops queries it for incident prefiguration.
**The single most important thing that generalizes is the failure mode.** In every
one of these manifolds, the fatal error is identical: *confident interpolation into
unexplored magnitude.* A crisis market, an unprecedented trade path, a novel asset,
a system state that has never occurred. All of them present as "familiar direction,
unfamiliar magnitude." All of them are the cosine-similarity-1.0 trap. **One guard
defends all of them**, and it is the magnitude/support gate of stage 2.
That is why this is worth building once, properly, as a shared kernel.
### A.8.4 A name, offered (the operator names things; I only propose)
The house speaks Kabbalah: MALKHUT is *kingdom* — the lowest sefirah, the
**ground**, where everything finally touches the world. Fitting, for execution.
The manifold-query engine is not a *thing that acts*; it is the faculty by which
the system **knows where it is**. In the tree that is **Da'at** — *knowledge* — the
hidden sefirah that is not counted among the ten, the one that emerges from the
others and unifies intellect with action. It is precisely the *knowing* that stands
between understanding and doing.
**DA'AT** — the manifold kernel. It knows where we are, and — more valuable — it
knows when it does not.
---
## A.9 Order of work (slots into §10)
| # | Task | Gate |
|---|---|---|
| A1 | **Cyclic encoding** of all phase coordinates `(sin, cos)` | Free, unconditional, and a prerequisite for the streak-phase study |
| A2 | **Transform + versioning**: rank/quantile → whiten → PCA; emit `metric_version` | A metric change invalidates the manifold; test that it says so |
| A3 | **Store `direction ⊕ magnitude` separately** per manifold cell | The whole crash-safety story is this one storage decision |
| A4 | **Stage 1 cosine retrieval** (matmul, GPU) | Measured µs at 10⁸ cells; deterministic ×2 |
| A5 | **Stage 2 gate**: magnitude envelope + support + Mahalanobis → `OUT_OF_DISTRIBUTION` | **Mutation litmus: feed a same-direction-10×-magnitude state; the system MUST refuse. If it recommends, the guard is not wired.** |
| A6 | **Stage 3 local model** (tangent-space fit or GP) → prediction + variance | Variance must grow away from data (test it) |
| A7 | **Risk gate honours OOD** → doctrinal fallback policy | Test: novel regime ⇒ no confident recommendation, ever |
| A8 | Component detection (cheap clustering); refuse cross-component interpolation | |
| A9 | **IFF EARNED**: persistent homology / mapper, offline, Mode 1, as a *gate-shaping diagnostic* | Only after A6's variance shows structured failure |
| A10 | Generalize to a shared kernel (DA'AT); second consumer = MARAS fingerprint | One engine, two manifolds, before claiming generality |
---
## A.10 The one-sentence summary
> **Cosine is the reflex; the gate is the judgment; the local model is the answer —
> and when the live world is a familiar shape at an unfamiliar size, the only
> honest output is "I do not know," because that is exactly the moment the market
> is trying to kill you.**
---
*Annex A written by **Claude (Opus 4.8)**, on Fable's watch, at HJ's order,
2026-07-13. The cosine proposal is the operator's; the three-stage decomposition,
the magnitude-blindness argument, the topology IFF-discipline, and the DA'AT
generalization are mine, and I stand behind them. Where I have asserted a
mathematical property (locally-Euclidean tangent spaces; GP variance growth away
from data; cosine's magnitude invariance) it is standard and checkable. Where I
have asserted something about OUR system (that `scalar_hash` is a hash reaching for
a fingerprint; that `conflict_level` is a latent OOD signal) it is an inference from
the schemas I read tonight, and it is flagged as such rather than dressed as fact.*
— Claude
---
*Written by Claude (Opus 4.8) on Fable's context, at HJ's order, 2026-07-13.
Every path, line number, row count, and measured value herein was verified against
the live system on the night of writing. Where a number was not verified, it says