malkhut(assets)+uv: system-wide asset directory + UV universe init

MALKHUT asset directory (malkhut/assets/): normalized canonical symbols,
KNOWN_EXCHANGES aux table (BINANCE/BINGX/BINGX_VST), per-exchange listing
status (TRADING/OFFLINE/UNKNOWN), JSON-backed, built for full-Binance-500
scale. Seeded with BLUE's NG7 feed universe (50 symbols) + live VST
contracts probe: 35 TRADING / 15 OFFLINE on VST (BAND, CELR, COS, CVC,
DENT, FUN, HOT, ICX, TFUEL, TUSD, USDC, WAN, WIN, XTZ, ZIL).
uv_asset_universe.init_asset_universe() = runtime tradable set for the
execution exchange; --onboard runs MALKHUT AssetCompiler for full taxonomy.
12 new tests, mutation-RED verified (status-filter + tradability guard).
This commit is contained in:
Codex
2026-07-11 21:48:05 +02:00
parent aaaf326abf
commit a062507656
7 changed files with 1687 additions and 0 deletions

View File

@@ -0,0 +1,73 @@
"""
Tests for uv_asset_universe — real classification kernel, fake CH/venue seams.
Mutation-sensitivity: flipping contract_is_tradable's status comparison, or
refresh_vst_listings' present/absent branch, goes RED here.
"""
from __future__ import annotations
import pytest
from uv_asset_universe import ( # noqa: I001 — wires MALKHUT_ROOT onto sys.path
contract_is_tradable,
import_blue_universe,
init_asset_universe,
refresh_vst_listings,
)
from malkhut.assets.directory import AssetDirectory, ListingStatus
def _dir(tmp_path):
return AssetDirectory(tmp_path / "assets.json")
CONTRACTS = [
{"symbol": "ETH-USDT", "status": 1, "apiStateOpen": "true"},
{"symbol": "TRX-USDT", "status": 1, "apiStateOpen": "true"},
{"symbol": "OLD-USDT", "status": 25, "apiStateOpen": "true"}, # delisted
{"symbol": "SHUT-USDT", "status": 1, "apiStateOpen": "false"}, # API entry closed
# BAND-USDT absent entirely (observed on VST 2026-07-11)
]
def test_contract_is_tradable():
assert contract_is_tradable(CONTRACTS[0]) is True
assert contract_is_tradable(CONTRACTS[2]) is False # status 25
assert contract_is_tradable(CONTRACTS[3]) is False # apiStateOpen false
def test_seed_and_refresh_marks_offline_and_absent(tmp_path):
d = _dir(tmp_path)
import_blue_universe(d, ["ETHUSDT", "TRXUSDT", "BANDUSDT", "OLDUSDT", "SHUTUSDT"])
assert d.symbols_for_exchange("BINANCE") == [
"BANDUSDT", "ETHUSDT", "OLDUSDT", "SHUTUSDT", "TRXUSDT"]
trading, offline = refresh_vst_listings(d, CONTRACTS)
assert (trading, offline) == (2, 3)
assert d.symbols_for_exchange("BINGX_VST") == ["ETHUSDT", "TRXUSDT"]
assert d.symbols_for_exchange("BINGX_VST", status=ListingStatus.OFFLINE) == [
"BANDUSDT", "OLDUSDT", "SHUTUSDT"]
# venue spelling preserved for tradable symbols
assert d.venue_symbol("ETHUSDT", "BINGX_VST") == "ETH-USDT"
def test_init_asset_universe_is_the_tradable_frozenset(tmp_path):
d = _dir(tmp_path)
import_blue_universe(d, ["ETHUSDT", "BANDUSDT"])
refresh_vst_listings(d, CONTRACTS)
uni = init_asset_universe("BINGX_VST", d)
assert uni == frozenset({"ETHUSDT"})
assert isinstance(uni, frozenset)
# the Binance view is unaffected by the VST stamp
assert init_asset_universe("BINANCE", d) == frozenset({"ETHUSDT", "BANDUSDT"})
def test_refresh_is_repeatable_and_recovers(tmp_path):
"""A symbol OFFLINE once must flip back to TRADING when the venue relists."""
d = _dir(tmp_path)
import_blue_universe(d, ["BANDUSDT"])
refresh_vst_listings(d, CONTRACTS) # BAND absent -> OFFLINE
assert d.symbols_for_exchange("BINGX_VST") == []
relisted = CONTRACTS + [{"symbol": "BAND-USDT", "status": 1, "apiStateOpen": "true"}]
refresh_vst_listings(d, relisted)
assert d.symbols_for_exchange("BINGX_VST") == ["BANDUSDT"]

View File

@@ -0,0 +1,203 @@
"""
UV asset-universe init — MALKHUT asset directory as the single source of truth.
Operator directive 2026-07-11: the MALKHUT asset store (malkhut/assets/) is
the system-wide known-asset universe. UV/VIOLET initialize their tradable
set from it against the CURRENT execution exchange, so a dead venue symbol
(e.g. BAND-USDT offline on VST) can never reach the kernel.
Flow:
1. import_blue_universe() — seed canonical symbols from the DOLPHIN NG7
scan feed (BINANCE listings; that feed IS the
Binance universe BLUE trades from).
2. refresh_vst_listings() — probe the execution venue's contracts endpoint
and stamp BINGX_VST TRADING/OFFLINE per symbol.
3. init_asset_universe() — the runtime call: returns the frozen tradable
set for the execution exchange. The promotion
bridge / seam refuses ENTER intents outside it.
Built for the full-Binance (~500+) universe north-star; today's feed carries
50 symbols — the store scales, the feed is the bottleneck.
CLI:
uv_asset_universe.py --seed-from-scans --refresh-vst --show
"""
from __future__ import annotations
import argparse
import json
import os
import sys
import urllib.request
from datetime import datetime, timezone
from typing import Any, Iterable
MALKHUT_ROOT = os.environ.get("MALKHUT_ROOT", "/mnt/dolphinng5_predict/MALKHUT")
if MALKHUT_ROOT not in sys.path:
sys.path.insert(0, MALKHUT_ROOT)
from malkhut.assets.directory import ( # noqa: E402
AssetDirectory,
ListingStatus,
normalize_symbol,
)
from uv_anomaly_sentinel import CHQuerier, CH_DB_UV, VST_BASE_URL # noqa: E402
FEED_SOURCE = "dolphin_ng7_scan_feed"
PROBE_SOURCE = "vst_contracts_probe"
def _now_iso() -> str:
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
# ── step 1: seed from the scan feed (the BLUE/Binance universe) ────────────
def blue_universe_from_scans(ch: CHQuerier | None = None) -> list[str]:
ch = ch or CHQuerier()
rows = ch.query(
f"SELECT DISTINCT arrayJoin(JSONExtract(assets,'Array(String)')) AS a"
f" FROM {CH_DB_UV}.prime_scans ORDER BY a FORMAT JSONEachRow"
)
return [r["a"] for r in rows]
def import_blue_universe(directory: AssetDirectory, symbols: Iterable[str]) -> int:
return directory.import_symbols(
[normalize_symbol(s) for s in symbols],
"BINANCE",
status=ListingStatus.TRADING,
checked_at=_now_iso(),
source=FEED_SOURCE,
)
# ── step 2: stamp execution-venue listings from the contracts probe ───────
def fetch_vst_contracts(base_url: str = VST_BASE_URL, timeout_s: int = 15) -> list[dict[str, Any]]:
req = urllib.request.Request(f"{base_url}/openApi/swap/v2/quote/contracts")
body = urllib.request.urlopen(req, timeout=timeout_s).read().decode("utf-8")
parsed = json.loads(body)
return parsed.get("data") or []
def contract_is_tradable(contract: dict[str, Any]) -> bool:
"""BingX contracts row -> tradable? status 1 = online; apiStateOpen gates API entry."""
return (
int(contract.get("status") or 0) == 1
and str(contract.get("apiStateOpen", "true")).lower() == "true"
)
def refresh_vst_listings(
directory: AssetDirectory, contracts: list[dict[str, Any]]
) -> tuple[int, int]:
"""Stamp BINGX_VST status for every symbol already in the directory.
Present + tradable -> TRADING; absent or non-tradable -> OFFLINE.
Returns (trading, offline) counts over the directory.
"""
now = _now_iso()
tradable: dict[str, str] = {} # canonical -> venue symbol
for c in contracts:
vsym = str(c.get("symbol") or "")
if vsym and contract_is_tradable(c):
tradable[normalize_symbol(vsym)] = vsym
n_trading = n_offline = 0
for sym in list(directory.records):
if sym in tradable:
directory.set_listing(sym, "BINGX_VST", venue_symbol=tradable[sym],
status=ListingStatus.TRADING,
checked_at=now, source=PROBE_SOURCE)
n_trading += 1
else:
directory.set_listing(sym, "BINGX_VST",
status=ListingStatus.OFFLINE,
checked_at=now, source=PROBE_SOURCE)
n_offline += 1
return n_trading, n_offline
# ── step 2b: full-taxonomy onboarding via MALKHUT's AssetCompiler ──────────
def onboard_profiles(directory: AssetDirectory, *, compiler: Any = None,
out_path: str = "") -> tuple[int, int]:
"""Run MALKHUT's asset onboarder over every directory symbol lacking a
profile_ref; persist compiled taxonomy JSON beside the directory.
Returns (compiled, failed). Failures are noted on the record, not fatal —
the directory stays authoritative for existence/listing even when the
taxonomy fetch fails.
"""
from dataclasses import asdict
if compiler is None:
from malkhut.training.asset_compiler import AssetCompiler
compiler = AssetCompiler()
path = out_path or str(directory.path.parent / "compiled_profiles.json")
try:
with open(path, encoding="utf-8") as fh:
store = json.load(fh)
except (OSError, json.JSONDecodeError):
store = {}
done = failed = 0
for sym in sorted(directory.records):
rec = directory.records[sym]
if rec.profile_ref and sym in store:
continue
result = compiler.compile_and_register(sym)
if result.asset_profile is None:
failed += 1
rec.notes = (rec.notes + " | " if rec.notes else "") + \
f"onboard failed {_now_iso()}: {'; '.join(result.warnings)[:120]}"
continue
entry = {"profile": asdict(result.asset_profile), "_compiled_at": _now_iso()}
if result.asset_behavior is not None:
entry["behavior"] = asdict(result.asset_behavior)
store[sym] = json.loads(json.dumps(entry, default=str))
rec.profile_ref = sym
done += 1
with open(path, "w", encoding="utf-8") as fh:
json.dump(store, fh, indent=1, sort_keys=True, default=str)
return done, failed
# ── step 3: the runtime call ───────────────────────────────────────────────
def init_asset_universe(execution_exchange: str = "BINGX_VST",
directory: AssetDirectory | None = None) -> frozenset[str]:
"""Tradable canonical symbols for the execution exchange in use."""
directory = directory or AssetDirectory()
return frozenset(directory.symbols_for_exchange(execution_exchange))
def main(argv: list[str] | None = None) -> int:
p = argparse.ArgumentParser(description="UV asset-universe init (MALKHUT store)")
p.add_argument("--seed-from-scans", action="store_true")
p.add_argument("--refresh-vst", action="store_true")
p.add_argument("--onboard", action="store_true",
help="full-taxonomy compile via MALKHUT AssetCompiler (rate-limited)")
p.add_argument("--show", action="store_true")
a = p.parse_args(argv)
d = AssetDirectory()
if a.seed_from_scans:
n = import_blue_universe(d, blue_universe_from_scans())
d.save()
print(f"seeded {n} BINANCE listings from {FEED_SOURCE} -> {d.path}")
if a.refresh_vst:
t, o = refresh_vst_listings(d, fetch_vst_contracts())
d.save()
print(f"BINGX_VST refresh: {t} TRADING, {o} OFFLINE")
if o:
print("OFFLINE:", ", ".join(d.symbols_for_exchange(
"BINGX_VST", status=ListingStatus.OFFLINE)))
if a.onboard:
done, failed = onboard_profiles(d)
d.save()
print(f"onboarded taxonomy: {done} compiled, {failed} failed")
if a.show:
uni = init_asset_universe("BINGX_VST", d)
print(f"directory: {len(d)} assets @ {d.path}")
print(f"BINGX_VST tradable universe ({len(uni)}):", ", ".join(sorted(uni)))
return 0
if __name__ == "__main__":
sys.exit(main())

88
prod/docs/POST_FINISH.md Normal file
View File

@@ -0,0 +1,88 @@
# POST_FINISH — the parking lot (parked with love, not killed)
**Standing rule (operator + Fable, 2026-07-10):** until the FINISH LINE is
crossed, new scope lands HERE, not in code. Fable answers new scope with
"parked" — even to the operator. Especially to the operator.
## THE FINISH LINE (frozen — only these three count)
1. **Parity RESIDUAL = 0** over N windows (codex dossier machinery,
`uv/parity-certification`)
2. **72h clean on VST** on the merged stack (`a2b0fac`+): no §4 anomalies —
no orphan orders, no journal gaps, tripwire quiet
3. **Operator gate:** small real capital, separate account, BLUE untouched
## SEQUENCED (above parked — operator 2026-07-11: "it is costing us money")
- **Smart OB exec: PINK ExecutionRouter integration** — FIRST task upon parity
(item 1) completion, before/alongside items 2–3. Asset exists certified
(265 tests incl. live VST E2E, maker_both ran production). Slots below the
kernel at the venue-adapter seam — parity-invisible. Economics: taker RT
~0.10% → maker-both ~0.04%; smoke-run day-1 flips breakeven → +0.9%
equal-weight. Research: 90%+ of orders fill as maker ("move to us").
**CRITICAL READING:** `prod/docs/BingX_FILL_CHARACTERIZATION_AND_ADVANTAGES.md`.
**SOA ADJUDICATION (first sub-task, Fable):** TWO exec assets exist —
(a) PINK ExecutionRouter: maker/taker POLICY, 265 tests, maker_both ran live;
(b) BLUE `alpha_engine/execution/` **SmartPlacer**: OB-aware PLACEMENT
(fill_simulator w/ 30s bookDepth, signal_confidence_adapter bridging Alpha
conviction → placement, ob_reader, smartplacer_constants). Likely
complementary (Router = maker-vs-taker; SmartPlacer = where-in-book).
Adjudicate router ∪ placer ∪ union against the fill-characterization doc
before integrating either.
## PARKED (specced where noted; zero build until finish)
- **IMPETUS build** — spec `IMPETUS_SPEC.md` + annex
`IMPETUS_SPEC_ANNEX_UV_INTEGRATION.md` (steel-clock plug, ASEx placement,
last-tick table). Paper done; build parked.
- **MALKHUT hose adaptation** — Fable's job when unparked (memory:
malkhut-hose-adaptation-fable)
- **JIMINY / PINOCCHIO** — ASEx fleet scoreboard + false-green detector
- **TP/SL cadence ladder L2/L3** — pulse-host tick path, Rust reflex engine
(spec `UV_TPSL_ENGINE_AND_REMAINING_DEV_SPEC.md` §1)
- **305M ring upgrade** — producer-side cached index in `uv_steel::ring`
(PASS9, ~10 lines + bench)
- **wCQ wait-free MPMC** — L3 mailbox candidate (`305MM_OPS_ALGO_ANNEX_FIFO.md`)
- **★ IMPETUS WIRE-ATOMIC (operator 2026-07-11, "insane, write it down")** —
exchange-wire→zinc-shm ATOMIC: the venue packet's tick becomes visible to
shm readers as ONE indivisible act, no intermediating copies/queues/threads.
Concretely: kernel-bypass ingest (AF_XDP / DPDK / io_uring registered
buffers — NIC DMAs frames into user memory), parse-in-place off the wire
buffer, single seqlock commit into `impetus_last` as the SAME pinned
thread's next instruction. Latency floor becomes NIC+parse (~µs), and the
atomicity contract extends end-to-end: a reader can never observe a tick
the wire didn't deliver, nor miss one it did (A2/A5 doctrine stretched to
the physical boundary). Prereqs: IMPETUS v1 shipped; WS/stream adapters
(annex R2); pinned-emit thread (annex R4); flat-C last-tick table (R3) —
each is a rung TOWARD this. TLS termination path (operator follow-up: the
sought term was "homomorphic" — FHE itself is 1e6-1e9x too slow, wrong
universe): SOA answers preserving wire-atomicity, in preference order:
(1) NIC inline TLS RX offload (ConnectX/kTLS-offload — silicon decrypts
AES-GCM at wire speed BEFORE DMA; buffer lands already-plaintext; CPU never
sees ciphertext); (2) kernel kTLS (decrypt in-kernel, zero-copy up);
(3) VAES/AES-NI decrypt-IN-PLACE (plaintext overwrites ciphertext in the
same buffer, ~1-4 cycles/byte — "decoded where it lies").
- **`uv_steel` crate extraction** — pulse-host modules → shared workspace crate
- **PASS9 mass-test regeneration** — far-end assertions, no broken codegen
- **BLUE trade_events dedup DDL** — operator-blessed write into dolphin ns
- **Vendor reconciliation** — T0-DEVIATION port upstream + vendor_sync
--from-committed-HEAD hardening
- **UvExecRunner consolidation** — route legacy direct async path through the
ASEx owner or retire it (execution-gate dossier §6.6)
- **Journal EXIT-labeling cleanup** — no-entry scans mislabeled EXIT (cosmetic)
- **Fleet identity hygiene** — stale zellij names, fossil handles, split bus
- **★ SHARP TODO (operator 2026-07-11): 375-branch upside study** — realized
upside under favorable-extension vs realized loss; left-tail cure hypothesis
(losses huge incl. scan-cadence SL misses; winners cramped). Depends on:
MAX_HOLD full-modulation port + ingress journaling
(UV_EXIT_DOCTRINE_PARITY_DESIGN.md D1.2). OB-cascade cause tuning rides the
same data.
## Not parked (allowed work)
Anything that directly serves items 1–3, bug fixes on the live path, and
operator-ordered exceptions (which must say "unpark X" explicitly).
## DESIRABLE (north-star, operator 2026-07-11 — not parked, no build yet)
- **Full-universe picking**: system should someday select from the ENTIRE
Binance (~500-asset, as minimum reference) universe, not the 50-symbol
NG7 feed slice — research indicates full-universe selection improves
ROI/WR/joy. MALKHUT asset directory (malkhut/assets/) is built to that
scale; the feed is the current bottleneck, not the store.