malkhut(assets)+uv: system-wide asset directory + UV universe init
MALKHUT asset directory (malkhut/assets/): normalized canonical symbols, KNOWN_EXCHANGES aux table (BINANCE/BINGX/BINGX_VST), per-exchange listing status (TRADING/OFFLINE/UNKNOWN), JSON-backed, built for full-Binance-500 scale. Seeded with BLUE's NG7 feed universe (50 symbols) + live VST contracts probe: 35 TRADING / 15 OFFLINE on VST (BAND, CELR, COS, CVC, DENT, FUN, HOT, ICX, TFUEL, TUSD, USDC, WAN, WIN, XTZ, ZIL). uv_asset_universe.init_asset_universe() = runtime tradable set for the execution exchange; --onboard runs MALKHUT AssetCompiler for full taxonomy. 12 new tests, mutation-RED verified (status-filter + tradability guard).
This commit is contained in:
17
MALKHUT/malkhut/assets/__init__.py
Normal file
17
MALKHUT/malkhut/assets/__init__.py
Normal file
@@ -0,0 +1,17 @@
|
|||||||
|
from malkhut.assets.directory import (
|
||||||
|
KNOWN_EXCHANGES,
|
||||||
|
AssetDirectory,
|
||||||
|
AssetRecord,
|
||||||
|
ExchangeListing,
|
||||||
|
ListingStatus,
|
||||||
|
normalize_symbol,
|
||||||
|
)
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"KNOWN_EXCHANGES",
|
||||||
|
"AssetDirectory",
|
||||||
|
"AssetRecord",
|
||||||
|
"ExchangeListing",
|
||||||
|
"ListingStatus",
|
||||||
|
"normalize_symbol",
|
||||||
|
]
|
||||||
1010
MALKHUT/malkhut/assets/asset_directory.json
Normal file
1010
MALKHUT/malkhut/assets/asset_directory.json
Normal file
File diff suppressed because it is too large
Load Diff
206
MALKHUT/malkhut/assets/directory.py
Normal file
206
MALKHUT/malkhut/assets/directory.py
Normal file
@@ -0,0 +1,206 @@
|
|||||||
|
"""
|
||||||
|
MALKHUT Asset Directory — the system-wide known-asset universe store.
|
||||||
|
|
||||||
|
This is the normalization layer UNDER the rich MALKHUT taxonomy
|
||||||
|
(`training/asset_classification.AssetProfile`): a record here says an asset
|
||||||
|
EXISTS, what its canonical symbol is, and WHICH exchanges list it — nothing
|
||||||
|
more. Most taxonomy fields stay empty until the AssetCompiler (or a human)
|
||||||
|
fills them; a record may link to a full AssetProfile when one exists.
|
||||||
|
|
||||||
|
Design (operator directive 2026-07-11):
|
||||||
|
- MALKHUT's asset store is the overall system asset store (BLUE / VIOLET /
|
||||||
|
UV consume it) — multi-exchange operation is the destination.
|
||||||
|
- `KNOWN_EXCHANGES` is the aux "known exchanges" table.
|
||||||
|
- Per-exchange listing carries the venue-local symbol and a TRADING /
|
||||||
|
OFFLINE / UNKNOWN status, so an execution layer can ask
|
||||||
|
`symbols_for_exchange("BINGX_VST")` and never submit a dead symbol.
|
||||||
|
- Storage is a JSON file (versioned, language-agnostic, GraalVM-safe,
|
||||||
|
no DB dependency). Path via $MALKHUT_ASSET_DIR_PATH or package-local
|
||||||
|
`asset_directory.json`.
|
||||||
|
|
||||||
|
Canonical symbol normalization: venue variants ("BAND-USDT", "band_usdt",
|
||||||
|
"BANDUSDT") all normalize to "BANDUSDT" (uppercase, separators stripped).
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
from dataclasses import dataclass, field, asdict
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Dict, Iterable, List, Optional
|
||||||
|
|
||||||
|
SCHEMA_VERSION = 1
|
||||||
|
|
||||||
|
# ─── Known exchanges (aux table) ───────────────────────────────────────────
|
||||||
|
# key → human description. BINGX_VST is deliberately separate from BINGX:
|
||||||
|
# the testnet universe differs from live (observed 2026-07-11: BAND/CELR
|
||||||
|
# offline on VST while live on Binance).
|
||||||
|
KNOWN_EXCHANGES: Dict[str, str] = {
|
||||||
|
"BINANCE": "Binance USDT-M futures/spot (DOLPHIN NG7 scan-feed universe)",
|
||||||
|
"BINGX": "BingX perpetual swap, live",
|
||||||
|
"BINGX_VST": "BingX perpetual swap, VST demo/testnet",
|
||||||
|
}
|
||||||
|
|
||||||
|
_STATUSES = ("TRADING", "OFFLINE", "UNKNOWN")
|
||||||
|
|
||||||
|
|
||||||
|
class ListingStatus:
|
||||||
|
TRADING = "TRADING"
|
||||||
|
OFFLINE = "OFFLINE"
|
||||||
|
UNKNOWN = "UNKNOWN"
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_symbol(symbol: str) -> str:
|
||||||
|
"""Canonical form: uppercase, separators stripped. 'BAND-USDT' -> 'BANDUSDT'."""
|
||||||
|
return symbol.replace("-", "").replace("_", "").replace("/", "").strip().upper()
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class ExchangeListing:
|
||||||
|
venue_symbol: str # symbol as the venue spells it
|
||||||
|
status: str = ListingStatus.UNKNOWN # TRADING / OFFLINE / UNKNOWN
|
||||||
|
last_checked: str = "" # ISO-8601 UTC, "" = never verified
|
||||||
|
source: str = "" # who asserted this (feed, API probe, human)
|
||||||
|
|
||||||
|
def __post_init__(self) -> None:
|
||||||
|
if self.status not in _STATUSES:
|
||||||
|
raise ValueError(f"unknown listing status {self.status!r}; use {_STATUSES}")
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class AssetRecord:
|
||||||
|
symbol: str # canonical (normalized)
|
||||||
|
base: str = "" # e.g. BAND
|
||||||
|
quote: str = "" # e.g. USDT
|
||||||
|
exchanges: Dict[str, ExchangeListing] = field(default_factory=dict)
|
||||||
|
profile_ref: str = "" # key into ASSET_PROFILES when taxonomy exists
|
||||||
|
notes: str = ""
|
||||||
|
|
||||||
|
def listed_on(self, exchange: str) -> bool:
|
||||||
|
lst = self.exchanges.get(exchange)
|
||||||
|
return lst is not None and lst.status == ListingStatus.TRADING
|
||||||
|
|
||||||
|
|
||||||
|
def _default_path() -> Path:
|
||||||
|
env = os.environ.get("MALKHUT_ASSET_DIR_PATH", "")
|
||||||
|
if env:
|
||||||
|
return Path(env)
|
||||||
|
return Path(__file__).resolve().parent / "asset_directory.json"
|
||||||
|
|
||||||
|
|
||||||
|
class AssetDirectory:
|
||||||
|
"""Load / mutate / persist the asset universe. All symbols canonical."""
|
||||||
|
|
||||||
|
def __init__(self, path: Optional[Path] = None) -> None:
|
||||||
|
self.path = Path(path) if path else _default_path()
|
||||||
|
self.records: Dict[str, AssetRecord] = {}
|
||||||
|
if self.path.exists():
|
||||||
|
self.load()
|
||||||
|
|
||||||
|
# ── persistence ───────────────────────────────────────────────────
|
||||||
|
def load(self) -> None:
|
||||||
|
raw = json.loads(self.path.read_text(encoding="utf-8"))
|
||||||
|
self.records = {}
|
||||||
|
for sym, rec in raw.get("assets", {}).items():
|
||||||
|
exchanges = {
|
||||||
|
ex: ExchangeListing(**lst) for ex, lst in rec.get("exchanges", {}).items()
|
||||||
|
}
|
||||||
|
self.records[sym] = AssetRecord(
|
||||||
|
symbol=sym,
|
||||||
|
base=rec.get("base", ""),
|
||||||
|
quote=rec.get("quote", ""),
|
||||||
|
exchanges=exchanges,
|
||||||
|
profile_ref=rec.get("profile_ref", ""),
|
||||||
|
notes=rec.get("notes", ""),
|
||||||
|
)
|
||||||
|
|
||||||
|
def save(self) -> None:
|
||||||
|
payload = {
|
||||||
|
"schema_version": SCHEMA_VERSION,
|
||||||
|
"known_exchanges": KNOWN_EXCHANGES,
|
||||||
|
"assets": {
|
||||||
|
sym: {
|
||||||
|
"base": r.base,
|
||||||
|
"quote": r.quote,
|
||||||
|
"exchanges": {ex: asdict(l) for ex, l in sorted(r.exchanges.items())},
|
||||||
|
"profile_ref": r.profile_ref,
|
||||||
|
"notes": r.notes,
|
||||||
|
}
|
||||||
|
for sym, r in sorted(self.records.items())
|
||||||
|
},
|
||||||
|
}
|
||||||
|
tmp = self.path.with_suffix(".json.tmp")
|
||||||
|
tmp.write_text(json.dumps(payload, indent=1, sort_keys=True), encoding="utf-8")
|
||||||
|
tmp.replace(self.path)
|
||||||
|
|
||||||
|
# ── mutation ──────────────────────────────────────────────────────
|
||||||
|
def upsert(self, symbol: str, *, base: str = "", quote: str = "") -> AssetRecord:
|
||||||
|
sym = normalize_symbol(symbol)
|
||||||
|
rec = self.records.get(sym)
|
||||||
|
if rec is None:
|
||||||
|
if not base and not quote and sym.endswith("USDT"):
|
||||||
|
base, quote = sym[:-4], "USDT"
|
||||||
|
rec = AssetRecord(symbol=sym, base=base, quote=quote)
|
||||||
|
self.records[sym] = rec
|
||||||
|
return rec
|
||||||
|
|
||||||
|
def set_listing(
|
||||||
|
self,
|
||||||
|
symbol: str,
|
||||||
|
exchange: str,
|
||||||
|
*,
|
||||||
|
venue_symbol: str = "",
|
||||||
|
status: str = ListingStatus.UNKNOWN,
|
||||||
|
checked_at: str = "",
|
||||||
|
source: str = "",
|
||||||
|
) -> None:
|
||||||
|
if exchange not in KNOWN_EXCHANGES:
|
||||||
|
raise ValueError(
|
||||||
|
f"unknown exchange {exchange!r}; add it to KNOWN_EXCHANGES first"
|
||||||
|
)
|
||||||
|
rec = self.upsert(symbol)
|
||||||
|
rec.exchanges[exchange] = ExchangeListing(
|
||||||
|
venue_symbol=venue_symbol or rec.symbol,
|
||||||
|
status=status,
|
||||||
|
last_checked=checked_at,
|
||||||
|
source=source,
|
||||||
|
)
|
||||||
|
|
||||||
|
def import_symbols(
|
||||||
|
self, symbols: Iterable[str], exchange: str, *,
|
||||||
|
status: str = ListingStatus.TRADING, checked_at: str = "", source: str = "",
|
||||||
|
) -> int:
|
||||||
|
"""Bulk-import a symbol list as listings on one exchange. Idempotent."""
|
||||||
|
n = 0
|
||||||
|
for s in symbols:
|
||||||
|
self.set_listing(
|
||||||
|
s, exchange, venue_symbol=s, status=status,
|
||||||
|
checked_at=checked_at, source=source,
|
||||||
|
)
|
||||||
|
n += 1
|
||||||
|
return n
|
||||||
|
|
||||||
|
# ── queries ───────────────────────────────────────────────────────
|
||||||
|
def symbols_for_exchange(
|
||||||
|
self, exchange: str, *, status: str = ListingStatus.TRADING
|
||||||
|
) -> List[str]:
|
||||||
|
if exchange not in KNOWN_EXCHANGES:
|
||||||
|
raise ValueError(f"unknown exchange {exchange!r}")
|
||||||
|
return sorted(
|
||||||
|
sym for sym, r in self.records.items()
|
||||||
|
if (l := r.exchanges.get(exchange)) is not None and l.status == status
|
||||||
|
)
|
||||||
|
|
||||||
|
def venue_symbol(self, symbol: str, exchange: str) -> str:
|
||||||
|
"""Venue-local spelling for a canonical symbol ('' if unlisted)."""
|
||||||
|
rec = self.records.get(normalize_symbol(symbol))
|
||||||
|
if rec is None:
|
||||||
|
return ""
|
||||||
|
lst = rec.exchanges.get(exchange)
|
||||||
|
return lst.venue_symbol if lst else ""
|
||||||
|
|
||||||
|
def get(self, symbol: str) -> Optional[AssetRecord]:
|
||||||
|
return self.records.get(normalize_symbol(symbol))
|
||||||
|
|
||||||
|
def __len__(self) -> int:
|
||||||
|
return len(self.records)
|
||||||
90
MALKHUT/malkhut/tests/test_asset_directory.py
Normal file
90
MALKHUT/malkhut/tests/test_asset_directory.py
Normal file
@@ -0,0 +1,90 @@
|
|||||||
|
"""
|
||||||
|
Tests for malkhut.assets.directory — the system-wide asset universe store.
|
||||||
|
|
||||||
|
Mutation-sensitivity: breaking normalize_symbol's uppercasing, listed_on's
|
||||||
|
status comparison, or symbols_for_exchange's filter goes RED here.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from malkhut.assets.directory import (
|
||||||
|
KNOWN_EXCHANGES,
|
||||||
|
AssetDirectory,
|
||||||
|
ExchangeListing,
|
||||||
|
ListingStatus,
|
||||||
|
normalize_symbol,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_normalize_symbol_variants():
|
||||||
|
assert normalize_symbol("BAND-USDT") == "BANDUSDT"
|
||||||
|
assert normalize_symbol("band_usdt") == "BANDUSDT"
|
||||||
|
assert normalize_symbol(" eth/usdt ") == "ETHUSDT"
|
||||||
|
assert normalize_symbol("BANDUSDT") == "BANDUSDT"
|
||||||
|
|
||||||
|
|
||||||
|
def test_known_exchanges_table_has_the_three_venues():
|
||||||
|
assert {"BINANCE", "BINGX", "BINGX_VST"} <= set(KNOWN_EXCHANGES)
|
||||||
|
|
||||||
|
|
||||||
|
def test_listing_status_validated():
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
ExchangeListing(venue_symbol="X-USDT", status="LISTED") # not a valid status
|
||||||
|
|
||||||
|
|
||||||
|
def test_set_listing_rejects_unknown_exchange(tmp_path):
|
||||||
|
d = AssetDirectory(tmp_path / "d.json")
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
d.set_listing("BTCUSDT", "KRAKEN", status=ListingStatus.TRADING)
|
||||||
|
|
||||||
|
|
||||||
|
def test_upsert_splits_base_quote_for_usdt():
|
||||||
|
d = AssetDirectory.__new__(AssetDirectory)
|
||||||
|
d.records = {}
|
||||||
|
rec = d.upsert("BANDUSDT")
|
||||||
|
assert rec.base == "BAND" and rec.quote == "USDT"
|
||||||
|
|
||||||
|
|
||||||
|
def test_import_and_query_roundtrip(tmp_path):
|
||||||
|
p = tmp_path / "dir.json"
|
||||||
|
d = AssetDirectory(p)
|
||||||
|
n = d.import_symbols(
|
||||||
|
["ETHUSDT", "BANDUSDT", "CELRUSDT"], "BINANCE",
|
||||||
|
status=ListingStatus.TRADING, checked_at="2026-07-11T20:00:00Z",
|
||||||
|
source="dolphin_ng7_scan_feed",
|
||||||
|
)
|
||||||
|
assert n == 3
|
||||||
|
d.set_listing("BANDUSDT", "BINGX_VST", venue_symbol="BAND-USDT",
|
||||||
|
status=ListingStatus.OFFLINE, source="vst_contracts_probe")
|
||||||
|
d.set_listing("ETHUSDT", "BINGX_VST", venue_symbol="ETH-USDT",
|
||||||
|
status=ListingStatus.TRADING, source="vst_contracts_probe")
|
||||||
|
d.save()
|
||||||
|
|
||||||
|
d2 = AssetDirectory(p) # reload from disk
|
||||||
|
assert len(d2) == 3
|
||||||
|
assert d2.symbols_for_exchange("BINANCE") == ["BANDUSDT", "CELRUSDT", "ETHUSDT"]
|
||||||
|
# BAND is OFFLINE on VST → excluded from the tradable set
|
||||||
|
assert d2.symbols_for_exchange("BINGX_VST") == ["ETHUSDT"]
|
||||||
|
assert d2.symbols_for_exchange("BINGX_VST", status=ListingStatus.OFFLINE) == ["BANDUSDT"]
|
||||||
|
assert d2.venue_symbol("ETHUSDT", "BINGX_VST") == "ETH-USDT"
|
||||||
|
assert d2.venue_symbol("BANDUSDT", "BINGX") == "" # never probed on live BingX
|
||||||
|
assert d2.get("band-usdt").listed_on("BINANCE") is True
|
||||||
|
assert d2.get("band-usdt").listed_on("BINGX_VST") is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_import_is_idempotent(tmp_path):
|
||||||
|
d = AssetDirectory(tmp_path / "d.json")
|
||||||
|
d.import_symbols(["ETHUSDT"], "BINANCE")
|
||||||
|
d.import_symbols(["ETHUSDT"], "BINANCE")
|
||||||
|
assert len(d) == 1
|
||||||
|
assert d.symbols_for_exchange("BINANCE") == ["ETHUSDT"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_listing_on_second_exchange_preserves_first(tmp_path):
|
||||||
|
d = AssetDirectory(tmp_path / "d.json")
|
||||||
|
d.import_symbols(["ETHUSDT"], "BINANCE")
|
||||||
|
d.set_listing("ETHUSDT", "BINGX_VST", venue_symbol="ETH-USDT",
|
||||||
|
status=ListingStatus.TRADING)
|
||||||
|
rec = d.get("ETHUSDT")
|
||||||
|
assert set(rec.exchanges) == {"BINANCE", "BINGX_VST"}
|
||||||
73
prod/clean_arch/violet/uv/test_uv_asset_universe.py
Normal file
73
prod/clean_arch/violet/uv/test_uv_asset_universe.py
Normal file
@@ -0,0 +1,73 @@
|
|||||||
|
"""
|
||||||
|
Tests for uv_asset_universe — real classification kernel, fake CH/venue seams.
|
||||||
|
|
||||||
|
Mutation-sensitivity: flipping contract_is_tradable's status comparison, or
|
||||||
|
refresh_vst_listings' present/absent branch, goes RED here.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from uv_asset_universe import ( # noqa: I001 — wires MALKHUT_ROOT onto sys.path
|
||||||
|
contract_is_tradable,
|
||||||
|
import_blue_universe,
|
||||||
|
init_asset_universe,
|
||||||
|
refresh_vst_listings,
|
||||||
|
)
|
||||||
|
from malkhut.assets.directory import AssetDirectory, ListingStatus
|
||||||
|
|
||||||
|
|
||||||
|
def _dir(tmp_path):
|
||||||
|
return AssetDirectory(tmp_path / "assets.json")
|
||||||
|
|
||||||
|
|
||||||
|
CONTRACTS = [
|
||||||
|
{"symbol": "ETH-USDT", "status": 1, "apiStateOpen": "true"},
|
||||||
|
{"symbol": "TRX-USDT", "status": 1, "apiStateOpen": "true"},
|
||||||
|
{"symbol": "OLD-USDT", "status": 25, "apiStateOpen": "true"}, # delisted
|
||||||
|
{"symbol": "SHUT-USDT", "status": 1, "apiStateOpen": "false"}, # API entry closed
|
||||||
|
# BAND-USDT absent entirely (observed on VST 2026-07-11)
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_contract_is_tradable():
|
||||||
|
assert contract_is_tradable(CONTRACTS[0]) is True
|
||||||
|
assert contract_is_tradable(CONTRACTS[2]) is False # status 25
|
||||||
|
assert contract_is_tradable(CONTRACTS[3]) is False # apiStateOpen false
|
||||||
|
|
||||||
|
|
||||||
|
def test_seed_and_refresh_marks_offline_and_absent(tmp_path):
|
||||||
|
d = _dir(tmp_path)
|
||||||
|
import_blue_universe(d, ["ETHUSDT", "TRXUSDT", "BANDUSDT", "OLDUSDT", "SHUTUSDT"])
|
||||||
|
assert d.symbols_for_exchange("BINANCE") == [
|
||||||
|
"BANDUSDT", "ETHUSDT", "OLDUSDT", "SHUTUSDT", "TRXUSDT"]
|
||||||
|
|
||||||
|
trading, offline = refresh_vst_listings(d, CONTRACTS)
|
||||||
|
assert (trading, offline) == (2, 3)
|
||||||
|
assert d.symbols_for_exchange("BINGX_VST") == ["ETHUSDT", "TRXUSDT"]
|
||||||
|
assert d.symbols_for_exchange("BINGX_VST", status=ListingStatus.OFFLINE) == [
|
||||||
|
"BANDUSDT", "OLDUSDT", "SHUTUSDT"]
|
||||||
|
# venue spelling preserved for tradable symbols
|
||||||
|
assert d.venue_symbol("ETHUSDT", "BINGX_VST") == "ETH-USDT"
|
||||||
|
|
||||||
|
|
||||||
|
def test_init_asset_universe_is_the_tradable_frozenset(tmp_path):
|
||||||
|
d = _dir(tmp_path)
|
||||||
|
import_blue_universe(d, ["ETHUSDT", "BANDUSDT"])
|
||||||
|
refresh_vst_listings(d, CONTRACTS)
|
||||||
|
uni = init_asset_universe("BINGX_VST", d)
|
||||||
|
assert uni == frozenset({"ETHUSDT"})
|
||||||
|
assert isinstance(uni, frozenset)
|
||||||
|
# the Binance view is unaffected by the VST stamp
|
||||||
|
assert init_asset_universe("BINANCE", d) == frozenset({"ETHUSDT", "BANDUSDT"})
|
||||||
|
|
||||||
|
|
||||||
|
def test_refresh_is_repeatable_and_recovers(tmp_path):
|
||||||
|
"""A symbol OFFLINE once must flip back to TRADING when the venue relists."""
|
||||||
|
d = _dir(tmp_path)
|
||||||
|
import_blue_universe(d, ["BANDUSDT"])
|
||||||
|
refresh_vst_listings(d, CONTRACTS) # BAND absent -> OFFLINE
|
||||||
|
assert d.symbols_for_exchange("BINGX_VST") == []
|
||||||
|
relisted = CONTRACTS + [{"symbol": "BAND-USDT", "status": 1, "apiStateOpen": "true"}]
|
||||||
|
refresh_vst_listings(d, relisted)
|
||||||
|
assert d.symbols_for_exchange("BINGX_VST") == ["BANDUSDT"]
|
||||||
203
prod/clean_arch/violet/uv/uv_asset_universe.py
Normal file
203
prod/clean_arch/violet/uv/uv_asset_universe.py
Normal file
@@ -0,0 +1,203 @@
|
|||||||
|
"""
|
||||||
|
UV asset-universe init — MALKHUT asset directory as the single source of truth.
|
||||||
|
|
||||||
|
Operator directive 2026-07-11: the MALKHUT asset store (malkhut/assets/) is
|
||||||
|
the system-wide known-asset universe. UV/VIOLET initialize their tradable
|
||||||
|
set from it against the CURRENT execution exchange, so a dead venue symbol
|
||||||
|
(e.g. BAND-USDT offline on VST) can never reach the kernel.
|
||||||
|
|
||||||
|
Flow:
|
||||||
|
1. import_blue_universe() — seed canonical symbols from the DOLPHIN NG7
|
||||||
|
scan feed (BINANCE listings; that feed IS the
|
||||||
|
Binance universe BLUE trades from).
|
||||||
|
2. refresh_vst_listings() — probe the execution venue's contracts endpoint
|
||||||
|
and stamp BINGX_VST TRADING/OFFLINE per symbol.
|
||||||
|
3. init_asset_universe() — the runtime call: returns the frozen tradable
|
||||||
|
set for the execution exchange. The promotion
|
||||||
|
bridge / seam refuses ENTER intents outside it.
|
||||||
|
|
||||||
|
Built for the full-Binance (~500+) universe north-star; today's feed carries
|
||||||
|
50 symbols — the store scales, the feed is the bottleneck.
|
||||||
|
|
||||||
|
CLI:
|
||||||
|
uv_asset_universe.py --seed-from-scans --refresh-vst --show
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import urllib.request
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from typing import Any, Iterable
|
||||||
|
|
||||||
|
MALKHUT_ROOT = os.environ.get("MALKHUT_ROOT", "/mnt/dolphinng5_predict/MALKHUT")
|
||||||
|
if MALKHUT_ROOT not in sys.path:
|
||||||
|
sys.path.insert(0, MALKHUT_ROOT)
|
||||||
|
|
||||||
|
from malkhut.assets.directory import ( # noqa: E402
|
||||||
|
AssetDirectory,
|
||||||
|
ListingStatus,
|
||||||
|
normalize_symbol,
|
||||||
|
)
|
||||||
|
|
||||||
|
from uv_anomaly_sentinel import CHQuerier, CH_DB_UV, VST_BASE_URL # noqa: E402
|
||||||
|
|
||||||
|
FEED_SOURCE = "dolphin_ng7_scan_feed"
|
||||||
|
PROBE_SOURCE = "vst_contracts_probe"
|
||||||
|
|
||||||
|
|
||||||
|
def _now_iso() -> str:
|
||||||
|
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||||
|
|
||||||
|
|
||||||
|
# ── step 1: seed from the scan feed (the BLUE/Binance universe) ────────────
|
||||||
|
def blue_universe_from_scans(ch: CHQuerier | None = None) -> list[str]:
|
||||||
|
ch = ch or CHQuerier()
|
||||||
|
rows = ch.query(
|
||||||
|
f"SELECT DISTINCT arrayJoin(JSONExtract(assets,'Array(String)')) AS a"
|
||||||
|
f" FROM {CH_DB_UV}.prime_scans ORDER BY a FORMAT JSONEachRow"
|
||||||
|
)
|
||||||
|
return [r["a"] for r in rows]
|
||||||
|
|
||||||
|
|
||||||
|
def import_blue_universe(directory: AssetDirectory, symbols: Iterable[str]) -> int:
|
||||||
|
return directory.import_symbols(
|
||||||
|
[normalize_symbol(s) for s in symbols],
|
||||||
|
"BINANCE",
|
||||||
|
status=ListingStatus.TRADING,
|
||||||
|
checked_at=_now_iso(),
|
||||||
|
source=FEED_SOURCE,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ── step 2: stamp execution-venue listings from the contracts probe ───────
|
||||||
|
def fetch_vst_contracts(base_url: str = VST_BASE_URL, timeout_s: int = 15) -> list[dict[str, Any]]:
|
||||||
|
req = urllib.request.Request(f"{base_url}/openApi/swap/v2/quote/contracts")
|
||||||
|
body = urllib.request.urlopen(req, timeout=timeout_s).read().decode("utf-8")
|
||||||
|
parsed = json.loads(body)
|
||||||
|
return parsed.get("data") or []
|
||||||
|
|
||||||
|
|
||||||
|
def contract_is_tradable(contract: dict[str, Any]) -> bool:
|
||||||
|
"""BingX contracts row -> tradable? status 1 = online; apiStateOpen gates API entry."""
|
||||||
|
return (
|
||||||
|
int(contract.get("status") or 0) == 1
|
||||||
|
and str(contract.get("apiStateOpen", "true")).lower() == "true"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def refresh_vst_listings(
|
||||||
|
directory: AssetDirectory, contracts: list[dict[str, Any]]
|
||||||
|
) -> tuple[int, int]:
|
||||||
|
"""Stamp BINGX_VST status for every symbol already in the directory.
|
||||||
|
|
||||||
|
Present + tradable -> TRADING; absent or non-tradable -> OFFLINE.
|
||||||
|
Returns (trading, offline) counts over the directory.
|
||||||
|
"""
|
||||||
|
now = _now_iso()
|
||||||
|
tradable: dict[str, str] = {} # canonical -> venue symbol
|
||||||
|
for c in contracts:
|
||||||
|
vsym = str(c.get("symbol") or "")
|
||||||
|
if vsym and contract_is_tradable(c):
|
||||||
|
tradable[normalize_symbol(vsym)] = vsym
|
||||||
|
n_trading = n_offline = 0
|
||||||
|
for sym in list(directory.records):
|
||||||
|
if sym in tradable:
|
||||||
|
directory.set_listing(sym, "BINGX_VST", venue_symbol=tradable[sym],
|
||||||
|
status=ListingStatus.TRADING,
|
||||||
|
checked_at=now, source=PROBE_SOURCE)
|
||||||
|
n_trading += 1
|
||||||
|
else:
|
||||||
|
directory.set_listing(sym, "BINGX_VST",
|
||||||
|
status=ListingStatus.OFFLINE,
|
||||||
|
checked_at=now, source=PROBE_SOURCE)
|
||||||
|
n_offline += 1
|
||||||
|
return n_trading, n_offline
|
||||||
|
|
||||||
|
|
||||||
|
# ── step 2b: full-taxonomy onboarding via MALKHUT's AssetCompiler ──────────
|
||||||
|
def onboard_profiles(directory: AssetDirectory, *, compiler: Any = None,
|
||||||
|
out_path: str = "") -> tuple[int, int]:
|
||||||
|
"""Run MALKHUT's asset onboarder over every directory symbol lacking a
|
||||||
|
profile_ref; persist compiled taxonomy JSON beside the directory.
|
||||||
|
|
||||||
|
Returns (compiled, failed). Failures are noted on the record, not fatal —
|
||||||
|
the directory stays authoritative for existence/listing even when the
|
||||||
|
taxonomy fetch fails.
|
||||||
|
"""
|
||||||
|
from dataclasses import asdict
|
||||||
|
if compiler is None:
|
||||||
|
from malkhut.training.asset_compiler import AssetCompiler
|
||||||
|
compiler = AssetCompiler()
|
||||||
|
path = out_path or str(directory.path.parent / "compiled_profiles.json")
|
||||||
|
try:
|
||||||
|
with open(path, encoding="utf-8") as fh:
|
||||||
|
store = json.load(fh)
|
||||||
|
except (OSError, json.JSONDecodeError):
|
||||||
|
store = {}
|
||||||
|
done = failed = 0
|
||||||
|
for sym in sorted(directory.records):
|
||||||
|
rec = directory.records[sym]
|
||||||
|
if rec.profile_ref and sym in store:
|
||||||
|
continue
|
||||||
|
result = compiler.compile_and_register(sym)
|
||||||
|
if result.asset_profile is None:
|
||||||
|
failed += 1
|
||||||
|
rec.notes = (rec.notes + " | " if rec.notes else "") + \
|
||||||
|
f"onboard failed {_now_iso()}: {'; '.join(result.warnings)[:120]}"
|
||||||
|
continue
|
||||||
|
entry = {"profile": asdict(result.asset_profile), "_compiled_at": _now_iso()}
|
||||||
|
if result.asset_behavior is not None:
|
||||||
|
entry["behavior"] = asdict(result.asset_behavior)
|
||||||
|
store[sym] = json.loads(json.dumps(entry, default=str))
|
||||||
|
rec.profile_ref = sym
|
||||||
|
done += 1
|
||||||
|
with open(path, "w", encoding="utf-8") as fh:
|
||||||
|
json.dump(store, fh, indent=1, sort_keys=True, default=str)
|
||||||
|
return done, failed
|
||||||
|
|
||||||
|
|
||||||
|
# ── step 3: the runtime call ───────────────────────────────────────────────
|
||||||
|
def init_asset_universe(execution_exchange: str = "BINGX_VST",
|
||||||
|
directory: AssetDirectory | None = None) -> frozenset[str]:
|
||||||
|
"""Tradable canonical symbols for the execution exchange in use."""
|
||||||
|
directory = directory or AssetDirectory()
|
||||||
|
return frozenset(directory.symbols_for_exchange(execution_exchange))
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv: list[str] | None = None) -> int:
|
||||||
|
p = argparse.ArgumentParser(description="UV asset-universe init (MALKHUT store)")
|
||||||
|
p.add_argument("--seed-from-scans", action="store_true")
|
||||||
|
p.add_argument("--refresh-vst", action="store_true")
|
||||||
|
p.add_argument("--onboard", action="store_true",
|
||||||
|
help="full-taxonomy compile via MALKHUT AssetCompiler (rate-limited)")
|
||||||
|
p.add_argument("--show", action="store_true")
|
||||||
|
a = p.parse_args(argv)
|
||||||
|
|
||||||
|
d = AssetDirectory()
|
||||||
|
if a.seed_from_scans:
|
||||||
|
n = import_blue_universe(d, blue_universe_from_scans())
|
||||||
|
d.save()
|
||||||
|
print(f"seeded {n} BINANCE listings from {FEED_SOURCE} -> {d.path}")
|
||||||
|
if a.refresh_vst:
|
||||||
|
t, o = refresh_vst_listings(d, fetch_vst_contracts())
|
||||||
|
d.save()
|
||||||
|
print(f"BINGX_VST refresh: {t} TRADING, {o} OFFLINE")
|
||||||
|
if o:
|
||||||
|
print("OFFLINE:", ", ".join(d.symbols_for_exchange(
|
||||||
|
"BINGX_VST", status=ListingStatus.OFFLINE)))
|
||||||
|
if a.onboard:
|
||||||
|
done, failed = onboard_profiles(d)
|
||||||
|
d.save()
|
||||||
|
print(f"onboarded taxonomy: {done} compiled, {failed} failed")
|
||||||
|
if a.show:
|
||||||
|
uni = init_asset_universe("BINGX_VST", d)
|
||||||
|
print(f"directory: {len(d)} assets @ {d.path}")
|
||||||
|
print(f"BINGX_VST tradable universe ({len(uni)}):", ", ".join(sorted(uni)))
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
88
prod/docs/POST_FINISH.md
Normal file
88
prod/docs/POST_FINISH.md
Normal file
@@ -0,0 +1,88 @@
|
|||||||
|
# POST_FINISH — the parking lot (parked with love, not killed)
|
||||||
|
|
||||||
|
**Standing rule (operator + Fable, 2026-07-10):** until the FINISH LINE is
|
||||||
|
crossed, new scope lands HERE, not in code. Fable answers new scope with
|
||||||
|
"parked" — even to the operator. Especially to the operator.
|
||||||
|
|
||||||
|
## THE FINISH LINE (frozen — only these three count)
|
||||||
|
1. **Parity RESIDUAL = 0** over N windows (codex dossier machinery,
|
||||||
|
`uv/parity-certification`)
|
||||||
|
2. **72h clean on VST** on the merged stack (`a2b0fac`+): no §4 anomalies —
|
||||||
|
no orphan orders, no journal gaps, tripwire quiet
|
||||||
|
3. **Operator gate:** small real capital, separate account, BLUE untouched
|
||||||
|
|
||||||
|
## SEQUENCED (above parked — operator 2026-07-11: "it is costing us money")
|
||||||
|
- **Smart OB exec: PINK ExecutionRouter integration** — FIRST task upon parity
|
||||||
|
(item 1) completion, before/alongside items 2–3. Asset exists certified
|
||||||
|
(265 tests incl. live VST E2E, maker_both ran production). Slots below the
|
||||||
|
kernel at the venue-adapter seam — parity-invisible. Economics: taker RT
|
||||||
|
~0.10% → maker-both ~0.04%; smoke-run day-1 flips breakeven → +0.9%
|
||||||
|
equal-weight. Research: 90%+ of orders fill as maker ("move to us").
|
||||||
|
**CRITICAL READING:** `prod/docs/BingX_FILL_CHARACTERIZATION_AND_ADVANTAGES.md`.
|
||||||
|
**SOA ADJUDICATION (first sub-task, Fable):** TWO exec assets exist —
|
||||||
|
(a) PINK ExecutionRouter: maker/taker POLICY, 265 tests, maker_both ran live;
|
||||||
|
(b) BLUE `alpha_engine/execution/` **SmartPlacer**: OB-aware PLACEMENT
|
||||||
|
(fill_simulator w/ 30s bookDepth, signal_confidence_adapter bridging Alpha
|
||||||
|
conviction → placement, ob_reader, smartplacer_constants). Likely
|
||||||
|
complementary (Router = maker-vs-taker; SmartPlacer = where-in-book).
|
||||||
|
Adjudicate router ∪ placer ∪ union against the fill-characterization doc
|
||||||
|
before integrating either.
|
||||||
|
|
||||||
|
## PARKED (specced where noted; zero build until finish)
|
||||||
|
- **IMPETUS build** — spec `IMPETUS_SPEC.md` + annex
|
||||||
|
`IMPETUS_SPEC_ANNEX_UV_INTEGRATION.md` (steel-clock plug, ASEx placement,
|
||||||
|
last-tick table). Paper done; build parked.
|
||||||
|
- **MALKHUT hose adaptation** — Fable's job when unparked (memory:
|
||||||
|
malkhut-hose-adaptation-fable)
|
||||||
|
- **JIMINY / PINOCCHIO** — ASEx fleet scoreboard + false-green detector
|
||||||
|
- **TP/SL cadence ladder L2/L3** — pulse-host tick path, Rust reflex engine
|
||||||
|
(spec `UV_TPSL_ENGINE_AND_REMAINING_DEV_SPEC.md` §1)
|
||||||
|
- **305M ring upgrade** — producer-side cached index in `uv_steel::ring`
|
||||||
|
(PASS9, ~10 lines + bench)
|
||||||
|
- **wCQ wait-free MPMC** — L3 mailbox candidate (`305MM_OPS_ALGO_ANNEX_FIFO.md`)
|
||||||
|
- **★ IMPETUS WIRE-ATOMIC (operator 2026-07-11, "insane, write it down")** —
|
||||||
|
exchange-wire→zinc-shm ATOMIC: the venue packet's tick becomes visible to
|
||||||
|
shm readers as ONE indivisible act, no intermediating copies/queues/threads.
|
||||||
|
Concretely: kernel-bypass ingest (AF_XDP / DPDK / io_uring registered
|
||||||
|
buffers — NIC DMAs frames into user memory), parse-in-place off the wire
|
||||||
|
buffer, single seqlock commit into `impetus_last` as the SAME pinned
|
||||||
|
thread's next instruction. Latency floor becomes NIC+parse (~µs), and the
|
||||||
|
atomicity contract extends end-to-end: a reader can never observe a tick
|
||||||
|
the wire didn't deliver, nor miss one it did (A2/A5 doctrine stretched to
|
||||||
|
the physical boundary). Prereqs: IMPETUS v1 shipped; WS/stream adapters
|
||||||
|
(annex R2); pinned-emit thread (annex R4); flat-C last-tick table (R3) —
|
||||||
|
each is a rung TOWARD this. TLS termination path (operator follow-up: the
|
||||||
|
sought term was "homomorphic" — FHE itself is 1e6-1e9x too slow, wrong
|
||||||
|
universe): SOA answers preserving wire-atomicity, in preference order:
|
||||||
|
(1) NIC inline TLS RX offload (ConnectX/kTLS-offload — silicon decrypts
|
||||||
|
AES-GCM at wire speed BEFORE DMA; buffer lands already-plaintext; CPU never
|
||||||
|
sees ciphertext); (2) kernel kTLS (decrypt in-kernel, zero-copy up);
|
||||||
|
(3) VAES/AES-NI decrypt-IN-PLACE (plaintext overwrites ciphertext in the
|
||||||
|
same buffer, ~1-4 cycles/byte — "decoded where it lies").
|
||||||
|
- **`uv_steel` crate extraction** — pulse-host modules → shared workspace crate
|
||||||
|
- **PASS9 mass-test regeneration** — far-end assertions, no broken codegen
|
||||||
|
- **BLUE trade_events dedup DDL** — operator-blessed write into dolphin ns
|
||||||
|
- **Vendor reconciliation** — T0-DEVIATION port upstream + vendor_sync
|
||||||
|
--from-committed-HEAD hardening
|
||||||
|
- **UvExecRunner consolidation** — route legacy direct async path through the
|
||||||
|
ASEx owner or retire it (execution-gate dossier §6.6)
|
||||||
|
- **Journal EXIT-labeling cleanup** — no-entry scans mislabeled EXIT (cosmetic)
|
||||||
|
- **Fleet identity hygiene** — stale zellij names, fossil handles, split bus
|
||||||
|
|
||||||
|
- **★ SHARP TODO (operator 2026-07-11): 375-branch upside study** — realized
|
||||||
|
upside under favorable-extension vs realized loss; left-tail cure hypothesis
|
||||||
|
(losses huge incl. scan-cadence SL misses; winners cramped). Depends on:
|
||||||
|
MAX_HOLD full-modulation port + ingress journaling
|
||||||
|
(UV_EXIT_DOCTRINE_PARITY_DESIGN.md D1.2). OB-cascade cause tuning rides the
|
||||||
|
same data.
|
||||||
|
|
||||||
|
## Not parked (allowed work)
|
||||||
|
Anything that directly serves items 1–3, bug fixes on the live path, and
|
||||||
|
operator-ordered exceptions (which must say "unpark X" explicitly).
|
||||||
|
|
||||||
|
## DESIRABLE (north-star, operator 2026-07-11 — not parked, no build yet)
|
||||||
|
- **Full-universe picking**: system should someday select from the ENTIRE
|
||||||
|
Binance (~500-asset, as minimum reference) universe, not the 50-symbol
|
||||||
|
NG7 feed slice — research indicates full-universe selection improves
|
||||||
|
ROI/WR/joy. MALKHUT asset directory (malkhut/assets/) is built to that
|
||||||
|
scale; the feed is the current bottleneck, not the store.
|
||||||
Reference in New Issue
Block a user