sentiment-engine/MALKHUT/malkhut/training/_DATA_STORAGE_SCHEMA_FORMATS.md

MALKHUT Asset Store — Data Storage Schema & Formats

System-wide asset universe. Used by BLUE, VIOLET, UV, and all downstream systems.

This document defines the data model, storage formats, and query interfaces for the MALKHUT asset classification and exchange registry. Other agents use this to:

• Understand what data is stored and where
• Import assets from other systems (e.g., BLUE's Binance universe)
• Query the asset universe by any dimension
• Extend the store with new exchanges or assets

Quick Reference

asset_classification.py   → AssetProfile, ExchangeProfile, query functions
asset_behavior.py         → AssetBehavior (10-dimension behavior model)
asset_compiler.py         → Auto-fetch from Binance/BingX API
parallel_eval.py          → Parallel episode evaluation
cma_trainer.py            → ScenarioFactory (uses both stores)

---

1. ExchangeProfile — Exchange Metadata

Frozen dataclass. One entry per exchange.

@dataclass(frozen=True, slots=True)
class ExchangeProfile:
    exchange_id: str          # "binance", "bingx", "bybit"
    display_name: str         # "Binance"
    has_spot: bool
    has_perps: bool
    has_options: bool
    api_base_url: str         # REST root
    ws_base_url: str          # WebSocket root ("" if N/A)
    default_taker_fee_bps: float
    default_maker_fee_bps: float
    typical_latency_ms: float

Storage: EXCHANGE_PROFILES: Dict[str, ExchangeProfile] in asset_classification.py.

Pre-defined: binance, bingx, bybit.

How to add a new exchange:

from malkhut.training.asset_classification import EXCHANGE_PROFILES, ExchangeProfile

EXCHANGE_PROFILES["okx"] = ExchangeProfile(
    exchange_id="okx", display_name="OKX",
    has_spot=True, has_perps=True, has_options=True,
    api_base_url="https://www.okx.com",
    ws_base_url="wss://ws.okx.com:8443/ws/v5/public",
    default_taker_fee_bps=0.1, default_maker_fee_bps=-0.02,
    typical_latency_ms=60,
)

---

2. AssetProfile — Per-Asset Classification

Frozen dataclass. One entry per symbol. Multi-label on Sector and TokenRole.

@dataclass(frozen=True, slots=True)
class AssetProfile:
    # --- Layer 1: Canonical identity (venue-independent) ---
    symbol: str                          # "BTCUSDT" — internal canonical ID
    base_asset: str                      # "BTC" — asset without quote
    name: str                            # "Bitcoin" — human-readable
    unified_symbol: str                  # "BTC/USDT" — CCXT format
    quote_currency: str                  # "USDT" — settlement currency

    # --- Layer 2: Cross-system identifiers ---
    coingecko_id: str                    # "bitcoin" — CoinGecko slug
    cmc_id: int                          # 1 — CoinMarketCap numeric ID
    blockchain: str                      # "bitcoin" — native chain
    contract_address: str                # "0x..." for ERC-20, "" for native coins

    # Fundamental (intrinsic, never change)
    sectors: tuple[Sector, ...]          # ("CURRENCY",)
    token_roles: tuple[TokenRole, ...]   # ("STORE_OF_VALUE",)
    supply_model: SupplyModel            # FIXED_CAP | DISINFLATIONARY | INFLATIONARY | BURN_MECHANISM
    consensus: ConsensusFamily           # POW | POS | DPOS
    smart_contracts: SmartContractCapability  # FULL | PARTIAL | NONE

    # Technical (invariant market-structure)
    market_cap_tier: MarketCapTier       # MEGA | LARGE | MID | SMALL | MICRO
    volatility_profile: VolatilityProfile # LOW | MEDIUM | HIGH | EXTREME
    liquidity_profile: LiquidityProfile   # DEEP | NORMAL | THIN | ILLIQUID
    derivative_access: DerivativeAccess   # PERPS_AND_OPTIONS | PERPS_ONLY | NONE

    # Execution parameters (exchange-set)
    tick_size: float
    lot_size: float
    price_decimals: int
    maker_fee_bps: float
    taker_fee_bps: float

    # Order-book fingerprint (long-run averages)
    typical_spread_bps: float
    typical_depth_usd: float
    typical_daily_volume_usd: float

    # Structural flags
    has_funding: bool = False
    has_options: bool = False

    # --- Layer 3: Exchange membership ---
    exchanges: tuple[str, ...] = ("binance",)  # which venues trade this

Storage: ASSET_PROFILES: Dict[str, AssetProfile] in asset_classification.py.

Multi-label rules:

• sectors and token_roles are tuples (ordered). First element = primary label.
• supply_model, consensus, smart_contracts = single enum (inherently singular).
• exchanges = tuple of strings (which venues list the asset).

Query functions:

Function	Returns
get_asset_profile(symbol)	Single profile or None
list_assets()	All symbols
get_asset_by_coingecko_id(id)	Profile by CoinGecko slug
get_asset_by_cmc_id(id)	Profile by CMC numeric ID
get_assets_by_base_asset(base)	All profiles for a base asset
get_assets_by_blockchain(chain)	Assets on a given chain
get_assets_by_unified_symbol(sym)	Profile by CCXT unified symbol
get_assets_by_sector(sector)	Assets in ANY of the queried sector
get_assets_by_token_role(role)	Assets with ANY of the queried role
get_assets_by_supply(model)	Assets with given supply model
get_assets_by_consensus(family)	Assets with given consensus
get_assets_by_market_cap(tier)	Assets in market cap band
get_assets_by_volatility(vol)	Assets in vol band
get_assets_by_liquidity(liq)	Assets in liquidity band
get_assets_by_derivatives(access)	Assets with given derivative access
get_gas_tokens()	All gas tokens
get_pov_assets()	PoW assets (forced selling)
get_shortable_assets()	All shortable assets
get_multi_sector_assets()	Assets in >1 sector
get_multi_role_assets()	Assets with >1 role
get_assets_on_exchange(exchange_id)	Assets traded on given exchange
get_common_assets(ex_a, ex_b)	Assets on BOTH exchanges
get_exchange_for_asset(symbol)	Which exchanges trade this asset

---

3. AssetBehavior — Per-Asset Behavior Model

Frozen dataclass. 10 orthogonal dimensions. Research-validated.

@dataclass(frozen=True, slots=True)
class AssetBehavior:
    symbol: str
    depth: DepthProfile        # book shape: amplitude, alpha, fragility
    spread: SpreadProfile       # normal spread, stress multiplier
    flow: FlowProfile           # order rate, sizes, cancel ratio
    vol: VolatilityProfile      # ann vol, GARCH params, half-life
    intraday: IntradayProfile   # peak/trough hours, ratio
    weekend: WeekendProfile     # vol/volume/spread multipliers
    correlation: CorrelationProfile  # ETH beta, BTC corr (normal vs crash)
    market_maker: MarketMakerProfile  # inventory, pull speed, margins
    liquidation: LiquidationProfile   # OI/MCap, trigger %, cascade
    funding: FundingProfile     # mean/std, positive %, basis
    retail: RetailProfile       # retail ratio, inst gap
    bingx: BingxProfile         # BingX-specific multiplier, latency
    template_name: str = ""     # which template this came from
    reference_price: float = 0.0  # last known mid-price

Storage: ASSET_BEHAVIORS: Dict[str, AssetBehavior] in asset_behavior.py.

3 templates: institutional_blue_chip, mid_cap_l1, retail_meme.

Query functions: same pattern as AssetProfile — get_behavior(), get_behaviors_by_template(), etc.

---

4. Auto-Compilation (AssetCompiler)

Auto-fetches from Binance/BingX public API, computes profiles.

compiler = AssetCompiler()
result = compiler.compile("XRPUSDT")  # ~6s, rate-limited
compiler.register(result)  # adds to ASSET_PROFILES + ASSET_BEHAVIORS

What it auto-fetches:

Endpoint	Computes
/api/v3/ticker/24hr	Price reference, daily volume
/api/v3/depth?limit=100	Spread, depth amplitude, decay α
/api/v3/klines?interval=1h&limit=168	Annualized vol, order flow stats
/api/v3/exchangeInfo	Tick size, lot size, price decimals
/fapi/v1/fundingRate	Funding rate mean/std
/fapi/v1/openInterest	OI/MCap ratio

Known classifications: 28 pre-defined assets. Unknowns get heuristic defaults.

---

5. Exchange-Asset Mapping Pattern

The mapping follows a many-to-many relationship:

Asset (BTCUSDT) ──exchanges──> (binance, bingx, bybit)
Exchange (binance) ──assets──> (BTCUSDT, ETHUSDT, SOLUSDT, ...)

In code: AssetProfile.exchanges is a tuple of exchange_id strings. For querying: get_assets_on_exchange(id), get_common_assets(a, b).

When importing from BLUE/VIOLET/UV:

1. Get the full symbol list from the source system
2. For each symbol, check if it already exists in ASSET_PROFILES
   • If yes: add the new exchange_id to the exchanges tuple
   • If no: create a minimal profile with the exchange's default fees
3. The _profile() helper and AssetProfile.from_template() handle creation

---

6. Data Flow Diagram

┌──────────────────┐     ┌──────────────────┐
│  Binance API     │     │  BingX API       │
│  (public, R/O)   │     │  (public, R/O)   │
└────────┬─────────┘     └────────┬─────────┘
         │                        │
         ▼                        ▼
┌────────────────────────────────────────────┐
│          AssetCompiler                      │
│  auto-fetch → compute → CompileResult      │
└────────────────────┬───────────────────────┘
                     │
                     ▼
┌────────────────────────────────────────────┐
│     ASSET_PROFILES (Dict[str, AssetProfile])│
│     ASSET_BEHAVIORS (Dict[str, AssetBehavior])│
│     EXCHANGE_PROFILES (Dict[str, ExchangeProfile])│
│                                             │
│  System-wide store:                         │
│  BLUE ──imports──→ this store               │
│  VIOLET ──imports──→ this store             │
│  UV ──imports──→ this store                 │
│  MALKHUT ──uses──→ this store               │
│  ScenarioFactory ──reads──→ this store      │
└────────────────────┬───────────────────────┘
                     │
                     ▼
┌────────────────────────────────────────────┐
│        ScenarioFactory                      │
│  behavior-driven scenarios                 │
│  auto-compile unknown assets               │
│  label queries (sector/role/template)       │
│  exchange-aware scenario generation         │
└────────────────────────────────────────────┘

---

7. File Locations

File	Purpose	Lines
malkhut/training/asset_classification.py	Enums, AssetProfile, ExchangeProfile, queries	~600
malkhut/training/asset_behavior.py	10-dimension behavior model, templates	~400
malkhut/training/asset_compiler.py	Binance/BingX auto-fetch, compile	~520
malkhut/training/parallel_eval.py	Parallel episode evaluation	~91
malkhut/training/cma_trainer.py	ScenarioFactory, CMA-ES trainer	~1340
malkhut/tests/test_asset_classification.py	190 classification tests	~355
malkhut/tests/test_parallel_eval.py	16 parallel eval tests	~225

---

8. Constants and Enums Reference

Sector (multi-label)

CURRENCY, LAYER1, LAYER2, DEFI, ORACLE, EXCHANGE, MEME, PRIVACY, STORAGE, GAMING_NFT

TokenRole (multi-label)

GAS, STORE_OF_VALUE, GOVERNANCE, UTILITY, MEME, EXCHANGE_FEE

SupplyModel (single)

FIXED_CAP, DISINFLATIONARY, INFLATIONARY, BURN_MECHANISM

ConsensusFamily (single)

POW, POS, DPOS

SmartContractCapability (single)

FULL, PARTIAL, NONE

MarketCapTier (single)

MEGA (>$500B), LARGE ($50-500B), MID ($5-50B), SMALL ($500M-5B), MICRO (<$500M)

VolatilityProfile (single)

LOW (<30%), MEDIUM (30-80%), HIGH (80-150%), EXTREME (>150%)

LiquidityProfile (single)

DEEP (>$100M), NORMAL ($10-100M), THIN ($1-10M), ILLIQUID (<$1M)

DerivativeAccess (single)

PERPS_AND_OPTIONS, PERPS_ONLY, NONE