sentiment-engine/MALKHUT/malkhut/training/_DATA_STORAGE_SCHEMA_FORMATS.md

# MALKHUT Asset Store — Data Storage Schema & Formats

**System-wide asset universe.** Used by BLUE, VIOLET, UV, and all downstream systems.

This document defines the data model, storage formats, and query interfaces for the
MALKHUT asset classification and exchange registry. Other agents use this to:

- Understand what data is stored and where
- Import assets from other systems (e.g., BLUE's Binance universe)
- Query the asset universe by any dimension
- Extend the store with new exchanges or assets

## Quick Reference

```
asset_classification.py   → AssetProfile, ExchangeProfile, query functions
asset_behavior.py         → AssetBehavior (10-dimension behavior model)
asset_compiler.py         → Auto-fetch from Binance/BingX API
parallel_eval.py          → Parallel episode evaluation
cma_trainer.py            → ScenarioFactory (uses both stores)
```

---

## 1. ExchangeProfile — Exchange Metadata

**Frozen dataclass. One entry per exchange.**

```python
@dataclass(frozen=True, slots=True)
class ExchangeProfile:
    exchange_id: str          # "binance", "bingx", "bybit"
    display_name: str         # "Binance"
    has_spot: bool
    has_perps: bool
    has_options: bool
    api_base_url: str         # REST root
    ws_base_url: str          # WebSocket root ("" if N/A)
    default_taker_fee_bps: float
    default_maker_fee_bps: float
    typical_latency_ms: float
```

**Storage:** `EXCHANGE_PROFILES: Dict[str, ExchangeProfile]` in `asset_classification.py`.

**Pre-defined:** `binance`, `bingx`, `bybit`.

**How to add a new exchange:**

```python
from malkhut.training.asset_classification import EXCHANGE_PROFILES, ExchangeProfile

EXCHANGE_PROFILES["okx"] = ExchangeProfile(
    exchange_id="okx", display_name="OKX",
    has_spot=True, has_perps=True, has_options=True,
    api_base_url="https://www.okx.com",
    ws_base_url="wss://ws.okx.com:8443/ws/v5/public",
    default_taker_fee_bps=0.1, default_maker_fee_bps=-0.02,
    typical_latency_ms=60,
)
```

---

## 2. AssetProfile — Per-Asset Classification

**Frozen dataclass. One entry per symbol. Multi-label on Sector and TokenRole.**

```python
@dataclass(frozen=True, slots=True)
class AssetProfile:
    # --- Layer 1: Canonical identity (venue-independent) ---
    symbol: str                          # "BTCUSDT" — internal canonical ID
    base_asset: str                      # "BTC" — asset without quote
    name: str                            # "Bitcoin" — human-readable
    unified_symbol: str                  # "BTC/USDT" — CCXT format
    quote_currency: str                  # "USDT" — settlement currency

    # --- Layer 2: Cross-system identifiers ---
    coingecko_id: str                    # "bitcoin" — CoinGecko slug
    cmc_id: int                          # 1 — CoinMarketCap numeric ID
    blockchain: str                      # "bitcoin" — native chain
    contract_address: str                # "0x..." for ERC-20, "" for native coins

    # Fundamental (intrinsic, never change)
    sectors: tuple[Sector, ...]          # ("CURRENCY",)
    token_roles: tuple[TokenRole, ...]   # ("STORE_OF_VALUE",)
    supply_model: SupplyModel            # FIXED_CAP | DISINFLATIONARY | INFLATIONARY | BURN_MECHANISM
    consensus: ConsensusFamily           # POW | POS | DPOS
    smart_contracts: SmartContractCapability  # FULL | PARTIAL | NONE

    # Technical (invariant market-structure)
    market_cap_tier: MarketCapTier       # MEGA | LARGE | MID | SMALL | MICRO
    volatility_profile: VolatilityProfile # LOW | MEDIUM | HIGH | EXTREME
    liquidity_profile: LiquidityProfile   # DEEP | NORMAL | THIN | ILLIQUID
    derivative_access: DerivativeAccess   # PERPS_AND_OPTIONS | PERPS_ONLY | NONE

    # Execution parameters (exchange-set)
    tick_size: float
    lot_size: float
    price_decimals: int
    maker_fee_bps: float
    taker_fee_bps: float

    # Order-book fingerprint (long-run averages)
    typical_spread_bps: float
    typical_depth_usd: float
    typical_daily_volume_usd: float

    # Structural flags
    has_funding: bool = False
    has_options: bool = False

    # --- Layer 3: Exchange membership ---
    exchanges: tuple[str, ...] = ("binance",)  # which venues trade this
```

**Storage:** `ASSET_PROFILES: Dict[str, AssetProfile]` in `asset_classification.py`.

**Multi-label rules:**
- `sectors` and `token_roles` are **tuples** (ordered). First element = primary label.
- `supply_model`, `consensus`, `smart_contracts` = **single enum** (inherently singular).
- `exchanges` = **tuple of strings** (which venues list the asset).

**Query functions:**

| Function | Returns |
|----------|---------|
| `get_asset_profile(symbol)` | Single profile or None |
| `list_assets()` | All symbols |
| `get_asset_by_coingecko_id(id)` | Profile by CoinGecko slug |
| `get_asset_by_cmc_id(id)` | Profile by CMC numeric ID |
| `get_assets_by_base_asset(base)` | All profiles for a base asset |
| `get_assets_by_blockchain(chain)` | Assets on a given chain |
| `get_assets_by_unified_symbol(sym)` | Profile by CCXT unified symbol |
| `get_assets_by_sector(sector)` | Assets in ANY of the queried sector |
| `get_assets_by_token_role(role)` | Assets with ANY of the queried role |
| `get_assets_by_supply(model)` | Assets with given supply model |
| `get_assets_by_consensus(family)` | Assets with given consensus |
| `get_assets_by_market_cap(tier)` | Assets in market cap band |
| `get_assets_by_volatility(vol)` | Assets in vol band |
| `get_assets_by_liquidity(liq)` | Assets in liquidity band |
| `get_assets_by_derivatives(access)` | Assets with given derivative access |
| `get_gas_tokens()` | All gas tokens |
| `get_pov_assets()` | PoW assets (forced selling) |
| `get_shortable_assets()` | All shortable assets |
| `get_multi_sector_assets()` | Assets in >1 sector |
| `get_multi_role_assets()` | Assets with >1 role |
| `get_assets_on_exchange(exchange_id)` | Assets traded on given exchange |
| `get_common_assets(ex_a, ex_b)` | Assets on BOTH exchanges |
| `get_exchange_for_asset(symbol)` | Which exchanges trade this asset |

---

## 3. AssetBehavior — Per-Asset Behavior Model

**Frozen dataclass. 10 orthogonal dimensions. Research-validated.**

```python
@dataclass(frozen=True, slots=True)
class AssetBehavior:
    symbol: str
    depth: DepthProfile        # book shape: amplitude, alpha, fragility
    spread: SpreadProfile       # normal spread, stress multiplier
    flow: FlowProfile           # order rate, sizes, cancel ratio
    vol: VolatilityProfile      # ann vol, GARCH params, half-life
    intraday: IntradayProfile   # peak/trough hours, ratio
    weekend: WeekendProfile     # vol/volume/spread multipliers
    correlation: CorrelationProfile  # ETH beta, BTC corr (normal vs crash)
    market_maker: MarketMakerProfile  # inventory, pull speed, margins
    liquidation: LiquidationProfile   # OI/MCap, trigger %, cascade
    funding: FundingProfile     # mean/std, positive %, basis
    retail: RetailProfile       # retail ratio, inst gap
    bingx: BingxProfile         # BingX-specific multiplier, latency
    template_name: str = ""     # which template this came from
    reference_price: float = 0.0  # last known mid-price
```

**Storage:** `ASSET_BEHAVIORS: Dict[str, AssetBehavior]` in `asset_behavior.py`.

**3 templates:** `institutional_blue_chip`, `mid_cap_l1`, `retail_meme`.

**Query functions:** same pattern as AssetProfile — `get_behavior()`, `get_behaviors_by_template()`, etc.

---

## 4. Auto-Compilation (AssetCompiler)

**Auto-fetches from Binance/BingX public API, computes profiles.**

```python
compiler = AssetCompiler()
result = compiler.compile("XRPUSDT")  # ~6s, rate-limited
compiler.register(result)  # adds to ASSET_PROFILES + ASSET_BEHAVIORS
```

**What it auto-fetches:**

| Endpoint | Computes |
|----------|----------|
| `/api/v3/ticker/24hr` | Price reference, daily volume |
| `/api/v3/depth?limit=100` | Spread, depth amplitude, decay α |
| `/api/v3/klines?interval=1h&limit=168` | Annualized vol, order flow stats |
| `/api/v3/exchangeInfo` | Tick size, lot size, price decimals |
| `/fapi/v1/fundingRate` | Funding rate mean/std |
| `/fapi/v1/openInterest` | OI/MCap ratio |

**Known classifications:** 28 pre-defined assets. Unknowns get heuristic defaults.

---

## 5. Exchange-Asset Mapping Pattern

The mapping follows a many-to-many relationship:

```
Asset (BTCUSDT) ──exchanges──> (binance, bingx, bybit)
Exchange (binance) ──assets──> (BTCUSDT, ETHUSDT, SOLUSDT, ...)
```

**In code:** `AssetProfile.exchanges` is a tuple of `exchange_id` strings.
**For querying:** `get_assets_on_exchange(id)`, `get_common_assets(a, b)`.

**When importing from BLUE/VIOLET/UV:**

1. Get the full symbol list from the source system
2. For each symbol, check if it already exists in `ASSET_PROFILES`
   - If yes: add the new exchange_id to the `exchanges` tuple
   - If no: create a minimal profile with the exchange's default fees
3. The `_profile()` helper and `AssetProfile.from_template()` handle creation

---

## 6. Data Flow Diagram

```
┌──────────────────┐     ┌──────────────────┐
│  Binance API     │     │  BingX API       │
│  (public, R/O)   │     │  (public, R/O)   │
└────────┬─────────┘     └────────┬─────────┘
         │                        │
         ▼                        ▼
┌────────────────────────────────────────────┐
│          AssetCompiler                      │
│  auto-fetch → compute → CompileResult      │
└────────────────────┬───────────────────────┘
                     │
                     ▼
┌────────────────────────────────────────────┐
│     ASSET_PROFILES (Dict[str, AssetProfile])│
│     ASSET_BEHAVIORS (Dict[str, AssetBehavior])│
│     EXCHANGE_PROFILES (Dict[str, ExchangeProfile])│
│                                             │
│  System-wide store:                         │
│  BLUE ──imports──→ this store               │
│  VIOLET ──imports──→ this store             │
│  UV ──imports──→ this store                 │
│  MALKHUT ──uses──→ this store               │
│  ScenarioFactory ──reads──→ this store      │
└────────────────────┬───────────────────────┘
                     │
                     ▼
┌────────────────────────────────────────────┐
│        ScenarioFactory                      │
│  behavior-driven scenarios                 │
│  auto-compile unknown assets               │
│  label queries (sector/role/template)       │
│  exchange-aware scenario generation         │
└────────────────────────────────────────────┘
```

---

## 7. File Locations

| File | Purpose | Lines |
|------|---------|-------|
| `malkhut/training/asset_classification.py` | Enums, AssetProfile, ExchangeProfile, queries | ~600 |
| `malkhut/training/asset_behavior.py` | 10-dimension behavior model, templates | ~400 |
| `malkhut/training/asset_compiler.py` | Binance/BingX auto-fetch, compile | ~520 |
| `malkhut/training/parallel_eval.py` | Parallel episode evaluation | ~91 |
| `malkhut/training/cma_trainer.py` | ScenarioFactory, CMA-ES trainer | ~1340 |
| `malkhut/tests/test_asset_classification.py` | 190 classification tests | ~355 |
| `malkhut/tests/test_parallel_eval.py` | 16 parallel eval tests | ~225 |

---

## 8. Constants and Enums Reference

### Sector (multi-label)
`CURRENCY`, `LAYER1`, `LAYER2`, `DEFI`, `ORACLE`, `EXCHANGE`, `MEME`, `PRIVACY`, `STORAGE`, `GAMING_NFT`

### TokenRole (multi-label)
`GAS`, `STORE_OF_VALUE`, `GOVERNANCE`, `UTILITY`, `MEME`, `EXCHANGE_FEE`

### SupplyModel (single)
`FIXED_CAP`, `DISINFLATIONARY`, `INFLATIONARY`, `BURN_MECHANISM`

### ConsensusFamily (single)
`POW`, `POS`, `DPOS`

### SmartContractCapability (single)
`FULL`, `PARTIAL`, `NONE`

### MarketCapTier (single)
`MEGA` (>$500B), `LARGE` ($50-500B), `MID` ($5-50B), `SMALL` ($500M-5B), `MICRO` (<$500M)

### VolatilityProfile (single)
`LOW` (<30%), `MEDIUM` (30-80%), `HIGH` (80-150%), `EXTREME` (>150%)

### LiquidityProfile (single)
`DEEP` (>$100M), `NORMAL` ($10-100M), `THIN` ($1-10M), `ILLIQUID` (<$1M)

### DerivativeAccess (single)
`PERPS_AND_OPTIONS`, `PERPS_ONLY`, `NONE`