siloqy/SYSTEM_BIBLE.md

# DOLPHIN Trading System Bible

**Version:** 1.0.0  
**Last Updated:** 2026-03-24  
**Status:** Production Ready - Paper Trading Operational

---

## 1. System Overview

The DOLPHIN Trading System is a production-grade algorithmic trading platform implementing clean hexagonal architecture for high-frequency eigenvalue-based trading strategies.

### 1.1 Current Architecture

```
┌─────────────────────────────────────────────────────────────────────────┐
│                         DOLPHIN TRADING SYSTEM                          │
│                              Version 1.0.0                               │
├─────────────────────────────────────────────────────────────────────────┤
│  PRESENTATION LAYER                                                      │
│  ├── Paper Trading CLI (paper_trade.py)                                 │
│  ├── System Monitor (status.py)                                         │
│  └── Trading Dashboard (future)                                         │
├─────────────────────────────────────────────────────────────────────────┤
│  CORE BUSINESS LOGIC (PORTS)                                             │
│  ├── DataFeedPort          - Abstract market data interface             │
│  ├── TradingPort           - Abstract execution interface               │
│  └── StrategyPort          - Abstract strategy interface                │
├─────────────────────────────────────────────────────────────────────────┤
│  ADAPTER LAYER                                                           │
│  ├── HazelcastDataFeed     - Live data from DolphinNG6                  │
│  ├── PaperTradeExecutor    - Simulated order execution                  │
│  └── (Future: BinanceDataFeed, LiveExecutor)                            │
├─────────────────────────────────────────────────────────────────────────┤
│  DOMAIN LOGIC                                                            │
│  ├── TradingEngine         - Position sizing, risk management           │
│  ├── SignalProcessor       - Eigenvalue signal generation               │
│  ├── PortfolioManager      - Position tracking, PnL calculation         │
│  └── ACBIntegration        - Adaptive Circuit Breaker                   │
└─────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────┐
│  INFRASTRUCTURE                                                          │
│  ├── Hazelcast Cluster     - Single Source of Truth (localhost:5701)    │
│  ├── Scan Bridge Service   - Arrow file watcher → Hazelcast             │
│  ├── Arrow File Storage    - /mnt/ng6_data/arrow_scans/                 │
│  └── Nautilus Trader       - Execution framework (future integration)   │
└─────────────────────────────────────────────────────────────────────────┘
```

### 1.2 Key Metrics

| Metric | Value |
|--------|-------|
| Architecture Pattern | Hexagonal (Clean Architecture) |
| Data Latency | ~5 seconds (DolphinNG6 pulse) |
| Assets Tracked | 50 (BTC, ETH, BNB, etc.) |
| Eigenvalue Windows | 50, 150, 300, 750 periods |
| Current BTC Price | $71,281.03 |
| Paper Trade Volume | 0.001 BTC per signal |

---

## 2. Version History

### v1.0.0 (2026-03-24) - CURRENT
**Codename:** Clean Slate

#### Major Features
- ✅ Clean hexagonal architecture implementation
- ✅ Hazelcast DataFeed adapter (live data from DolphinNG6)
- ✅ Scan Bridge Service (Arrow → Hazelcast bridge)
- ✅ Paper trading engine with PnL tracking
- ✅ Mean reversion strategy on velocity divergence
- ✅ 23 round-trip trades executed in testing

#### Components Added
- `prod/clean_arch/ports/data_feed.py` - Port interfaces
- `prod/clean_arch/adapters/hazelcast_feed.py` - Hazelcast adapter
- `prod/clean_arch/core/trading_engine.py` - Business logic
- `prod/scan_bridge_service.py` - Infrastructure bridge
- `prod/clean_arch/paper_trade.py` - Trading CLI

#### Infrastructure
- Hazelcast cluster at localhost:5701
- Scan bridge watching /mnt/ng6_data/arrow_scans/
- 6,500+ Arrow scan files processed

---

## 3. Architecture Principles

### 3.1 Hexagonal Architecture (Ports & Adapters)

The system follows strict dependency rules:
```
Core (inner) → Adapters (outer)
     ↑
   Ports (interfaces)
```

**Rule:** Dependencies only point inward. Core knows nothing about Hazelcast, Arrow files, or Binance.

### 3.2 Single Source of Truth

All market data comes from Hazelcast `DOLPHIN_FEATURES` map:
- Price + eigenvalues written atomically by DolphinNG6
- No synchronization issues
- Guaranteed consistent timestamps

### 3.3 File Timestamp vs Scan Number

The Scan Bridge uses **file modification time** (mtime) instead of scan numbers because:
- DolphinNG6 resets scan counters on restarts
- mtime always correctly identifies the latest file
- Handles NG6 crashes and restarts gracefully

---

## 4. Subsystems

### 4.1 Data Flow Architecture

```
┌─────────────┐     ┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│  DolphinNG6 │────▶│ Arrow Files │────▶│ Scan Bridge │────▶│  Hazelcast  │
│  (Trading)  │     │  (Storage)  │     │  (Service)  │     │   (SSOT)    │
└─────────────┘     └─────────────┘     └─────────────┘     └──────┬──────┘
     5s pulse                                                          │
                                                                      ▼
┌─────────────┐     ┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│   Binance   │◀────│  Nautilus   │◀────│   Trading   │◀────│ Hazelcast   │
│  (Paper)    │     │   Trader    │     │   Engine    │     │  DataFeed   │
│             │     │  (Future)   │     │  (Core)     │     │ (Adapter)   │
└─────────────┘     └─────────────┘     └─────────────┘     └─────────────┘
```

### 4.2 Component Details

#### Scan Bridge Service
**File:** `prod/scan_bridge_service.py`
**Purpose:** Watch Arrow files and push to Hazelcast
**Key Features:**
- Uses `watchdog` for file system events
- Parses JSON columns (`assets_json`, `asset_prices_json`)
- Handles numpy types in serialization
- Rolls to new day directory automatically

**Startup:**
```bash
cd /mnt/dolphinng5_predict/prod
source /home/dolphin/siloqy_env/bin/activate
python3 scan_bridge_service.py
```

#### Hazelcast DataFeed Adapter
**File:** `prod/clean_arch/adapters/hazelcast_feed.py`
**Purpose:** Implement DataFeedPort using Hazelcast
**Key Methods:**
- `connect()` - Connect to Hazelcast cluster
- `get_latest_snapshot(symbol)` - Get MarketSnapshot
- `subscribe_snapshots(callback)` - Subscribe to updates

**Data Structure:**
```python
MarketSnapshot(
    timestamp=datetime,
    symbol="BTCUSDT",
    price=71281.03,
    eigenvalues=[...],  # 50 loadings
    velocity_divergence=-0.0058,
    scan_number=7315
)
```

#### Trading Engine
**File:** `prod/clean_arch/core/trading_engine.py`
**Purpose:** Pure business logic, no external dependencies
**Responsibilities:**
- Position sizing
- Signal processing
- Risk management
- PnL calculation

---

## 5. Trading Strategy

### 5.1 Current Strategy: Velocity Divergence Mean Reversion

**Logic:**
```python
if velocity_divergence < BUY_THRESHOLD:  # -0.01
    BUY 0.001 BTC
elif velocity_divergence > SELL_THRESHOLD:  # 0.01
    SELL position
```

**Rationale:** Extreme negative velocity divergence suggests oversold conditions.

### 5.2 Signal Sources

| Signal | Source | Description |
|--------|--------|-------------|
| `velocity_divergence` | DolphinNG6 | Rate of change divergence |
| `instability_composite` | DolphinNG6 | Market instability metric |
| `w50_lambda_max` | DolphinNG6 | Max eigenvalue (50-period) |
| `asset_loadings` | DolphinNG6 | Eigenvector components |

---

## 6. Operations Guide

### 6.1 Starting the System

```bash
# 1. Ensure Hazelcast is running
docker ps | grep hazelcast

# 2. Start Scan Bridge
cd /mnt/dolphinng5_predict/prod
source /home/dolphin/siloqy_env/bin/activate
python3 scan_bridge_service.py &

# 3. Verify data flow
python3 clean_arch/status.py

# 4. Run paper trading
python3 clean_arch/paper_trade.py --duration 60
```

### 6.2 Monitoring

**Check Status:**
```bash
cd /mnt/dolphinng5_predict/prod/clean_arch
python3 status.py
```

**Expected Output:**
```
🐬 DOLPHIN PAPER TRADING STATUS
⚡ HAZELCAST: CONNECTED
   Scan: #7315
   Assets: 50
   Prices: 50
   BTC Price: $71,281.03
```

### 6.3 Troubleshooting

| Issue | Solution |
|-------|----------|
| No Hazelcast connection | Check `docker ps`, restart container |
| No data in Hz | Restart scan bridge, check Arrow files |
| Scan number mismatch | Normal - bridge uses mtime, not scan # |
| Zero eigenvalues | Check `asset_loadings` field exists |

---

## 7. Evolution Roadmap

### Phase 1: Hazelcast Feed (COMPLETE - v1.0.0)
- ✅ Clean architecture
- ✅ Hazelcast adapter
- ✅ Paper trading
- ✅ Scan bridge service

### Phase 2: Direct Market Feed (v1.1.0)
- [ ] Binance WebSocket adapter
- [ ] Local eigenvalue computation
- [ ] Reduced latency (<100ms)

### Phase 3: Rust Kernel (v2.0.0)
- [ ] Port core to Rust
- [ ] Python adapter layer only
- [ ] Microsecond latency

### Phase 4: Live Trading (v3.0.0)
- [ ] Live order execution
- [ ] Risk management integration
- [ ] Production deployment

---

## 8. File Reference

| Path | Purpose | Version Added |
|------|---------|---------------|
| `prod/clean_arch/ports/data_feed.py` | Port interfaces | v1.0.0 |
| `prod/clean_arch/adapters/hazelcast_feed.py` | Hz adapter | v1.0.0 |
| `prod/clean_arch/core/trading_engine.py` | Business logic | v1.0.0 |
| `prod/scan_bridge_service.py` | Arrow bridge | v1.0.0 |
| `prod/clean_arch/paper_trade.py` | Trading CLI | v1.0.0 |
| `prod/clean_arch/status.py` | Status monitor | v1.0.0 |

---

## 9. Environment

**Python Environment:** `/home/dolphin/siloqy_env`
**Python Version:** 3.12.12
**Key Dependencies:**
- nautilus_trader 1.219.0
- hazelcast-python-client 5.6.0
- pyarrow
- watchdog

**Data Locations:**
- Arrow files: `/mnt/ng6_data/arrow_scans/YYYY-MM-DD/`
- Hazelcast: `localhost:5701`
- Logs: `/tmp/scan_bridge.log`

---

## 10. Team & Ownership

**System Architect:** Clean Architecture Pattern  
**Data Source:** DolphinNG6 (Eigenvalue Engine)  
**Infrastructure:** Hazelcast Cluster  
**Trading Engine:** Hexagonal Core  

---

*This document is the single source of truth for the DOLPHIN Trading System architecture and operations. All changes must be documented with version updates.*

**Document Version:** 1.0.0  
**Last Modified:** 2026-03-24  
**Next Review:** On Phase 2 completion

---

## 11. DITAv2 / PINK — Accounting Architecture & Known Fixes

### 11.1 Fee Sign Convention (CRITICAL — 2026-06-08)

The Rust K-account uses **POSITIVE = cost, NEGATIVE = rebate** for all fee buckets.
BingX VST uses the **opposite convention**: the `"n"` (commission) field is **NEGATIVE for costs, POSITIVE for rebates**.

The sign translation is applied at `bingx_user_stream.py:_normalise_order()`:
```python
raw_fee = _safe_float(o.get("n") or 0.0)
fee = -raw_fee   # BingX cost (neg) → kernel cost (pos)
```

**Never apply this translation twice.** The kernel's `apply_fill_settled` / `apply_predicted_fill` always expect kernel convention (positive = cost). If a second boundary layer exists, it must pass fees already in kernel convention.

### 11.2 Opening-Fee Settlement (Defect B — fixed 2026-06-08)

BingX ENTER fills report filled quantity only in field `"z"` (cumFilledQty), not `"l"` (lastFilledQty). The `_normalise_order` fallback:
```python
_last_qty = _safe_float(o.get("l") or 0.0)
_cum_qty  = _safe_float(o.get("z") or o.get("cumFilledQty") or 0.0)
fill_qty  = _last_qty if _last_qty > 0.0 else _cum_qty
```
Ensures `apply_predicted_fill` receives non-zero `fill_qty` for ENTER fills, so the opening-leg fee prediction lands in `k_taker_fees`.

### 11.3 Reconcile / Capital Freeze Logic

The Rust kernel classifies K-vs-E divergence:

| `reconcile_status` | delta range | `capital_frozen` |
|---|---|---|
| `OK` | < 0.01 USDT | False |
| `WARN` | 0.01–20 USDT | False (unsettled but tradeable) |
| `ERROR` | ≥ 20 USDT | True (unexplained gap, HALT ENTERs) |

**WARN does NOT freeze.** During a live trade, the ENTER predicted fee temporarily pushes delta into ERROR range (seed balance not yet updated by exchange). The EXIT's realized PnL brings delta back into WARN (< 20), then the `ACCOUNT_UPDATE` event brings it to OK. Freezing on WARN would permanently block all trading after the first trade.

Only `ERROR` (≥ 20 USDT, unexplained) warrants a freeze.

### 11.4 Acceptance Criterion

After every round-trip trade: `|k_capital − bingx_wallet_balance| < 1.0 USDT` and `capital_frozen = False`.

Verified numerically against BingX ledger in `test_fee_sign_accounting.py` (34 tests, all green as of 2026-06-08).

### 11.5 Test File

`prod/clean_arch/dita_v2/test_fee_sign_accounting.py` — 34 tests covering:
- Prove-Broken: characterises the pre-fix buggy behaviour
- Prove-Fixed: T-2 and T-1 ground-truth round-trips
- Boundary layer: BingX `_normalise_order` sign + fill_qty
- Edge cases: rebates, partials, multiple fills