hjnormey/siloqy
Public Access
1
0
Fork 0
Code Issues 7 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

PINK: fix fee-sign bug + WARN-unfreeze — 451/451 tests green

Browse Source 
Defect A (fee sign): bingx_user_stream._normalise_order flipped to
  fee = -raw_fee so BingX negative-n costs arrive as positive kernel
  costs.  k_maker_rebates no longer accumulates phantom rebates.

Defect B (opening fee dropped): fill_qty now falls back to "z"
  (cumFilledQty) when "l" (lastFilledQty) is zero/absent, so
  apply_predicted_fill computes a non-zero opening-leg fee.

Architectural fix (WARN unfreezes): lib.rs reconcile() now unfreezes
  capital_frozen on WARN as well as OK.  WARN (0.01-20 USDT delta) is
  normal in-flight settlement — only ERROR (≥20, unexplained) should
  halt ENTERs.  The old keep-state logic trapped the kernel permanently
  frozen after the first trade's ENTER predicted-fee phase pushed delta
  briefly into ERROR.

Acceptance criterion: |k_capital - bingx_balance| < 1 USDT, frozen=False
after every round-trip trade — verified numerically against T-1/T-2
ground truth from the CRITICAL doc.

Docs: CRITICAL_AGENT-TODO_ACCOUNTING_BUGFIX.md §12-13 (fix record),
      CAPITAL_BOOKKEEPING_DESIGN.md §8 (kernel spec), SYSTEM_BIBLE §11.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Codex
2026-06-08 11:08:31 +02:00
parent 7e83a5c5c5
commit e38ec77221
8 changed files with 2455 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files

385
SYSTEM_BIBLE.md Normal file
View File

@@ -0,0 +1,385 @@
# DOLPHIN Trading System Bible
**Version:** 1.0.0
**Last Updated:** 2026-03-24
**Status:** Production Ready - Paper Trading Operational
---
## 1. System Overview
The DOLPHIN Trading System is a production-grade algorithmic trading platform implementing clean hexagonal architecture for high-frequency eigenvalue-based trading strategies.
### 1.1 Current Architecture
```
┌─────────────────────────────────────────────────────────────────────────┐
│ DOLPHIN TRADING SYSTEM │
│ Version 1.0.0 │
├─────────────────────────────────────────────────────────────────────────┤
│ PRESENTATION LAYER │
│ ├── Paper Trading CLI (paper_trade.py) │
│ ├── System Monitor (status.py) │
│ └── Trading Dashboard (future) │
├─────────────────────────────────────────────────────────────────────────┤
│ CORE BUSINESS LOGIC (PORTS) │
│ ├── DataFeedPort - Abstract market data interface │
│ ├── TradingPort - Abstract execution interface │
│ └── StrategyPort - Abstract strategy interface │
├─────────────────────────────────────────────────────────────────────────┤
│ ADAPTER LAYER │
│ ├── HazelcastDataFeed - Live data from DolphinNG6 │
│ ├── PaperTradeExecutor - Simulated order execution │
│ └── (Future: BinanceDataFeed, LiveExecutor) │
├─────────────────────────────────────────────────────────────────────────┤
│ DOMAIN LOGIC │
│ ├── TradingEngine - Position sizing, risk management │
│ ├── SignalProcessor - Eigenvalue signal generation │
│ ├── PortfolioManager - Position tracking, PnL calculation │
│ └── ACBIntegration - Adaptive Circuit Breaker │
└─────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────┐
│ INFRASTRUCTURE │
│ ├── Hazelcast Cluster - Single Source of Truth (localhost:5701) │
│ ├── Scan Bridge Service - Arrow file watcher → Hazelcast │
│ ├── Arrow File Storage - /mnt/ng6_data/arrow_scans/ │
│ └── Nautilus Trader - Execution framework (future integration) │
└─────────────────────────────────────────────────────────────────────────┘
```
### 1.2 Key Metrics
| Metric | Value |
|--------|-------|
| Architecture Pattern | Hexagonal (Clean Architecture) |
| Data Latency | ~5 seconds (DolphinNG6 pulse) |
| Assets Tracked | 50 (BTC, ETH, BNB, etc.) |
| Eigenvalue Windows | 50, 150, 300, 750 periods |
| Current BTC Price | $71,281.03 |
| Paper Trade Volume | 0.001 BTC per signal |
---
## 2. Version History
### v1.0.0 (2026-03-24) - CURRENT
**Codename:** Clean Slate
#### Major Features
- ✅ Clean hexagonal architecture implementation
- ✅ Hazelcast DataFeed adapter (live data from DolphinNG6)
- ✅ Scan Bridge Service (Arrow → Hazelcast bridge)
- ✅ Paper trading engine with PnL tracking
- ✅ Mean reversion strategy on velocity divergence
- ✅ 23 round-trip trades executed in testing
#### Components Added
- `prod/clean_arch/ports/data_feed.py` - Port interfaces
- `prod/clean_arch/adapters/hazelcast_feed.py` - Hazelcast adapter
- `prod/clean_arch/core/trading_engine.py` - Business logic
- `prod/scan_bridge_service.py` - Infrastructure bridge
- `prod/clean_arch/paper_trade.py` - Trading CLI
#### Infrastructure
- Hazelcast cluster at localhost:5701
- Scan bridge watching /mnt/ng6_data/arrow_scans/
- 6,500+ Arrow scan files processed
---
## 3. Architecture Principles
### 3.1 Hexagonal Architecture (Ports & Adapters)
The system follows strict dependency rules:
```
Core (inner) → Adapters (outer)
↑
Ports (interfaces)
```
**Rule:** Dependencies only point inward. Core knows nothing about Hazelcast, Arrow files, or Binance.
### 3.2 Single Source of Truth
All market data comes from Hazelcast `DOLPHIN_FEATURES` map:
- Price + eigenvalues written atomically by DolphinNG6
- No synchronization issues
- Guaranteed consistent timestamps
### 3.3 File Timestamp vs Scan Number
The Scan Bridge uses **file modification time** (mtime) instead of scan numbers because:
- DolphinNG6 resets scan counters on restarts
- mtime always correctly identifies the latest file
- Handles NG6 crashes and restarts gracefully
---
## 4. Subsystems
### 4.1 Data Flow Architecture
```
┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ DolphinNG6 │────▶│ Arrow Files │────▶│ Scan Bridge │────▶│ Hazelcast │
│ (Trading) │ │ (Storage) │ │ (Service) │ │ (SSOT) │
└─────────────┘ └─────────────┘ └─────────────┘ └──────┬──────┘
5s pulse │
▼
┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ Binance │◀────│ Nautilus │◀────│ Trading │◀────│ Hazelcast │
│ (Paper) │ │ Trader │ │ Engine │ │ DataFeed │
│ │ │ (Future) │ │ (Core) │ │ (Adapter) │
└─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘
```
### 4.2 Component Details
#### Scan Bridge Service
**File:** `prod/scan_bridge_service.py`
**Purpose:** Watch Arrow files and push to Hazelcast
**Key Features:**
- Uses `watchdog` for file system events
- Parses JSON columns (`assets_json`, `asset_prices_json`)
- Handles numpy types in serialization
- Rolls to new day directory automatically
**Startup:**
```bash
cd /mnt/dolphinng5_predict/prod
source /home/dolphin/siloqy_env/bin/activate
python3 scan_bridge_service.py
```
#### Hazelcast DataFeed Adapter
**File:** `prod/clean_arch/adapters/hazelcast_feed.py`
**Purpose:** Implement DataFeedPort using Hazelcast
**Key Methods:**
- `connect()` - Connect to Hazelcast cluster
- `get_latest_snapshot(symbol)` - Get MarketSnapshot
- `subscribe_snapshots(callback)` - Subscribe to updates
**Data Structure:**
```python
MarketSnapshot(
timestamp=datetime,
symbol="BTCUSDT",
price=71281.03,
eigenvalues=[...], # 50 loadings
velocity_divergence=-0.0058,
scan_number=7315
)
```
#### Trading Engine
**File:** `prod/clean_arch/core/trading_engine.py`
**Purpose:** Pure business logic, no external dependencies
**Responsibilities:**
- Position sizing
- Signal processing
- Risk management
- PnL calculation
---
## 5. Trading Strategy
### 5.1 Current Strategy: Velocity Divergence Mean Reversion
**Logic:**
```python
if velocity_divergence < BUY_THRESHOLD: # -0.01
BUY 0.001 BTC
elif velocity_divergence > SELL_THRESHOLD: # 0.01
SELL position
```
**Rationale:** Extreme negative velocity divergence suggests oversold conditions.
### 5.2 Signal Sources
| Signal | Source | Description |
|--------|--------|-------------|
| `velocity_divergence` | DolphinNG6 | Rate of change divergence |
| `instability_composite` | DolphinNG6 | Market instability metric |
| `w50_lambda_max` | DolphinNG6 | Max eigenvalue (50-period) |
| `asset_loadings` | DolphinNG6 | Eigenvector components |
---
## 6. Operations Guide
### 6.1 Starting the System
```bash
# 1. Ensure Hazelcast is running
docker ps | grep hazelcast
# 2. Start Scan Bridge
cd /mnt/dolphinng5_predict/prod
source /home/dolphin/siloqy_env/bin/activate
python3 scan_bridge_service.py &
# 3. Verify data flow
python3 clean_arch/status.py
# 4. Run paper trading
python3 clean_arch/paper_trade.py --duration 60
```
### 6.2 Monitoring
**Check Status:**
```bash
cd /mnt/dolphinng5_predict/prod/clean_arch
python3 status.py
```
**Expected Output:**
```
🐬 DOLPHIN PAPER TRADING STATUS
⚡ HAZELCAST: CONNECTED
Scan: #7315
Assets: 50
Prices: 50
BTC Price: $71,281.03
```
### 6.3 Troubleshooting
| Issue | Solution |
|-------|----------|
| No Hazelcast connection | Check `docker ps`, restart container |
| No data in Hz | Restart scan bridge, check Arrow files |
| Scan number mismatch | Normal - bridge uses mtime, not scan # |
| Zero eigenvalues | Check `asset_loadings` field exists |
---
## 7. Evolution Roadmap
### Phase 1: Hazelcast Feed (COMPLETE - v1.0.0)
- ✅ Clean architecture
- ✅ Hazelcast adapter
- ✅ Paper trading
- ✅ Scan bridge service
### Phase 2: Direct Market Feed (v1.1.0)
- [ ] Binance WebSocket adapter
- [ ] Local eigenvalue computation
- [ ] Reduced latency (<100ms)
### Phase 3: Rust Kernel (v2.0.0)
- [ ] Port core to Rust
- [ ] Python adapter layer only
- [ ] Microsecond latency
### Phase 4: Live Trading (v3.0.0)
- [ ] Live order execution
- [ ] Risk management integration
- [ ] Production deployment
---
## 8. File Reference
| Path | Purpose | Version Added |
|------|---------|---------------|
| `prod/clean_arch/ports/data_feed.py` | Port interfaces | v1.0.0 |
| `prod/clean_arch/adapters/hazelcast_feed.py` | Hz adapter | v1.0.0 |
| `prod/clean_arch/core/trading_engine.py` | Business logic | v1.0.0 |
| `prod/scan_bridge_service.py` | Arrow bridge | v1.0.0 |
| `prod/clean_arch/paper_trade.py` | Trading CLI | v1.0.0 |
| `prod/clean_arch/status.py` | Status monitor | v1.0.0 |
---
## 9. Environment
**Python Environment:** `/home/dolphin/siloqy_env`
**Python Version:** 3.12.12
**Key Dependencies:**
- nautilus_trader 1.219.0
- hazelcast-python-client 5.6.0
- pyarrow
- watchdog
**Data Locations:**
- Arrow files: `/mnt/ng6_data/arrow_scans/YYYY-MM-DD/`
- Hazelcast: `localhost:5701`
- Logs: `/tmp/scan_bridge.log`
---
## 10. Team & Ownership
**System Architect:** Clean Architecture Pattern
**Data Source:** DolphinNG6 (Eigenvalue Engine)
**Infrastructure:** Hazelcast Cluster
**Trading Engine:** Hexagonal Core
---
*This document is the single source of truth for the DOLPHIN Trading System architecture and operations. All changes must be documented with version updates.*
**Document Version:** 1.0.0
**Last Modified:** 2026-03-24
**Next Review:** On Phase 2 completion
---
## 11. DITAv2 / PINK — Accounting Architecture & Known Fixes
### 11.1 Fee Sign Convention (CRITICAL — 2026-06-08)
The Rust K-account uses **POSITIVE = cost, NEGATIVE = rebate** for all fee buckets.
BingX VST uses the **opposite convention**: the `"n"` (commission) field is **NEGATIVE for costs, POSITIVE for rebates**.
The sign translation is applied at `bingx_user_stream.py:_normalise_order()`:
```python
raw_fee = _safe_float(o.get("n") or 0.0)
fee = -raw_fee # BingX cost (neg) → kernel cost (pos)
```
**Never apply this translation twice.** The kernel's `apply_fill_settled` / `apply_predicted_fill` always expect kernel convention (positive = cost). If a second boundary layer exists, it must pass fees already in kernel convention.
### 11.2 Opening-Fee Settlement (Defect B — fixed 2026-06-08)
BingX ENTER fills report filled quantity only in field `"z"` (cumFilledQty), not `"l"` (lastFilledQty). The `_normalise_order` fallback:
```python
_last_qty = _safe_float(o.get("l") or 0.0)
_cum_qty = _safe_float(o.get("z") or o.get("cumFilledQty") or 0.0)
fill_qty = _last_qty if _last_qty > 0.0 else _cum_qty
```
Ensures `apply_predicted_fill` receives non-zero `fill_qty` for ENTER fills, so the opening-leg fee prediction lands in `k_taker_fees`.
### 11.3 Reconcile / Capital Freeze Logic
The Rust kernel classifies K-vs-E divergence:
| `reconcile_status` | delta range | `capital_frozen` |
|---|---|---|
| `OK` | < 0.01 USDT | False |
| `WARN` | 0.01–20 USDT | False (unsettled but tradeable) |
| `ERROR` | ≥ 20 USDT | True (unexplained gap, HALT ENTERs) |
**WARN does NOT freeze.** During a live trade, the ENTER predicted fee temporarily pushes delta into ERROR range (seed balance not yet updated by exchange). The EXIT's realized PnL brings delta back into WARN (< 20), then the `ACCOUNT_UPDATE` event brings it to OK. Freezing on WARN would permanently block all trading after the first trade.
Only `ERROR` (≥ 20 USDT, unexplained) warrants a freeze.
### 11.4 Acceptance Criterion
After every round-trip trade: `|k_capital − bingx_wallet_balance| < 1.0 USDT` and `capital_frozen = False`.
Verified numerically against BingX ledger in `test_fee_sign_accounting.py` (34 tests, all green as of 2026-06-08).
### 11.5 Test File
`prod/clean_arch/dita_v2/test_fee_sign_accounting.py` — 34 tests covering:
- Prove-Broken: characterises the pre-fix buggy behaviour
- Prove-Fixed: T-2 and T-1 ground-truth round-trips
- Boundary layer: BingX `_normalise_order` sign + fill_qty
- Edge cases: rebates, partials, multiple fills