initial: import DOLPHIN baseline 2026-04-21 from dolphinng5_predict working tree
Includes core prod + GREEN/BLUE subsystems: - prod/ (BLUE harness, configs, scripts, docs) - nautilus_dolphin/ (GREEN Nautilus-native impl + dvae/ preserved) - adaptive_exit/ (AEM engine + models/bucket_assignments.pkl) - Observability/ (EsoF advisor, TUI, dashboards) - external_factors/ (EsoF producer) - mc_forewarning_qlabs_fork/ (MC regime/envelope) Excludes runtime caches, logs, backups, and reproducible artifacts per .gitignore.
This commit is contained in:
hjnormey
228
prod/clean_arch/README.md
Executable file
228
prod/clean_arch/README.md
Executable file
@@ -0,0 +1,228 @@
|
||||
# DOLPHIN Clean Architecture - Paper Trading
|
||||
|
||||
## ✅ Status: OPERATIONAL
|
||||
|
||||
The clean hexagonal architecture paper trading system is now running with live data from Hazelcast.
|
||||
|
||||
---
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ PAPER TRADING SYSTEM │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ PORTS (Interfaces) │
|
||||
│ ├── DataFeedPort - Abstract data feed interface │
|
||||
│ └── TradingPort - Abstract trading interface │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ ADAPTERS (Implementations) │
|
||||
│ ├── HazelcastDataFeed - Reads from DolphinNG6 via Hz │
|
||||
│ └── PaperTradingExecutor - Simulated order execution │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ CORE (Business Logic) │
|
||||
│ ├── TradingEngine - Position sizing, risk management │
|
||||
│ ├── SignalProcessor - Eigenvalue-based signals │
|
||||
│ └── PortfolioManager - Position tracking, PnL │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ INFRASTRUCTURE │
|
||||
│ ├── Hazelcast Cluster - Single source of truth │
|
||||
│ ├── Scan Bridge Service - Arrow → Hazelcast bridge │
|
||||
│ └── Arrow Files - DolphinNG6 output │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
### 1. Single Source of Truth (Hazelcast)
|
||||
- **Problem**: Price and eigenvalue data need to be perfectly synchronized
|
||||
- **Solution**: DolphinNG6 writes both to Hazelcast atomically
|
||||
- **Benefit**: No sync issues, consistent data for trading decisions
|
||||
|
||||
### 2. File Timestamp vs Scan Number
|
||||
- **Problem**: DolphinNG6 resets scan counters on restarts
|
||||
- **Solution**: Bridge uses file modification time (mtime) not scan_number
|
||||
- **Benefit**: Always gets latest data even after NG6 restarts
|
||||
|
||||
### 3. Hexagonal Architecture
|
||||
- **Benefit**: Core logic is adapter-agnostic
|
||||
- **Future**: Can swap Hazelcast adapter for direct Binance WebSocket
|
||||
- **Testing**: Easy to mock data feeds for unit tests
|
||||
|
||||
---
|
||||
|
||||
## Components
|
||||
|
||||
### DataFeedPort (`ports/data_feed.py`)
|
||||
Abstract interface for market data:
|
||||
```python
|
||||
class DataFeedPort(ABC):
|
||||
@abstractmethod
|
||||
async def get_latest_snapshot(self, symbol: str) -> MarketSnapshot:
|
||||
...
|
||||
```
|
||||
|
||||
### HazelcastDataFeed (`adapters/hazelcast_feed.py`)
|
||||
Implementation reading from Hazelcast:
|
||||
- Connects to `DOLPHIN_FEATURES` map
|
||||
- Reads `latest_eigen_scan` key
|
||||
- Returns `MarketSnapshot` with price + eigenvalues
|
||||
|
||||
### TradingEngine (`core/trading_engine.py`)
|
||||
Pure business logic:
|
||||
- Position sizing based on eigenvalues
|
||||
- Risk management
|
||||
- ACB (Adaptive Circuit Breaker) integration
|
||||
|
||||
---
|
||||
|
||||
## Data Flow
|
||||
|
||||
```
|
||||
┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
|
||||
│ DolphinNG6 │────▶│ Arrow Files │────▶│ Scan Bridge │────▶│ Hazelcast │
|
||||
│ (Trading) │ │ (Storage) │ │ (Service) │ │ (SSOT) │
|
||||
└─────────────┘ └─────────────┘ └─────────────┘ └──────┬──────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
|
||||
│ Binance │◀────│ Nautilus │◀────│ Trading │◀────│ Hazelcast │
|
||||
│ Futures │ │ Trader │ │ Engine │ │ DataFeed │
|
||||
│ (Paper) │ │ (Adapter) │ │ (Core) │ │ (Adapter) │
|
||||
└─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Running the System
|
||||
|
||||
### 1. Start Hazelcast (if not running)
|
||||
```bash
|
||||
cd /mnt/dolphinng5_predict
|
||||
docker-compose up -d hazelcast
|
||||
```
|
||||
|
||||
### 2. Start Scan Bridge Service
|
||||
```bash
|
||||
cd /mnt/dolphinng5_predict/prod
|
||||
source /home/dolphin/siloqy_env/bin/activate
|
||||
python3 scan_bridge_service.py
|
||||
```
|
||||
|
||||
### 3. Check Status
|
||||
```bash
|
||||
cd /mnt/dolphinng5_predict/prod/clean_arch
|
||||
python3 status.py
|
||||
```
|
||||
|
||||
### 4. Test Adapter
|
||||
```bash
|
||||
python3 -c "
|
||||
import asyncio
|
||||
import sys
|
||||
sys.path.insert(0, '/mnt/dolphinng5_predict/prod/clean_arch')
|
||||
from adapters.hazelcast_feed import HazelcastDataFeed
|
||||
|
||||
async def test():
|
||||
feed = HazelcastDataFeed({'hazelcast': {'cluster': 'dolphin', 'host': 'localhost:5701'}})
|
||||
await feed.connect()
|
||||
snap = await feed.get_latest_snapshot('BTCUSDT')
|
||||
print(f'BTC: \${snap.price:,.2f} | Eigenvalues: {len(snap.eigenvalues)}')
|
||||
await feed.disconnect()
|
||||
|
||||
asyncio.run(test())
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Current Status
|
||||
|
||||
| Component | Status | Notes |
|
||||
|-----------|--------|-------|
|
||||
| Hazelcast Cluster | ✅ Running | localhost:5701 |
|
||||
| Scan Bridge | ⚠️ Manual start | Run: `python3 scan_bridge_service.py` |
|
||||
| Arrow Files | ✅ Present | ~6500 files, latest #7320+ |
|
||||
| Hazelcast Data | ✅ Valid | 50 assets, 50 prices |
|
||||
| DataFeed Adapter | ✅ Working | BTC @ $71,281 |
|
||||
| Trading Engine | 🔄 Ready | Core logic implemented |
|
||||
| Nautilus Trader | 🔄 Ready | Integration pending |
|
||||
|
||||
---
|
||||
|
||||
## Evolution Path
|
||||
|
||||
### Phase 1: Hazelcast Feed (CURRENT)
|
||||
- Uses DolphinNG6 eigenvalue calculations
|
||||
- Single source of truth via Hazelcast
|
||||
- Bridge service watches Arrow files
|
||||
|
||||
### Phase 2: Direct Binance Feed (NEXT)
|
||||
- Replace Hazelcast adapter with Binance WebSocket
|
||||
- Compute eigenvalues locally
|
||||
- Lower latency
|
||||
|
||||
### Phase 3: Rust Kernel (FUTURE)
|
||||
- Port core trading logic to Rust
|
||||
- Python adapter layer only
|
||||
- Maximum performance
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### No data in Hazelcast
|
||||
```bash
|
||||
# Check if bridge is running
|
||||
ps aux | grep scan_bridge
|
||||
|
||||
# Check latest Arrow files
|
||||
ls -lt /mnt/ng6_data/arrow_scans/$(date +%Y-%m-%d)/ | head -5
|
||||
|
||||
# Manually push latest
|
||||
python3 << 'EOF'
|
||||
# (see scan_bridge_service.py for manual push code)
|
||||
EOF
|
||||
```
|
||||
|
||||
### Hazelcast connection refused
|
||||
```bash
|
||||
# Check if Hazelcast is running
|
||||
docker ps | grep hazelcast
|
||||
|
||||
# Check logs
|
||||
docker logs dolphin-hazelcast
|
||||
```
|
||||
|
||||
### Scan number mismatch
|
||||
- This is normal - DolphinNG6 resets counters
|
||||
- Bridge uses file timestamps, not scan numbers
|
||||
- Always gets latest data
|
||||
|
||||
---
|
||||
|
||||
## File Locations
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `prod/clean_arch/ports/data_feed.py` | Abstract interfaces (PORTS) |
|
||||
| `prod/clean_arch/adapters/hazelcast_feed.py` | Hazelcast adapter |
|
||||
| `prod/clean_arch/core/trading_engine.py` | Business logic |
|
||||
| `prod/scan_bridge_service.py` | Arrow → Hazelcast bridge |
|
||||
| `prod/clean_arch/status.py` | Status check |
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
✅ **Clean architecture implemented**
|
||||
✅ **Hazelcast data feed working**
|
||||
✅ **Live market data flowing**
|
||||
✅ **Ready for trading logic integration**
|
||||
|
||||
Next step: Connect TradingEngine to Nautilus Trader for paper trading execution.
|
||||
Reference in New Issue
Block a user