initial: import DOLPHIN baseline 2026-04-21 from dolphinng5_predict working tree

Includes core prod + GREEN/BLUE subsystems: - prod/ (BLUE harness, configs, scripts, docs) - nautilus_dolphin/ (GREEN Nautilus-native impl + dvae/ preserved) - adaptive_exit/ (AEM engine + models/bucket_assignments.pkl) - Observability/ (EsoF advisor, TUI, dashboards) - external_factors/ (EsoF producer) - mc_forewarning_qlabs_fork/ (MC regime/envelope) Excludes runtime caches, logs, backups, and reproducible artifacts per .gitignore.
2026-04-21 16:58:38 +02:00
commit 01c19662cb
643 changed files with 260241 additions and 0 deletions
--- a/nautilus_dolphin/CRITICAL_PRICE_UPDATE.md
+++ b/nautilus_dolphin/CRITICAL_PRICE_UPDATE.md
@@ -0,0 +1,293 @@
+# CRITICAL UPDATE: Actual Price Data Integration
+
+**Date**: 2026-02-18  
+**Status**: COMPLETE  
+**Priority**: CRITICAL
+
+---
+
+## Summary
+
+The eigenvalue JSON files **DO contain actual price data** in `pricing_data.current_prices`. The system has been updated to use these **REAL prices** instead of synthetic/generated prices.
+
+---
+
+## Price Data Structure in JSON Files
+
+```json
+{
+  "scan_number": 22284,
+  "timestamp": "2026-01-01T16:00:05.291658",
+  "windows": { ... },
+  "pricing_data": {
+    "current_prices": {
+      "BTCUSDT": 87967.06,
+      "ETHUSDT": 2985.16,
+      "BNBUSDT": 857.94,
+      ...
+    },
+    "price_changes": {
+      "BTCUSDT": 1.1367892858475202e-05,
+      ...
+    },
+    "volatility": {
+      "BTCUSDT": 0.04329507905570856,
+      ...
+    },
+    "per_asset_correlation": { ... }
+  }
+}
+```
+
+---
+
+## Files Modified
+
+### 1. `nautilus/data_adapter.py`
+
+**Changes**:
+- Added `_extract_prices()` method to get actual prices from `pricing_data.current_prices`
+- Added `_extract_price_changes()` method
+- Added `_extract_volatility()` method
+- Added `_extract_regime_data()` method for regime signals
+- Updated `generate_bars()` to use **ACTUAL prices** instead of synthetic
+- Updated `get_signal_metadata()` to include actual price in signals
+- Updated metadata to indicate `price_source: 'actual_from_json'`
+
+**Key Code**:
+```python
+def _extract_prices(self, data: dict) -> Dict[str, float]:
+    """Extract ACTUAL current prices from scan data."""
+    prices = {}
+    pricing_data = data.get('pricing_data', {})
+    current_prices = pricing_data.get('current_prices', {})
+    
+    for asset, price in current_prices.items():
+        if isinstance(price, (int, float)):
+            prices[asset] = float(price)
+    return prices
+```
+
+---
+
+### 2. `signal_generator/generator.py`
+
+**Changes**:
+- Added `_extract_prices()` method using actual price data
+- Updated `_extract_vel_div()` to compute from window tracking data
+- Added `_extract_volatility()` method
+- Added `_extract_price_changes()` method
+- Updated `_generate_signals_from_scan()` to include actual price in signals
+
+**Key Code**:
+```python
+def _generate_signals_from_scan(self, scan_data: Dict) -> List[Dict]:
+    # Get ACTUAL prices and vel_div
+    prices = self._extract_prices(scan_data)
+    vel_div_data = self._extract_vel_div(scan_data)
+    
+    for asset, vel_div in vel_div_data.items():
+        if vel_div < self.VEL_DIV_THRESHOLD:
+            price = prices.get(asset, 0.0)  # ACTUAL price
+            
+            signal = {
+                'timestamp': timestamp,
+                'asset': asset,
+                'direction': self.SIGNAL_DIRECTION,
+                'vel_div': vel_div,
+                'strength': strength,
+                'price': price,  # ACTUAL price from JSON
+                'volatility': vol,
+            }
+```
+
+---
+
+### 3. `signal_generator/enricher.py`
+
+**Changes**:
+- Added `_extract_price_data()` method to get actual price, price_change, volatility
+- Updated `_compute_direction_confirm()` to use **actual price changes** from `pricing_data.price_changes`
+- Added momentum threshold check (0.75bps)
+- Updated `enrich()` to use actual price data for direction confirmation
+
+**Key Code**:
+```python
+def _extract_price_data(self, scan_data: Dict, asset: str) -> Dict[str, Any]:
+    """Extract ACTUAL price data for asset."""
+    pricing_data = scan_data.get('pricing_data', {})
+    return {
+        'price': pricing_data.get('current_prices', {}).get(asset, 0.0),
+        'price_change': pricing_data.get('price_changes', {}).get(asset, 0.0),
+        'volatility': pricing_data.get('volatility', {}).get(asset, 0.1),
+    }
+
+def _compute_direction_confirm(self, signal, price_data, asset_data) -> bool:
+    """Compute direction confirmation using ACTUAL price changes."""
+    price_change = price_data.get('price_change', 0.0)
+    change_bps = abs(price_change) * 10000
+    
+    if change_bps < self.MOMENTUM_THRESHOLD_BPS:  # 0.75bps
+        return False
+        
+    if signal['direction'] == 'SHORT':
+        return price_change < 0
+    else:
+        return price_change > 0
+```
+
+---
+
+### 4. `nautilus/strategy.py`
+
+**Changes**:
+- Updated `_execute_entry()` to use **actual price from signal** when available
+- Falls back to quote tick only for live trading
+- Logs price source for transparency
+
+**Key Code**:
+```python
+def _execute_entry(self, signal_data: dict):
+    # Get price: Use ACTUAL price from signal (validation) or quote (live)
+    signal_price = signal_data.get('price')
+    if signal_price and signal_price > 0:
+        price = float(signal_price)
+        price_source = "signal"  # ACTUAL from eigenvalue JSON
+    else:
+        quote = self.cache.quote_tick(instrument_id)
+        price = float(quote.bid if direction == 'SHORT' else quote.ask)
+        price_source = "quote"  # Live market data
+    
+    self.log.info(
+        f"Entry order: {asset} {direction}, price=${price:.2f} "
+        f"(source: {price_source})"
+    )
+```
+
+---
+
+## Validation Impact
+
+### Before (Synthetic Prices)
+```python
+# Generated synthetic prices from vel_div
+drift = -vel_div_value * 0.01
+noise = random.gauss(0, abs(vel_div_value) * 0.005)
+price = base_price * (1 + drift + noise)  # SYNTHETIC!
+```
+
+### After (Actual Prices)
+```python
+# Extract ACTUAL prices from JSON
+prices = pricing_data.get('current_prices', {})
+price = prices.get('BTCUSDT')  # 87967.06 - ACTUAL!
+```
+
+### Benefits
+1. **Accurate Validation**: VBT and Nautilus use identical prices
+2. **Real P&L Calculation**: Based on actual market prices
+3. **Proper Slippage**: Measured against real prices
+4. **Faithful Backtests**: Reflects actual historical conditions
+
+---
+
+## Price Flow Architecture
+
+```
+eigenvalue JSON
+├── pricing_data.current_prices ──────┐
+├── pricing_data.price_changes ───────┤
+└── pricing_data.volatility ──────────┤
+                                      ▼
+┌─────────────────────────────────────────────────┐
+│ SignalGenerator                                 │
+│ - _extract_prices() → ACTUAL prices             │
+│ - _extract_vel_div() → velocity divergence      │
+│ - signal['price'] = actual_price               │
+└────────────────────┬────────────────────────────┘
+                     │ signal with ACTUAL price
+                     ▼
+┌─────────────────────────────────────────────────┐
+│ RedisSignalPublisher                            │
+│ - Publish signal with price                     │
+└────────────────────┬────────────────────────────┘
+                     │
+                     ▼
+┌─────────────────────────────────────────────────┐
+│ SignalBridgeActor                               │
+│ - Consume from Redis                            │
+│ - Publish to Nautilus bus                       │
+└────────────────────┬────────────────────────────┘
+                     │
+                     ▼
+┌─────────────────────────────────────────────────┐
+│ DolphinExecutionStrategy                        │
+│ - signal_data['price'] → ACTUAL price          │
+│ - Use for position sizing & order entry         │
+│ - Fallback to quote only if no signal price     │
+└─────────────────────────────────────────────────┘
+```
+
+---
+
+## Validation Checklist
+
+- [x] Data adapter extracts actual prices from JSON
+- [x] Signal generator includes actual price in signals
+- [x] Enricher uses actual price changes for direction confirmation
+- [x] Strategy uses signal price (actual) when available
+- [x] OHLC bars constructed from actual prices
+- [x] Trade P&L calculated from actual prices
+- [x] Comparator validates entry/exit prices within 0.1%
+
+---
+
+## Testing
+
+### Unit Test
+```python
+def test_actual_price_extraction():
+    adapter = JSONEigenvalueDataAdapter(eigenvalues_dir)
+    scan_data = adapter.load_scan_file(sample_file)
+    
+    prices = adapter._extract_prices(scan_data)
+    assert 'BTCUSDT' in prices
+    assert prices['BTCUSDT'] > 0  # Actual price, not synthetic
+    assert isinstance(prices['BTCUSDT'], float)
+```
+
+### Integration Test
+```python
+def test_signal_contains_actual_price():
+    generator = SignalGenerator(eigenvalues_dir)
+    scan_data = generator._load_scan_file(sample_file)
+    signals = generator._generate_signals_from_scan(scan_data)
+    
+    for signal in signals:
+        assert 'price' in signal
+        assert signal['price'] > 0  # ACTUAL price
+        assert signal['price_source'] == 'actual_json'
+```
+
+---
+
+## Migration Notes
+
+For existing deployments:
+1. No configuration changes required
+2. Price source automatically detected (signal vs quote)
+3. Backward compatible - falls back to quotes if no signal price
+
+---
+
+## Documentation Updates
+
+- [x] `data_adapter.py` - Docstrings updated
+- [x] `generator.py` - Comments added
+- [x] `enricher.py` - Documentation updated
+- [x] `strategy.py` - Logging includes price source
+- [x] This document created
+
+---
+
+**CRITICAL**: Both VBT and Nautilus now use **identical actual prices** from `pricing_data.current_prices`, ensuring accurate validation comparison.