siloqy/SILOQY_DOLPHIN_Integration_Engineering_Spect_1.0.md

SILOQY DOLPHIN Integration Engineering Specification

Document Version: 1.0
Date: August 31, 2025
Target Audience: SILOQY Component Developers
Framework: Nautilus Trader Integration Architecture

Executive Summary

This document provides the complete technical specification for integrating with the SILOQY DOLPHIN market regime detection system within the Nautilus Trader framework. It details the message bus architecture, actor lifecycle management, data flow patterns, and exact implementation requirements for new SILOQY components.

Table of Contents

1. Architecture Overview

2. Nautilus Message Bus Communication

3. Actor Lifecycle and Execution Model

4. SILOQY Topic Hierarchy and Data Structures

5. Producer/Consumer Patterns

6. Component Integration Patterns

7. Error Handling and Recovery

8. Performance Optimization Guidelines

9. Testing and Validation Requirements

10. Code Examples and Templates

Architecture Overview

System Components

The SILOQY DOLPHIN system operates within a single Nautilus TradingNode instance with four core actors:

1. SILOQYSymbolDiscoveryActor: Symbol discovery and candle reconstruction

2. SILOQYMainActor: Real-time data ingestion and WebSocket management

3. DOLPHINRegimeActor: Market regime detection and analysis

4. SILOQYNormalizerActor: Tick normalization and format standardization

Data Flow Architecture

External Market Data → SILOQYMainActor → MessageBus → [DOLPHINRegimeActor, SILOQYNormalizerActor] 
                                                   ↓
                                          Custom SILOQY Components

Shared Node Pattern

Critical Design Principle: New SILOQY components MUST integrate into the existing TradingNode instance rather than creating separate nodes. This ensures:

• Shared message bus access - Unified actor lifecycle management - Consistent resource allocation - Optimal performance through single-process architecture

Nautilus Message Bus Communication

Message Bus Access Pattern

All SILOQY actors gain access to the Nautilus MessageBus through the standard Nautilus actor initialization:

class MySILOQYComponent(Actor):
    def __init__(self, config: MyComponentConfig):
        super().__init__(config)
        # MessageBus is available as self.msgbus after Nautilus initialization
    
    def on_start(self) -> None:
        # Subscribe to topics ONLY in on_start()
        if hasattr(self, 'msgbus') and self.msgbus:
            self.msgbus.subscribe("SILOQY.MY.TOPIC", self.handle_my_data)

Message Publishing Protocol

High-Performance Publishing Pattern:

def publish_siloqy_data(self, topic: str, data: Any) -> None:
    """Standard SILOQY publishing pattern"""
    try:
        if hasattr(self, 'msgbus') and self.msgbus:
            self.msgbus.publish(topic, data)
        else:
            self.log.error("MessageBus not available for publishing")
    except Exception as e:
        self.log.error(f"Failed to publish to {topic}: {e}")

Subscription Management

Thread-Safe Subscription Pattern:

def on_start(self) -> None:
    """Subscribe to required SILOQY topics"""
    if hasattr(self, 'msgbus') and self.msgbus:
        # Subscribe to core SILOQY data streams
        self.msgbus.subscribe(RAW_TOPIC, self.handle_raw_tick)
        self.msgbus.subscribe(REGIME_TOPIC, self.handle_regime_update)
        self.msgbus.subscribe("SILOQY.CUSTOM.TOPIC", self.handle_custom_data)
        
        self.log.info("Subscribed to SILOQY message bus topics")
    else:
        self.log.error("MessageBus not available during initialization")

Message Bus Timing Guarantees

• Publication Latency: < 1μs for tuple data

• Delivery Guarantee: Synchronous delivery to all subscribers

• Thread Safety: Fully thread-safe for concurrent access

• Memory Management: Zero-copy for immutable data types

Actor Lifecycle and Execution Model

Nautilus Actor Initialization Sequence

1. Construction Phase:

   def __init__(self, config: ActorConfig):
       super().__init__(config)  # MUST call super().__init__()
       # Initialize component-specific state
       # MessageBus NOT available yet
   

2. Registration Phase:

   # Handled automatically by TradingNode.build()
   # Actor registered with Nautilus kernel
   # MessageBus reference established
   

3. Startup Phase:

   def on_start(self) -> None:
       # MessageBus NOW available as self.msgbus
       # Subscribe to required topics
       # Initialize async tasks using ActorExecutor
   

4. Execution Phase:

   # Actor receives messages via subscribed handlers
   # Actor publishes data via self.msgbus.publish()
   # Long-running tasks managed by ActorExecutor
   

5. Shutdown Phase:

   def on_stop(self) -> None:
       # Clean up component-specific resources
       # Nautilus handles ActorExecutor cleanup automatically
   

ActorExecutor Task Management

Preferred Pattern for Async Operations:

def on_start(self) -> None:
    if hasattr(self, '_executor') and self._executor:
        self.log.info("Using Nautilus ActorExecutor")
        future = self._executor.submit(self.async_initialization())
    else:
        self.log.warning("No executor available, falling back to asyncio")
        asyncio.create_task(self.async_initialization())

async def async_initialization(self):
    # Long-running async operations
    # Managed by Nautilus ActorExecutor
    pass

Actor State Management

Thread-Safe State Pattern:

class MySILOQYActor(Actor):
    def __init__(self, config):
        super().__init__(config)
        
        # Pre-allocated arrays for high-frequency data (DOLPHIN pattern)
        self.data_arrays = np.zeros(config.max_items, dtype=np.float64)
        self.symbol_mapping = {}
        self.active_count = 0
        
        # Thread-safe collections for shared state
        self.message_queue = deque(maxlen=10000)
        self.state_lock = threading.Lock()

SILOQY Topic Hierarchy and Data Structures

Core SILOQY Topics

# Primary data flow topics
RAW_TOPIC = "SILOQY.RAW.TICKS"           # Raw market tick data
STRUCTURED_TOPIC = "SILOQY.STRUCTURED.TICKS"  # Normalized SiloqyTradeTick objects
REGIME_TOPIC = "DOLPHIN.REGIME.RESULTS"   # Market regime detection results

# Discovery and initialization topics  
SYMBOLS_DISCOVERED_TOPIC = "SILOQY.SYMBOLS.DISCOVERED"  # Symbol discovery completion
CANDLES_INITIAL_TOPIC = "SILOQY.CANDLES.INITIAL"        # Candle reconstruction results

# Custom component topics (examples)
HD_VECTOR_TOPIC = "SILOQY.HD.VECTORS"     # Hyperdimensional vector data
TEMPORAL_TOPIC = "SILOQY.TEMPORAL.PATTERNS"  # Temporal memory patterns
ANALYSIS_TOPIC = "SILOQY.ANALYSIS.RESULTS"   # Custom analysis results

Topic Naming Convention

Standard Format: SILOQY.{COMPONENT}.{DATA_TYPE}.{VARIANT}

Examples: - SILOQY.SENTIMENT.ANALYSIS.REALTIME - SILOQY.CORRELATION.MATRIX.UPDATED - SILOQY.RISK.METRICS.CALCULATED - SILOQY.ML.PREDICTIONS.GENERATED

Data Structure Specifications

Raw Tick Data Format (RAW_TOPIC)

# Tuple format: (symbol, price, size, ts_event, open_price, candle_open_time)
raw_tick_tuple = (
    "BTCUSDT",              # str: Symbol identifier
    45000.12345678,         # float: Current price (8 decimal precision)
    1.23456789,             # float: Trade quantity (8 decimal precision)  
    1693489234567,          # int: Event timestamp (milliseconds)
    44950.87654321,         # float: Current candle open price (8 decimal precision)
    1693488900000           # int: Candle open time (milliseconds)
)

Structured Tick Data Format (STRUCTURED_TOPIC)

# SiloqyTradeTick object with additional SILOQY fields
structured_tick = SiloqyTradeTick(
    instrument_id="BTCUSDT.BINANCE",    # InstrumentId: Nautilus format
    price=Price(45000.12345678, 8),     # Price: Nautilus price object
    size=Quantity(1.23456789, 8),       # Quantity: Nautilus quantity object
    ts_event=1693489234567000000,       # int: Nanosecond timestamp
    open_price=44950.87654321,          # float: SILOQY candle open price
    candle_open_time=1693488900000,     # int: SILOQY candle open time
    exchange="BINANCE",                 # str: SILOQY exchange identifier
    tick_side=None,                     # Optional[str]: Order side
    liquidity_flag=None                 # Optional[str]: Liquidity classification
)

Regime Detection Results (REGIME_TOPIC)

# Tuple format: (timestamp, regime, bull_ratio, bear_ratio, sideways_ratio, analyzed, total, confidence)
regime_tuple = (
    1693489234567,          # int: Detection timestamp (milliseconds)
    "BULL",                 # str: Regime type ["BULL", "BEAR", "SIDEWAYS"]
    0.67,                   # float: Bull ratio (0.0-1.0)
    0.25,                   # float: Bear ratio (0.0-1.0) 
    0.08,                   # float: Sideways ratio (0.0-1.0)
    150,                    # int: Symbols analyzed
    180,                    # int: Total symbols available
    0.82                    # float: Confidence score (0.0-1.0)
)

Custom Data Structure Template

@dataclass
class SILOQYCustomData:
    """Template for custom SILOQY data structures"""
    component_id: str
    timestamp: int
    data_type: str
    payload: Dict[str, Any]
    confidence: float = 1.0
    metadata: Dict[str, Any] = field(default_factory=dict)
    
    def to_message_tuple(self) -> Tuple:
        """Convert to message bus tuple format"""
        return (
            self.component_id,
            self.timestamp,
            self.data_type,
            json.dumps(self.payload),
            self.confidence,
            json.dumps(self.metadata)
        )

Producer/Consumer Patterns

High-Frequency Data Producer Pattern

class SILOQYHighFrequencyProducer(Actor):
    """Template for high-frequency data producers"""
    
    def __init__(self, config):
        super().__init__(config)
        self.publish_count = 0
        self.last_publish_time = 0
        
    def produce_data_efficiently(self, data: Any, topic: str) -> None:
        """High-performance data publishing"""
        try:
            # Validate data before publishing
            if self._validate_data(data):
                self.msgbus.publish(topic, data)
                self.publish_count += 1
                
                # Performance monitoring
                if self.publish_count % 1000 == 0:
                    self._log_performance_metrics()
            else:
                self.log.warning(f"Invalid data rejected for topic {topic}")
                
        except Exception as e:
            self.log.error(f"Publishing failed for topic {topic}: {e}")
    
    def _validate_data(self, data: Any) -> bool:
        """Implement component-specific validation"""
        # Override in derived classes
        return True
    
    def _log_performance_metrics(self) -> None:
        """Log performance metrics"""
        current_time = time.time()
        if self.last_publish_time > 0:
            duration = current_time - self.last_publish_time
            rate = 1000 / duration if duration > 0 else 0
            self.log.info(f"Publishing rate: {rate:.1f} messages/sec")
        self.last_publish_time = current_time

Data Consumer Pattern

class SILOQYDataConsumer(Actor):
    """Template for SILOQY data consumers"""
    
    def __init__(self, config):
        super().__init__(config)
        self.processed_count = 0
        self.processing_queue = deque(maxlen=10000)
        self.batch_size = 100
        
    def on_start(self) -> None:
        """Subscribe to required SILOQY topics"""
        if hasattr(self, 'msgbus') and self.msgbus:
            # Subscribe to multiple SILOQY data streams
            self.msgbus.subscribe(RAW_TOPIC, self.handle_raw_tick)
            self.msgbus.subscribe(REGIME_TOPIC, self.handle_regime_update)
            self.msgbus.subscribe("SILOQY.CUSTOM.DATA", self.handle_custom_data)
            
            self.log.info("SILOQY consumer subscriptions established")
    
    def handle_raw_tick(self, data: Tuple) -> None:
        """Handle raw tick data with efficient processing"""
        try:
            # Parse data
            symbol, price, size, ts_event, open_price, candle_open_time = data
            
            # Add to processing queue
            self.processing_queue.append({
                'type': 'raw_tick',
                'symbol': symbol,
                'price': price,
                'size': size,
                'timestamp': ts_event,
                'open_price': open_price,
                'candle_open_time': candle_open_time
            })
            
            # Batch processing for efficiency
            if len(self.processing_queue) >= self.batch_size:
                self._process_batch()
                
        except Exception as e:
            self.log.error(f"Raw tick processing error: {e}")
    
    def handle_regime_update(self, data: Tuple) -> None:
        """Handle DOLPHIN regime updates"""
        try:
            timestamp, regime, bull_ratio, bear_ratio, sideways_ratio, analyzed, total, confidence = data
            
            # Process regime change
            self._on_regime_change(regime, bull_ratio, bear_ratio, confidence)
            
        except Exception as e:
            self.log.error(f"Regime update processing error: {e}")
    
    def _process_batch(self) -> None:
        """Process queued data in batches for efficiency"""
        batch = list(self.processing_queue)
        self.processing_queue.clear()
        
        # Process batch with single operation
        self._bulk_process_data(batch)
        self.processed_count += len(batch)
    
    def _bulk_process_data(self, batch: List[Dict]) -> None:
        """Override in derived classes for bulk processing"""
        pass
    
    def _on_regime_change(self, regime: str, bull_ratio: float, bear_ratio: float, confidence: float) -> None:
        """Override in derived classes for regime-based logic"""
        pass

Actor Lifecycle and Execution Model

TradingNode Initialization Sequence

1. Node Creation:

   trading_config = TradingNodeConfig(
       trader_id=TraderId("SILOQY-PRODUCTION"),
       actors=[
           # All SILOQY actor configurations
           symbol_discovery_config,
           main_actor_config,
           regime_actor_config,
           normalizer_config,
           custom_component_config  # New components added here
       ]
   )
   node = TradingNode(config=trading_config)
   

2. Build Phase:

   node.build()  # Creates all actors, establishes MessageBus
   

3. Startup Phase:

   node.run()    # Calls on_start() on all actors in dependency order
   

Actor Dependency Management

Initialization Order (Critical): 1. SILOQYSymbolDiscoveryActor - Must complete first 2. SILOQYMainActor - Depends on symbol discovery 3. DOLPHINRegimeActor - Depends on main actor tick data 4. SILOQYNormalizerActor - Depends on raw tick data 5. Custom Components - Typically depend on normalized data or regime results

Async Task Management with ActorExecutor

ActorExecutor Usage Pattern:

def on_start(self) -> None:
    # Start long-running operations via ActorExecutor
    if hasattr(self, '_executor') and self._executor:
        self.log.info("Using Nautilus ActorExecutor for task management")
        future = self._executor.submit(self.long_running_operation())
    else:
        # Fallback for development/testing
        asyncio.create_task(self.long_running_operation())

async def long_running_operation(self):
    """Long-running async operation managed by Nautilus"""
    while self.is_running:
        try:
            # Perform periodic operations
            await self.process_periodic_task()
            await asyncio.sleep(1.0)
        except asyncio.CancelledError:
            break
        except Exception as e:
            self.log.error(f"Long-running operation error: {e}")

Actor State Synchronization

Synchronization Pattern for Actor Dependencies:

class DependentSILOQYActor(Actor):
    def __init__(self, config):
        super().__init__(config)
        self.dependency_ready = asyncio.Event()
        self.symbols = []
        self.candle_data = {}
    
    def on_start(self) -> None:
        # Subscribe to dependency notifications
        self.msgbus.subscribe(SYMBOLS_DISCOVERED_TOPIC, self.handle_symbols_ready)
        self.msgbus.subscribe(CANDLES_INITIAL_TOPIC, self.handle_candles_ready)
        
        # Start dependent operations
        if hasattr(self, '_executor') and self._executor:
            future = self._executor.submit(self.wait_for_dependencies())
    
    async def wait_for_dependencies(self):
        """Wait for required data before starting operations"""
        await asyncio.wait_for(self.dependency_ready.wait(), timeout=120.0)
        # Now safe to start component-specific operations
        await self.start_component_operations()
    
    def handle_symbols_ready(self, data):
        symbols, timestamp = data
        self.symbols = symbols
        self._check_dependencies_complete()
    
    def handle_candles_ready(self, data):
        candles, timestamp = data
        self.candle_data = candles
        self._check_dependencies_complete()
    
    def _check_dependencies_complete(self):
        if self.symbols and self.candle_data:
            self.dependency_ready.set()

Component Integration Patterns

Adding New SILOQY Components

Step 1: Component Configuration

class MySILOQYComponentConfig(ActorConfig):
    """Configuration for custom SILOQY component"""
    component_name: str = "MyComponent"
    update_interval_ms: int = 1000
    enable_hd_processing: bool = True
    custom_parameters: Dict[str, Any] = field(default_factory=dict)

Step 2: Actor Implementation

class MySILOQYComponent(Actor):
    def __init__(self, config: MySILOQYComponentConfig):
        super().__init__(config)
        self.component_name = config.component_name
        self.update_interval_ms = config.update_interval_ms
        self.enable_hd_processing = config.enable_hd_processing
        
    def on_start(self) -> None:
        # Subscribe to required SILOQY data streams
        self.msgbus.subscribe(STRUCTURED_TOPIC, self.handle_structured_tick)
        self.msgbus.subscribe(REGIME_TOPIC, self.handle_regime_update)
        
        # Start component-specific operations
        if hasattr(self, '_executor') and self._executor:
            future = self._executor.submit(self.start_processing())
    
    def handle_structured_tick(self, tick: SiloqyTradeTick) -> None:
        """Process incoming SILOQY tick data"""
        # Component-specific processing
        result = self.process_siloqy_tick(tick)
        
        # Publish results to custom topic
        if result:
            self.msgbus.publish("SILOQY.MYCOMPONENT.RESULTS", result)

Step 3: Registration with TradingNode

my_component_config = ImportableActorConfig(
    actor_path="my_module:MySILOQYComponent",
    config_path="my_module:MySILOQYComponentConfig",
    config={
        "component_id": "MY-SILOQY-COMPONENT",
        "component_name": "CustomAnalyzer",
        "update_interval_ms": 500,
        "enable_hd_processing": True
    }
)

# Add to existing TradingNode configuration
trading_config = TradingNodeConfig(
    trader_id=TraderId("SILOQY-EXTENDED"),
    actors=[
        symbol_discovery_config,
        main_actor_config,
        regime_actor_config,
        normalizer_config,
        my_component_config  # Add custom component here
    ]
)

Inter-Component Communication Patterns

Direct Communication via MessageBus:

# Component A publishes analysis results
def publish_analysis_result(self, analysis_data: Dict):
    custom_data = {
        'component_id': self.id,
        'analysis_type': 'correlation_matrix',
        'timestamp': int(time.time() * 1000),
        'results': analysis_data,
        'confidence': 0.95
    }
    self.msgbus.publish("SILOQY.CORRELATION.ANALYSIS", custom_data)

# Component B consumes analysis results
def on_start(self):
    self.msgbus.subscribe("SILOQY.CORRELATION.ANALYSIS", self.handle_correlation_data)

def handle_correlation_data(self, data: Dict):
    # Process correlation analysis from other component
    pass

Request/Response Pattern:

# Component A requests data
def request_analysis(self, symbol: str, timeframe: str):
    request_data = {
        'request_id': str(uuid.uuid4()),
        'requesting_component': str(self.id),
        'symbol': symbol,
        'timeframe': timeframe,
        'response_topic': f"SILOQY.RESPONSE.{self.id}"
    }
    self.msgbus.publish("SILOQY.REQUEST.ANALYSIS", request_data)

# Component B responds to requests
def handle_analysis_request(self, request_data: Dict):
    # Process request
    analysis_result = self.perform_analysis(request_data['symbol'], request_data['timeframe'])
    
    # Send response
    response = {
        'request_id': request_data['request_id'],
        'result': analysis_result,
        'timestamp': int(time.time() * 1000)
    }
    self.msgbus.publish(request_data['response_topic'], response)

Error Handling and Recovery

Message Processing Error Handling

Standard Error Handling Pattern:

def handle_siloqy_message(self, data: Any) -> None:
    """Standard SILOQY message handling with error recovery"""
    try:
        # Validate message structure
        if not self._validate_message_structure(data):
            self.log.warning(f"Invalid message structure rejected")
            self._record_validation_failure("structure")
            return
        
        # Process message
        result = self._process_message_content(data)
        
        # Publish result if successful
        if result:
            self.msgbus.publish(self.output_topic, result)
            
    except ValidationError as e:
        self.log.warning(f"Message validation failed: {e}")
        self._record_validation_failure("validation")
    except ProcessingError as e:
        self.log.error(f"Message processing failed: {e}")
        self._record_processing_failure()
    except Exception as e:
        self.log.error(f"Unexpected error in message handling: {e}")
        self._record_unexpected_failure()

def _record_validation_failure(self, failure_type: str) -> None:
    """Record validation failures for monitoring"""
    if not hasattr(self, 'validation_failures'):
        self.validation_failures = defaultdict(int)
    self.validation_failures[failure_type] += 1

Actor Recovery Patterns

Graceful Degradation Pattern:

class ResilientSILOQYActor(Actor):
    def __init__(self, config):
        super().__init__(config)
        self.error_count = 0
        self.max_errors = 100
        self.recovery_mode = False
    
    def handle_processing_error(self, error: Exception) -> None:
        """Handle processing errors with graceful degradation"""
        self.error_count += 1
        
        if self.error_count > self.max_errors:
            self.log.error("Error threshold exceeded, entering recovery mode")
            self.recovery_mode = True
            # Reduce processing frequency or disable non-critical features
        
        # Reset error count periodically
        if time.time() % 300 == 0:  # Every 5 minutes
            self.error_count = max(0, self.error_count - 10)
            if self.error_count < 10:
                self.recovery_mode = False
                self.log.info("Exiting recovery mode")

Performance Optimization Guidelines

Zero-Allocation Patterns (DOLPHIN Style)

class HighPerformanceSILOQYActor(Actor):
    """High-performance actor using zero-allocation patterns"""
    
    def __init__(self, config):
        super().__init__(config)
        
        # Pre-allocate arrays for high-frequency data
        self.max_symbols = config.max_symbols
        self.price_data = np.zeros(self.max_symbols, dtype=np.float64)
        self.volume_data = np.zeros(self.max_symbols, dtype=np.float64)
        self.timestamp_data = np.zeros(self.max_symbols, dtype=np.int64)
        
        # Symbol mapping for O(1) lookups
        self.symbol_to_index = {}
        self.index_to_symbol = {}
        self.active_symbols = 0
    
    def process_tick_zero_allocation(self, symbol: str, price: float, volume: float, timestamp: int):
        """Process tick with zero memory allocation"""
        # Get or assign symbol index
        if symbol not in self.symbol_to_index:
            if self.active_symbols >= self.max_symbols:
                self.log.error(f"Symbol capacity exceeded: {self.max_symbols}")
                return
            
            idx = self.active_symbols
            self.symbol_to_index[symbol] = idx
            self.index_to_symbol[idx] = symbol
            self.active_symbols += 1
        else:
            idx = self.symbol_to_index[symbol]
        
        # Update arrays in-place (zero allocation)
        self.price_data[idx] = price
        self.volume_data[idx] = volume
        self.timestamp_data[idx] = timestamp

Message Bus Performance Optimization

Efficient Data Serialization:

# Prefer tuples over complex objects for high-frequency data
efficient_message = (symbol, price, volume, timestamp)  # Fast
inefficient_message = {'symbol': symbol, 'price': price, 'volume': volume}  # Slower

# Use pre-allocated buffers for repeated operations
class BufferedSILOQYActor(Actor):
    def __init__(self, config):
        super().__init__(config)
        self.message_buffer = [None] * 1000  # Pre-allocated buffer
        self.buffer_index = 0
    
    def publish_efficiently(self, data):
        # Reuse buffer slots to minimize garbage collection
        self.message_buffer[self.buffer_index] = data
        self.msgbus.publish(self.topic, self.message_buffer[self.buffer_index])
        self.buffer_index = (self.buffer_index + 1) % 1000

Testing and Validation Requirements

Unit Test Requirements for SILOQY Components

Mandatory Test Coverage:

class TestMySILOQYComponent(unittest.TestCase):
    """Required test patterns for all SILOQY components"""
    
    def test_message_bus_integration(self):
        """Test MessageBus subscription and publishing"""
        # Verify component can subscribe to required topics
        # Verify component can publish to output topics
        # Test message handling with valid and invalid data
        pass
    
    def test_actor_lifecycle(self):
        """Test complete actor lifecycle"""
        # Test initialization with TradingNode
        # Test on_start() behavior
        # Test on_stop() behavior
        # Test resource cleanup
        pass
    
    def test_data_validation(self):
        """Test input data validation"""
        # Test with valid SILOQY data structures
        # Test with malformed data
        # Test boundary conditions
        # Test error recovery
        pass
    
    def test_performance_requirements(self):
        """Test performance under load"""
        # Test high-frequency message processing
        # Test memory usage stability
        # Test latency requirements
        pass

Integration Test Requirements

MessageBus Integration Test Template:

def test_full_integration_with_dolphin(self):
    """Test integration with existing DOLPHIN system"""
    # Create TradingNode with DOLPHIN + custom component
    node = self._create_test_node_with_custom_component()
    
    # Verify component receives DOLPHIN data
    collector = TestDataCollector()
    node.kernel.msgbus.subscribe("SILOQY.CUSTOM.OUTPUT", collector.collect_data)
    
    # Inject test data through DOLPHIN pipeline
    test_ticks = self._generate_test_tick_data()
    for tick in test_ticks:
        node.kernel.msgbus.publish(RAW_TOPIC, tick)
    
    # Verify custom component processed DOLPHIN regime data
    time.sleep(2.0)
    self.assertGreater(len(collector.collected_data), 0)

Code Examples and Templates

Complete Component Template

#!/usr/bin/env python3
"""
SILOQY Custom Component Template
Template for integrating new components with SILOQY DOLPHIN system
"""

import time
import asyncio
import numpy as np
from typing import Dict, List, Optional, Any
from collections import deque, defaultdict
from dataclasses import dataclass, field

from nautilus_trader.common.actor import Actor
from nautilus_trader.common.config import ActorConfig
from nautilus_trader.common.component import Logger

# SILOQY imports
from nautilus_actor_test_implementation_5x_modified import (
    RAW_TOPIC, STRUCTURED_TOPIC, REGIME_TOPIC,
    SiloqyTradeTick, MarketRegime
)

@dataclass
class MySILOQYComponentConfig(ActorConfig):
    """Configuration for custom SILOQY component"""
    component_name: str = "CustomComponent"
    processing_interval_ms: int = 1000
    enable_regime_tracking: bool = True
    max_history_size: int = 10000
    custom_parameters: Dict[str, Any] = field(default_factory=dict)

class MySILOQYComponent(Actor):
    """Template for custom SILOQY components"""
    
    def __init__(self, config: MySILOQYComponentConfig):
        super().__init__(config)
        
        # Component configuration
        self.component_name = config.component_name
        self.processing_interval_ms = config.processing_interval_ms
        self.enable_regime_tracking = config.enable_regime_tracking
        self.max_history_size = config.max_history_size
        
        # High-performance data structures
        self.tick_history = deque(maxlen=self.max_history_size)
        self.regime_history = deque(maxlen=100)
        self.current_regime = MarketRegime.SIDEWAYS
        
        # Performance metrics
        self.processed_ticks = 0
        self.processed_regimes = 0
        self.last_performance_log = time.time_ns()
        
        # State management
        self.is_processing = False
        self.dependency_ready = asyncio.Event()
        
        self.log.info(f"Custom SILOQY component '{self.component_name}' initialized")
    
    def on_start(self) -> None:
        """Start component and subscribe to SILOQY data streams"""
        self.log.info(f"Starting SILOQY component: {self.component_name}")
        
        if hasattr(self, 'msgbus') and self.msgbus:
            # Subscribe to core SILOQY data streams
            self.msgbus.subscribe(STRUCTURED_TOPIC, self.handle_structured_tick)
            
            if self.enable_regime_tracking:
                self.msgbus.subscribe(REGIME_TOPIC, self.handle_regime_update)
            
            # Subscribe to custom topics if needed
            self.msgbus.subscribe("SILOQY.CUSTOM.INPUT", self.handle_custom_input)
            
            self.log.info("SILOQY MessageBus subscriptions established")
        
        # Start async processing
        if hasattr(self, '_executor') and self._executor:
            future = self._executor.submit(self.start_async_processing())
        else:
            asyncio.create_task(self.start_async_processing())
    
    def on_stop(self) -> None:
        """Stop component processing"""
        self.log.info(f"Stopping SILOQY component: {self.component_name}")
        self.is_processing = False
        # Nautilus ActorExecutor handles task cleanup
    
    async def start_async_processing(self):
        """Start component-specific async processing"""
        self.is_processing = True
        self.log.info("Custom SILOQY component async processing started")
        
        while self.is_processing:
            try:
                # Perform periodic processing
                await self.perform_periodic_analysis()
                await asyncio.sleep(self.processing_interval_ms / 1000.0)
                
            except asyncio.CancelledError:
                break
            except Exception as e:
                self.log.error(f"Periodic processing error: {e}")
                await asyncio.sleep(1.0)
    
    def handle_structured_tick(self, tick: SiloqyTradeTick) -> None:
        """Handle incoming structured tick data"""
        try:
            # Add to history
            self.tick_history.append({
                'symbol': str(tick.instrument_id),
                'price': float(tick.price),
                'size': float(tick.size),
                'timestamp': tick.ts_event,
                'open_price': tick.open_price,
                'exchange': tick.exchange
            })
            
            self.processed_ticks += 1
            
            # Process tick for component-specific logic
            result = self.process_tick_data(tick)
            
            # Publish results if any
            if result:
                self.msgbus.publish(f"SILOQY.{self.component_name.upper()}.RESULTS", result)
            
            # Performance logging
            now = time.time_ns()
            if now - self.last_performance_log > 10_000_000_000:  # Every 10 seconds
                self.log.info(f"Component {self.component_name}: processed {self.processed_ticks} ticks")
                self.last_performance_log = now
                
        except Exception as e:
            self.log.error(f"Structured tick processing error: {e}")
    
    def handle_regime_update(self, data: Tuple) -> None:
        """Handle DOLPHIN regime updates"""
        try:
            timestamp, regime, bull_ratio, bear_ratio, sideways_ratio, analyzed, total, confidence = data
            
            # Update current regime
            old_regime = self.current_regime
            self.current_regime = MarketRegime(regime)
            
            # Add to history
            self.regime_history.append({
                'regime': regime,
                'bull_ratio': bull_ratio,
                'bear_ratio': bear_ratio,
                'confidence': confidence,
                'timestamp': timestamp
            })
            
            self.processed_regimes += 1
            
            # React to regime changes
            if old_regime != self.current_regime:
                self.on_regime_change(old_regime, self.current_regime, confidence)
            
        except Exception as e:
            self.log.error(f"Regime update processing error: {e}")
    
    def handle_custom_input(self, data: Any) -> None:
        """Handle custom input data"""
        # Override in derived classes
        pass
    
    def process_tick_data(self, tick: SiloqyTradeTick) -> Optional[Dict]:
        """Override in derived classes for tick processing"""
        # Implement component-specific tick processing
        return None
    
    async def perform_periodic_analysis(self) -> None:
        """Override in derived classes for periodic analysis"""
        # Implement component-specific periodic operations
        pass
    
    def on_regime_change(self, old_regime: MarketRegime, new_regime: MarketRegime, confidence: float) -> None:
        """Override in derived classes for regime change handling"""
        self.log.info(f"Regime change detected: {old_regime.value} → {new_regime.value} (confidence: {confidence:.1%})")

Component Registration Template

def create_siloqy_trading_node_with_custom_components():
    """Template for creating TradingNode with custom SILOQY components"""
    
    # Core SILOQY components (required)
    symbol_discovery_config = ImportableActorConfig(
        actor_path="nautilus_actor_test_implementation_5x_modified:SILOQYSymbolDiscoveryActor",
        config_path="nautilus_actor_test_implementation_5x_modified:SILOQYSymbolDiscoveryConfig",
        config={
            "component_id": "SILOQY-SYMBOL-DISCOVERY",
            "symbols": [],
            "candle_interval_ms": 15 * 60 * 1000,
            "throttle_mode": True,  # Safe for development
            "throttle_rate_limit_seconds": 10.0,
            "max_symbols_throttled": 100
        }
    )
    
    main_actor_config = ImportableActorConfig(
        actor_path="nautilus_actor_test_implementation_5x_modified:SILOQYMainActor",
        config_path="nautilus_actor_test_implementation_5x_modified:SILOQYMainActorConfig",
        config={
            "component_id": "SILOQY-MAIN-ACTOR",
            "candle_interval_ms": 15 * 60 * 1000,
            "throttle_mode": True,
            "enable_real_data": False  # Set to True for production
        }
    )
    
    regime_actor_config = ImportableActorConfig(
        actor_path="nautilus_actor_test_implementation_5x_modified:DOLPHINRegimeActor",
        config_path="nautilus_actor_test_implementation_5x_modified:DOLPHINRegimeActorConfig",
        config={
            "component_id": "DOLPHIN-REGIME-ACTOR",
            "max_symbols": 5000,
            "ticks_per_analysis": 1000
        }
    )
    
    normalizer_config = ImportableActorConfig(
        actor_path="nautilus_actor_test_implementation_5x_modified:SILOQYNormalizerActor",
        config_path="nautilus_actor_test_implementation_5x_modified:SILOQYNormalizerConfig",
        config={
            "component_id": "SILOQY-NORMALIZER"
        }
    )
    
    # Custom SILOQY components (add new components here)
    custom_component_config = ImportableActorConfig(
        actor_path="my_custom_module:MySILOQYComponent",
        config_path="my_custom_module:MySILOQYComponentConfig",
        config={
            "component_id": "MY-CUSTOM-SILOQY-COMPONENT",
            "component_name": "CustomAnalyzer",
            "processing_interval_ms": 1000,
            "enable_regime_tracking": True,
            "max_history_size": 10000
        }
    )
    
    # Complete TradingNode configuration
    trading_config = TradingNodeConfig(
        trader_id=TraderId("SILOQY-EXTENDED-SYSTEM"),
        actors=[
            symbol_discovery_config,
            main_actor_config,
            regime_actor_config,
            normalizer_config,
            custom_component_config  # Add all custom components here
        ],
        data_clients={},  # External data clients if needed
        exec_clients={}   # Execution clients if needed
    )
    
    # Create and return node
    node = TradingNode(config=trading_config)
    return node

Advanced Integration Patterns

Custom Data Type Integration

from nautilus_trader.core.data import Data

class SILOQYCustomData(Data):
    """Custom SILOQY data type for Nautilus integration"""
    
    def __init__(self, component_id: str, analysis_data: Dict, timestamp_ns: int):
        self.component_id = component_id
        self.analysis_data = analysis_data
        super().__init__(ts_event=timestamp_ns, ts_init=timestamp_ns)
    
    def __repr__(self) -> str:
        return f"SILOQYCustomData(component={self.component_id}, data_keys={list(self.analysis_data.keys())})"

# Usage in component
def publish_custom_analysis(self, analysis_result: Dict):
    custom_data = SILOQYCustomData(
        component_id=str(self.id),
        analysis_data=analysis_result,
        timestamp_ns=int(time.time_ns())
    )
    
    # Publish as Nautilus Data object
    if hasattr(self, 'publish_data'):
        self.publish_data(custom_data)
    else:
        # Fallback to message bus
        self.msgbus.publish("SILOQY.CUSTOM.ANALYSIS", custom_data)

Component Coordination Patterns

Master/Worker Pattern:

class SILOQYMasterComponent(Actor):
    """Master component that coordinates worker components"""
    
    def __init__(self, config):
        super().__init__(config)
        self.worker_topics = [
            "SILOQY.WORKER.CORRELATION",
            "SILOQY.WORKER.SENTIMENT", 
            "SILOQY.WORKER.MOMENTUM"
        ]
        self.worker_results = {}
    
    def distribute_work(self, work_data: Dict):
        """Distribute work to worker components"""
        for topic in self.worker_topics:
            work_package = {
                'master_id': str(self.id),
                'work_data': work_data,
                'response_topic': f"SILOQY.MASTER.{self.id}.RESULTS"
            }
            self.msgbus.publish(topic, work_package)
    
    def on_start(self):
        # Subscribe to worker responses
        self.msgbus.subscribe(f"SILOQY.MASTER.{self.id}.RESULTS", self.handle_worker_result)
        self.msgbus.subscribe(REGIME_TOPIC, self.handle_regime_change)

class SILOQYWorkerComponent(Actor):
    """Worker component that processes distributed tasks"""
    
    def on_start(self):
        self.msgbus.subscribe("SILOQY.WORKER.CORRELATION", self.handle_work_assignment)
    
    def handle_work_assignment(self, work_package: Dict):
        """Process assigned work and return results"""
        try:
            result = self.process_work(work_package['work_data'])
            
            response = {
                'worker_id': str(self.id),
                'result': result,
                'timestamp': int(time.time() * 1000)
            }
            
            self.msgbus.publish(work_package['response_topic'], response)
            
        except Exception as e:
            self.log.error(f"Work processing failed: {e}")

Production Deployment Checklist

Pre-Deployment Requirements

Configuration Validation: - [ ] throttle_mode: False for production performance - [ ] enable_real_data: True for live market data

• max_symbols_throttled: 5000+ for full market coverage - [ ] ticks_per_analysis: 1000 for stable regime detection

Component Integration Validation: - [ ] All custom components pass integration tests - [ ] MessageBus topic subscriptions verified - [ ] Actor dependency order confirmed - [ ] Error handling and recovery tested - [ ] Performance requirements met

System Resource Validation: - [ ] Memory allocation sufficient for all actors - [ ] CPU capacity adequate for real-time processing - [ ] Network bandwidth adequate for WebSocket streams - [ ] Storage capacity planned (if using persistent storage)

Production Configuration Template

def create_production_siloqy_node():
    """Production-ready SILOQY TradingNode configuration"""
    
    # Production symbol discovery
    symbol_discovery_config = ImportableActorConfig(
        actor_path="nautilus_actor_test_implementation_5x_modified:SILOQYSymbolDiscoveryActor",
        config_path="nautilus_actor_test_implementation_5x_modified:SILOQYSymbolDiscoveryConfig",
        config={
            "component_id": "SILOQY-SYMBOL-DISCOVERY-PROD",
            "symbols": [],  # Dynamic discovery
            "candle_interval_ms": 15 * 60 * 1000,
            "throttle_mode": False,  # PRODUCTION: Full performance
            "throttle_rate_limit_seconds": 2.5,  # Standard rate limit
            "max_symbols_throttled": 5000  # Full symbol coverage
        }
    )
    
    # Production main actor with real data
    main_actor_config = ImportableActorConfig(
        actor_path="nautilus_actor_test_implementation_5x_modified:SILOQYMainActor",
        config_path="nautilus_actor_test_implementation_5x_modified:SILOQYMainActorConfig",
        config={
            "component_id": "SILOQY-MAIN-ACTOR-PROD",
            "candle_interval_ms": 15 * 60 * 1000,
            "throttle_mode": False,  # PRODUCTION: Full performance
            "enable_real_data": True  # PRODUCTION: Real WebSocket data
        }
    )
    
    # Production DOLPHIN regime detection
    regime_actor_config = ImportableActorConfig(
        actor_path="nautilus_actor_test_implementation_5x_modified:DOLPHINRegimeActor",
        config_path="nautilus_actor_test_implementation_5x_modified:DOLPHINRegimeActorConfig",
        config={
            "component_id": "DOLPHIN-REGIME-ACTOR-PROD",
            "max_symbols": 5000,  # Full capacity
            "ticks_per_analysis": 1000  # Stable regime detection
        }
    )
    
    return TradingNodeConfig(
        trader_id=TraderId("SILOQY-PRODUCTION"),
        actors=[symbol_discovery_config, main_actor_config, regime_actor_config, normalizer_config],
        data_clients={},
        exec_clients={}
    )

Appendix: Message Bus Topic Reference

Complete Topic Hierarchy

SILOQY Root Topics:
├── SILOQY.RAW.TICKS              # Raw market tick data (tuple format)
├── SILOQY.STRUCTURED.TICKS       # Normalized SiloqyTradeTick objects
├── SILOQY.SYMBOLS.DISCOVERED     # Symbol discovery completion
├── SILOQY.CANDLES.INITIAL        # Candle reconstruction results
│
DOLPHIN Topics:
├── DOLPHIN.REGIME.RESULTS        # Market regime detection results
│
Custom Component Topics (Examples):
├── SILOQY.HD.VECTORS            # Hyperdimensional vector data
├── SILOQY.TEMPORAL.PATTERNS     # Temporal memory patterns  
├── SILOQY.CORRELATION.MATRIX    # Symbol correlation analysis
├── SILOQY.SENTIMENT.ANALYSIS    # Market sentiment data
├── SILOQY.RISK.METRICS         # Risk calculation results
├── SILOQY.ML.PREDICTIONS       # Machine learning predictions
├── SILOQY.ANALYSIS.REQUESTS    # Analysis request coordination
├── SILOQY.ANALYSIS.RESPONSES   # Analysis response delivery

Performance Characteristics by Topic

Topic

Frequency

Latency Requirement

Data Size

Thread Safety

RAW_TOPIC

1000+ msg/sec

< 5μs

200 bytes

Full

STRUCTURED_TOPIC

1000+ msg/sec

< 10μs

500 bytes

Full

REGIME_TOPIC

1-10 msg/sec

< 100μs

300 bytes

Full

Custom Topics

Variable

< 1ms

Variable

Full

This document provides the complete technical foundation for integrating new SILOQY components with the DOLPHIN market regime detection system within the Nautilus framework. All patterns, data structures, and lifecycle management approaches are based on the proven implementation that has processed 289+ million ticks in production operation.