DOLPHIN/prod/services/README.md

Dolphin Userland Services

Server-grade service management without root! Uses systemd --user for reliability.

🚀 Quick Start

# Check status
./service_manager.py status

# Start all services
./service_manager.py start

# View logs
./service_manager.py logs exf -f

📋 Service Overview

Service	File	Description	Interval
exf	dolphin-exf.service	External Factors (aggressive)	0.5s
ob	dolphin-ob.service	Order Book Streamer	500ms
watchdog	dolphin-watchdog.service	Survival Stack	10s
mc	dolphin-mc.timer	MC-Forewarner	4h

🔧 Service Manager Commands

# Status
./service_manager.py status          # All services
./service_manager.py status exf      # Specific service

# Control
./service_manager.py start           # Start all
./service_manager.py stop            # Stop all
./service_manager.py restart exf     # Restart specific

# Logs
./service_manager.py logs exf        # Last 50 lines
./service_manager.py logs exf -f     # Follow
./service_manager.py logs exf -n 100 # Last 100 lines

# Auto-start on boot
./service_manager.py enable          # Enable all
./service_manager.py disable         # Disable all

# After editing .service files
./service_manager.py reload          # Reload systemd

🏗️ Creating a New Service

Option 1: Full Service Base (Recommended)

#!/usr/bin/env python3
from services.service_base import ServiceBase

class MyService(ServiceBase):
    def __init__(self):
        super().__init__(
            name='my-service',
            check_interval=30,
            max_retries=3,
            notify_systemd=True
        )
    
    async def run_cycle(self):
        # Your logic here
        await do_work()
        await asyncio.sleep(1)  # Cycle interval
    
    async def health_check(self) -> bool:
        # Optional: custom health check
        return True

if __name__ == '__main__':
    MyService().run()

Create systemd service file:

cat > ~/.config/systemd/user/dolphin-my.service << 'SERVICEFILE'
[Unit]
Description=My Service
After=network.target

[Service]
Type=notify
ExecStart=/usr/bin/python3 /path/to/my_service.py
Restart=always
RestartSec=5
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=default.target
SERVICEFILE

# Enable and start
systemctl --user daemon-reload
systemctl --user enable dolphin-my.service
systemctl --user start dolphin-my.service

Option 2: Simple Scheduled Task

from services.service_base import run_scheduled

def my_task():
    print("Running...")

run_scheduled(my_task, interval_seconds=60, name='my-task')

📊 Features

Automatic

• Restart on crash: Services auto-restart with backoff
• Health checks: Built-in monitoring
• Structured logging: JSON to systemd journal
• Resource limits: Memory/CPU quotas
• Graceful shutdown: SIGTERM handling

Retry Logic (Tenacity)

@ServiceBase.retry_with_backoff
async def fetch_data(self):
    # Automatically retries with exponential backoff
    pass

Health Check Endpoint

Services expose health via Hazelcast or file:

async def health_check(self) -> bool:
    return self.last_update > time.time() - 2.0

📝 Logging

All services log structured JSON:

{
  "timestamp": "2024-03-25T15:30:00",
  "level": "INFO",
  "service": "exf",
  "message": "Indicators updated"
}

View logs:

# All services
journalctl --user -f

# Specific service
journalctl --user -u dolphin-exf -f

🔍 Monitoring

# Service status
systemctl --user status

# Resource usage
systemctl --user show dolphin-exf --property=MemoryCurrent,CPUUsageNSec

# Recent failures
systemctl --user --failed

🛠️ Troubleshooting

Issue	Solution
Service won't start	Check journalctl --user -u dolphin-exf
High memory usage	Adjust MemoryMax= in .service file
Restart loop	Check exit code: systemctl --user status exf
Logs not showing	Ensure StandardOutput=journal
Permission denied	Service files must be in ~/.config/systemd/user/

🔄 Service Dependencies

exf -> hazelcast
ob  -> hazelcast, exf
watchdog -> hazelcast, exf, ob
mc -> hazelcast (timer-triggered)

Configured via After= and Wants= in service files.