DOLPHIN/prod/services/INDUSTRIAL_FRAMEWORKS.md

Industrial-Grade Service Frameworks

🏆 Recommendation: Supervisor

Supervisor is the industry standard for process management in Python deployments.

Why Supervisor?

• ✅ Battle-tested: Used by millions of production systems
• ✅ Mature: 20+ years of development
• ✅ Simple: INI-style configuration
• ✅ Reliable: Handles crashes, restarts, logging automatically
• ✅ Web UI: Built-in web interface for monitoring
• ✅ API: XML-RPC API for programmatic control

---

Quick Start: Supervisor

# 1. Start supervisor and all services
cd /mnt/dolphinng5_predict/prod/supervisor
./supervisorctl.sh start

# 2. Check status
./supervisorctl.sh status

# 3. View logs
./supervisorctl.sh logs exf
./supervisorctl.sh logs ob_streamer

# 4. Restart a service
./supervisorctl.sh ctl restart exf

# 5. Stop everything
./supervisorctl.sh stop

---

Alternative: Circus (Mozilla)

Circus is Mozilla's Python process & socket manager.

Pros:

• ✅ Python-native (easier to extend)
• ✅ Built-in statistics (CPU, memory per process)
• ✅ Socket management
• ✅ Web dashboard

Cons:

• ❌ Less widely used than Supervisor
• ❌ Smaller community

# Install
pip install circus

# Run
circusd circus.ini

---

Alternative: Honcho (Python Foreman)

Honcho is a Python port of Ruby's Foreman.

Pros:

• ✅ Very simple (Procfile-based)
• ✅ Good for development
• ✅ Easy to understand

Cons:

• ❌ Less production features
• ❌ No auto-restart on crash

# Procfile
exf: python -m external_factors.realtime_exf_service
ob: python -m services.ob_stream_service
watchdog: python -m services.system_watchdog_service

# Run
honcho start

---

Comparison Table

Feature	Supervisor	Circus	Honcho	Custom Code
Auto-restart	✅	✅	❌	✅ (if built)
Web UI	✅	✅	❌	❌
Log rotation	✅	✅	❌	⚠️ (manual)
Resource limits	✅	✅	❌	⚠️ (partial)
API	✅ XML-RPC	✅	❌	❌
Maturity	⭐⭐⭐	⭐⭐	⭐⭐	⭐
Ease of use	⭐⭐	⭐⭐	⭐⭐⭐	⭐⭐

---

Our Setup: Supervisor

Location: /mnt/dolphinng5_predict/prod/supervisor/

Config: dolphin-supervisord.conf

Services managed:

• exf - External Factors (0.5s)
• ob_streamer - Order Book (0.5s)
• watchdog - Survival Stack (10s)
• mc_forewarner - MC-Forewarner (4h)

Features enabled:

• Auto-restart with backoff
• Separate stdout/stderr logs
• Log rotation (50MB, 10 backups)
• Process groups
• Event listeners (alerts)

---

Integration with Existing Code

Your existing service code works unchanged with Supervisor:

# Your existing service (works with Supervisor)
class ExFService:
    def run(self):
        while True:
            self.fetch_indicators()
            self.push_to_hz()
            time.sleep(0.5)

# Supervisor handles:
# - Starting it
# - Restarting if it crashes
# - Logging stdout/stderr
# - Monitoring

No code changes needed!

---

Web Dashboard

Supervisor includes a web interface:

[inet_http_server]
port=0.0.0.0:9001
username=user
password=pass

Then visit: http://localhost:9001

---

Summary

Use Case	Recommendation
Production trading system	Supervisor ✅
Development/Testing	Honcho
Need sockets + stats	Circus
Maximum control	Custom + systemd

We recommend Supervisor for Dolphin production.

---

CHANGE LOG - All Modifications Made

Session: 2026-03-25 (Current Session)

1. Supervisor Installation

Command executed:

pip install supervisor

Result: Supervisor 4.3.0 installed

---

2. Directory Structure Created

/mnt/dolphinng5_predict/prod/supervisor/
├── dolphin-supervisord.conf     # Main supervisor configuration
├── supervisorctl.sh             # Control wrapper script
├── logs/                        # Log directory (created)
└── run/                         # PID/socket directory (created)

---

3. Configuration File: dolphin-supervisord.conf

Location: /mnt/dolphinng5_predict/prod/supervisor/dolphin-supervisord.conf

Contents:

• [supervisord] section with logging, pidfile, environment
• [unix_http_server] for supervisorctl communication
• [rpcinterface:supervisor] for API
• [supervisorctl] client configuration
• [program:exf] - External Factors service (0.5s)
• [program:ob_streamer] - Order Book Streamer (0.5s)
• [program:watchdog] - Survival Stack Watchdog (10s)
• [program:mc_forewarner] - MC-Forewarner (4h)
• [eventlistener:crashmail] - Alert on crashes
• [group:dolphin] - Group all programs

Key settings:

• autostart=true - All services start with supervisor
• autorestart=true - Auto-restart on crash
• startretries=3 - 3 restart attempts
• stdout_logfile_maxbytes=50MB - Log rotation
• rlimit_as=512MB - Memory limit per service

---

4. Control Script: supervisorctl.sh

Location: /mnt/dolphinng5_predict/prod/supervisor/supervisorctl.sh

Commands implemented:

• start - Start supervisord and all services
• stop - Stop all services and supervisord
• restart - Restart all services
• status - Show service status
• logs [service] - Show logs (last 50 lines)
• ctl [cmd] - Pass through to supervisorctl

Usage:

./supervisorctl.sh start
./supervisorctl.sh status
./supervisorctl.sh logs exf

---

5. Python Libraries Installed

Via pip:

• supervisor==4.3.0 - Main process manager
• tenacity==9.1.4 - Retry logic (previously installed)
• schedule==1.2.2 - Task scheduling (previously installed)

System packages checked:

• supervisor.noarch available via dnf (not installed, using pip)

---

6. Alternative Architectures (Previously Created)

6.1 Custom Supervisor (Pure Python)

Location: /mnt/dolphinng5_predict/prod/services/supervisor.py

Features:

• ServiceComponent base class
• DolphinSupervisor manager
• Thread-based component management
• Built-in health monitoring
• Example components: ExF, OB, Watchdog, MC

Status: Available but NOT primary (Supervisor preferred)

---

6.2 Systemd User Services

Location: ~/.config/systemd/user/

Files created:

• dolphin-exf.service - External Factors
• dolphin-ob.service - Order Book
• dolphin-watchdog.service - Watchdog
• dolphin-mc.service + dolphin-mc.timer - MC-Forewarner
• dolphin-supervisor.service - Custom supervisor (optional)
• dolphin-test.service - Test service

Control script: /mnt/dolphinng5_predict/prod/services/service_manager.py

---

7. Service Base Class (Boilerplate)

Location: /mnt/dolphinng5_predict/prod/services/service_base.py

Features:

• ServiceBase abstract class
• Automatic retries with tenacity
• Structured JSON logging
• Health check endpoints
• Graceful shutdown handling
• Systemd notify support
• run_scheduled() helper

Status: Available for custom implementations

---

8. Documentation Files Created

File	Location	Purpose
INDUSTRIAL_FRAMEWORKS.md	/mnt/dolphinng5_predict/prod/services/	This document - framework comparison
ARCHITECTURE_CHOICE.md	/mnt/dolphinng5_predict/prod/services/	Architecture options comparison
README.md	/mnt/dolphinng5_predict/prod/services/	General services documentation
dolphin-supervisord.conf	/mnt/dolphinng5_predict/prod/supervisor/	Supervisor configuration

---

9. kimi.json Updated

Change: Associated session with ops directory

Before:

{
  "path": "/mnt/dolphinng5_predict/prod/ops",
  "kaos": "local",
  "last_session_id": null
}

After:

{
  "path": "/mnt/dolphinng5_predict/prod/ops",
  "kaos": "local",
  "last_session_id": "c23a69c5-ba4a-41c4-8624-05114e8fd9ea"
}

---

10. Session Backup

Session backed up: c23a69c5-ba4a-41c4-8624-05114e8fd9ea

• Original location: ~/.kimi/sessions/9330f053b5f85e950222ed1fed8f6f02/
• Backup location 1: /mnt/dolphinng5_predict/prod/ops/kimi_session_backup/
• Backup location 2: /mnt/vids/
• Markdown transcript: KIMI_Session_Rearch_Services-Prefect.md (684KB)

---

Summary: What to Use

For Production Trading System:

Recommended: SUPERVISOR

cd /mnt/dolphinng5_predict/prod/supervisor
./supervisorctl.sh start
./supervisorctl.sh status

Why: Battle-tested, 20+ years, web UI, API, log rotation

For Simplicity / No Extra Deps:

Alternative: SYSTEMD --user

systemctl --user start dolphin-exf
systemctl --user start dolphin-ob
systemctl --user start dolphin-watchdog

Why: Built-in, no pip installs, OS-integrated

For Full Control:

Alternative: Custom Python

systemctl --user start dolphin-supervisor  # Custom one

Why: Educational, customizable, no external deps

---

Files Modified/Created Summary

New Directories:

1. /mnt/dolphinng5_predict/prod/supervisor/
2. /mnt/dolphinng5_predict/prod/supervisor/logs/
3. /mnt/dolphinng5_predict/prod/supervisor/run/
4. /mnt/dolphinng5_predict/prod/ops/kimi_session_backup/

New Files:

1. /mnt/dolphinng5_predict/prod/supervisor/dolphin-supervisord.conf
2. /mnt/dolphinng5_predict/prod/supervisor/supervisorctl.sh
3. /mnt/dolphinng5_predict/prod/services/INDUSTRIAL_FRAMEWORKS.md (this file)
4. /mnt/dolphinng5_predict/prod/services/ARCHITECTURE_CHOICE.md
5. /mnt/dolphinng5_predict/prod/services/supervisor.py (custom impl)
6. /mnt/dolphinng5_predict/prod/services/service_base.py (boilerplate)
7. /mnt/dolphinng5_predict/prod/services/service_manager.py (systemd ctl)
8. /mnt/dolphinng5_predict/prod/ops/KIMI_Session_Rearch_Services-Prefect.md
9. /mnt/dolphinng5_predict/prod/ops/SESSION_INFO.txt
10. /mnt/dolphinng5_predict/prod/ops/resume_session.sh

Modified Files:

1. ~/.config/systemd/user/dolphin-*.service (6 services)
2. ~/.config/systemd/user/dolphin-mc.timer
3. ~/.kimi/kimi.json (session association)

---

Current Status

✅ Supervisor 4.3.0 installed and configured ✅ 6 systemd user services configured (backup option) ✅ Custom supervisor available (educational) ✅ Service base class with retries/logging (boilerplate) ✅ All documentation complete ✅ Session backed up to multiple locations

Ready for: Production deployment