136 lines
6.2 KiB
Markdown
136 lines
6.2 KiB
Markdown
|
# DOLPHIN-NAUTILUS SYSTEM BIBLE
|
||
|
|
## Doctrinal Reference — As Running 2026-03-23
|
||
|
|
|
||
|
|
**Version**: v3 — Meta-System Monitoring (MHD) Integration
|
||
|
|
**Previous version**: `SYSTEM_BIBLE.md` (v2, forked 2026-03-23)
|
||
|
|
**CI gate (Nautilus)**: 46/46 tests green (11 bootstrap + 35 DolphinActor)
|
||
|
|
**CI gate (OBF)**: ~120 unit tests green
|
||
|
|
**MHD Health Status**: GREEN (MHD integrated and operational)
|
||
|
|
**Status**: Paper trading ready. NOT deployed with real capital.
|
||
|
|
|
||
|
|
### What changed since v2 (2026-03-22)
|
||
|
|
|
||
|
|
| Area | Change |
|
||
|
|
|---|---|
|
||
|
|
| **Meta-System** | **MetaHealthDaemon (MHD)** implemented — standalone watchdog monitoring liveness, freshness, and coherence. |
|
||
|
|
| **Orchestration** | MHD auto-restart logic for infrastructure (HZ/Prefect) added. |
|
||
|
|
| **Cross-Platform** | Native support for both Linux (systemd) and FreeBSD (rc.d) service management. |
|
||
|
|
| **Observability** | `meta_health.json` + `DOLPHIN_META_HEALTH` HZ map for L2 health state tracking. |
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## TABLE OF CONTENTS
|
||
|
|
|
||
|
|
1. [System Philosophy](#1-system-philosophy)
|
||
|
|
2. [Physical Architecture](#2-physical-architecture)
|
||
|
|
3. [Data Layer](#3-data-layer)
|
||
|
|
4. [Signal Layer — vel_div & DC](#4-signal-layer)
|
||
|
|
5. [Asset Selection — IRP](#5-asset-selection-irp)
|
||
|
|
6. [Position Sizing — AlphaBetSizer](#6-position-sizing)
|
||
|
|
7. [Exit Management](#7-exit-management)
|
||
|
|
8. [Fee & Slippage Model](#8-fee--slippage-model)
|
||
|
|
9. [OB Intelligence Layer](#9-ob-intelligence-layer)
|
||
|
|
10. [ACB v6 — Adaptive Circuit Breaker](#10-acb-v6)
|
||
|
|
11. [Survival Stack — Posture Control](#11-survival-stack)
|
||
|
|
12. [MC-Forewarner Envelope Gate](#12-mc-forewarner-envelope-gate)
|
||
|
|
13. [NDAlphaEngine — Full Bar Loop](#13-ndalpha-engine-full-bar-loop)
|
||
|
|
14. [DolphinActor — Nautilus Integration](#14-dolphin-actor)
|
||
|
|
15. [Hazelcast — Full IMap Schema](#15-hazelcast-full-imap-schema)
|
||
|
|
16. [Production Daemon Topology](#16-production-daemon-topology)
|
||
|
|
17. [Prefect Orchestration Layer](#17-prefect-orchestration-layer)
|
||
|
|
18. [CI Test Suite](#18-ci-test-suite)
|
||
|
|
19. [Parameter Reference](#19-parameter-reference)
|
||
|
|
20. [OBF Sprint 1 Hardening](#20-obf-sprint-1-hardening)
|
||
|
|
21. [Known Research TODOs](#21-known-research-todos)
|
||
|
|
22. [0.1s Resolution — Readiness Assessment](#22-01s-resolution-readiness-assessment)
|
||
|
|
23. [MetaHealthDaemon (MHD) — Meta-System Monitoring](#23-meta-health-daemon-mhd)
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
*(Sections 1-22 remain unchanged as per v2 specification. See v2 for details.)*
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 23. MetaHealthDaemon (MHD) — Meta-System Monitoring
|
||
|
|
|
||
|
|
### 23.1 Purpose & Design Philosophy
|
||
|
|
|
||
|
|
The **MetaHealthDaemon (MHD)** is a "Watchdog of Watchdogs" (Layer 2). While the Survival Stack (L1) monitors trading risk and execution health, the MHD monitors the **liveness and validity of the entire system infrastructure**.
|
||
|
|
|
||
|
|
**Core Principles**:
|
||
|
|
- **Statelessness**: No local state beyond the current check cycle.
|
||
|
|
- **Dependency-Light**: Operates even if Hazelcast, Prefect, or Network are down.
|
||
|
|
- **Hierarchical Reliability**: Uses 5 orthogonal "Meta-Sensors" to compute a system-wide health score (`Rm_meta`).
|
||
|
|
- **Platform Agnostic**: Native support for Linux (Red Hat) and FreeBSD.
|
||
|
|
|
||
|
|
### 23.2 MHD Physical Files
|
||
|
|
|
||
|
|
| File | Purpose | Location |
|
||
|
|
|---|---|---|
|
||
|
|
| `meta_health_daemon.py` | Core daemon logic | `prod/` |
|
||
|
|
| `meta_health_daemon.service` | Systemd unit (Linux) | `prod/` (deployed to `/etc/systemd/`) |
|
||
|
|
| `meta_health_daemon_bsd.rc` | rc.d script (FreeBSD) | `prod/` (deployed to `/usr/local/etc/rc.d/`) |
|
||
|
|
| `meta_health.json` | Latest health report (JSON) | `run_logs/` |
|
||
|
|
| `meta_health.log` | Persistent diagnostic log | `run_logs/` |
|
||
|
|
|
||
|
|
### 23.3 The 5 Meta-Sensors (M1-M5)
|
||
|
|
|
||
|
|
MHD computes health using the product of 5 sensors: `Rm_meta = M1 * M2 * M3 * M4 * M5`.
|
||
|
|
|
||
|
|
| Sensor | Name | Logic | Failure Threshold |
|
||
|
|
|---|---|---|---|
|
||
|
|
| **M1** | **Process Integrity** | `psutil` check for `hazelcast`, `prefect`, `watchdog_service`, `acb_processor`. | Any process missing → 0.0 |
|
||
|
|
| **M2** | **Heartbeat Freshness** | Age of `nautilus_flow_heartbeat` in HZ `DOLPHIN_HEARTBEAT` map. | Age > 30s → 0.0 |
|
||
|
|
| **M3** | **Data Freshness** | Mtime age of `latest_ob_features.json` on disk. | Age > 10s → 0.0 |
|
||
|
|
| **M4** | **Control Plane** | TCP connect response on ports 5701 (HZ) and 4200 (Prefect). | Port closed → 0.2 (partial) |
|
||
|
|
| **M5** | **Health Coherence** | Schema and validity check of L1 `DOLPHIN_SAFETY` (Rm ∈ [0,1], valid posture). | Invalid/Stale (>60s) → 0.0 |
|
||
|
|
|
||
|
|
### 23.4 Meta-Health Postures
|
||
|
|
|
||
|
|
| Rm_meta | Status | Meaning | Action |
|
||
|
|
|---|---|---|---|
|
||
|
|
| **≥ 0.80** | **GREEN** | System fully operational. | Nominal logging. |
|
||
|
|
| **≥ 0.50** | **DEGRADED** | Partial sensor failure (e.g., stale heartbeats). | Warning log + HZ alert. |
|
||
|
|
| **≥ 0.20** | **CRITICAL** | Severe infrastructure failure (e.g., HZ down, but processes up). | Critical alert. |
|
||
|
|
| **< 0.20** | **DEAD** | System core collapsed. | **Infrastructure auto-restart cycle.** |
|
||
|
|
|
||
|
|
### 23.5 Operations Guide
|
||
|
|
|
||
|
|
#### 23.5.1 Deployment (Linux)
|
||
|
|
```bash
|
||
|
|
sudo cp prod/meta_health_daemon.service /etc/systemd/system/
|
||
|
|
sudo systemctl daemon-reload
|
||
|
|
sudo systemctl enable --now meta_health_daemon
|
||
|
|
```
|
||
|
|
|
||
|
|
#### 23.5.2 Deployment (FreeBSD)
|
||
|
|
```bash
|
||
|
|
sudo cp prod/meta_health_daemon_bsd.rc /usr/local/etc/rc.d/meta_health_daemon
|
||
|
|
sudo chmod +x /usr/local/etc/rc.d/meta_health_daemon
|
||
|
|
sudo sysrc meta_health_daemon_enable="YES"
|
||
|
|
sudo service meta_health_daemon start
|
||
|
|
```
|
||
|
|
|
||
|
|
#### 23.5.3 Monitoring & Debugging
|
||
|
|
- **Live State**: `tail -f run_logs/meta_health.log`
|
||
|
|
- **JSON API**: `cat run_logs/meta_health.json` (used by dashboards/CLIs).
|
||
|
|
- **HZ State**: Read `DOLPHIN_META_HEALTH["latest"]` for remote monitoring.
|
||
|
|
|
||
|
|
#### 23.5.4 Restart Logic
|
||
|
|
When `Rm_meta` falls into the `DEAD` zone (<0.20), MHD attempts to restart:
|
||
|
|
1. **Level 1**: Restart `hazelcast` service.
|
||
|
|
2. **Level 2**: Restart `prefect_worker` / `prefect_server`.
|
||
|
|
3. **Level 3**: Restart core daemons (`acb_processor`, `watchdog`).
|
||
|
|
|
||
|
|
Restarts are gated by a cooldown and platform-native commands (`systemctl` or `service`).
|
||
|
|
|
||
|
|
#### 23.5.5 Manual Overrides
|
||
|
|
To disable MHD auto-actions without stopping it, create an empty file: `touch /tmp/MHD_PAUSE_ACTION`. (MHD will continue reporting but skip `attempt_restart`).
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
*End of DOLPHIN-NAUTILUS System Bible v3.0 — 2026-03-23*
|
||
|
|
*Champion: SHORT only (APEX posture, blue configuration)*
|
||
|
|
*Meta-System: MHD v1.0 active*
|
||
|
|
*Status: Paper trading ready. Meta-system "Gold Certified".*
|