diff --git a/prod/docs/VIOLET_FULL_SYSTEM_ASSESSMENT.md b/prod/docs/VIOLET_FULL_SYSTEM_ASSESSMENT.md new file mode 100644 index 00000000..b72b27af --- /dev/null +++ b/prod/docs/VIOLET_FULL_SYSTEM_ASSESSMENT.md @@ -0,0 +1,425 @@ +# VIOLET Full System Assessment + +This is the current source-faithful assessment of how the full VIOLET system works in this checkout, mapped against the BLUE doctrine in `prod/docs/SYSTEM_BIBLE_v7.md` and the VIOLET spec files. The current `SYSTEM_BIBLE_v7.md` file is a `v7.2` document in this tree; the BLUE v7 / v7.1 lineage is treated here as the same doctrinal source family. + +Scope: +- start from launcher and invocation points +- trace the BLUE-equivalent layers through the current VIOLET codepaths +- identify exactly where DITAv2 executes +- isolate ASEx involvement +- list the data inputs and input sites +- describe the VIOLET-only support systems +- record worktree provenance for the files and file-groups that implement the path + +What follows is a map of the live code, not a proposal. + +## 1. Canonical sources + +BLUE doctrine and system behavior: +- `prod/docs/SYSTEM_BIBLE_v7.md` +- `prod/docs/DITA_V2_KERNEL_REFERENCE.md` + +VIOLET doctrine and rollout intent: +- `prod/docs/VIOLET_DEV_SPEC_AND_PLAN.md` +- `prod/docs/VIOLET_DEV_SPEC_AND_PLAN_SOAK.md` +- `prod/docs/VIOLET_DEV_SPEC_AND_PLAN_SOAK__DEV_STATUS_20260627.md` +- `prod/docs/VIOLET_PASS_MM1_V4_READINESS_REPORTER.md` +- `prod/docs/ASEx_INTEGRATION_STATUS.md` +- `prod/docs/DITA_V2_OPERATOR_PLAYBOOK.md` + +Key implementation files: +- `prod/launch_dolphin_violet.py` +- `prod/launch_dolphin_violet_v4.py` +- `prod/clean_arch/violet/v4_execution_runner.py` +- `prod/clean_arch/violet/v4_arming.py` +- `prod/clean_arch/violet/v4_readiness.py` +- `prod/clean_arch/violet/shadow_live_factors.py` +- `prod/clean_arch/violet/live_blue_source.py` +- `prod/clean_arch/violet/live_factor_source.py` +- `prod/clean_arch/violet/shadow_journal.py` +- `prod/clean_arch/violet/divergence.py` +- `prod/clean_arch/violet/observe_guard.py` +- `prod/clean_arch/violet/pass2_8_soak_runner.py` +- `prod/clean_arch/violet/layer_parity_harness.py` +- `prod/clickhouse/violet/apply_violet_ddl.py` +- `prod/clickhouse/violet/*.sql` +- `prod/clean_arch/dita_v2/launcher.py` +- `prod/clean_arch/dita_v2/rust_backend.py` +- `prod/clean_arch/dita_v2/bingx_venue.py` +- `prod/clean_arch/dita_v2/mock_venue.py` +- `prod/clean_arch/dita_v2/control.py` +- `prod/clean_arch/dita_v2/projection.py` +- `prod/clean_arch/dita_v2/hazelcast_projection.py` +- `prod/supervisor/dolphin-supervisord.conf` +- `prod/supervisor/run_with_dolphin_env.sh` + +## 2. Invocation points + +### 2.1 Observe-only VIOLET + +Primary entrypoint: +- `prod/launch_dolphin_violet.py` + +What it does: +- fixes the VIOLET namespace before anything else reads env +- preflights all `dolphin_violet.*` tables with `SELECT 0 ... LIMIT 0` +- if tables are missing, it idles dark and names the DDL applier +- builds a DITAv2 launcher bundle, then wraps the venue in `ObserveOnlyVenue` +- writes VIOLET persistence and Hazelcast state into VIOLET-only maps and tables +- starts a divergence monitor and an optional muted shadow path +- never calls the execution kernel for live order placement + +Observed behavior: +- `dolphin_violet` is the observe-only runtime +- it is designed to be safe when BLUE is present because it reads BLUE state but does not write BLUE tables + +### 2.2 VIOLET live V4 execution + +Wrapper entrypoint: +- `prod/launch_dolphin_violet_v4.py` + +Live runner: +- `prod/clean_arch/violet/v4_execution_runner.py` + +What it does: +- consumes NG7 scans from Hazelcast +- converts scans into VIOLET decisions +- converts decisions into `ExecIntent` +- converts `ExecIntent` into DITAv2 `KernelIntent` +- submits the kernel intent to the DITAv2 execution kernel +- requires arming gates and VST-only settings before live execution is allowed + +Current runtime note: +- in this checkout the observe-only service is running +- the V4 execution service is registered but remains stopped because arming is not GO + +### 2.3 DITAv2 operator surface + +General DITAv2 launcher: +- `prod/launch_dita_v2.py` + +Operator playbook: +- `prod/docs/DITA_V2_OPERATOR_PLAYBOOK.md` + +This is the lower-level DITAv2 surface. VIOLET uses the same launcher bundle construction, but VIOLET adds its own scan ingestion, decision logic, shadow journal, arming gate, and execution wrapper on top. + +## 3. BLUE layers and VIOLET equivalents + +The BLUE system bible describes a layered trading stack. VIOLET mirrors that structure, but with its own namespaces and additional arming/execution scaffolding. + +| BLUE layer / concern | VIOLET equivalent | Main files | Input sites | Effect | +|---|---|---|---|---| +| Invocation / process control | VIOLET launcher and supervisor surface | `prod/launch_dolphin_violet.py`, `prod/launch_dolphin_violet_v4.py`, `prod/supervisor/dolphin-supervisord.conf`, `prod/supervisor/run_with_dolphin_env.sh` | env, supervisor, shell wrapper | selects observe-only or live V4 posture | +| Scan ingest / feature feed | Hazelcast NG7 scan source | `prod/clean_arch/violet/v4_execution_runner.py`, `prod/clean_arch/violet/live_blue_source.py` | `DOLPHIN_FEATURES.latest_eigen_scan` | provides scan payloads and BLUE live factors | +| Posture / capital sensing | BLUE posture and capital readback | `prod/clean_arch/violet/live_blue_source.py`, `prod/clean_arch/violet/live_factor_source.py` | `DOLPHIN_STATE_BLUE.latest_nautilus`, `DOLPHIN_STATE_BLUE.engine_snapshot` | feeds posture-aware factorization | +| Alpha / conviction / sizing | VIOLET decision engine and sizing helpers | `prod/clean_arch/violet/decision_engine.py`, `prod/clean_arch/violet/sizing.py`, `prod/clean_arch/violet/alpha_wrappers.py`, `prod/clean_arch/violet/v4_execution_runner.py` | scan payload, live BLUE factor source | produces `ExecIntent` from BLUE-faithful decision logic | +| Shadow audit / decision logging | Shadow journal and parity surfaces | `prod/clean_arch/violet/shadow_journal.py`, `prod/clean_arch/violet/parity_report.py` | decision rows, scan numbers | writes `dolphin_violet.violet_decisions` | +| Divergence / feed hygiene | VIOLET feed divergence monitor | `prod/clean_arch/violet/divergence.py` | scan stream and runtime samples | writes `dolphin_violet.violet_feed_divergence` | +| Execution boundary | DITAv2 kernel bridge | `prod/clean_arch/violet/v4_execution_runner.py`, `prod/clean_arch/dita_v2/launcher.py`, `prod/clean_arch/dita_v2/rust_backend.py`, `prod/clean_arch/dita_v2/bingx_venue.py` | `ExecIntent`, arming report, BingX env | runs actual DITAv2 execution path | +| Safety / no-touch guard | Observe-only venue wrapper | `prod/clean_arch/violet/observe_guard.py` | venue object | prevents any order placement in dark mode | +| Schema / persistence | VIOLET-only ClickHouse schema | `prod/clickhouse/violet/*.sql`, `prod/clickhouse/violet/apply_violet_ddl.py` | DDL files, table probes | creates and verifies `dolphin_violet.*` only | +| Readiness / arming | V4 arming gate | `prod/clean_arch/violet/v4_readiness.py`, `prod/clean_arch/violet/v4_arming.py` | soak reports, env, keys | decides whether V4 may arm | + +## 4. Exact execution flow + +### 4.1 Observe-only launcher flow + +`prod/launch_dolphin_violet.py` does this in order: +1. applies VIOLET env names +2. probes `dolphin_violet` tables +3. if tables are missing, logs and idles dark +4. if dark divergence is enabled or keys exist, starts the divergence task +5. if no VIOLET keys are present, idles dark +6. otherwise constructs a DITAv2 launcher bundle +7. wraps the venue in `ObserveOnlyVenue` +8. wires VIOLET persistence and Hazelcast state writers +9. builds the muted shadow path +10. runs the observe-only loop + +Important point: +- it does not submit live orders +- it does not execute `runtime.step()` +- it is a read-mostly / audit / shadow surface, not the live trading boundary + +### 4.2 Live V4 execution flow + +`prod/clean_arch/violet/v4_execution_runner.py` is the live boundary. The chain is: +1. `run_live()` loads env and runtime options +2. it applies VIOLET runtime env and optimizations +3. it builds the real DITAv2 bundle +4. it builds the shadow/live-factor source +5. it connects to Hazelcast NG7 scan source +6. it waits for a scan payload from `DOLPHIN_FEATURES.latest_eigen_scan` +7. `process_scan_payload()` parses and validates the payload +8. the scan number is deduplicated in guarded state +9. live factors are read from BLUE state and feature maps +10. the decision step produces a VIOLET decision +11. the decision is transformed to `ExecIntent` +12. the intent may be capped by configured notional limits +13. `ExecIntent` becomes DITAv2 `KernelIntent` +14. the runner mutates submission state +15. `bundle.kernel.process_intent_async(kernel_intent)` is awaited +16. the runner records outcome or error in guarded state + +The exact DITAv2 execution point is step 15. That is where VIOLET hands off from its own decision plane into DITAv2 execution. + +### 4.3 Where DITAv2 runs + +DITAv2 is constructed by `prod/clean_arch/dita_v2/launcher.py` as a launcher bundle: +- kernel +- control plane +- projection +- zinc plane +- venue adapter + +In VIOLET dark mode: +- the venue is wrapped in `ObserveOnlyVenue` +- the kernel exists, but the venue refuses live order placement + +In V4 execution mode: +- the same bundle is built with BingX VST configuration +- the intent is submitted through the kernel into the venue adapter +- that is the only place VIOLET is allowed to reach live execution + +## 5. Input data and input sites + +### 5.1 BLUE live inputs consumed by VIOLET + +VIOLET reads these BLUE surfaces as inputs: +- `DOLPHIN_STATE_BLUE.latest_nautilus` +- fallback `DOLPHIN_STATE_BLUE.engine_snapshot` +- `DOLPHIN_FEATURES.esof_latest` +- fallback `DOLPHIN_FEATURES.esof_advisor_latest` +- `DOLPHIN_FEATURES.mc_forewarner_latest` +- `DOLPHIN_FEATURES.latest_eigen_scan` + +These are read-only input sites. VIOLET should not mutate BLUE state maps. + +### 5.2 VIOLET scan input + +The active scan input for VIOLET is: +- Hazelcast map `DOLPHIN_FEATURES` +- key `latest_eigen_scan` + +That is the scan source the live runner consumes. The older scan bridge path is superseded in this layout. + +### 5.3 VIOLET persistence inputs + +VIOLET writes into: +- `dolphin_violet.policy_events` +- `dolphin_violet.trade_reconstruction` +- `dolphin_violet.trade_exit_legs` +- `dolphin_violet.position_state` +- `dolphin_violet.anomaly_events` +- `dolphin_violet.account_events` +- `dolphin_violet.status_snapshots` +- `dolphin_violet.trade_events` +- `dolphin_violet.v7_decision_events` +- `dolphin_violet.violet_feed_divergence` +- `dolphin_violet.violet_decisions` + +These are VIOLET-only tables. They are not BLUE tables and should not be used to backfill BLUE state directly. + +## 6. ASEx involvement + +ASEx exists in the tree, but the current VIOLET mainline use is narrow. + +Confirmed current VIOLET codepath: +- `prod/clean_arch/violet/v4_execution_runner.py` optionally imports: + - `asex.guarded.ASExGuardedState` + - `asex.worker.ASExWorker` +- if ASEx is unavailable, the runner falls back to a local guarded state implementation +- the ASEx wrapper is used for guarded serial state around scan/order lifecycle counters + +What this means: +- ASEx is not the execution engine +- ASEx is not the DITAv2 kernel +- ASEx is not the default VIOLET decision path +- it is an optional serial-state wrapper in the live runner boundary + +Cross-check from the integration status doc: +- ASEx is installed as an editable package +- VIOLET main has zero ASEx imports outside the live runner boundary +- DITAv2 main has zero ASEx imports +- the earlier PASS9 adapter work exists in another worktree and is not merged here + +Operational conclusion: +- ASEx is currently peripheral, not foundational, in this checkout +- the only current VIOLET runtime touchpoint is the guarded state wrapper inside `v4_execution_runner.py` + +## 7. VIOLET-only support systems + +### 7.1 Read-only dark guard + +`prod/clean_arch/violet/observe_guard.py` implements `ObserveOnlyVenue`. + +Purpose: +- wraps a venue +- raises on order placement +- allows the launcher to run a full decision stack without touching a live venue + +### 7.2 Shadow journal + +`prod/clean_arch/violet/shadow_journal.py` + +Purpose: +- persists executed shadow decisions to `dolphin_violet.violet_decisions` +- keeps the shadow path auditable +- matches the ClickHouse DDL in `prod/clickhouse/violet/22_violet_decisions.sql` + +### 7.3 Feed divergence monitor + +`prod/clean_arch/violet/divergence.py` + +Purpose: +- records feed divergence into `dolphin_violet.violet_feed_divergence` +- gives VIOLET its own drift surface separate from BLUE + +### 7.4 V4 readiness and arming + +`prod/clean_arch/violet/v4_readiness.py` +`prod/clean_arch/violet/v4_arming.py` + +Purpose: +- combine soak readiness, credential checks, namespace isolation, and launcher mode checks +- fail closed when the readiness report is not GO +- keep V4 from arming unless the environment says it may + +Current observed behavior in this checkout: +- arming is not GO +- the live runner therefore remains stopped + +### 7.5 Soak runner and parity harness + +`prod/clean_arch/violet/pass2_8_soak_runner.py` +`prod/clean_arch/violet/layer_parity_harness.py` + +Purpose: +- run the DARK -> MOCK -> VST canary progression used in VIOLET soak work +- compare VIOLET behavior against the expected layer parity surface +- keep execution preparation separate from the live V4 runner + +## 8. Storage and schema + +`prod/clickhouse/violet/apply_violet_ddl.py` is the schema applier for VIOLET. + +Important characteristics: +- it only targets `dolphin_violet` +- it probes table existence before use +- it does not create BLUE tables +- it applies each SQL file as a separate statement path + +Relevant SQL set: +- `00_create_database.sql` +- `01_policy_events.sql` +- `02_trade_reconstruction.sql` +- `03_trade_exit_legs.sql` +- `04_position_state.sql` +- `05_anomaly_events.sql` +- `06_account_events.sql` +- `07_status_snapshots.sql` +- `08_trade_events.sql` +- `09_v7_decision_events.sql` +- `10_adaptive_exit_shadow.sql` +- `11_fee_settled_events.sql` +- `12_sc_bucket_gauge_shadow.sql` +- `13_sc_threshold_advisor_shadow.sql` +- `20_violet_feed_divergence.sql` +- `22_violet_decisions.sql` + +The important storage boundary is simple: +- VIOLET uses `dolphin_violet.*` +- BLUE uses `dolphin.*` and `DOLPHIN_*` control/state maps +- VIOLET should not write into BLUE storage + +## 9. BLUE-equivalent layer mapping, expanded + +### 9.1 Data / scan layer + +BLUE uses `DOLPHIN_FEATURES.latest_eigen_scan` as the canonical scan source. + +VIOLET uses the same scan feed as its input, but via the live runner and shadow source: +- `prod/clean_arch/violet/v4_execution_runner.py` +- `prod/clean_arch/violet/shadow_live_factors.py` +- `prod/clean_arch/violet/live_blue_source.py` + +### 9.2 Signal / conviction layer + +BLUE has a large signal layer family in the system bible. + +VIOLET's equivalent is: +- `decision_engine.py` +- `sizing.py` +- `alpha_wrappers.py` +- `shadow_live_factors.py` +- `live_factor_source.py` + +This is where the BLUE-like live factors are folded into VIOLET conviction and sizing. + +### 9.3 Execution / accounting layer + +BLUE's execution/accounting boundary is the DITA family in the bible. + +VIOLET's equivalent is: +- `prod/clean_arch/dita_v2/launcher.py` +- `prod/clean_arch/dita_v2/rust_backend.py` +- `prod/clean_arch/dita_v2/bingx_venue.py` +- `prod/clean_arch/dita_v2/mock_venue.py` +- `prod/clean_arch/violet/v4_execution_runner.py` + +This is the actual order-facing path when V4 is armed. + +### 9.4 Observability / divergence + +BLUE's observability surfaces are mirrored in VIOLET by: +- `shadow_journal.py` +- `divergence.py` +- `v4_readiness.py` +- `v4_arming.py` +- `parity_report.py` + +These keep the VIOLET branch auditable without mutating BLUE. + +## 10. Provenance appendix + +Worktree state here means the current checkout state observed during this assessment. + +| File group | Worktree status | Last-known committer | Responsible actor | +|---|---|---|---| +| `prod/launch_dolphin_violet.py` | modified | Codex, `fb344318aa849fde13b934bdba7337819cc42743` | current local edit in this session | +| `prod/clickhouse/violet/apply_violet_ddl.py` | modified | Codex, `99e529c32ab353027fd8e53e5219d5c949ec06ce` | current local edit in this session | +| `prod/supervisor/dolphin-supervisord.conf` | modified | Codex, `02ecc55c1670019682b193a239c72ab0f66ab1fc` | current local edit in this session | +| `prod/clean_arch/violet/v4_execution_runner.py` | untracked | new file in worktree | Codex | +| `prod/clean_arch/violet/v4_arming.py` | untracked | new file in worktree | Codex | +| `prod/launch_dolphin_violet_v4.py` | untracked in this checkout view | new file in worktree | Codex | +| `prod/launch_dita_v2.py` | untracked in this checkout view | new file in worktree | Codex | +| `prod/supervisor/run_with_dolphin_env.sh` | untracked in this checkout view | new file in worktree | Codex | +| `prod/clean_arch/dita_v2/launcher.py` | clean in this worktree during this assessment | not revalidated here | upstream DITAv2 owner / baseline | +| `prod/clean_arch/dita_v2/rust_backend.py` | clean in this worktree during this assessment | not revalidated here | upstream DITAv2 owner / baseline | +| `prod/clean_arch/dita_v2/bingx_venue.py` | clean in this worktree during this assessment | not revalidated here | upstream DITAv2 owner / baseline | +| `prod/clean_arch/dita_v2/mock_venue.py` | clean in this worktree during this assessment | not revalidated here | upstream DITAv2 owner / baseline | +| `prod/clean_arch/violet/shadow_live_factors.py` | clean in this worktree during this assessment | not revalidated here | upstream VIOLET owner / baseline | +| `prod/clean_arch/violet/live_blue_source.py` | clean in this worktree during this assessment | not revalidated here | upstream VIOLET owner / baseline | +| `prod/clean_arch/violet/live_factor_source.py` | clean in this worktree during this assessment | not revalidated here | upstream VIOLET owner / baseline | +| `prod/clean_arch/violet/shadow_journal.py` | clean in this worktree during this assessment | not revalidated here | upstream VIOLET owner / baseline | +| `prod/clean_arch/violet/divergence.py` | clean in this worktree during this assessment | not revalidated here | upstream VIOLET owner / baseline | +| `prod/clean_arch/violet/observe_guard.py` | clean in this worktree during this assessment | not revalidated here | upstream VIOLET owner / baseline | +| `prod/clean_arch/violet/v4_readiness.py` | clean in this worktree during this assessment | not revalidated here | upstream VIOLET owner / baseline | +| `prod/docs/SYSTEM_BIBLE_v7.md` | clean in this worktree during this assessment | not revalidated here | doctrine baseline | +| `prod/docs/DITA_V2_KERNEL_REFERENCE.md` | clean in this worktree during this assessment | not revalidated here | doctrine baseline | +| `prod/docs/VIOLET_DEV_SPEC_AND_PLAN*.md` | clean in this worktree during this assessment | not revalidated here | doctrine baseline | + +Notes: +- for files modified or added in this session, the responsible actor is the current local worktree edit path +- for clean baseline files, I did not re-resolve the exact last commit for every single file in this pass +- the assessment is still source-faithful because the runtime path is anchored to the specific files above and the current worktree status is explicit + +## 11. Bottom line + +VIOLET in this checkout is a BLUE-faithful, VIOLET-namespaced stack with three distinct surfaces: +- observe-only VIOLET, which mirrors the full decision plane but blocks venue writes +- shadow / readiness / divergence, which audit and gate behavior +- live V4 execution, which feeds scan payloads into DITAv2 only after arming passes + +The decisive boundary is the handoff from `ExecIntent` into `KernelIntent` and then `bundle.kernel.process_intent_async(...)`. +That is where VIOLET stops being analysis and becomes actual execution through DITAv2.