hjnormey/siloqy
Public Access
1
0
Fork 0
Code Issues 7 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

docs: add VIOLET full system assessment

Browse Source
This commit is contained in:
Codex
2026-06-27 14:08:55 +02:00
parent 37748cf180
commit 1db4802ee8
1 changed files with 425 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

425
prod/docs/VIOLET_FULL_SYSTEM_ASSESSMENT.md Normal file
View File

@@ -0,0 +1,425 @@
# VIOLET Full System Assessment
This is the current source-faithful assessment of how the full VIOLET system works in this checkout, mapped against the BLUE doctrine in `prod/docs/SYSTEM_BIBLE_v7.md` and the VIOLET spec files. The current `SYSTEM_BIBLE_v7.md` file is a `v7.2` document in this tree; the BLUE v7 / v7.1 lineage is treated here as the same doctrinal source family.
Scope:
- start from launcher and invocation points
- trace the BLUE-equivalent layers through the current VIOLET codepaths
- identify exactly where DITAv2 executes
- isolate ASEx involvement
- list the data inputs and input sites
- describe the VIOLET-only support systems
- record worktree provenance for the files and file-groups that implement the path
What follows is a map of the live code, not a proposal.
## 1. Canonical sources
BLUE doctrine and system behavior:
- `prod/docs/SYSTEM_BIBLE_v7.md`
- `prod/docs/DITA_V2_KERNEL_REFERENCE.md`
VIOLET doctrine and rollout intent:
- `prod/docs/VIOLET_DEV_SPEC_AND_PLAN.md`
- `prod/docs/VIOLET_DEV_SPEC_AND_PLAN_SOAK.md`
- `prod/docs/VIOLET_DEV_SPEC_AND_PLAN_SOAK__DEV_STATUS_20260627.md`
- `prod/docs/VIOLET_PASS_MM1_V4_READINESS_REPORTER.md`
- `prod/docs/ASEx_INTEGRATION_STATUS.md`
- `prod/docs/DITA_V2_OPERATOR_PLAYBOOK.md`
Key implementation files:
- `prod/launch_dolphin_violet.py`
- `prod/launch_dolphin_violet_v4.py`
- `prod/clean_arch/violet/v4_execution_runner.py`
- `prod/clean_arch/violet/v4_arming.py`
- `prod/clean_arch/violet/v4_readiness.py`
- `prod/clean_arch/violet/shadow_live_factors.py`
- `prod/clean_arch/violet/live_blue_source.py`
- `prod/clean_arch/violet/live_factor_source.py`
- `prod/clean_arch/violet/shadow_journal.py`
- `prod/clean_arch/violet/divergence.py`
- `prod/clean_arch/violet/observe_guard.py`
- `prod/clean_arch/violet/pass2_8_soak_runner.py`
- `prod/clean_arch/violet/layer_parity_harness.py`
- `prod/clickhouse/violet/apply_violet_ddl.py`
- `prod/clickhouse/violet/*.sql`
- `prod/clean_arch/dita_v2/launcher.py`
- `prod/clean_arch/dita_v2/rust_backend.py`
- `prod/clean_arch/dita_v2/bingx_venue.py`
- `prod/clean_arch/dita_v2/mock_venue.py`
- `prod/clean_arch/dita_v2/control.py`
- `prod/clean_arch/dita_v2/projection.py`
- `prod/clean_arch/dita_v2/hazelcast_projection.py`
- `prod/supervisor/dolphin-supervisord.conf`
- `prod/supervisor/run_with_dolphin_env.sh`
## 2. Invocation points
### 2.1 Observe-only VIOLET
Primary entrypoint:
- `prod/launch_dolphin_violet.py`
What it does:
- fixes the VIOLET namespace before anything else reads env
- preflights all `dolphin_violet.*` tables with `SELECT 0 ... LIMIT 0`
- if tables are missing, it idles dark and names the DDL applier
- builds a DITAv2 launcher bundle, then wraps the venue in `ObserveOnlyVenue`
- writes VIOLET persistence and Hazelcast state into VIOLET-only maps and tables
- starts a divergence monitor and an optional muted shadow path
- never calls the execution kernel for live order placement
Observed behavior:
- `dolphin_violet` is the observe-only runtime
- it is designed to be safe when BLUE is present because it reads BLUE state but does not write BLUE tables
### 2.2 VIOLET live V4 execution
Wrapper entrypoint:
- `prod/launch_dolphin_violet_v4.py`
Live runner:
- `prod/clean_arch/violet/v4_execution_runner.py`
What it does:
- consumes NG7 scans from Hazelcast
- converts scans into VIOLET decisions
- converts decisions into `ExecIntent`
- converts `ExecIntent` into DITAv2 `KernelIntent`
- submits the kernel intent to the DITAv2 execution kernel
- requires arming gates and VST-only settings before live execution is allowed
Current runtime note:
- in this checkout the observe-only service is running
- the V4 execution service is registered but remains stopped because arming is not GO
### 2.3 DITAv2 operator surface
General DITAv2 launcher:
- `prod/launch_dita_v2.py`
Operator playbook:
- `prod/docs/DITA_V2_OPERATOR_PLAYBOOK.md`
This is the lower-level DITAv2 surface. VIOLET uses the same launcher bundle construction, but VIOLET adds its own scan ingestion, decision logic, shadow journal, arming gate, and execution wrapper on top.
## 3. BLUE layers and VIOLET equivalents
The BLUE system bible describes a layered trading stack. VIOLET mirrors that structure, but with its own namespaces and additional arming/execution scaffolding.
| BLUE layer / concern | VIOLET equivalent | Main files | Input sites | Effect |
|---|---|---|---|---|
| Invocation / process control | VIOLET launcher and supervisor surface | `prod/launch_dolphin_violet.py`, `prod/launch_dolphin_violet_v4.py`, `prod/supervisor/dolphin-supervisord.conf`, `prod/supervisor/run_with_dolphin_env.sh` | env, supervisor, shell wrapper | selects observe-only or live V4 posture |
| Scan ingest / feature feed | Hazelcast NG7 scan source | `prod/clean_arch/violet/v4_execution_runner.py`, `prod/clean_arch/violet/live_blue_source.py` | `DOLPHIN_FEATURES.latest_eigen_scan` | provides scan payloads and BLUE live factors |
| Posture / capital sensing | BLUE posture and capital readback | `prod/clean_arch/violet/live_blue_source.py`, `prod/clean_arch/violet/live_factor_source.py` | `DOLPHIN_STATE_BLUE.latest_nautilus`, `DOLPHIN_STATE_BLUE.engine_snapshot` | feeds posture-aware factorization |
| Alpha / conviction / sizing | VIOLET decision engine and sizing helpers | `prod/clean_arch/violet/decision_engine.py`, `prod/clean_arch/violet/sizing.py`, `prod/clean_arch/violet/alpha_wrappers.py`, `prod/clean_arch/violet/v4_execution_runner.py` | scan payload, live BLUE factor source | produces `ExecIntent` from BLUE-faithful decision logic |
| Shadow audit / decision logging | Shadow journal and parity surfaces | `prod/clean_arch/violet/shadow_journal.py`, `prod/clean_arch/violet/parity_report.py` | decision rows, scan numbers | writes `dolphin_violet.violet_decisions` |
| Divergence / feed hygiene | VIOLET feed divergence monitor | `prod/clean_arch/violet/divergence.py` | scan stream and runtime samples | writes `dolphin_violet.violet_feed_divergence` |
| Execution boundary | DITAv2 kernel bridge | `prod/clean_arch/violet/v4_execution_runner.py`, `prod/clean_arch/dita_v2/launcher.py`, `prod/clean_arch/dita_v2/rust_backend.py`, `prod/clean_arch/dita_v2/bingx_venue.py` | `ExecIntent`, arming report, BingX env | runs actual DITAv2 execution path |
| Safety / no-touch guard | Observe-only venue wrapper | `prod/clean_arch/violet/observe_guard.py` | venue object | prevents any order placement in dark mode |
| Schema / persistence | VIOLET-only ClickHouse schema | `prod/clickhouse/violet/*.sql`, `prod/clickhouse/violet/apply_violet_ddl.py` | DDL files, table probes | creates and verifies `dolphin_violet.*` only |
| Readiness / arming | V4 arming gate | `prod/clean_arch/violet/v4_readiness.py`, `prod/clean_arch/violet/v4_arming.py` | soak reports, env, keys | decides whether V4 may arm |
## 4. Exact execution flow
### 4.1 Observe-only launcher flow
`prod/launch_dolphin_violet.py` does this in order:
1. applies VIOLET env names
2. probes `dolphin_violet` tables
3. if tables are missing, logs and idles dark
4. if dark divergence is enabled or keys exist, starts the divergence task
5. if no VIOLET keys are present, idles dark
6. otherwise constructs a DITAv2 launcher bundle
7. wraps the venue in `ObserveOnlyVenue`
8. wires VIOLET persistence and Hazelcast state writers
9. builds the muted shadow path
10. runs the observe-only loop
Important point:
- it does not submit live orders
- it does not execute `runtime.step()`
- it is a read-mostly / audit / shadow surface, not the live trading boundary
### 4.2 Live V4 execution flow
`prod/clean_arch/violet/v4_execution_runner.py` is the live boundary. The chain is:
1. `run_live()` loads env and runtime options
2. it applies VIOLET runtime env and optimizations
3. it builds the real DITAv2 bundle
4. it builds the shadow/live-factor source
5. it connects to Hazelcast NG7 scan source
6. it waits for a scan payload from `DOLPHIN_FEATURES.latest_eigen_scan`
7. `process_scan_payload()` parses and validates the payload
8. the scan number is deduplicated in guarded state
9. live factors are read from BLUE state and feature maps
10. the decision step produces a VIOLET decision
11. the decision is transformed to `ExecIntent`
12. the intent may be capped by configured notional limits
13. `ExecIntent` becomes DITAv2 `KernelIntent`
14. the runner mutates submission state
15. `bundle.kernel.process_intent_async(kernel_intent)` is awaited
16. the runner records outcome or error in guarded state
The exact DITAv2 execution point is step 15. That is where VIOLET hands off from its own decision plane into DITAv2 execution.
### 4.3 Where DITAv2 runs
DITAv2 is constructed by `prod/clean_arch/dita_v2/launcher.py` as a launcher bundle:
- kernel
- control plane
- projection
- zinc plane
- venue adapter
In VIOLET dark mode:
- the venue is wrapped in `ObserveOnlyVenue`
- the kernel exists, but the venue refuses live order placement
In V4 execution mode:
- the same bundle is built with BingX VST configuration
- the intent is submitted through the kernel into the venue adapter
- that is the only place VIOLET is allowed to reach live execution
## 5. Input data and input sites
### 5.1 BLUE live inputs consumed by VIOLET
VIOLET reads these BLUE surfaces as inputs:
- `DOLPHIN_STATE_BLUE.latest_nautilus`
- fallback `DOLPHIN_STATE_BLUE.engine_snapshot`
- `DOLPHIN_FEATURES.esof_latest`
- fallback `DOLPHIN_FEATURES.esof_advisor_latest`
- `DOLPHIN_FEATURES.mc_forewarner_latest`
- `DOLPHIN_FEATURES.latest_eigen_scan`
These are read-only input sites. VIOLET should not mutate BLUE state maps.
### 5.2 VIOLET scan input
The active scan input for VIOLET is:
- Hazelcast map `DOLPHIN_FEATURES`
- key `latest_eigen_scan`
That is the scan source the live runner consumes. The older scan bridge path is superseded in this layout.
### 5.3 VIOLET persistence inputs
VIOLET writes into:
- `dolphin_violet.policy_events`
- `dolphin_violet.trade_reconstruction`
- `dolphin_violet.trade_exit_legs`
- `dolphin_violet.position_state`
- `dolphin_violet.anomaly_events`
- `dolphin_violet.account_events`
- `dolphin_violet.status_snapshots`
- `dolphin_violet.trade_events`
- `dolphin_violet.v7_decision_events`
- `dolphin_violet.violet_feed_divergence`
- `dolphin_violet.violet_decisions`
These are VIOLET-only tables. They are not BLUE tables and should not be used to backfill BLUE state directly.
## 6. ASEx involvement
ASEx exists in the tree, but the current VIOLET mainline use is narrow.
Confirmed current VIOLET codepath:
- `prod/clean_arch/violet/v4_execution_runner.py` optionally imports:
- `asex.guarded.ASExGuardedState`
- `asex.worker.ASExWorker`
- if ASEx is unavailable, the runner falls back to a local guarded state implementation
- the ASEx wrapper is used for guarded serial state around scan/order lifecycle counters
What this means:
- ASEx is not the execution engine
- ASEx is not the DITAv2 kernel
- ASEx is not the default VIOLET decision path
- it is an optional serial-state wrapper in the live runner boundary
Cross-check from the integration status doc:
- ASEx is installed as an editable package
- VIOLET main has zero ASEx imports outside the live runner boundary
- DITAv2 main has zero ASEx imports
- the earlier PASS9 adapter work exists in another worktree and is not merged here
Operational conclusion:
- ASEx is currently peripheral, not foundational, in this checkout
- the only current VIOLET runtime touchpoint is the guarded state wrapper inside `v4_execution_runner.py`
## 7. VIOLET-only support systems
### 7.1 Read-only dark guard
`prod/clean_arch/violet/observe_guard.py` implements `ObserveOnlyVenue`.
Purpose:
- wraps a venue
- raises on order placement
- allows the launcher to run a full decision stack without touching a live venue
### 7.2 Shadow journal
`prod/clean_arch/violet/shadow_journal.py`
Purpose:
- persists executed shadow decisions to `dolphin_violet.violet_decisions`
- keeps the shadow path auditable
- matches the ClickHouse DDL in `prod/clickhouse/violet/22_violet_decisions.sql`
### 7.3 Feed divergence monitor
`prod/clean_arch/violet/divergence.py`
Purpose:
- records feed divergence into `dolphin_violet.violet_feed_divergence`
- gives VIOLET its own drift surface separate from BLUE
### 7.4 V4 readiness and arming
`prod/clean_arch/violet/v4_readiness.py`
`prod/clean_arch/violet/v4_arming.py`
Purpose:
- combine soak readiness, credential checks, namespace isolation, and launcher mode checks
- fail closed when the readiness report is not GO
- keep V4 from arming unless the environment says it may
Current observed behavior in this checkout:
- arming is not GO
- the live runner therefore remains stopped
### 7.5 Soak runner and parity harness
`prod/clean_arch/violet/pass2_8_soak_runner.py`
`prod/clean_arch/violet/layer_parity_harness.py`
Purpose:
- run the DARK -> MOCK -> VST canary progression used in VIOLET soak work
- compare VIOLET behavior against the expected layer parity surface
- keep execution preparation separate from the live V4 runner
## 8. Storage and schema
`prod/clickhouse/violet/apply_violet_ddl.py` is the schema applier for VIOLET.
Important characteristics:
- it only targets `dolphin_violet`
- it probes table existence before use
- it does not create BLUE tables
- it applies each SQL file as a separate statement path
Relevant SQL set:
- `00_create_database.sql`
- `01_policy_events.sql`
- `02_trade_reconstruction.sql`
- `03_trade_exit_legs.sql`
- `04_position_state.sql`
- `05_anomaly_events.sql`
- `06_account_events.sql`
- `07_status_snapshots.sql`
- `08_trade_events.sql`
- `09_v7_decision_events.sql`
- `10_adaptive_exit_shadow.sql`
- `11_fee_settled_events.sql`
- `12_sc_bucket_gauge_shadow.sql`
- `13_sc_threshold_advisor_shadow.sql`
- `20_violet_feed_divergence.sql`
- `22_violet_decisions.sql`
The important storage boundary is simple:
- VIOLET uses `dolphin_violet.*`
- BLUE uses `dolphin.*` and `DOLPHIN_*` control/state maps
- VIOLET should not write into BLUE storage
## 9. BLUE-equivalent layer mapping, expanded
### 9.1 Data / scan layer
BLUE uses `DOLPHIN_FEATURES.latest_eigen_scan` as the canonical scan source.
VIOLET uses the same scan feed as its input, but via the live runner and shadow source:
- `prod/clean_arch/violet/v4_execution_runner.py`
- `prod/clean_arch/violet/shadow_live_factors.py`
- `prod/clean_arch/violet/live_blue_source.py`
### 9.2 Signal / conviction layer
BLUE has a large signal layer family in the system bible.
VIOLET's equivalent is:
- `decision_engine.py`
- `sizing.py`
- `alpha_wrappers.py`
- `shadow_live_factors.py`
- `live_factor_source.py`
This is where the BLUE-like live factors are folded into VIOLET conviction and sizing.
### 9.3 Execution / accounting layer
BLUE's execution/accounting boundary is the DITA family in the bible.
VIOLET's equivalent is:
- `prod/clean_arch/dita_v2/launcher.py`
- `prod/clean_arch/dita_v2/rust_backend.py`
- `prod/clean_arch/dita_v2/bingx_venue.py`
- `prod/clean_arch/dita_v2/mock_venue.py`
- `prod/clean_arch/violet/v4_execution_runner.py`
This is the actual order-facing path when V4 is armed.
### 9.4 Observability / divergence
BLUE's observability surfaces are mirrored in VIOLET by:
- `shadow_journal.py`
- `divergence.py`
- `v4_readiness.py`
- `v4_arming.py`
- `parity_report.py`
These keep the VIOLET branch auditable without mutating BLUE.
## 10. Provenance appendix
Worktree state here means the current checkout state observed during this assessment.
| File group | Worktree status | Last-known committer | Responsible actor |
|---|---|---|---|
| `prod/launch_dolphin_violet.py` | modified | Codex, `fb344318aa849fde13b934bdba7337819cc42743` | current local edit in this session |
| `prod/clickhouse/violet/apply_violet_ddl.py` | modified | Codex, `99e529c32ab353027fd8e53e5219d5c949ec06ce` | current local edit in this session |
| `prod/supervisor/dolphin-supervisord.conf` | modified | Codex, `02ecc55c1670019682b193a239c72ab0f66ab1fc` | current local edit in this session |
| `prod/clean_arch/violet/v4_execution_runner.py` | untracked | new file in worktree | Codex |
| `prod/clean_arch/violet/v4_arming.py` | untracked | new file in worktree | Codex |
| `prod/launch_dolphin_violet_v4.py` | untracked in this checkout view | new file in worktree | Codex |
| `prod/launch_dita_v2.py` | untracked in this checkout view | new file in worktree | Codex |
| `prod/supervisor/run_with_dolphin_env.sh` | untracked in this checkout view | new file in worktree | Codex |
| `prod/clean_arch/dita_v2/launcher.py` | clean in this worktree during this assessment | not revalidated here | upstream DITAv2 owner / baseline |
| `prod/clean_arch/dita_v2/rust_backend.py` | clean in this worktree during this assessment | not revalidated here | upstream DITAv2 owner / baseline |
| `prod/clean_arch/dita_v2/bingx_venue.py` | clean in this worktree during this assessment | not revalidated here | upstream DITAv2 owner / baseline |
| `prod/clean_arch/dita_v2/mock_venue.py` | clean in this worktree during this assessment | not revalidated here | upstream DITAv2 owner / baseline |
| `prod/clean_arch/violet/shadow_live_factors.py` | clean in this worktree during this assessment | not revalidated here | upstream VIOLET owner / baseline |
| `prod/clean_arch/violet/live_blue_source.py` | clean in this worktree during this assessment | not revalidated here | upstream VIOLET owner / baseline |
| `prod/clean_arch/violet/live_factor_source.py` | clean in this worktree during this assessment | not revalidated here | upstream VIOLET owner / baseline |
| `prod/clean_arch/violet/shadow_journal.py` | clean in this worktree during this assessment | not revalidated here | upstream VIOLET owner / baseline |
| `prod/clean_arch/violet/divergence.py` | clean in this worktree during this assessment | not revalidated here | upstream VIOLET owner / baseline |
| `prod/clean_arch/violet/observe_guard.py` | clean in this worktree during this assessment | not revalidated here | upstream VIOLET owner / baseline |
| `prod/clean_arch/violet/v4_readiness.py` | clean in this worktree during this assessment | not revalidated here | upstream VIOLET owner / baseline |
| `prod/docs/SYSTEM_BIBLE_v7.md` | clean in this worktree during this assessment | not revalidated here | doctrine baseline |
| `prod/docs/DITA_V2_KERNEL_REFERENCE.md` | clean in this worktree during this assessment | not revalidated here | doctrine baseline |
| `prod/docs/VIOLET_DEV_SPEC_AND_PLAN*.md` | clean in this worktree during this assessment | not revalidated here | doctrine baseline |
Notes:
- for files modified or added in this session, the responsible actor is the current local worktree edit path
- for clean baseline files, I did not re-resolve the exact last commit for every single file in this pass
- the assessment is still source-faithful because the runtime path is anchored to the specific files above and the current worktree status is explicit
## 11. Bottom line
VIOLET in this checkout is a BLUE-faithful, VIOLET-namespaced stack with three distinct surfaces:
- observe-only VIOLET, which mirrors the full decision plane but blocks venue writes
- shadow / readiness / divergence, which audit and gate behavior
- live V4 execution, which feeds scan payloads into DITAv2 only after arming passes
The decisive boundary is the handoff from `ExecIntent` into `KernelIntent` and then `bundle.kernel.process_intent_async(...)`.
That is where VIOLET stops being analysis and becomes actual execution through DITAv2.