docs: add v20r1 FLIGHT13 System Test & Ops Guide

Covers v20r1 identity (pid 790059->790060, zinc uv_flight13_v20r1, lease /tmp/flight13_hl_testnet_single_writer.lock), config chain, FORCE_ENGAGE firehose (u=exec_urgency, capacity=1, 30/hr SHORT 14 syms), exit/max-hold, venue-truth via unsigned HL /info, monitoring runbook, relaunch/stop procedures, and troubleshooting (incl. sub-$10 residual dust and stale HL metadata). Testnet-only; no system code/data modified.
2026-08-26 21:30:11 +02:00
parent 8255c807e9
commit 23cd98105c
1 changed files with 428 additions and 0 deletions
--- a/prod/docs/FLIGHT13_SYSTEM_TEST_AND_OPS_GUIDE_v20r1_20260826.md
+++ b/prod/docs/FLIGHT13_SYSTEM_TEST_AND_OPS_GUIDE_v20r1_20260826.md
@@ -0,0 +1,428 @@
+# FLIGHT13 v20r1 — System Test & Operations Guide
+
+> Operational runbook for the **v20r1** test campaign of the FLIGHT13 testnet harness.
+> Generated 2026-08-26 from live observation of the running v20r1 process.
+> Audience: an operator/agent that must **start, monitor, stop, or relaunch** v20r1 without altering its code or data.
+>
+> **Ground truth over memory.** Every path below was verified against the live host at write time. If a path or version differs, re-check with `h5i recall context` / `h5i recall context relevant <file>` before acting.
+>
+> ⚠️ **Secrets hygiene.** No secret *values* are reproduced here. Credentials live in protected files on the host and are redacted in transit. Only **public identifiers** (the HL testnet signer address) are quoted, because they are required for the unsigned `/info` venue-truth checks and are already emitted in plaintext by the runner's own startup log.
+
+---
+
+## 1. What v20r1 actually is (and what it is NOT)
+
+v20r1 is **not** BLUE (in-memory, no exchange exposure) and **not** the legacy BingX-VST PINK path. It is a **FLIGHT13 / DITAv3.49 single-slot testnet campaign executing on HyperLiquid TESTNET** (`api.hyperliquid-testnet.xyz`).
+
+| Role | Reality | Venue |
+|---|---|---|
+| v20r1 execution venue | **HyperLiquid TESTNET** (real testnet mechanics, demo funds) | `api.hyperliquid-testnet.xyz` |
+| Orderbook data feed | BingX **VST** (testnet) | `open-api-vst.bingx.com` |
+| Money risk | Testnet-real (burn-in evidence integrity). Not a BLUE bookkeeping artifact. | HL VST |
+| Mainnet capability | **None.** Hard gate aborts unless `is_mainnet=False` + `allow_mainnet=False` + `keystore=hl_testnet`. | none |
+
+The launch hard-gate (`/root/uv-wt/f13-hl/prod/launch_flight13_testnet.sh`, lines 89–106) resolves the config and **refuses to start** unless `(not is_mainnet) and ("testnet" in http) and (keystore=="hl_testnet") and (not allow_mainnet)`.
+
+**Single-slot, single-writer.** One kernel slot (`UV_EXEC_MAX_SLOTS=1`, `UV_EXEC_SLOT_ID=0`). A process-local exclusive writer lease prevents two FLIGHT runners from sharing the account (see §5).
+
+---
+
+## 2. Component map (exact paths)
+
+Verified paths on host `linux/amd64`:
+
+| Concern | Path |
+|---|---|
+| **Launcher (entry point)** | `/root/uv-wt/f13-hl/prod/launch_flight13_testnet.sh` |
+| **Boot strap (signal handlers + lease + kernel run)** | `/root/uv-wt/f13-hl/.f13_boot.py` |
+| **Kernel / TPSL ticker / FORCE_ENGAGE** | `/root/uv-wt/f13-hl/prod/clean_arch/violet/uv/blue_prime/runner.py` |
+| **FORCE_ENGAGE config object** | `/root/uv-wt/f13-hl/prod/clean_arch/violet/uv/exec/force_engage.py` |
+| **Exec control plane (zinc knobs)** | `/root/uv-wt/f13-hl/prod/clean_arch/violet/uv/exec/control_plane.py` |
+| **Exec guardrails (max_cost, ceilings, sizes)** | `/root/uv-wt/f13-hl/prod/clean_arch/violet/uv/exec/exec_control.py` |
+| **Single-writer lease (flock)** | `/root/uv-wt/f13-hl/prod/clean_arch/violet/uv/exec/flight13_testnet_lease.py` |
+| **Venue truth (stuck-order resolution)** | `/root/uv-wt/f13-hl/prod/clean_arch/violet/uv/exec/venue_truth.py` |
+| **HL HTTP /info client (unsigned reads)** | `/root/uv-wt/f13-hl/prod/hl/http.py` |
+| **HL venue adapter (orders/positions/cancel/submit)** | `/root/uv-wt/f13-hl/prod/clean_arch/dita_v2/hl_venue.py` |
+| **HL user stream (WS orderUpdates+fills, REST clearinghouseState)** | `/root/uv-wt/f13-hl/prod/clean_arch/dita_v2/hl_user_stream.py` |
+| **Canonical armed config** | `/root/flight13_testnet_armed.env` |
+| **Campaign config (this run)** | `/root/uv-wt/f13-hl/prod/configs/flight13_testnet_single_slot_multi_asset_sweep.env` |
+| Campaign sources → inert shadow baseline | `/root/uv-wt/f13-hl/prod/configs/flight13_testnet_single_slot_shadow.env` |
+| **BingX VST creds source** | `/root/.dolphin_sub_mainnet.env` (perms `0600`; do NOT cat) |
+| **Kernel Python** | `/home/dolphin/siloqy_env/bin/python3` (CPython 3.12.14) |
+| **Runtime toolkit** | `rtk` on `PATH` (`which rtk` → `/root/.local/bin/rtk`; binary, not shell) |
+
+---
+
+## 3. Runtime identity of v20r1 (how to recognise the live incarnation)
+
+| Artifact | Value / Path |
+|---|---|
+| Parent wrapper PID | **790059** — `timeout --signal=TERM 8h /root/uv-wt/f13-hl/prod/launch_flight13_testnet.sh` (PPID 1, detached) |
+| Kernel PID | **790060** — `/home/dolphin/siloqy_env/bin/python3 /root/uv-wt/f13-hl/.f13_boot.py` (parent 790059, same SID) |
+| Lifetime budget | 8h wall (SIGTERM at expiry). Started ~16:20 CEST; kills ~00:20 CEST. |
+| **Namespace (Zinc prefix)** | `uv_flight13_v20r1` (fresh; corrects the inherited `uv_flight13_single_slot` namespace) |
+| Control-plane instance | `f13_testnet_v20r1` (`UV_EXEC_INSTANCE`) |
+| Exclusive-writer lease | `/tmp/flight13_hl_testnet_single_writer.lock` — `{"owner":"flight13-standard-runner","pid":790060,...}` (flock, released on 790060 exit) |
+| Zinc shared-RAM planes (live shm) | `/dev/shm/zinc_uv_flight13_v20r1_flight_exec_knobs`, `/dev/shm/zinc_uv_flight13_v20r1_blue_prime_hooks`, `/dev/shm/zinc_uv_flight13_v20r1_state` |
+| HL signer (testnet) | `0x6af790574E5E9EA067FB4Ce7458B429039cE3f96` (keystore sealed, never on disk) |
+| Capital seed | K-capital anchored := venue wallet **1145.8935** USDT at first balance (DITAv2.7 ATLAS ANCHOR) |
+
+Confirm membership at any time:
+```bash
+rtk ps -p 790059,790060 -o pid,ppid,sid,stat,etime,args
+cat /tmp/flight13_hl_testnet_single_writer.lock
+ls -l /dev/shm/zinc_uv_flight13_v20r1_*
+```
+
+---
+
+## 4. Evidence spools (this campaign)
+
+`/root/dolphin_logs/uv_journal_runs/f13_terminal_slot_release_v20r1_20260826/`
+
+```
+campaign.env              # exports UV_ZINC_PREFIX=uv_flight13_v20r1, fresh spool paths; sources the sweep config
+runner.log                # human text log (CEST). tpsl_ticker @0.20s poll.
+journal.jsonl             # structured audit, TSV: <id json>\t<body json>
+journal.jsonl.ack         # seq/ack sidecar for the async journal writer
+telemetry/
+    f7_slippage.jsonl     # REAL venue order friction (every ENTER fill): fills/cancels/slippage/fees
+    f9_forced_friction.jsonl  # FORCE_ENGAGE-only friction (separate spool; rows tagged forced_engage=true)
+```
+
+**Journal tables** (5): `dolphin_uv.prime_scans`, `dolphin_uv.tp_exit_ingress`, `dolphin_uv.max_hold_ingress`, `dolphin_uv.prime_decisions`, `dolphin_uv.exec_journal`.
+
+**Timezones matter.** `runner.log` timestamps are **CEST (local)`; `journal.jsonl` timestamps are **UTC**. They describe the same events (e.g. scan #184524 ↔ bar #184524).
+
+**Journal format** is two tab-separated JSON blobs per line. Decode the first few keys safely:
+```bash
+tail -n 4 journal.jsonl | while IFS=$'\t' read -r id body; do
+  python3 -c "import sys,json; d=json.loads('''$id'''); b=json.loads('''$body'''); print(d.get('table'),b.get('trade_id'),b.get('kind'),b.get('asset'),b.get('suppress_reason'))"
+done
+```
+
+---
+
+## 5. Configuration chain (how the env is actually assembled)
+
+```
+/root/flight13_testnet_armed.env                      # F13_CONFIG default — canonical armed HL-testnet config
+  └─ (sourced by launch script: . "$CFG")
+/root/uv-wt/f13-hl/prod/configs/flight13_testnet_single_slot_multi_asset_sweep.env
+  source → flight13_testnet_single_slot_shadow.env        # INERT baseline: UV_FORCE_ENGAGE=0, UV_PROMOTED=0
+  └─ sets UV_ZINC_PREFIX=uv_flight13_single_slot_sweep, UV_EXEC_INSTANCE=f13_testnet_single_slot_sweep
+         UV_FORCE_ENGAGE=1, UV_FORCE_ENGAGE_SYMBOLS=…, UV_FORCE_ENGAGE_SIDE=SHORT, PER_HOUR=30, MAX_PER_HOUR=36,
+         UV_FORCE_ENGAGE_SWEEP=1, sweep buckets HEALTHY/MID/THIN
+```
+
+The **campaign.env** in the spool dir is what makes v20r1 *v20r1*: it re-`export`s `UV_ZINC_PREFIX=uv_flight13_v20r1`, `UV_EXEC_INSTANCE=f13_testnet_v20r1`, and points `UV_JOURNAL_SPOOL`/`UV_TELEMETRY_SPOOL` at the run-specific spool. **This re-export is the "fresh namespace" correction** — without it the shadow's `uv_flight13_single_slot_sweep` prefix collides with stale shm from prior runs.
+
+**BingX VST creds** are pulled *individually* (not sourced wholesale) from `/root/.dolphin_sub_mainnet.env`:
+```bash
+eval "$(grep -aE '^(export +)?BINGX_(API_KEY|API_SECRET|SECRET_KEY)=' "$KEYS_SRC" | sed -E 's/^export +//; s/^/export /')"
+```
+Only `BINGX_*` lines; mainnet flags in that file are never imported. CH creds (`DOLPHIN_CH_URL/USER/PASS`) come from the armed env, imported only if not already set.
+
+### 5.1 Key runtime knobs (armed values, from the boot banner)
+
+```
+exec.max_cost_bps            = 7.0       (guardrail)
+exec.maker_fee_bps           = 1.5       (tunable)
+exec.taker_fee_bps           = 4.5       (tunable)
+exec.ttl_min_ms / ttl_max_ms = 750 / 60000
+exec.maker_offset_bps        = 1.0
+exec.default_urgency         = 0.5
+exec.market_urgency_threshold= 0.9
+reprice.abandon_edge_floor_bps = 1.5
+reprice.max_chase_seconds    = 60.0
+reprice.max_chase_concession_bps = 6.0
+brain.fill_hazard_per_s      = 0.011
+brain.expected_adverse_bps   = 2.5
+```
+These are written into `/dev/shm/zinc_uv_flight13_v20r1_flight_exec_knobs` (writable, 50 knobs).
+
+---
+
+## 6. How it runs (arm sequence, in order)
+
+From `runner.log` head (CEST) the boot order is:
+
+1. `[GATE]` HL env=testnet, is_mainnet=False, keystore=hl_testnet, http=…/testnet. `FORCE_ENGAGE enabled=True per_hour=30.0`. `✅ TESTNET CONFIRMED`.
+2. `slot-capacity=1` — one authoritative value. **ATLAS parity gate ARMED** — ENTER on a symbol the venue already holds is refused; positions capped at 1 (kernel slots).
+3. **PREEMPTION gate ARMED** — ENTERs counted for slot contention.
+4. **REST-CEILING HALT ARMED** — ENTERs refuse while any resting ENTER has out-rested the 12-bar (132 s) ceiling with cancels failing. (`#59 Path 1` — now *acted on*, not merely logged.)
+5. `VENUE MODE: VST (testnet) host=open-api-vst.bingx.com` and `HL VENUE MODE: testnet host=api.hyperliquid-testnet.xyz`.
+6. HL **keystore: agent signer loaded** (address `0x6af7…f96`) — key sealed, never on disk. `NonceManager: loaded last_nonce=…`.
+7. `FLIGHT f13.14-94-ge663160-dirty on DITAv3.49 (fix-level 24)`.
+8. `I/O loop owner started (M2 async seam) — running`. `HL runner workarounds W1 boundary armed`.
+9. `userFills snapshot rows=30 recovered_current_run=0` (WS orderUpdates+userFills subscribed).
+10. `VENUE READINESS: GREEN — 7 checks`.
+11. `ARMED — readiness GREEN, direct ASEx future targets REAL DITAv3.49 kernel (HYPERLIQUID testnet)`.
+12. `VENUE-TRUTH armed (#59 Site 1c)`; `DITAv3.49 BOUNDED RECONCILIATION OWNER armed`; `RATE GOVERNOR armed`; `SOFT CALIBRATION: OK`; `TP/SL price authority = Hyperliquid resolved-book only`.
+13. Control plane `zinc:uv_flight13_v20r1_flight_exec_knobs` bound (writable); 50 knobs resolved.
+14. `UV EXEC: TP REST-ON-ENTRY wired`; `FILL-DRIVEN exit arming ENABLED`; `UV UNIVERSE: 156 canonical assets`.
+
+**Arming is fail-safe:** a missing control plane → knobs on guardrails (degraded run, not broken). A non-GREEN venue readiness aborts.
+
+---
+
+## 7. The FORCE_ENGAGE subsystem (the synthetic short firehose)
+
+`/root/uv-wt/f13-hl/prod/clean_arch/violet/uv/exec/force_engage.py` (`ForceEngageConfig`).
+
+- `UV_FORCE_ENGAGE=1`; `UV_FORCE_ENGAGE_SIDE=SHORT`; `UV_FORCE_ENGAGE_PER_HOUR=30`, `MAX_PER_HOUR=36`, `MAX_CONCURRENT=1`.
+- Symbols: `BTCUSDT,ETHUSDT,SOLUSDT,DOGEUSDT,AVAXUSDT,NEARUSDT,ATOMUSDT,AAVEUSDT,ETCUSDT,XLMUSDT,FETUSDT,HBARUSDT,ZECUSDT,DASHUSDT`.
+- Sweep buckets: `HEALTHY` (BTC,ETH,SOL,DOGE), `MID` (AVAX,NEAR,ATOM,AAVE,ETC), `THIN` (XLM,FET,HBAR,ZEC,DASH).
+- Fires **Poisson-spaced** synthetic SHORT *signals* at ~30/hr. They are injected **only** when BLUE produced no real entry that scan (`not out["has_entry"]`) — they never clobber an organic entry.
+
+### 7.1 The `u=` field is NOT size (common misconception)
+
+In `FORCE_ENGAGE: SYNTHETIC ENTER <SYM> SHORT @ <px> u=<x> cell=<y> …` (`runner.py` log format, line ~2756):
+
+- `u=<x>` = **`exec_urgency`** ∈ [0.0, 0.95] (0.0 = maker-passive LIMIT, 0.95 = near-taker). It is **not** the order quantity.
+- `cell=<status|u>` = the liquidity cell (`healthy`/`mid`/`thin` × a utilization).
+- **Order size** is the canonical bridge sizing (capital × conviction), e.g. the observed ATOM force-enter was **15.02 ATOM** (`PROMOTION: … size=15.0200 lev=1 raw_conv=1.0000`), independent of the `u` value.
+
+So `u=0.0` force-engages **do** place real venue orders (passive maker LIMITs).
+
+### 7.2 `in_flight` and `total`
+
+- `total` = cumulative synthetic signals injected.
+- `in_flight` = synthetic signals that **promoted** (actually submitted to the venue) and are not yet terminal. **This is the live force-engage order count** — suppressed signals (`CapacityExceeded`, `ParityViolation`, not-tradable) release their slot immediately (`_force_engage.note_closed`) and do **not** increment resting orders.
+
+Therefore: `in_flight` ≈ force-engage orders currently resting on the venue, and `in_flight + open positions` are linked by the **capacity=1** gate.
+
+---
+
+## 8. Exit, max-hold, and terminal logic
+
+- **TP/SL**: reduce-only maker TP posted on entry fill at `tp_effective_pct` ≈ 0.28% (`_exec_tpsl_...` TP REST-ON-ENTRY). SL is the **catastrophic floor** (configured 1.2% on ATOM). Exits are **fill-driven** (arm on WS fill); `UV_FILL_DRIVEN_EXIT` is the kill-switch. Fill router is a **flattener** — an unmatched fill is closed reduce-only against venue truth.
+- **MAX_HOLD TERMINAL**: a position held past **125 modulated bars (1375 s)** flips the ceiling to *advisory* and logs `MAX_HOLD TERMINAL: … held=Ns limit=1375.0s — ceiling now ADVISORY, crossing to completion`. This log fires **every scan (~12 s)** while the position remains. The actual cross-exit is subject to the gates below.
+
+> ### 8.1 Known stuck condition: sub-$-10 residuals (LIVE in v20r1)
+> `force-c3760f497e` ATOM SHORT entered 15.02 @ 1.5262; partial exits left **3.87 ATOM** (~**$5.87**). A reduce-only TP on the residual was **VENUE_REJECTED: "Order must have minimum value of $10"** (HL min order value). The MAX_HOLD terminal then logs "crossing to completion" every ~12 s (`openOrders=0`, position persists ~45+ min) because **any** cross of the < $10 residual is likewise rejected.
+>
+> - **Money risk: none** — tiny, *profitable* testnet position (ROE +10–16%).
+> - **Integrity risk: real** — an unclosed force-engage terminal record + noisy CRITICAL log storm.
+> - **Do NOT mutate history / do NOT force-exit via the ledger** — this is a *real testnet* position, not a BLUE bookkeeping artifact. Per the AGENTS.md separation: BLUE stranded trades are fixed via `capital_update_ledger` + CH position_state; this HL-testnet residual needs a **venue-side** resolution.
+> - **Deferred decision (NOT NOW).** The mooted fix class: an **IOC/FOK reduce that fills-to-min-and-cancels-the-rest**, or **accumulate-to-min-then-close**, or a platform reduce path that bypasses the per-order $10 floor. Record the design choice before touching `max_hold`/`_smart_exit` in `runner.py`. Flagged as an `h5i NOTE` for the next session.
+
+---
+
+## 9. Venue-truth & independent verification (the "no inference" rule)
+
+The kernel never infers order fate from a *rejection of an action*; it asks the venue directly. `HlVenueTruthSupplier` (`venue_truth.py`) and `hl_user_stream.py` perform **unsigned `/info` reads** (they do not charge HL's per-address account bucket).
+
+| What | `/info` call | Fields of interest |
+|---|---|---|
+| Positions | `{"type":"clearinghouseState","user":"0x6af7…f96"}` | `assetPositions[].position.{coin,szi,entryPx,returnOnEquity,liquidationPx,marginUsed}`, `marginSummary.accountValue`, `withdrawable` |
+| Resting orders | `{"type":"openOrders","user":"0x6af7…f96"}` | list of `coin, side, px, oid, cloid` |
+| Recent fills | `{"type":"userFills","user":"0x6af7…f96"}` (`startTime` optional) | `user`,`oid`,`side`,`px`,`sz`,`fee`,`time` |
+| Mid prices | `{"type":"allMids"}` | mark prices |
+
+**One-liner independent check** (cross-checks the kernel's own `open_orders`/`positions`):
+```bash
+A=0x6af790574E5E9EA067FB4Ce7458B429039cE3f96
+curl -s --max-time 12 -X POST https://api.hyperliquid-testnet.xyz/info -H 'Content-Type: application/json' \
+  -d "{\"type\":\"openOrders\",\"user\":\"$A\"}" | python3 -m json.tool
+curl -s --max-time 12 -X POST https://api.hyperliquid-testnet.xyz/info -H 'Content-Type: application/json' \
+  -d "{\"type\":\"clearinghouseState\",\"user\":\"$A\"}" | python3 -m json.tool
+```
+- `[]` open orders + `positions=[]` ⇒ venue flat (the kernel's `V20R1_VENUE_CHECK positions=[] open_orders=0`).
+- `None`/error on a side ⇒ treat as **UNKNOWN** (the supplier propagates `None` precisely so a failed read is never mistaken for an empty book — see the 234-hidden-fills incident note in `venue_truth.py`).
+
+**Stuck-order policy** (`venue_truth.py` `apply_stuck_truth`):
+- ENTER + gone → slot FREE (drop tracking). This is what un-sticks a single-slot kernel held `SLOT_BUSY`.
+- EXIT + gone (unfilled) → position still open → **cross reduce-only first**, then drop (never drop an open position).
+- FILLED → hand to fill path; a venue-truth-discovered fill the E-feed missed is a **CRITICAL feed gap**.
+- RESTING past ceiling → one bounded retry cancel; if cancels fail → **HALT_AND_ALARM** (ENTERs halted).
+- UNKNOWN/REQUERY → do nothing; next cadence re-queries.
+
+Cloid derivation is deterministic: `wire_cloid = sha256(client_id)[:32]` (`hl_venue._cloid_for`), so truth checks survive a process restart that wipes `_cloid_map` (task #60).
+
+---
+
+## 10. Monitoring runbook
+
+A reusable one-shot helper is deployed at **`/tmp/v20r1_monitor.sh`** (sandboxed, *not* part of the system repo). It prints: process, lease, heartbeat tail, FORCE_ENGAGE cadence, the independent HL venue truth, and the most recent CRITICAL/WARNING/REFUSING lines.
+
+```bash
+# One-shot health snapshot
+/tmp/v20r1_monitor.sh
+```
+
+### 10.1 Healthy signals (what "GREEN" looks like in real time)
+
+In `runner.log` / via the helper, expect every ~12–13 s:
+- `TPSL HEARTBEAT: polls=… ticks=… tp=… sl=… prices={…}` (poll=0.20 s; polls climb ~60/s).
+- `BAR #… watched=1 dt=12.1s avg=12.0s bars={'force-c3': n} px={…}` (one watched asset; the force-c3 cell).
+- FORCE_ENGAGE `in_flight=1 total=N` cycling (≤ `MAX_CONCURRENT=1`); non-promoted refused with `CapacityExceeded` are normal while a position is open.
+- Independent `clearinghouseState` returns the expected position (or `[]` when flat); `openOrders` matches `in_flight`.
+
+### 10.2 What to verify during a monitor pass
+
+| Check | Where | Healthy | Action if not |
+|---|---|---|---|
+| Process alive & not D state | `rtk ps -p 790059,790060 -o pid,stat,etime` | `Sl`/`Ss`, etime < 8h | restart (§12) |
+| Lease held by 790060 | `cat /tmp/flight13_hl_testnet_single_writer.lock` | `pid:790060` | stale lock if pid gone → `rm` (see §12) |
+| Zninc shm present | `ls /dev/shm/zinc_uv_flight13_v20r1_*` | 3 files (knobs/hooks/state) | launch regressed to in-mem? relaunch (§12) |
+| TPSL ticker alive | `runner.log` recent `TPSL HEARTBEAT` | < 60 s stale | check I/O loop / CIFS (§11) |
+| Venue readiness | startup banner `VENUE READINESS: GREEN — 7 checks` | GREEN | do not let it trade |
+| Venue truth reconciles | helper §9 curl | positions/openOrders agree with log | investigate venue-truth read failure |
+| FORCE_ENGAGE within rate | `in_flight<=1`, `total` ~ 30/hr | ≤ cap 36/hr, ≤1 in_flight | see §11 |
+| Reconcile armed | `DITAv3.49 BOUNDED RECONCILIATION OWNER armed` | present | relaunch |
+
+### 10.3 Continuous-tail during a watch
+
+```bash
+rtk tail -n 160 /root/dolphin_logs/uv_journal_runs/f13_terminal_slot_release_v20r1_20260826/runner.log
+grep -nE "CRITICAL|ERROR|REFUSING|stale_entry_ttl|CAPACITY|MAX_HOLD|VENUE REJECTED|stale.*metadata" \
+  /root/dolphin_logs/uv_journal_runs/f13_terminal_slot_release_v20r1_20260826/runner.log | tail -n 20
+# journal: force-engage fills (real venue orders) — sanity that size ≠ conviction
+python3 -c "import json;rows=[json.loads(l.split('\t')[1]) for l in open('/root/dolphin_logs/uv_journal_runs/f13_terminal_slot_release_v20r1_20260826/telemetry/f7_slippage.jsonl')];[print(r['trade_id'],r['asset'],r['leg'],r['triage'],r.get('fill_px'),r.get('slippage_bps')) for r in rows]"
+```
+
+---
+
+## 11. Troubleshooting catalog
+
+| Symptom | Meaning | Remedy |
+|---|---|---|
+| `HlInstrumentProvider: serving Ns-stale metadata (ttl=300s)` | HL instrument metadata not refreshing — symptom of the noted **CIFS/host I/O latency**. Tolerable while the test universe is stable; blocks only if a listed/delisted coin changes. | Watch; if it grows unbounded or a symbol 404s on the venue, escalate host I/O. Do **not** override metadata by hand. |
+| `MAX_HOLD TERMINAL … crossing to completion` **every ~12 s, position persists** | The held position (usually a force-engage residual) is below HL's **$10 min order value** → every cross-exit is rejected → `openOrders=0`, position lingers. See §8.1. | **NOT a stop-the-world fault.** Confirm residual < $10 via `clearinghouseState`. Do not mutate history. Defer design decision (§8.1 NOTE); escalate to operator if it blocks the next slot's arm. |
+| `TP REST-ON-ENTRY REJECTED … Order must have minimum value of $10` | Same min-order floor; the reduce-only TP on a sub-min residual can't rest. | Same as above — residual is sub-min by construction. |
+| `CapacityExceeded: 1 open position(s) against capacity 1` on a FORCE_ENGAGE ENTER | One position already open (cap=1). This is the parity gate working; subsequent force-engages are correctly refused. | Normal. Wait for the open position to exit (TP/SL or max-hold terminal). |
+| `ZECUSDT not tradable on execution venue — suppressed` | ZEC not in the HL testnet configured universe at that moment. | Normal; the symbol is dropped from the live book, not a crash. |
+| `stale_entry_ttl` / `triage=abandoned` | A maker LIMIT force-engage didn't fill within the stale-TTL (~53–60 s) → cancelled. | Normal churn of the firehose. |
+| `BOOK-BLIND HALT` / `NEITHER book lane serves` | WS book cache down → new-enter risk refused; **exits unaffected**. | REST touch fallback (~100–200 ms). Investigate `WS book cache start ERROR`. |
+| Stale lease (`/tmp/flight13_hl_testnet_single_writer.lock` with a dead pid) | A previous launch did not release the flock cleanly. | Only after confirming the pid is dead: `rm -f /tmp/flight13_hl_testnet_single_writer.lock` (the kernel auto-releases on process exit; the file is just the metadata carrier). **Never** remove while a live writer holds it. |
+| `REFUSING TO LAUNCH: resolved config is NOT HL testnet` | The hard gate caught a non-testnet resolution. | Fix the env (keystore/testnet flags) and relaunch; do not bypass the gate. |
+
+---
+
+## 12. Relaunch procedure (clean restart of v20r1)
+
+The launch script **refuses to reuse a journal spool** (`UV_JOURNAL_SPOOL` must be a fresh file, not a dir, not pre-existing) and the lease is single-writer. Sequence:
+
+1. **Stop the live writer.**
+   ```bash
+   rtk ps -p 790059,790060 -o pid,stat,etime,args   # confirm ownership
+   kill 790059                                      # SIGTERM to the 8h timeout wrapper
+   # .f13_boot.py installs SIGTERM/SIGINT -> SystemExit(128+signum); the runner's
+   # finally-block cancels open orders before exit. Wait for both pids gone.
+   rtk ps -p 790059,790060 -o pid,stat,etime,args || echo "stopped"
+   ```
+   - The `flock` is released by the kernel when 790060 exits. If the lock file still shows a *dead* pid, `rm -f /tmp/flight13_hl_testnet_single_writer.lock`.
+   - **Before killing, flatten via the venue** if a position is open and ≥ $10 (see §13). The sub-$10 residual cannot be venue-closed (§8.1) — record it and let the next session decide.
+
+2. **Prepare a fresh campaign spool** (timestamped, never reuse):
+   ```bash
+   D=20260826T17xx
+   mkdir -p /root/dolphin_logs/uv_journal_runs/f13_terminal_slot_release_v20r1_$D
+   : >  /root/dolphin_logs/uv_journal_runs/f13_terminal_slot_release_v20r1_$D/journal.jsonl
+   cp /root/dolphin_logs/uv_journal_runs/f13_terminal_slot_release_v20r1_20260826/campaign.env \
+      /root/dolphin_logs/uv_journal_runs/f13_terminal_slot_release_v20r1_$D/campaign.env
+   # EDIT the copied campaign.env: bump UV_ZINC_PREFIX/UV_EXEC_INSTANCE to a fresh
+   # namespace, and repoint UV_JOURNAL_SPOOL + UV_TELEMETRY_SPOOL to $D. Then:
+   ```
+   The campaign.env you want to mirror:
+   ```bash
+   export UV_ZINC_PREFIX=uv_flight13_v20r1
+   export UV_EXEC_INSTANCE=f13_testnet_v20r1
+   export UV_EXEC_MAX_SLOTS=1
+   export UV_EXEC_SLOT_ID=0
+   export UV_JOURNAL_SPOOL=/root/dolphin_logs/uv_journal_runs/f13_terminal_slot_release_v20r1_<NEW>/journal.jsonl
+   export UV_TELEMETRY_SPOOL=/root/dolphin_logs/uv_journal_runs/f13_terminal_slot_release_v20r1_<NEW>/telemetry
+   source "/root/uv-wt/f13-hl/prod/configs/flight13_testnet_single_slot_multi_asset_sweep.env"
+   # then RE-export the v20r1 namespace + spool (override the shadow's uv_flight13_single_slot_sweep)
+   export UV_ZINC_PREFIX=uv_flight13_v20r1
+   export UV_EXEC_INSTANCE=f13_testnet_v20r1
+   export UV_JOURNAL_SPOOL=/root/dolphin_logs/uv_journal_runs/f13_terminal_slot_release_v20r1_<NEW>/journal.jsonl
+   export UV_TELEMETRY_SPOOL=/root/dolphin_logs/uv_journal_runs/f13_terminal_slot_release_v20r1_<NEW>/telemetry
+   ```
+
+3. **Arm and launch** (the hard gate self-verifies testnet):
+   ```bash
+   cd /root/uv-wt/f13-hl
+   source /root/dolphin_logs/uv_journal_runs/f13_terminal_slot_release_v20r1_<NEW>/campaign.env
+   export F13_CONFIG=/root/flight13_testnet_armed.env
+   /root/uv-wt/f13-hl/prod/launch_flight13_testnet.sh
+   ```
+   Expected at arm: `[GATE] ✅ TESTNET CONFIRMED`, `VENUE READINESS: GREEN — 7 checks`, `control plane … zinc:uv_flight13_v20r1_flight_exec_knobs (writable=True)`.
+
+4. **Detach for monitoring** (as this run was):
+   ```bash
+   nohup /root/uv-wt/f13-hl/prod/launch_flight13_testnet.sh \
+     > /root/dolphin_logs/uv_journal_runs/f13_terminal_slot_release_v20r1_<NEW>/runner.log 2>&1 &
+   ```
+   (The `timeout --signal=TERM 8h …` wrapper may be applied by your launcher to bound lifetime.)
+
+5. **Verify** with `/tmp/v20r1_monitor.sh` (update the PID column if the wrapper changes).
+
+> 🔒 The launch script **never** sources mainnet flags from `/root/.dolphin_sub_mainnet.env`; it imports only the three `DOLPHIN_CH_*` values and the three `BINGX_*` VST values. Do not shortcut this by `source`-ing the cred file wholesale.
+
+---
+
+## 13. Stop / flatten / clean-shutdown
+
+- **Graceful**: `kill 790059` (SIGTERM). `.f13_boot.py`'s `_terminate_cleanly` raises `SystemExit(128+SIGTERM)`, and the runner's shutdown finally-block **cancels resting orders** before exit. Watch the tail for the cancel ACKs.
+- **Venue flatten (testnet-real position)**: verify on the venue first (§9), then let the kernel's normal TP/SL + max-hold terminal close it. For a **≥ $10** position you can also reduce via the HL testnet directly. For a **< $10 residual** (§8.1): it is not venue-closeable at the current size — do **not** invent a bookkeeping fix (that is the BLUE-data-pollution anti-pattern). Record it; the next slot or a size-accumulate is the clean close path.
+- **After stop**: confirm `790059`,`790060` gone, lease released, no resting orders (`openOrders` `[]` via §9). A leftover position is **foreign fill contamination** — surface it.
+
+---
+
+## 14. Safety invariants (do not violate)
+
+1. **Testnet-only gate** is hard. Never launch with mainnet creds/keystore.
+2. **Single writer** per account (flock lease). Two FLIGHT runners ⇒ foreign fills the other classifies as unresolved.
+3. **Capacity=1** positions (ATLAS parity) — do not weaken the one-position cap.
+4. **Fresh journal spool** every run (launch refuses reuse) — preserves audit provenance.
+5. **`openOrders` is a separate `/info` read, not a `clearinghouseState` field** — reading `state.openOrders` always yields `[]` (see `hl_venue.py` comment at 1148). Use the `openOrders` type.
+6. **Never mutate history.** BLUE stranded-trade corrections go through `capital_update_ledger` + CH `dolphin.position_state` terminal row. This run is HL-testnet (not BLUE bookkeeping) — keep the venue and the ledger in agreement via the **bounded reconciler**, not by editing spools.
+7. **Do not edit vendored copies** (VENDOR.lock). Edit canonical sources, then `scripts/vendor_sync.sh`.
+8. `FORCE_ENGAGE` **must be OFF on mainnet** (per `BingX_RATE_LIMITS_SPEC.md`); it is testnet-fenced here by the gate.
+
+---
+
+## 15. Quick reference
+
+```text
+# v20r1 runtime
+PID   : 790059 (8h timeout wrapper) -> 790060 (.f13_boot.py)
+LEASE : /tmp/flight13_hl_testnet_single_writer.lock  (flock, owner=flight13-standard-runner, pid 790060)
+ZINC  : /dev/shm/zinc_uv_flight13_v20r1_{flight_exec_knobs,blue_prime_hooks,state}
+RUNNER: /home/dolphin/siloqy_env/bin/python3 /root/uv-wt/f13-hl/.f13_boot.py
+LAUNCH:/root/uv-wt/f13-hl/prod/launch_flight13_testnet.sh  (hard testnet gate)
+
+# spools
+ROOT  : /root/dolphin_logs/uv_journal_runs/f13_terminal_slot_release_v20r1_20260826
+  campaign.env, runner.log, journal.jsonl (+.ack), telemetry/{f7_slippage.jsonl, f9_forced_friction.jsonl}
+
+# venue truth (unsigned /info, HL testnet)
+POST https://api.hyperliquid-testnet.xyz/info  {"type":"clearinghouseState","user":"0x6af790574E5E9EA067FB4Ce7458B429039cE3f96"}
+                                                    {"type":"openOrders","user":"<same>"}
+                                                    {"type":"userFills","user":"<same>"}
+                                                    {"type":"allMids"}
+
+# monitor (one-shot)
+/tmp/v20r1_monitor.sh
+
+# config chain
+/root/flight13_testnet_armed.env
+  -> /root/uv-wt/f13-hl/prod/configs/flight13_testnet_single_slot_multi_asset_sweep.env
+       -> flight13_testnet_single_slot_shadow.env  (inert baseline: UV_FORCE_ENGAGE=0)
+  campaign.env overrides namespace -> uv_flight13_v20r1 + fresh spool
+```
+
+---
+
+## 16. Glossary
+
+- **FORCE_ENGAGE**: synthetic SHORT signal firehose (30/hr) that exercises the real exec path; not organic intent. `u=` = exec_urgency, not size.
+- **`in_flight` vs `total`**: `total` = signals fired; `in_flight` = signals that promoted to a resting venue order (suppressed ones release the slot immediately).
+- **cell**: liquidity-quality bucket the symbol resolved into (`healthy`/`mid`/`thin`); `thin` (e.g. ZEC) is most often suppressed.
+- **REST-CEILING HALT**: ENTERs refuse while a resting ENTER out-rested 132 s with failing cancels.
+- **MAX_HOLD TERMINAL**: position held past 125 modulated bars (1375 s) → ceiling advisory → "crossing to completion".
+- **VENUE-TRUTH / "stuck-order resolution" (#59 Site 1c)**: independent unsigned `/info` reads decide an order's true fate; the kernel never infers from action rejection.
+- **BOUNDED RECONCILIATION (DITAv3.49)**: slot-scoped position reconciliation on (re)arm; cold-start has no REST-fill economics (`TAIL_UNAVAILABLE_COLD`).
+- **BLUE vs PINK vs FLIGHT13**: BLUE = in-memory bookkeeping only; PINK = real BingX-VST orders; FLIGHT13 v20r1 = real HyperLiquid-testnet orders via the `blue_prime.runner` kernel with `DITA_HL_*` plumbing. (Per AGENTS.md, BLUE/PINK share a BingX key; here BingX is VST-only for OB data, HL is execution.)