80 lines
3.4 KiB
Markdown
80 lines
3.4 KiB
Markdown
|
# PINK-on-DITAv2 Fault Taxonomy & Operator Response
|
||
|
|
|
||
|
|
## Fault Classes
|
||
|
|
|
||
|
|
### RATE_LIMITED
|
||
|
|
**Kernel code**: `KernelDiagnosticCode.RATE_LIMITED`
|
||
|
|
**Severity**: WARNING
|
||
|
|
**Recovery**: Automatic — kernel retries on next step cycle.
|
||
|
|
|
||
|
|
Operator action: none required unless persistent (>10 min). If persistent, check BingX API limits at `/openApi/swap/v2/user/balance` directly. Reduce poll frequency via `DOLPHIN_PINK_POLL_INTERVAL_SEC` (default 1.0s).
|
||
|
|
|
||
|
|
### ORDER_REJECTED
|
||
|
|
**Kernel code**: `KernelDiagnosticCode.ORDER_REJECTED`
|
||
|
|
**Entry reject**: Slot returns to IDLE. Decision engine will re-evaluate on next cycle.
|
||
|
|
**Exit reject**: Slot stays in EXIT_WORKING. Decision engine will retry exit.
|
||
|
|
|
||
|
|
Operator action: check that instrument is tradeable on BingX VST. Symbol precision changes or contract suspensions can cause rejects. Inspect `outcome.details` for venue reason text.
|
||
|
|
|
||
|
|
### EXIT_ORDER_REJECTED
|
||
|
|
**Kernel code**: `KernelDiagnosticCode.EXIT_ORDER_REJECTED`
|
||
|
|
**Slot state**: EXIT_WORKING. The kernel will retry via process_intent(EXIT) on the next step where the decision engine produces an exit signal.
|
||
|
|
|
||
|
|
Operator action: if position remains open past `DOLPHIN_MAX_HOLD_BARS` (default 250), manually flatten via `pink_ctl.py` or direct BingX REST.
|
||
|
|
|
||
|
|
### CANCEL_REJECTED
|
||
|
|
**Kernel code**: `KernelDiagnosticCode.CANCEL_REJECTED`
|
||
|
|
**Slot state**: Unchanged. Cancel is retried on the next cycle.
|
||
|
|
|
||
|
|
Operator action: check open orders on BingX. If the order filled between cancel attempt and rejection, the slot will converge on the next reconcile cycle.
|
||
|
|
|
||
|
|
### NO_ACTIVE_EXIT_ORDER
|
||
|
|
**Kernel code**: `KernelDiagnosticCode.NO_ACTIVE_EXIT_ORDER`
|
||
|
|
**Cause**: Exit intent processed but no working exit order exists (usually because it filled between decision and execution).
|
||
|
|
|
||
|
|
Operator action: none — the fill event will converge the slot to CLOSED on the next `on_venue_event` or reconcile.
|
||
|
|
|
||
|
|
### STALE_STATE_RECONCILE
|
||
|
|
**Kernel code**: `KernelDiagnosticCode.STALE_STATE_RECONCILING`
|
||
|
|
**Slot state**: STALE_STATE_RECONCILING. Normal event progression is blocked until reconciliation completes.
|
||
|
|
|
||
|
|
Operator action: if the slot stays in this state for >30s, the exchange snapshot may be inconsistent. Run `pink_ctl.py restart` to force full restart reconcile.
|
||
|
|
|
||
|
|
### DUPLICATE_EVENT
|
||
|
|
**Kernel code**: `KernelDiagnosticCode.DUPLICATE_EVENT`
|
||
|
|
**Severity**: INFO
|
||
|
|
**Effect**: Event is dropped. No capital or state change. Idempotency via `seen_event_ids` on the slot.
|
||
|
|
|
||
|
|
Operator action: none.
|
||
|
|
|
||
|
|
### RATE_LIMITED (persistent cycle)
|
||
|
|
**Detection**: Consecutive RATE_LIMITED outcomes with no successful exchange interaction.
|
||
|
|
**Anomaly row origin**: `ditav2_kernel`
|
||
|
|
|
||
|
|
Operator action: check exchange API status. If the rate limit window is known, set `DITA_V2_RATE_LIMIT_COOLDOWN_SEC` in env.
|
||
|
|
|
||
|
|
## Diagnostic Surface
|
||
|
|
|
||
|
|
All fault codes appear in:
|
||
|
|
- `KernelOutcome.diagnostic_code` (programmatic)
|
||
|
|
- `KernelOutcome.severity` (INFO/WARNING/ERROR/CRITICAL)
|
||
|
|
- `KernelOutcome.details` (structured payload with reason, retry_after_ms, etc.)
|
||
|
|
|
||
|
|
## Log Paths
|
||
|
|
|
||
|
|
- Runtime: `/tmp/dolphin_logs/supervisor/dolphin_live_pink.log`
|
||
|
|
- Kernel: `/tmp/dolphin_logs/supervisor/dolphin_live_pink-error.log`
|
||
|
|
|
||
|
|
## Recovery Tools
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# Check DITAv2 health
|
||
|
|
python /mnt/dolphinng5_predict/prod/ops/pink_ctl.py ditav2-status
|
||
|
|
|
||
|
|
# Full restart reconcile
|
||
|
|
python /mnt/dolphinng5_predict/prod/ops/pink_ctl.py restart
|
||
|
|
|
||
|
|
# Namespace isolation check
|
||
|
|
python /mnt/dolphinng5_predict/prod/ops/pink_ctl.py mode-verify
|
||
|
|
```
|