siloqy/prod/docs/MALFORMED_OPEN_RESTORE_BUG.md

# MALFORMED_OPEN_RESTORE_BUG

## Summary

BLUE was repeatedly rehydrating after startup because `dolphin.position_state` contained stale `OPEN` rows with zero effective size.

The restore path treated those rows as fatal:

- it selected the latest `OPEN` row per `trade_id`
- it accepted that row even when `quantity` or `notional` had been driven to `0`
- it hard-stopped on `position_state row invalid quantity ...`
- `supervisord` then restarted the trader
- the next startup read the same bad row again

That created a restart loop.

This was observed most clearly on the `2026-06-11` BLUE window. The recurring bad row was the legacy `ATOMUSDT` leg `1a3d2f9c`, which was persisted as:

- `status = OPEN`
- `quantity = 0`
- `notional = 0`
- `bars_held = 34`

That row is not a live position. It is a stale snapshot that should have been treated as tombstoned history.

## Root Cause

The bad rows were self-inflicted by the partial-retract path in `nautilus_event_trader.py`.

Before the fix:

1. `_apply_internal_retract()` shrank the live position.
2. It wrote a new `position_state` row with `status="OPEN"` for the remaining leg.
3. If the remaining size rounded to zero, the row still existed as an `OPEN` snapshot.
4. A later startup restore could pick that row and treat it as authoritative.

That is enough to leave behind `OPEN` rows with:

- `quantity = 0`
- `notional = 0`

These are not valid live positions, but they looked like one to the old restore logic.

There is a second contributing factor in the restore path:

- the restore code historically trusted the latest `OPEN` candidate too early
- zero-sized `OPEN` rows were only rejected after the row had already been chosen as the best candidate
- rejection used a hard failure path, which made the process exit instead of trying the next sane source

That means the persistence bug and the restore policy bug reinforced each other.

## Observable Symptoms

- repeated `restore candidate parse failed from capital_update_ledger: 'list' object has no attribute 'get'`
- repeated `position_state row invalid quantity for trade ...: 0.0`
- `RESTORE HALT`
- immediate restart by `supervisord`

The chain-token mismatch logs were a separate warning. They were not the restart trigger.

The capital-ledger parse warning is also distinct:

- it indicates the ledger file is list-shaped, not a dict
- it forces restore to rely more heavily on the other state surfaces
- it is noisy, but it is not what actually killed the process in this incident

## Fix Applied

Two changes were made.

### 1. Stop writing zero-sized `OPEN` rows

In `_apply_internal_retract()`:

- compute `remaining_qty`
- if the remaining size is effectively zero, treat the retract as a full close
- return the forced exit without emitting a new `position_state` row with `status="OPEN"`

This prevents the bad row from being created in the first place.

### 2. Make restore skip legacy bad `OPEN` rows

In `_restore_position_state()`:

- the ClickHouse restore query now filters `OPEN` rows with `quantity > 0 AND notional > 0`
- if an invalid candidate still appears, restore logs and rejects it instead of hard-halting the process
- restore falls back to HZ state or flat continuation rather than turning a stale row into a restart loop

This is important because the repository already contains stale history. The fix is not only to stop producing new malformed rows; it also has to prevent old rows from re-triggering the same failure path on the next reboot.

### 3. Keep the full-close path coherent

The retract path now computes `remaining_qty` explicitly and treats `remaining_notional <= 1e-9` or `remaining_qty <= 0.0` as a full close.

That means:

- a full retract does not leave a zero-size `OPEN` snapshot behind
- the exit is finalized as a close, not as a pseudo-open partial state
- the runtime slot is removed cleanly instead of being left in a half-closed limbo

## Verification Added

Regression tests were added for both sides:

- full-close retracts no longer emit zero-sized `OPEN` rows
- restore skips zero-sized `OPEN` candidates without setting `restore_failed`

The tests use the existing retract and restore harnesses:

- one test seeds a tiny short leg that collapses to zero on retract and asserts no `OPEN` zero-size row is written
- one test feeds a zero-sized `OPEN` `position_state` row into restore and asserts restore does not hard-halt

## Operational Impact

After this fix:

- stale zero-sized `OPEN` rows no longer restart BLUE
- malformed open snapshots are quarantined as legacy garbage
- the live runtime can continue from a sane source instead of bouncing on the same bad record

## What This Does Not Fix

This change does not rewrite historical ClickHouse rows already present in the warehouse.

It only changes:

- new retract writes
- restore selection and rejection policy
- restart behavior when the old garbage is encountered

If you want the historical ledger cleaned up, that is a separate reconciliation task. The current patch is intentionally conservative and only stops the bad row from causing further damage.