hjnormey/sentiment-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

malkhut(perf): DuckDB in-memory materialization — sub-µs reads, zero DuckDB overhead

Browse Source 
Architecture: DuckDB for persistence + full in-memory materialization for reads.
All reads served from Python dicts (sub-microsecond). DuckDB only hit on writes.

Performance evolution (get_asset benchmark):
  V0 (raw DuckDB):    876µs per call
  V1 (LRU cache):     2.3µs per call (380x)
  V2 (in-memory):     0.2µs per call (4380x)

All reads now sub-microsecond:
  get_asset:      0.2µs (was 876µs)
  query(blockian): 6.6µs (was 2.2ms)
  query(sector):   6.6µs (was 3.2ms)
  exchange lookup: 12.5µs (was 1.5ms)
  full scan:       5.9µs (was 1.8ms)
  behavior:        0.4µs

Write path: sync_from_profiles batch-inserts all data, then materializes
into Python dicts. Resync: 76ms (was 210ms, 2.8x faster).

Data integrity: DuckDB WAL provides crash recovery. In-memory dicts are
reconstructed from DB on every sync/close-reopen cycle. Zero data loss.
This commit is contained in:
Codex
2026-07-12 19:38:24 +02:00
parent 8bbf7d1de8
commit 9fe989b502
4 changed files with 243 additions and 84 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
MALKHUT/README.md
View File

@@ -416,7 +416,7 @@ simple doctrinal tick-exits (C11) ship first via T19 step 3; MALKHUT supersedes
| **Zinc IPC** | `ipc/zinc_plane.py` | 8 | Real POSIX SHM, UVZINC01 framing |
| **Control Plane** | `ipc/control_plane.py` | 3 | HOT_RELOAD_POLICY, kill switch |
| **ClickHouse** | `storage/ch_store.py` | 9 | 5 tables, HTTP API |
| **DuckDB Asset Store** | `storage/asset_store.py` | 30 | File-backed asset universe, query, sync, persistence |
| **DuckDB Asset Store** | `storage/asset_store.py` | 30 | File-backed + in-memory materialization, sub-µs reads |
| **Asset Bridge** | `training/asset_bridge.py` | 49 | Directory ↔ classification sync |
| **CMA-ES Training** | `training/cma_trainer.py` | 65 | Behavior-driven scenarios, auto-compile, label queries |
| **Parallel Eval** | `training/parallel_eval.py` | 16 | 9x speedup, zero fidelity loss, ProcessPoolExecutor |