hjnormey/sentiment-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
229 Commits 1 Branch 0 Tags
ada14f04b304773e088cc89a3197afe002771da0
Commit Graph

22 Commits

Author SHA1 Message Date
Codex
459215b7d8 malkhut(fix): wire workers into CMA training loop 
CMAESTrainer.train() now accepts workers parameter and passes it to
evaluate_candidate(), enabling parallel episode evaluation during
actual training (not just in tests/benchmarks).

Benchmark result: ProcessPoolExecutor is optimal (4.76x speedup).
Ray is slower (0.36x) due to head init + plasma overhead for 90 scenarios.
 2026-07-13 03:25:55 +02:00
Codex
c6b7a41bb4 malkhut(optim): vectorized reward + Ray parallel eval + VBT post-analysis 
1. Vectorized reward path (cwm/core.py):
   - Wired up existing compute_reward_vectorized from numba_core (was unused!)
   - Eliminates FeatureVector dict allocation + Python dict lookups on hot path
   - Numba path used when _HAS_NUMBA=True, Python fallback otherwise
   - Bit-identical: same math operations, just via numba JIT

2. Ray-based parallel eval (training/ray_eval.py):
   - Industrial multi-core execution via Ray (used by OpenAI/Anyscale)
   - ray.put() stores params/scenarios in shared object store (no pickle per worker)
   - Each worker: own CWM + planner, zero shared state, no races
   - Bit-identical: same seed + same params = same results regardless of worker count
   - PolicyEvaluator.evaluate_candidate: new use_ray=True parameter

3. VBT post-analysis (training/vbt_analysis.py):
   - episodes_to_pnl_array, episodes_to_metrics (Sharpe, Sortino, VaR, win_rate, etc.)
   - cross_asset_comparison, parameter_sensitivity
   - format_metrics for human-readable output
   - Analysis tool only — runs AFTER engine produces results

4. numba_core.py: added missing 'import math' for compute_reward_vectorized

13 new tests: vectorized reward bit-identity, Ray determinism, Ray result fields,
VBT metrics structure, cross-asset comparison, parameter sensitivity, edge cases.
Total: 1178 tests, 50 files, all green, zero regressions.
 2026-07-12 23:56:16 +02:00
Codex
4c30f664e3 malkhut(fix): DuckDB sync FK-safe delete order + full E2E verification 
Fixed foreign key constraint violation in sync_from_profiles:
  DELETE child tables (asset_exchanges, behavior_profiles) BEFORE
  parent tables (assets, exchanges).

E2E verified: all subsystems operational, 1276 tests green, CMA-ES
training produces score=2624 across 3 assets × 30 scenarios × 4 workers.
 2026-07-12 20:52:56 +02:00
Codex
9fe989b502 malkhut(perf): DuckDB in-memory materialization — sub-µs reads, zero DuckDB overhead 
Architecture: DuckDB for persistence + full in-memory materialization for reads.
All reads served from Python dicts (sub-microsecond). DuckDB only hit on writes.

Performance evolution (get_asset benchmark):
  V0 (raw DuckDB):    876µs per call
  V1 (LRU cache):     2.3µs per call (380x)
  V2 (in-memory):     0.2µs per call (4380x)

All reads now sub-microsecond:
  get_asset:      0.2µs (was 876µs)
  query(blockian): 6.6µs (was 2.2ms)
  query(sector):   6.6µs (was 3.2ms)
  exchange lookup: 12.5µs (was 1.5ms)
  full scan:       5.9µs (was 1.8ms)
  behavior:        0.4µs

Write path: sync_from_profiles batch-inserts all data, then materializes
into Python dicts. Resync: 76ms (was 210ms, 2.8x faster).

Data integrity: DuckDB WAL provides crash recovery. In-memory dicts are
reconstructed from DB on every sync/close-reopen cycle. Zero data loss.
 2026-07-12 19:38:24 +02:00
Codex
7f27ed22c8 malkhut: DuckDB file-backed asset store — schema, sync, queries, persistence 
DuckDB store (asset_store.py):
- 4 tables: assets (28 cols), exchanges (10 cols), asset_exchanges (junction),
  behavior_profiles (28 cols)
- Schema with indexes on blockchain, coingecko_id, cmc_id, exchange_id
- sync_from_profiles(): populate from in-memory dicts in one call
- Query API: query_assets(**filters), get_asset(), symbols_for_exchange(),
  assets_on_blockchain(), asset_count(), exchange_count()
- Case-insensitive exchange lookup
- File-backed persistence: data survives connection close/reopen
- Performance: <1s sync, 100 queries in <1s, 1000 gets in <1s

30 tests covering:
- Schema creation and table structure (3 tests)
- Sync from profiles (6 tests, idempotent)
- Asset CRUD (7 tests, query by blockchain/coingecko/sector)
- Exchange mapping (5 tests, case-insensitive)
- Behavior profiles (4 tests)
- Persistence across connections (2 tests)
- Performance baseline (3 tests)

Total: 1276 tests across 50 files, all green, zero regressions.
 2026-07-12 12:21:48 +02:00
Codex
019b620ab9 malkhut: asset bridge — directory ↔ classification integration 
asset_bridge.py: connects Fable's AssetDirectory (operational layer,
runtime-mutable, JSON-backed listing status) with our AssetProfile
(taxonomic layer, frozen, invariant classification).

Functions:
- sync_asset_to_profile(directory, symbol): sync one asset's TRADING
  exchanges from directory to AssetProfile.exchanges
- sync_exchanges_from_directory(directory): sync all matching assets
- get_universe_stats(directory): matched/unmatched counts

49 tests covering:
- normalize_symbol (7 edge cases)
- ExchangeListing validation (4 tests)
- AssetRecord listing queries (5 tests)
- AssetDirectory CRUD + persistence (14 tests)
- symbols_for_exchange filtering by status (4 tests)
- venue_symbol mapping (3 tests)
- Bridge sync (8 tests with state save/restore)
- Integration with ScenarioFactory + get_assets_on_exchange (4 tests)

Total: 1246 tests across 49 files, all green, zero regressions.
Fable's assets/ package preserved intact, interfaces retained.
 2026-07-12 10:45:37 +02:00
Codex
1f709af6d1 malkhut: three-layer identifier architecture for cross-system asset identification 
Layer 1 (canonical identity): symbol, base_asset, name, unified_symbol
  (CCXT format), quote_currency
Layer 2 (cross-system): coingecko_id, cmc_id, blockchain, contract_address
Layer 3 (exchange mapping): exchanges tuple

All 13 pre-defined assets migrated with accurate CoinGecko IDs, CMC IDs,
blockchains, and contract addresses (ERC-20 tokens).

7 new query functions: get_asset_by_coingecko_id, get_asset_by_cmc_id,
get_assets_by_base_asset, get_assets_by_blockchain,
get_assets_by_unified_symbol.

33 new tests covering: Layer 1 identity, Layer 2 cross-system identifiers,
identifier query functions, identifier consistency (uniqueness, derivation),
exchange registry.

Total: 1189 tests, 47 files, all green.

Based on research: CCXT BASE/QUOTE is de facto standard, CoinGecko ID
most widely used in crypto-native, ISO 24165 DTI emerging, FIGI for
institutional.
 2026-07-12 00:40:14 +02:00
Codex
d2d0c5e292 malkhut: multi-exchange asset universe + schema docs 
ExchangeProfile: standardized exchange metadata (fees, latency, capabilities).
3 pre-defined exchanges: Binance, BingX, Bybit.
AssetProfile.exchanges: tuple[str] — which venues trade each asset.
New query functions: get_assets_on_exchange, get_common_assets,
get_exchange_for_asset, get_exchange, list_exchanges.

_DATA_STORAGE_SCHEMA_FORMATS.md: comprehensive reference for agent
consumption — data model, storage format, query interfaces, data flow
diagram, enum reference, import patterns for BLUE/VIOLET/UV integration.

README updated: exchange registry section, package structure, subsystems
table.
 2026-07-11 22:37:15 +02:00
Codex
3efb8749fe malkhut(assets): full taxonomy onboarded for all 50 universe symbols (AssetCompiler, 0 failures) 2026-07-11 21:57:04 +02:00
Codex
a062507656 malkhut(assets)+uv: system-wide asset directory + UV universe init 
MALKHUT asset directory (malkhut/assets/): normalized canonical symbols,
KNOWN_EXCHANGES aux table (BINANCE/BINGX/BINGX_VST), per-exchange listing
status (TRADING/OFFLINE/UNKNOWN), JSON-backed, built for full-Binance-500
scale. Seeded with BLUE's NG7 feed universe (50 symbols) + live VST
contracts probe: 35 TRADING / 15 OFFLINE on VST (BAND, CELR, COS, CVC,
DENT, FUN, HOT, ICX, TFUEL, TUSD, USDC, WAN, WIN, XTZ, ZIL).
uv_asset_universe.init_asset_universe() = runtime tradable set for the
execution exchange; --onboard runs MALKHUT AssetCompiler for full taxonomy.
12 new tests, mutation-RED verified (status-filter + tradability guard).
 2026-07-11 21:48:05 +02:00
Codex
be0e1468da malkhut(perf): parallel episode eval — 9x single-eval speedup, zero fidelity loss 
- parallel_eval.py: ProcessPoolExecutor-based episode runner. Each worker
  gets its own CWM + planner instance. Zero shared state = embarrassingly
  parallel. Deterministic: same seed → same result.
- PolicyEvaluator.evaluate_candidate: new workers parameter (0=sequential,
  >1=parallel). Backward compatible: default workers=0.
- 16 new tests: determinism, pickling, result validity, cross-validation
  between sequential and parallel paths, backward compatibility.
- README: training performance table with speedup measurements.

Speedup results (3 assets × 30 scenarios = 90 scenarios):
  Sequential:  3.3s per eval  (1.0x)
  2 workers:   1.3s per eval  (2.6x)
  4 workers:   0.6s per eval  (5.9x)
  8 workers:   0.4s per eval  (9.1x)
  CMA-ES 48 evals: 125s → 85s (1.5x training speedup)

Note: CWM numba hot path was already wired (_HAS_NUMBA=True, 5.3µs/transition).
Bottleneck is MCTS planner (96% of eval time), not CWM.
 2026-07-11 20:19:32 +02:00
Codex
4c239f7774 malkhut(tests): 1140 test functions across 46 test files 
CWM (103): core mechanics, exhaustive edge cases, numba, exchange mechanics
Replay (118): exhaustive verification, microstructure, trajectory
Training (190): asset classification, phase0 extensive, pipeline, exhaustive
DSL (102): v2 syntax, expanded, new features
ASEx (33): validate-before-mutate, single-writer
Planner (48): MCTS, alternatives, hooks
Counterparties (19): 9 adversarial agent policies
Clock (30): event-driven reactor
BingX (28): venue adapter
IPC (8): Zinc SHM
Storage (9): ClickHouse
Risk (4): hard invariants
State (17): frozen dataclass invariants
Integration: E2E, concurrency, sync/async seams, hypothesis, fuzz, adversarial
 2026-07-11 10:46:12 +02:00
Codex
8af7e3bce8 malkhut(T9): smoke test launchers 
launch_smoke_test.py: 10-min quick smoke.
smoke_test_60min.py: 60-min full smoke with checkpoints.
 2026-07-11 10:41:31 +02:00
Codex
dd86174107 malkhut(T8): cognition pipeline + regime expansion + prod tooling 
Cognition pipeline (cognition.py): rate-limited, 8 sources, dedup, perm-run.
Regime expansion (regime_expansion.py): 200+ regimes from 4x4x4x4 dimensions.
News sources (news_sources.py): 12 industry-standard sources with ranking.
Monitor (monitor.py): metrics, health scoring, alerts, JSONL logging.
Cognition launcher (cognition_launcher.py): standalone long-run service.
Continuous pipeline (continuous_pipeline.py): forever-loop training runner.
 2026-07-11 10:39:03 +02:00
Codex
316d01079b malkhut(T7): UV Clock — event-driven reactor + staleness watchdog 
UVClock (clock/host.py): T19 event-driven reactor, asyncio dispatch.
Events (clock/events.py): Scan, Tick, Timer, BarFire, Stale.
Staleness watchdog (clock/staleness.py): T19 Staleness Law enforcement.
DeadNode reaper (clock/deadnode.py): iox2 orphan sweep on startup.
 2026-07-11 10:36:30 +02:00
Codex
ef2f8e8827 malkhut(T6): training core — CMA-ES trainer, registry, pipeline, selector 
CMA-ES trainer (cma_trainer.py): self-play pool, bootstrap CI, ScenarioFactory
with behavior-driven scenarios, auto-compile, label query interfaces.
Policy registry (registry.py): CANDIDATE → ACTIVE lifecycle.
Training pipeline (pipeline.py): bounded continuous learning loop + logger.
Strategy selector (selector.py): regime → strategy mapping, performance matrix.
 2026-07-11 10:33:56 +02:00
Codex
6fb55dadcb malkhut(T5): risk gate + ASEx + IPC + storage + venue adapter 
Risk gate (risk/gate.py): hard invariants — leverage, self-trade, post-only.
ASEx integration (execution/asex_integration.py): validate-before-mutate,
single-writer, zero-lock state serialization.
IPC (ipc/): Zinc SHM (zinc_plane.py) + control plane (control_plane.py),
UVZINC01 seqlock framing.
ClickHouse storage (storage/ch_store.py): 5 tables, HTTP API.
BingX venue adapter (venue/bingx/adapter.py): wraps DITAv2, order tracking.
 2026-07-11 10:31:31 +02:00
Codex
863a4cc8c9 malkhut(T4): Strategy DSL v2 + generator + supporting modules 
Strategy DSL v2 (dsl.py): 40+ action primitives, 40+ market sensors,
12 comparison operators, 16 builtins, full parser.
Strategy Generator (generator.py): genetic programming evolution —
crossover, mutation, tournament selection, pool management.
Supporting: discrepancy tracking, execution quality, hooks, feature
importance, observability, parallel eval, auto-rollback, stress testing,
structured observations, trajectory recording.
 2026-07-11 10:28:38 +02:00
Codex
4ffc8a601f malkhut(T3): planner + adversarial counterparties 
Planner: Decoupled UCB/UCT simultaneous-move MCTS (sm_mcts.py),
compact action space (action_menu.py), planner alternatives (alternatives.py).
Counterparties: 4+ adversarial agent ecology — ToxicTaker, LatencyArb,
MarketMaker, NoiseTrader + extended: LiquidationFlow, WhaleOrder,
MomentumFollower, SpoofDetector, QueueChaser.
 2026-07-11 10:26:01 +02:00
Codex
f943191d56 malkhut(T2): Code World Model — deterministic exchange simulator 
CWM core (core.py): price-time priority, sequential level consumption,
partial fills, queue position, latency injection, maker/taker fees.
Numba acceleration (numba_core.py): JIT hot loops, 1.8x fill speedup.
Replay verification (replay_verify.py): binary search, trajectory recording.
Supporting: adverse_selection, correlation, latency_model, multi_level,
queue_model, spread_dynamics, volatility, hftbacktest_validator.
 2026-07-11 10:23:44 +02:00
Codex
aa22529330 malkhut(T1): scaffold — frozen state model, actions, features, engine 
T1 scaffold: 42 frozen dataclasses (state.py), action model + PlannedPolicy +
RiskDecision (actions.py), 17-feature extraction (features.py),
FulfilmentEngine hot-path orchestrator (engine.py).
 2026-07-11 10:21:27 +02:00
Codex
981b469d51 malkhut: asset classification, behavior DSL, auto-compiler, behavior-driven scenarios 
Cambrian Explosion Phase 0 complete:
- Multi-label invariant asset taxonomy (13 assets, 10 sectors, 6 roles)
- Asset Behavior DSL: 10 orthogonal dimensions per asset, 3 composable
  templates, research-validated from live Binance/BingX API data
- Asset Compiler: auto-fetch from Binance public API, compute profiles,
  rate-limited (1 req/s), cached, 28 known classifications
- ScenarioFactory: all 30 scenario types use behavior-driven prices
  (BTC=$64K, ETH=$1.8K, SOL=$80, DOGE=$0.07) instead of hardcoded
  BTC prices. Auto-compiles unknown assets on demand.
- Label query interfaces: build_suite_for_sector/role/template/vol/labels
- 190 exhaustive tests for asset classification (up from 66)
- 1140 tests all green, CWM throughput 189K calls/s (121% of baseline)
- Comprehensive README: 1043 lines with full documentation

Research sources: Binance live REST API, BingX open API, CoinGlass,
academic literature (Bouchaud, Cont/Stoikov, Cartea/Jaimungal)
 2026-07-11 06:37:17 +02:00