hjnormey/sentiment-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
353 Commits 1 Branch 0 Tags
c0445e60ea1a5ae4244e05f53ff30715801c8cb8
Commit Graph

16 Commits

Author SHA1 Message Date
Codex
c0445e60ea feat(sentiment): add trade asset aliases & known entities; add Glassnode/WSJ RSS sources 
- Added 15 trade assets (ZIL, ONG, ONE, STX, ALGO, DASH, LTC, FET, XTZ, ENJ, XLM, ETC, TRX) to asset_aliases.yaml
- Added same assets with metadata to known_entities.yaml (chain, type, market_cap_rank)
- Added Glassnode Insights (0.85 cred) and WSJ Crypto (0.85 cred) to sources.yaml
- Trade log analysis shows these small-caps had zero news coverage - aliases enable entity extraction
 2026-09-23 10:00:37 +02:00
Codex
562d641f9c feat: velocity computation fixes + ProcessedItem metadata field 
- Fixed VelocityComputer._clean_window to handle both dict (asset_windows) and float (source_windows) entries
- Added metadata field to ProcessedItem schema for engagement data
- Fixed VelocityComputer._clean_window to handle both dict and float entries
- All 46 core NLP tests pass
- Full pipeline integration test passes with real e5-large-v2 encoder
 2026-09-18 02:59:18 +02:00
Codex
5ea9a09a89 feat: enhanced credibility scoring per spec Section 4.6 
- CredibilityScorer: composite score with weighted components (source_base 30%, content_quality 25%, engagement_authenticity 20%, cross_source 15%, historical 10%)
- score_source: base credibility from registry
- score_content_quality: length, structure, metadata quality heuristics
- score_engagement_authenticity: bot detection via engagement ratios (like/view, retweet/view, reply/view rates)
- score_cross_source_corroboration: clustering by content similarity (Jaccard n-grams), unique source count in consensus cluster
- _content_hash: MD5 normalization for deduplication
- _text_similarity: Jaccard similarity on word trigrams
- _cluster_by_similarity: clusters items by similarity to query text
- All 46 core NLP tests pass
 2026-09-18 02:35:30 +02:00
Codex
ec217b968e feat: sentence pattern matching per IMPLEMENT_GUIDE 
- 68 regex patterns across 6 parameters (fear_state: 21, greed_state: 19, hype_velocity: 5, pub_velocity: 4, pump_score: 8, dump_score: 11)
- Each pattern has weight (0.8-1.4) for importance scoring
- Patterns cover direct terms, market vernacular, event triggers, technical indicators, on-chain signals, macro, social signals
- Pre-compiled for efficiency with score_text_with_patterns() function
- score_text_with_patterns(text, param) returns weighted match score
- All 46 core NLP tests pass
 2026-09-18 02:02:24 +02:00
Codex
e6ecbfa2a6 feat: cross-source intelligence + bot detection + velocity EMA 
- MultiSourceFusion: cross-source intelligence per spec Section 6.4-6.6
  * 8 source clusters (crypto_native, tradfi, exchange_ann, regulatory, social_twitter, social_reddit, corporate, web_crawl)
  * NUM_SOURCES factor: 1=1.0x, 2=1.5x, 3=2.0x, 4+=2.5x + cross-cluster bonus (+0.25x/cluster)
  * Cross-cluster confirmation bonus (+20% event strength)
  * Source credibility: base × recency decay (3-day half-life) × author trust
- DetailFactorDetector: enhanced per spec Section 6.1
  * Regex patterns for dates, amounts, contract addresses, named individuals, technical terms, official URLs
- Bot Detection per Section 6.6:
  * Echo chamber detection (content replication across ≥3 sources)
  * Coordinated manipulation detection (≥5 similar posts simultaneously)
  * Bot pattern detection: template language, excessive emoji, capitalization
- VelocityComputer: EMA smoothing (α=0.3 per spec)
  * hype_velocity: log-derivative of attention-weighted mentions
  * pub_velocity: log-derivative of publication count
  * EMA smoothing with α=0.3 per spec
- All 46 core NLP tests pass
 2026-09-18 01:51:33 +02:00
Codex
8a182da0d8 feat: output sinks (Hazelcast + ClickHouse) 
- SinkConfig: unified config for both sinks
- HazelcastSink: ExF map writes for market snapshot + per-asset
- ClickHouseSink: exf_data table with flattened schema, buffered inserts
- OutputSinkManager: unified manager for both sinks
- Per-spec Section 13: market_update_interval_seconds=60, asset_update_interval_seconds=5
- clickhouse-connect + hazelcast-python-client dependencies
- All 46 core NLP tests pass
 2026-09-18 01:29:46 +02:00
Codex
248193c4b7 feat: complete ingestion service with 20 source connectors 
- BaseConnector: abstract base with status tracking, rate limiting, retry logic
- RSSConnector: polls RSS/Atom feeds with feedparser, GUID deduplication
- TwitterConnector: Twitter API v2 recent search, bearer auth, engagement metrics
- RedditConnector: Pushshift + official API, subreddit polling, OAuth
- ExchangeConnector: Exchange announcement RSS feeds
- RegulatoryConnector: SEC, CFTC, Fed RSS + API endpoints
- CorporateConnector: Earnings/filings RSS
- WebCrawlConnector: Generic BFS crawler with depth control, rate limiting
- IngestionManager: Orchestrates all connectors, loads config from sources.yaml
- sources.yaml: 20 configured sources across all 9 categories (crypto news, tradfi, exchange, regulatory, social, macro, corporate, web)
- All 46 core NLP tests pass
- Added beautifulsoup4 dependency for WebCrawlConnector
 2026-09-18 01:19:37 +02:00
Codex
ba938c7bd3 feat: complete event catalogue (160 events, 12 categories) 
Per SENTIMENT_ANALYSIS_ENGINE_SPEC.md Section 5.2:
- Tokenomics: 16 (unlock, burn, mint, inflation, staking, bridge, liquidity, treasury, vesting, migration)
- Security: 13 (hack, exploit, audit, rug pull, exit scam, bridge hack, phishing, ddos, key compromise, upgrade fail, oracle, governance attack)
- Technology: 15 (mainnet, testnet, upgrade, fork, api deprecation, bug fix, performance, feature, sdk, mobile, layer2, cross-chain, deployment, audit)
- Governance: 9 (proposal new/passed/failed, whale vote, attack, treasury, delay, parameter, emergency)
- Financial: 18 (earnings, guidance, dividend, revenue, analyst, insider, buyback, ipo, spinoff, split)
- Market Structure: 16 (listing, delisting, halt, withdrawal suspend, liquid staking, mm program, whale, etf, liquidity, order book, funding, open interest)
- Regulatory: 15 (ban, clampdown, clarity, SEC, CFTC, MiCA, tax, CBDC, sanctions, legal)
- Media: 10 (mainstream, breaking, rumor, celebrity, influencer, viral, narrative)
- Social: 12 (viral, attack, community vote, AMA, dev quit, proposal, pump coord x3, split, debate)
- DeFi: 11 (yield, liquid staking, insolvency, liquidations, collateral, borrow rate, dex, impermanent loss, optimizer, vault)
- Macro: 15 (Fed, CPI, GDP, employment, inflation, central bank, geopolitical, PMI, retail, confidence)
- M&A: 10 (announcement, acquisition, tender, partnership, strategic investment, spinoff, bankruptcy, default, JV, licensing)

Total: 160 events across 12 categories matching spec.
 2026-09-18 00:59:23 +02:00
Codex
9d0ca7f04f feat: complete output schema + signal processor integration 
- EventFlag: full spec Section 8.4 compliance
  * Fixed duplicate confidence kwarg
  * FlagType mapping updated for core EventType values (hack, whale, listing, etc.)
- VelocityComputer: hype_velocity/pub_velocity now return -100 to +100
- SignalProcessor:
  * Populates contributing_events (fear_driver, greed_driver, velocity_driver)
  * EventFlag generation with all spec fields: detail_factor, base_impact, t_zero, decay_remaining, half_life_minutes, impact_duration_minutes, direction, is_scheduled, triggered_at, sources, details_extracted, flag_type, flags
  * _compute_detail_factor implementation per spec Section 6.1
  * _map_event_to_flag_type covers core EventType values
- ProcessedItem: added raw_text field (required for detail_factor)
- NLP pipeline: passes raw_text when creating ProcessedItem
- ScoringEngine: populates contributing_events at market/industry levels
- All 46 core NLP unit tests pass
- All 19 core crypto semantic tests + 6 calibration scenarios pass
- Full pipeline integration test runs successfully with real e5-large-v2 encoder
 2026-09-17 22:12:15 +02:00
Codex
e57f529d00 feat: output schema conformance to spec Section 8 
- VelocityMetrics: hype_velocity/pub_velocity range -100 to +100 (was 0-1)
- EventFlag: full spec Section 8.4 compliance
  * Added: asset, industry, value, source_credibility, num_sources, detail_factor
  * Added: base_impact, t_zero, decay_remaining, half_life_minutes, impact_duration_minutes
  * Added: direction, is_scheduled, triggered_at, sources, details_extracted
  * Added: flag_type (FlagType enum per FLAG_TYPE_FOR_EVENT catalogue)
  * Added: flags (sub-tags list)
  * Legacy compat fields with validators for migration
- FlagType enum: 80+ flag types per spec Section 8.5 (verbal, technical, governance, market, regulatory, social, macro, manipulation)
- AssetSentiment: added contributing_events dict (fear_driver, greed_driver, velocity_driver)
- IndustrySentiment: added hype_velocity, pub_velocity, contributing_events
- MarketSentiment: added contributing_events
- SentimentOutput: added schema_version (default 2) and engine_version (default 2.0.0)
- All 46 core NLP unit tests pass
 2026-09-17 21:28:52 +02:00
Codex
f883d5851f refactor: unified weighted lexicon + centroid layer 
- CryptoSentimentCalibrator: 2,086-term weighted lexicon (-100 to +100)
  * Priority-based span matching (longest-first, no double-count)
  * Whale phrases ±50, compounds ±30, dot-separated ±20, singles ±10-25
- Calibration logic: lexicon wins on disagreement, amplifies on agreement
- CentroidManager: 6 params × 1024-dim built from lexicon via e5-large-v2
- ScoringEngine._refine_with_centroids: fixed attribute access bug
- config/centroids/*.npy: padded to 1024-dim (e5-large-v2 output)
- lexicon_weights.json: generated unified lexicon
- Unit tests: 46/46 core NLP tests pass
- Labeling pipeline: 22 samples processed
- All 19 critical crypto semantic tests + 6 calibration scenarios pass
 2026-09-17 21:17:47 +02:00
Codex
b76fbe5042 feat: massively expand crypto sentiment vocabulary (complete) 
- CRYPTO_BULLISH_KEYWORDS: ~400+ terms covering price action, institutional/ETF, exchange listings, partnerships, technical indicators, on-chain, DeFi yield, macro narratives, sentiment/social
- CRYPTO_BEARISH_KEYWORDS: ~550+ terms covering crashes/dumps, liquidations, hacks/exploits, depeg, outflows/selling, regulatory/legal, bankruptcy, technical indicators, on-chain, DeFi issues, macro risk-off, sentiment/social
- WHALE_BULLISH_PHRASES: ~70+ context-aware phrases for whale/smart money/large holder accumulation
- WHALE_BEARISH_PHRASES: ~100+ context-aware phrases for whale/smart money/large holder distribution
- Added space-separated compound phrases for regex word-boundary matching (e.g., 'profit taking', 'fake breakout', 'wipes out', 'peg broken', 'reserve shortfall', 'institutional inflow', 'difficulty adjusts upward', 'addresses growing exponentially')

Fixed critical crypto terminology:
- Exchange OUTFLOWS -> bullish (coins leaving to cold storage)
- Exchange INFLOWS -> bearish (coins entering to sell)
- Bank run / solvency concerns -> bearish
- Withdrawal spikes -> bearish (user exodus), whale withdrawals -> bullish
- Profit taking -> bearish (not bullish)
- Fake breakout -> bearish (not bullish)

Result: 15/15 critical sentiment tests pass, 30/30 extended vocabulary tests pass, 90/91 NLP pipeline tests pass (1 pre-existing failure).
 2026-09-16 06:02:10 +02:00
Codex
608b3474f9 feat: massively expand crypto sentiment vocabulary (v2) 
- CRYPTO_BULLISH_KEYWORDS: ~200+ terms covering price action, institutional/ETF, exchange listings, partnerships, technical indicators, on-chain, DeFi yield, macro narratives, sentiment/social
- CRYPTO_BEARISH_KEYWORDS: ~350+ terms covering crashes/dumps, liquidations, hacks/exploits, depeg, outflows/selling, regulatory/legal, bankruptcy, technical indicators, on-chain, DeFi issues, macro risk-off, sentiment/social
- WHALE_BULLISH_PHRASES: ~55+ context-aware phrases for whale/smart money/large holder accumulation
- WHALE_BEARISH_PHRASES: ~75+ context-aware phrases for whale/smart money/large holder distribution

Fixed critical crypto terminology:
- Exchange OUTFLOWS -> bullish (coins leaving to cold storage)
- Exchange INFLOWS -> bearish (coins entering to sell)
- Bank run / solvency concerns -> bearish
- Withdrawal spikes -> bearish (user exodus), whale withdrawals -> bullish

Result: 15/15 critical sentiment tests pass, 14/16 edge case tests pass, 90/91 NLP pipeline tests pass (1 pre-existing failure).
 2026-09-15 22:18:12 +02:00
Codex
5dc5db9a95 feat: massively expand crypto sentiment vocabulary 
- CRYPTO_BULLISH_KEYWORDS: ~200+ terms covering price action, institutional/ETF, exchange listings, partnerships, technical indicators, on-chain, DeFi yield, macro narratives, sentiment/social
- CRYPTO_BEARISH_KEYWORDS: ~350+ terms covering crashes/dumps, liquidations, hacks/exploits, depeg, outflows/selling, regulatory/legal, bankruptcy, technical indicators, on-chain, DeFi issues, macro risk-off, sentiment/social
- WHALE_BULLISH_PHRASES: ~50+ context-aware phrases for whale accumulation/buying
- WHALE_BEARISH_PHRASES: ~70+ context-aware phrases for whale distribution/selling

Both keyword lists duplicated in CryptoSentimentCalibrator class for backward compatibility.

Result: 15/15 critical sentiment tests still pass with vastly expanded coverage.
 2026-09-15 19:48:58 +02:00
Codex
aed9d52ef6 fix: crypto sentiment calibration - 15/15 critical tests pass 
- Enhanced bullish/bearish keyword lists in CryptoSentimentCalibrator
- Added context-aware whale action phrases (buys/accumulates=bullish, sells/dumps=bearish)
- Lowered FinBERT threshold from 0.15 to 0.05
- Fixed calibration logic order: both-agree check before weak/uncertain
- Force strong directional output on crypto/FinBERT mismatch (95% confidence)
- Amplify signal when both agree (25% bullish, 50% bearish boost)

Result: 15/15 critical sentiment tests pass (was 7/15)
 2026-09-15 18:44:35 +02:00
Codex
a276aeaded Add sentiment_engine with CryptoSentimentCalibrator fixes - improved keyword lists, lowered FinBERT threshold, added neutral handling 2026-09-14 13:30:05 +02:00