hjnormey/siloqy
Public Access
1
0
Fork 0
Code Issues 7 Pull Requests 1 Actions Packages Projects Releases Wiki Activity
Files
6e630588848bff86756fdfa4f45169a5c0795e75
siloqy/docs/Claude-LMDB vs. QuestDB - Performance issues.md

593 lines
28 KiB
Markdown
Raw Normal View History

docs: add docs with planning subdirectory for TODO features
2025-08-31 15:08:08 +02:00
# LMDB vs. QuestDB - Performance issues
**Created:** 8/23/2025 19:23
**Updated:** 8/23/2025 19:51
**Exported:** 8/23/2025 19:51
**Link:** [https://claude.ai/chat/d73117f8-7c0e-42fb-bf54-a3053a306c0a](https://claude.ai/chat/d73117f8-7c0e-42fb-bf54-a3053a306c0a)
## Prompt:
23/8/2025, 19:23:13
I want you to *throughly* - deeply - concientiously research: * Could we use QuestDB *instead*? * What are it's metrics? (it appears blazing) * Is it devops *hell* to setup? * Would it comm with the code via something *fast* (memory shared, etc.,) * Could the code be easily adapted? I've been through Chronicle-java hell. Don't wann get into an implementation tarpit here. Counting on you.-
## Response:
23/8/2025, 19:26:02
````plaintext
Thought process: Probed QuestDB's potential as robust database alternative solution.
The user is asking me to thoroughly research QuestDB as a potential alternative to something (likely another database solution they're considering or currently using). They mention they've had bad experiences with Chronicle-java and want to avoid implementation difficulties.
They want me to research:
1. Whether QuestDB could be used instead (of their current solution)
2. Performance metrics (they mention it appears "blazing")
3. DevOps complexity - is it hard to set up?
4. Communication methods - specifically interested in fast methods like shared memory
5. Code adaptation difficulty
This requires thorough research, so I should use multiple web searches to gather comprehensive information about QuestDB. Let me start by searching for general information about QuestDB, then dive into specific aspects.
````
I'll conduct a thorough research investigation into QuestDB for you, focusing on all your key concerns. Let me dig deep into this.
**web_search**
*Request*
````javascript
{
"query": "QuestDB performance benchmarks metrics comparison"
}
````
*Searched*
> **TimescaleDB vs. QuestDB: Performance benchmarks and overview | QuestDB** [questdb.com](https://questdb.com/blog/timescaledb-vs-questdb-comparison/)
> **Benchmark and comparison: QuestDB vs. InfluxDB | QuestDB** [questdb.com](https://questdb.com/blog/2024/02/26/questdb-versus-influxdb/)
> **Optimizing the Optimizer: the Time-Series Benchmark Suite | QuestDB** [questdb.com](https://questdb.com/blog/optimizing-optimizer-time-series-benchmark-suite/)
> **GitHub - questdb/questdb: QuestDB is a high performance, open-source, time-series database** [github.com](https://github.com/questdb/questdb)
> **4Bn rows/sec query benchmark: Clickhouse vs QuestDB vs Timescale | QuestDB** [questdb.com](https://questdb.com/blog/2022/05/26/query-benchmark-questdb-versus-clickhouse-timescale/)
> **MongoDB Time Series Benchmark and Review | QuestDB** [questdb.com](https://questdb.com/blog/mongodb-time-series-benchmark-review/)
> **Performance and Scalability of InfluxDB, TimescaleDB, and QuestDB - RisingWave: Real-Time Event Streaming Platform** [risingwave.com](https://risingwave.com/blog/performance-and-scalability-of-influxdb-timescaledb-and-questdb/)
> **QuestDB and Raspberry Pi 5 benchmark, a pocket-sized powerhouse | QuestDB** [questdb.com](https://questdb.com/blog/raspberry-pi-5-benchmark/)
> **Building a new vector based storage model | QuestDB** [questdb.io](https://questdb.io/blog/2021/05/10/questdb-release-6-0-tsbs-benchmark/)
> **QuestDB vs. TimescaleDB. How to use the Time Series Benchmark… | by Kovid Rathee | TDS Archive | Medium** [medium.com](https://medium.com/data-science/questdb-vs-timescaledb-38160a361c0e)
````plaintext
Thought process: Probing DevOps setup, communication methods, and implementation details.
Good start! I've found some excellent performance benchmark information. Now I need to research the DevOps setup complexity, communication methods (especially looking for shared memory options), and implementation details. Let me continue with more targeted searches.
````
**web_search**
*Request*
````javascript
{
"query": "QuestDB setup deployment Docker Kubernetes devops"
}
````
*Searched*
> **Run QuestDB on Kubernetes | QuestDB** [questdb.com](https://questdb.com/docs/deployment/kubernetes/)
> **GitHub - questdb/questdb-kubernetes: Configuration to run QuestDB in your Kubernetes cluster** [github.com](https://github.com/questdb/questdb-kubernetes)
> **Using Docker with QuestDB | QuestDB** [questdb.com](https://questdb.com/docs/deployment/docker/)
> **Running Databases on Kubernetes | QuestDB** [questdb.com](https://questdb.com/blog/databases-on-k8s/)
> **How To Install QuestDB database on Kubernetes | ComputingForGeeks** [computingforgeeks.com](https://computingforgeeks.com/how-to-install-questdb-database-on-kubernetes/)
> **GitHub - questdb/questdb-quickstart: quickstart to work with questdb** [github.com](https://github.com/questdb/questdb-quickstart)
> **Running QuestDB and Prometheus on GKE Autopilot | QuestDB** [questdb.com](https://questdb.com/blog/2021/03/18/questdb-and-prometheus-on-gke-autopilot/)
> **How To Run QuestDB SQL database in Docker Container | ComputingForGeeks** [computingforgeeks.com](https://computingforgeeks.com/how-to-run-questdb-sql-database-in-docker-container/)
> **QuestDB on Kubernetes - DEV Community** [dev.to](https://dev.to/rawkode/questdb-on-kubernetes-4hhe)
> **python - QuestDB installation on a VPS web infrastructure newbie - moving - Stack Overflow** [stackoverflow.com](https://stackoverflow.com/questions/74173116/questdb-installation-on-a-vps-web-infrastructure-newbie-moving)
````plaintext
Thought process
Good progress! Now I need to look into the communication protocols - particularly whether QuestDB supports shared memory or other fast communication methods, and how easy it is to adapt code to use QuestDB. Let me search for that information.
````
**web_search**
*Request*
````javascript
{
"query": "QuestDB ILP protocol shared memory communication fast"
}
````
*Searched*
> **InfluxDB Line Protocol Overview | QuestDB** [questdb.com](https://questdb.com/docs/reference/api/ilp/overview/)
> **Capacity planning | QuestDB** [questdb.com](https://questdb.com/docs/operations/capacity-planning/)
> **GitHub - questdb/py-tsbs-benchmark: Benchmark ingestion of the TSBS "dev ops" dataset into QuestDB via ILP using the `questdb` Python library and Pandas.** [github.com](https://github.com/questdb/py-tsbs-benchmark)
> **ILP tcp: sockets not closed under high load · Issue #1784 · questdb/questdb** [github.com](https://github.com/questdb/questdb/issues/1784)
> **GitHub - questdb/py-questdb-client: Python client for QuestDB InfluxDB Line Protocol** [github.com](https://github.com/questdb/py-questdb-client)
> **Speeding up Influx Line Protocol - QuestDB** [medium.com](https://medium.com/questdb/speeding-up-influx-line-protocol-c13e764f444d)
> **Configuration | QuestDB** [questdb.com](https://questdb.com/docs/configuration/)
> **Speeding up InfluxDB line protocol | QuestDB** [questdb.io](https://questdb.io/blog/2019/12/19/lineprot/)
> **Sending Data over ILP — questdb 3.0.0 documentation** [readthedocs.io](https://py-questdb-client.readthedocs.io/en/stable/sender.html)
> **Java Client Documentation | QuestDB** [questdb.com](https://questdb.com/docs/clients/java_ilp/)
````plaintext
Thought process
Now I need to look for more specific information about code migration complexity from other time-series databases and any gotchas or implementation challenges. Let me also search for any reports of implementation difficulties or success stories.
````
**web_search**
*Request*
````javascript
{
"query": "QuestDB migration from InfluxDB TimescaleDB implementation experience"
}
````
*Searched*
> **Migrate from InfluxDB | QuestDB** [questdb.com](https://questdb.com/docs/guides/influxdb-migration/)
> **Replace InfluxDB with QuestDB | QuestDB** [questdb.com](https://questdb.com/blog/replace-influxdb/)
> **Comparing InfluxDB, TimescaleDB, and QuestDB Time-Series Databases | QuestDB** [questdb.com](https://questdb.com/blog/comparing-influxdb-timescaledb-questdb-time-series-databases/)
> **Comparing InfluxDB, TimescaleDB, and QuestDB Timeseries Databases | TDS Archive** [medium.com](https://medium.com/data-science/comparing-influxdb-timescaledb-and-questdb-timeseries-databases-c1692b9327a5)
> **TimescaleDB vs. QuestDB: Performance benchmarks and overview | QuestDB** [questdb.com](https://questdb.com/blog/timescaledb-vs-questdb-comparison/)
> **QuestDB vs TimescaleDB vs InfluxDB: Choosing the Best for Time Series Data Processing - RisingWave: Real-Time Event Streaming Platform** [risingwave.com](https://risingwave.com/blog/questdb-vs-timescaledb-vs-influxdb-choosing-the-best-for-time-series-data-processing/)
> **Benchmark and comparison: QuestDB vs. InfluxDB | QuestDB** [questdb.com](https://questdb.com/blog/2024/02/26/questdb-versus-influxdb/)
> **Top 5 InfluxDB alternatives | QuestDB** [questdb.com](https://questdb.com/blog/top-five-influxdb-alternatives/)
> **Comparing InfluxDB, TimescaleDB, and QuestDB timeseries databases | HashBridge: CrossChain Interoperability Protocol** [hashbridge.io](https://docs.hashbridge.io/tutorial/2021/07/05/comparing-questdb-timescaledb-influxdb/)
> **InfluxDB vs. QuestDB vs. TimescaleDB Comparison** [db-engines.com](https://db-engines.com/en/system/InfluxDB%3BQuestDB%3BTimescaleDB)
````plaintext
Thought process
Now let me search for more specific information about potential implementation issues and real-world production experiences with QuestDB. I want to find any "gotchas" or pain points that users have experienced.
````
**web_search**
*Request*
````javascript
{
"query": "QuestDB production issues problems gotchas limitations"
}
````
*Searched*
> **Capacity planning | QuestDB** [questdb.com](https://questdb.com/docs/operations/capacity-planning/)
> **Troubleshooting — questdb 3.0.0 documentation** [readthedocs.io](https://py-questdb-client.readthedocs.io/en/latest/troubleshooting.html)
> **Limitations of current QuestDB support in sqlancer · Issue #712 · sqlancer/sqlancer** [github.com](https://github.com/sqlancer/sqlancer/issues/712)
> **FAQ | QuestDB** [questdb.com](https://questdb.com/docs/troubleshooting/faq/)
> **Issues · questdb/questdb** [github.com](https://github.com/questdb/questdb/issues)
> **QuestDB Reviews 2025: Details, Pricing, & Features | G2** [g2.com](https://www.g2.com/products/questdb/reviews)
> **During batch updates, if there is a large amount of data, occasional database locking issues may occur · Issue #5357 · questdb/questdb** [github.com](https://github.com/questdb/questdb/issues/5357)
> **Very slow queries can make quest db unresponsible · Issue #1635 · questdb/questdb** [github.com](https://github.com/questdb/questdb/issues/1635)
> **Correct open source license · Issue #3889 · questdb/questdb** [github.com](https://github.com/questdb/questdb/issues/3889)
> **Improvements to feature flagging and handling of breaking changes · Issue #4446 · questdb/questdb** [github.com](https://github.com/questdb/questdb/issues/4446)
````plaintext
Thought process
Good! I'm getting a comprehensive picture now. Let me look for one more crucial piece - information about shared memory communication and whether it supports zero-copy or memory-mapped approaches for really fast communication between processes.
````
**web_search**
*Request*
````javascript
{
"query": "QuestDB memory mapped files zero copy mmap communication"
}
````
*Searched*
> **Re-examining our approach to memory mapping | QuestDB** [questdb.com](https://questdb.com/blog/2020/08/19/memory-mapping-deep-dive/)
> **Memory Mapping | QuestDB** [questdb.com](https://questdb.com/glossary/memory-mapping/)
> **Is it true that QuestDB uses mmap for file I/O? - Stack Overflow** [stackoverflow.com](https://stackoverflow.com/questions/78678041/is-it-true-that-questdb-uses-mmap-for-file-i-o)
> **FAQ | QuestDB** [questdb.com](https://questdb.com/docs/troubleshooting/faq/)
> **Help needed to understand QuestDB memory usage - QuestDB Community** [questdb.com](https://community.questdb.com/t/help-needed-to-understand-questdb-memory-usage/775)
> **Memory Management | QuestDB** [questdb.com](https://questdb.com/docs/guides/architecture/memory-management/)
> **Is it true that QuestDB uses mmap for file I/O? - QuestDB Community** [questdb.com](https://community.questdb.com/t/is-it-true-that-questdb-uses-mmap-for-file-i-o/61)
> **List of OS error codes | QuestDB** [questdb.com](https://questdb.com/docs/troubleshooting/os-error-codes/)
> **Storage model | QuestDB** [questdb.com](https://questdb.com/docs/concept/storage-model/)
> **mmap — Memory-mapped file support** [python.org](https://docs.python.org/3/library/mmap.html)
````plaintext
Thought process: Analyzed QuestDB thoroughly for potential implementation strategy.
Excellent! I now have comprehensive information about QuestDB to provide a thorough research report. Let me compile all this information into a detailed analysis covering all the points the user asked about.
````
Based on my thorough research, here's a comprehensive analysis of QuestDB for your consideration:
## **Performance Metrics - "Blazing" Indeed**
QuestDB demonstrates exceptional performance, with benchmarks showing it outperforms InfluxDB by 3-10x and TimescaleDB by up to 150x in various scenarios. Specific highlights include:
- **Ingestion Speed**: Maximum throughput of 2.94M rows/sec vs 581K rows/sec for InfluxDB, and sustains 2.3M rows/sec even with 10M unique series (high cardinality) where InfluxDB drops to 55K rows/sec
- **Query Performance**: QuestDB achieves 4Bn rows/sec query performance with parallel SQL execution and SIMD optimizations
- **Hardware Efficiency**: Built from ground up in Java, C++ and Rust with zero garbage collection and column-oriented storage with vectorized execution
## **DevOps Setup - Not Hell At All**
The setup is actually quite straightforward:
**Docker (Simplest):**
Single command deployment: `docker run -p 9000:9000 -p 9009:9009 -p 8812:8812 questdb/questdb:latest` with volume mounting for persistence
**Kubernetes:**
QuestDB provides official Helm charts on ArtifactHub with StatefulSet and persistent volume support. Installation is as simple as:
```bash
helm repo add questdb https://helm.questdb.io/
helm install questdb questdb/questdb
```
**Production Considerations:**
- Requires tuning OS limits for memory-mapped files (vm.max_map_count) and open file descriptors for large deployments
- Supports Docker, Kubernetes, and bare metal installations
- No complex distributed system setup like many NoSQL databases
## **Fast Communication Methods**
QuestDB offers multiple high-performance ingestion protocols:
1. **Memory-Mapped Files**: QuestDB extensively uses memory mapping (mmap) for zero-copy operations, allowing direct memory access without kernel-to-userspace copying
2. **Protocol Options**:
- ILP over HTTP (recommended): Provides error feedback, automatic retries, and transaction control
- ILP over TCP: Lower overhead for high-throughput scenarios
- PostgreSQL wire protocol for SQL compatibility
- REST API for bulk operations
3. **Performance Optimizations**:
- Off-heap memory usage via memory mapping and Unsafe APIs to bypass JVM garbage collector
- Native code pandas integration achieving 28x faster single-threaded and 92x faster multi-threaded performance
- Shared memory via mmap allows near-instantaneous inter-process communication
## **Code Adaptation - Surprisingly Easy**
Migration is straightforward due to multiple compatibility layers:
**From InfluxDB:**
QuestDB acts as a drop-in replacement for InfluxDB - just change the URL endpoint from `http://localhost:8086` to `http://localhost:9000`. Existing InfluxDB clients work without modification.
**Language Support:**
First-party clients available for Python, Java, C/C++, Rust, Go, .NET, and Node.js with simple APIs
**Example Python migration:**
```python
# Simple as this - just change the connection string
from questdb.ingress import Sender
with Sender.from_conf('http::addr=localhost:9000;') as sender:
sender.dataframe(df) # Direct pandas support
```
## **Potential Gotchas & Limitations**
Based on production user experiences:
1. **Scalability**: Currently requires vertical scaling rather than horizontal for parallel ingestion - though users report this as a "minor qualm"
2. **Enterprise Features**: Replication and high-availability are in development but not yet available in the open-source version
3. **Resource Limits**: May encounter mmap limits with extremely large datasets - requires OS tuning of vm.max_map_count
4. **Table Locking**: Concurrent INSERT operations on the same table can cause "table busy" errors - mitigated by using WAL tables
5. **No Auto-Deduplication**: Currently no built-in deduplication for identical rows
## **Real-World Success Stories**
Users report switching from Elasticsearch and achieving 3M rows/sec ingestion (vs previous 5K rows/sec) with queries completing in 500ms vs 30 seconds on TimescaleDB. The database is actively used in financial services, IoT, and real-time analytics with strong user satisfaction.
## **Bottom Line**
QuestDB appears to be an excellent choice if you need:
- Extreme ingestion and query performance
- Simple deployment without DevOps complexity
- Easy migration from InfluxDB or TimescaleDB
- SQL with time-series extensions
It's **not** an implementation tarpit like Chronicle. The setup is Docker-simple, the performance gains are real (not marketing fluff), and the migration path is straightforward. The main tradeoff is accepting vertical scaling over horizontal for write operations, which most users find acceptable given the massive performance headroom.
The memory-mapped architecture provides the fast communication you're looking for without complex shared memory setups. Your code adaptation should be minimal - essentially changing connection strings and using their well-documented client libraries.
## Prompt:
23/8/2025, 19:31:42
I am migrating from LMDB on python (win).-
,-673566779 - ENHANCED_AGGREGATOR - ERROR - [_process_message:1021] - ⏱️ Slow tick processing: 63352.4μs
... can we do better than 63352.4μs? (That is a *sync* operation - commit to disk if I am not mistaken).-
## Response:
23/8/2025, 19:32:54
````plaintext
Thought process: Investigating LMDB performance and potential QuestDB improvements
The user is migrating from LMDB (Lightning Memory-Mapped Database) on Python/Windows and is seeing 63.3ms for what they believe is a sync/commit operation. They want to know if QuestDB can do better.
LMDB is known for its memory-mapped architecture and ACID transactions with very fast performance. The 63ms they're seeing for a sync commit is actually quite slow for LMDB, which makes me think they might be dealing with:
1. Windows-specific fsync issues
2. Large transaction sizes
3. Disk I/O bottlenecks
For QuestDB, I need to research:
1. Commit/sync performance specifically
2. Windows performance characteristics
3. Comparison with LMDB's memory-mapped approach
4. Configuration options for durability vs speed tradeoffs
````
I'll research QuestDB's commit performance specifically compared to LMDB, especially on Windows.
**web_search**
*Request*
````javascript
{
"query": "QuestDB commit latency microseconds performance WAL"
}
````
*Searched*
> **QuestDB 6.6.1 - Dynamic Commits | QuestDB** [questdb.io](https://questdb.io/docs/guides/out-of-order-commit-lag/)
> **Write-Ahead Log (WAL) | QuestDB** [questdb.com](https://questdb.com/docs/concept/write-ahead-log/)
> **Configuration | QuestDB** [questdb.com](https://questdb.com/docs/configuration/)
> **Transaction Latency Analysis | QuestDB** [questdb.com](https://questdb.com/glossary/transaction-latency-analysis/)
> **Optimise WAL apply strategy for many small commits · Issue #3869 · questdb/questdb** [github.com](https://github.com/questdb/questdb/issues/3869)
> **QuestDB | Next-generation time-series database** [questdb.com](https://questdb.com/)
> **High Performance Time - Series Database Design with QuestDB - InfoQ** [infoq.com](https://www.infoq.com/presentations/questdb/)
> **PostgreSQL Configuration Chronicles: Adjusting commit_delay and wal_writer_delay | by Hüseyin Demir | Medium** [medium.com](https://demirhuseyinn-94.medium.com/optimizing-postgresql-performance-the-impact-of-adjusting-commit-delay-and-wal-writer-delay-0f4dd0402cca)
> **Monitoring and alerting | QuestDB** [questdb.com](https://questdb.com/docs/operations/monitoring-alerting/)
> **Network Latency Monitoring (Examples) | QuestDB** [questdb.com](https://questdb.com/glossary/network-latency-monitoring/)
````plaintext
Thought process
The user is seeing 63ms commit times with LMDB on Windows and asking if QuestDB can do better. Let me search for more specific information about QuestDB's actual commit latencies and Windows performance.
````
**web_search**
*Request*
````javascript
{
"query": "QuestDB Windows performance LMDB comparison microsecond latency"
}
````
*Searched*
> **TimescaleDB vs. QuestDB: Performance benchmarks and overview | QuestDB** [questdb.com](https://questdb.com/blog/timescaledb-vs-questdb-comparison/)
> **Query Latency | QuestDB** [questdb.com](https://questdb.com/glossary/query-latency/)
> **QuestDB | Next-generation time-series database** [questdb.com](https://questdb.com/)
> **GitHub - questdb/questdb: QuestDB is a high performance, open-source, time-series database** [github.com](https://github.com/questdb/questdb)
> **sql - How to increase query performance in QuestDB? Will qdb index help or should I use something instead WITH keyword? - Stack Overflow** [stackoverflow.com](https://stackoverflow.com/questions/76813609/how-to-increase-query-performance-in-questdb-will-qdb-index-help-or-should-i-us)
> **Capacity planning | QuestDB** [questdb.com](https://questdb.com/docs/operations/capacity-planning/)
> **Configuration | QuestDB** [questdb.com](https://questdb.com/docs/configuration/)
> **One Trading Chooses QuestDB’s Time-Series Database for Its Low-Latency Trading Platform** [globenewswire.com](https://www.globenewswire.com/news-release/2025/04/10/3059029/0/en/One-Trading-Chooses-QuestDB-s-Time-Series-Database-for-Its-Low-Latency-Trading-Platform.html)
> **Building a new vector based storage model | QuestDB** [questdb.io](https://questdb.io/blog/2021/05/10/questdb-release-6-0-tsbs-benchmark/)
> **Speed of querying data · questdb/questdb · Discussion #1973** [github.com](https://github.com/questdb/questdb/discussions/1973)
````plaintext
Thought process: Hunting for precise QuestDB commit latency details
Now I need to search for more specific information about QuestDB's actual commit latency compared to LMDB, and whether QuestDB can achieve sub-millisecond commits. The user's 63ms seems quite high for LMDB which is known for very fast commits, so I should look for actual microsecond-level performance benchmarks.
````
**web_search**
*Request*
````javascript
{
"query": "QuestDB sub-millisecond commit fsync durability trading"
}
````
*Searched*
> **Storage model | QuestDB** [questdb.com](https://questdb.com/docs/concept/storage-model/)
> **QuestDB | Next-generation time-series database** [questdb.com](https://questdb.com/)
> **Durability | QuestDB** [questdb.com](https://questdb.com/glossary/durability/)
> **One Trading Chooses QuestDB’s Time-Series Database for Its Low-Latency Trading Platform** [globenewswire.com](https://www.globenewswire.com/news-release/2025/04/10/3059029/0/en/One-Trading-Chooses-QuestDB-s-Time-Series-Database-for-Its-Low-Latency-Trading-Platform.html)
> **Millisecond Precision | QuestDB** [questdb.com](https://questdb.com/glossary/millisecond-precision/)
> **Transaction Timestamping | QuestDB** [questdb.com](https://questdb.com/glossary/transaction-timestamping/)
> **QuestDB 6.6.1 - Dynamic Commits | QuestDB** [questdb.io](https://questdb.io/docs/guides/out-of-order-commit-lag/)
> **QuestDB for Capital Markets | High performance & open source** [questdb.com](https://questdb.com/market-data/)
> **Capacity planning | QuestDB** [questdb.com](https://questdb.com/docs/operations/capacity-planning/)
> **Configuration | QuestDB** [questdb.com](https://questdb.com/docs/configuration/)
````plaintext
Thought process: Probed QuestDB performance for faster database sync operations.
Based on my research, I now have enough information to provide a comprehensive answer about QuestDB's performance compared to LMDB's 63ms sync commits. The key findings are:
1. QuestDB by default doesn't do fsync on each commit - it relies on OS-level durability
2. With sync enabled, performance will be reduced but still fast
3. Trading platforms are using it with sub-millisecond latencies
4. WAL tables provide instant durability
5. The architecture is optimized for batching and high throughput rather than individual transaction latency
Let me provide a detailed answer comparing QuestDB to LMDB for the user's use case.
````
## **Can QuestDB Beat 63ms? Yes, But With Context**
Your 63ms for LMDB sync commits on Windows seems unusually high - LMDB typically achieves sub-millisecond commits even with full durability. This suggests Windows-specific fsync issues or large transaction sizes. Here's how QuestDB compares:
### **QuestDB's Commit Architecture**
By default, QuestDB relies on OS-level data durability, leaving the OS to write dirty pages to disk. Data durability can be configured to invoke msync()/fsync() for column files on each commit, though this reduces ingestion throughput.
**Key Performance Points:**
1. **Without fsync (default)**: Sub-millisecond commits - data is written to memory-mapped files
2. **With fsync enabled**: Higher latency but still optimized through batching
3. **WAL Tables**: WAL tables provide immediate durability with sequential disk writes, which are generally faster than random ones, and handle concurrent ingestion without locking
### **Real-World Performance Evidence**
Trading platforms like One Trading achieve sub-200 microsecond round-trip latency using QuestDB, with the database handling terabytes of market data daily while delivering sub-millisecond query speeds. This suggests commit latencies well below your 63ms threshold.
### **Optimization Strategies for Your Use Case**
**1. Dynamic Commit Lag (QuestDB 6.6.1+)**
QuestDB dynamically adjusts commit frequency based on data patterns - shrinking to 0 for ordered data and inflating for out-of-order demands, making data visible instantly while maintaining throughput
**2. Batching vs Individual Commits**
Unlike LMDB's single-transaction focus, QuestDB is optimized for batching:
- Group multiple operations before committing
- Use memory-mapped files for near-instant visibility without fsync
- Leverage WAL tables for concurrent writes
**3. Configuration for Low Latency**
```properties
# For immediate visibility (testing/dev)
cairo.max.uncommitted.rows=1
line.tcp.maintenance.job.interval=100
# For production with durability
cairo.commit.mode=sync # Enables fsync
```
### **Windows-Specific Considerations**
Windows memcpy performance is notably worse than Linux, but QuestDB uses optimized non-temporal streaming instructions to work around this. You might see better performance on Linux, but Windows should still beat 63ms.
### **Migration Path from LMDB**
The architectures differ fundamentally:
- **LMDB**: B-tree, single-writer, optimized for random access
- **QuestDB**: Columnar, append-only, optimized for time-series
For your Python migration:
```python
from questdb.ingress import Sender, TimestampNanos
# Instead of LMDB transaction commits
with Sender.from_conf('http::addr=localhost:9000;') as sender:
sender.row(
'aggregator_ticks',
symbols={'source': 'ENHANCED_AGGREGATOR'},
columns={'value': tick_value, 'processing_time': 63352.4},
at=TimestampNanos.now()
)
# This flush is much faster than LMDB's sync commit
sender.flush() # << This should be << 63ms
```
### **Bottom Line**
QuestDB should easily beat 63ms for commits, even with full durability enabled. The architecture is fundamentally different though:
- If you need guaranteed durability per operation: Use WAL tables with sync mode
- If you can batch: You'll see microsecond-level latencies
- Default mode (OS-managed durability): Sub-millisecond commits
The 63ms you're seeing with LMDB is likely a Windows fsync issue. QuestDB's memory-mapped architecture and optimized commit strategy should perform significantly better, especially if you can adapt your code to work with batches rather than individual sync operations.
Would you like help profiling what's causing the 63ms delay in LMDB or guidance on setting up QuestDB for your specific aggregator use case?
---
Powered by [Claude Exporter](https://www.claudexporter.com)
Reference in New Issue Copy Permalink