One of our products, Hermes, is an audit logs service. Currently in its prototype phase, Hermes uses a Go REST API server to ingest audit logs and send them to Loki.
We were trying out different databases, ingesters, and tools to see which are best suited for Hermes. The goal was to find a solution that could scale with high traffic without losing a single audit log and efficiently search through large amounts of data.
We decided to benchmark various combinations of ingesters (Vector, Fluentd, Fluent-Bit, etc.) and storage & query tools (MongoDB, ClickHouse, Elasticsearch, etc.).
The first round of benchmarks would be lightweight, with more extensive tests to follow once we picked the right tools for Hermes.
Hardware Configuration
The following tests and benchmarks were performed on a MacBook Pro (14-inch, 2021) with an Apple M1 Pro and 16 GB RAM. The tools being tested were dockerized with Docker Desktop running with 4 GB Memory, 4 CPUs, & 1 GB Swap.
Fluent Bit is a super-fast, lightweight, highly scalable logging and metrics processor and forwarder. It is the preferred choice for cloud and containerized environments. Source: fluent-bit website Clickhouse is the fastest OLAP database on earth. ClickHouse works 100–1000x faster than traditional approaches. Companies like Uber, Cloudflare, Spotify, and eBay use Clickhouse. Source: Clickhouse website
So few pointers before we go ahead,
- Fluent-bit is fast at ingesting logs/data, processing them, and sending them to a destination.
- ClickHouse is efficient at handling and querying data.
- Fluent-bit does not support ClickHouse by default.
- The Fluent-bit ecosystem allows users to write their plugins in Golang to add additional required support.
- For faster querying in ClickHouse, an efficient table schema with indexes, compression, etc., should be established.
ClickHouse Plugin for Fluent-bit
I developed a Fluent-bit output plugin for ClickHouse.
Fluent-Bit config
This config makes Fluent-bit ingest data via an HTTP server listening on port 8888 and sends the data to ClickHouse with the stated configuration.
ClickHouse config
I ramped up the number of concurrent requests/queries by modifying the config.xml. After multiple tests, I finalized the following config:
Load Testing Tool
I developed a load testing tool with Node.js that can be used to benchmark REST API-based endpoints of Fluent-bit.
Another tool to load test is the querying part of Clickhouse.
The Results
These results are dependent on the RAM allocated to the Docker engine, which in my case was 4 GiBs.
Ingester
Query
Conclusion
Ingester
- Fluent-bit can handle loads up to 2000 req/sec, but in the case of bigger batches, the speed decreases drastically (e.g., 200 X 10 & 300 X 10).
- For long-term, light batches, Fluent-bit performs consistently (e.g., 10 X 1000).
- Fluent-bit performs at average speeds with average loads (e.g., 50 X 50).
Query
- ClickHouse shows the best req/sec performance with an average load (50 X 50).
- ClickHouse's performance was quite satisfactory for all the different variations of records in the DB (1.1 mils, 50k, 25k, 10k, 2k, & 1k).
- ClickHouse was able to manage short-term high loads and long-term light loads efficiently (100 X 10 and 10 X 5000).
We will be posting more blogs regarding benchmarks, tools, etc., as we continue to build Hermes and many other dev tools. Please leave comments below.
Thank you!
