One of our products, Hermes is an audit logs service. Currently, Hermes is in the prototype phase and uses a Go REST API server to ingest audit logs and send them to Loki.
How Apple broke "Sign in with Apple" with an unannounced and silent redirect
One of our products, Hermes, is an audit logs service. Currently in its prototype phase, Hermes uses a Go REST API server to ingest audit logs and send them to Loki.
We were trying out different databases, ingesters, and tools to see which are best suited for Hermes. The goal was to find a solution that could scale with high traffic without losing a single audit log and efficiently search through large amounts of data.
We decided to benchmark various combinations of ingesters (Vector, Fluentd, Fluent-Bit, etc.) and storage & query tools (MongoDB, ClickHouse, Elasticsearch, etc.).
The first round of benchmarks would be lightweight, with more extensive tests to follow once we picked the right tools for Hermes.
The following tests and benchmarks were performed on a MacBook Pro (14-inch, 2021) with an Apple M1 Pro and 16 GB RAM. The tools being tested were dockerized with Docker Desktop running with 4 GB Memory, 4 CPUs, & 1 GB Swap.
Fluent Bit is a super-fast, lightweight, highly scalable logging and metrics processor and forwarder. It is the preferred choice for cloud and containerized environments. Source: fluent-bit website Clickhouse is the fastest OLAP database on earth. ClickHouse works 100–1000x faster than traditional approaches. Companies like Uber, Cloudflare, Spotify, and eBay use Clickhouse. Source: Clickhouse website
So few pointers before we go ahead,
I developed a Fluent-bit output plugin for ClickHouse.
This config makes Fluent-bit ingest data via an HTTP server listening on port 8888 and sends the data to ClickHouse with the stated configuration.
I ramped up the number of concurrent requests/queries by modifying the config.xml. After multiple tests, I finalized the following config:
I developed a load testing tool with Node.js that can be used to benchmark REST API-based endpoints of Fluent-bit.
Another tool to load test is the querying part of Clickhouse.
These results are dependent on the RAM allocated to the Docker engine, which in my case was 4 GiBs.
We will be posting more blogs regarding benchmarks, tools, etc., as we continue to build Hermes and many other dev tools. Please leave comments below.
Thank you!
