Kaflow Search – A desktop tool for locally indexed Kafka message search

Hacker News

Share card

Kaflow Search – A desktop tool for locally indexed Kafka message search

For a long time, I’ve used Kafka for services for some purpose. like tracing user action log, event sourcing propagation pipeline step processing and so on. Whenever I use Kafka, I need to set up kafka ecosystem such as, ELK ( Elasticsearch + Logstash + Kibana) for search Kafka messages for service operation, issue trace. and Kafka sink connector for some data forward other persistent DB, and ksqlDB( or Flink ) for count and monitor event stream. With additional infra and software One day, a thought suddenly crossed my mind. That is “Basically Kafka is processing message stream sequentially, with a consumer. It means If I don’t know what the partition and offset of “needle” is, I have to find needle sequentially from beginning. but if I know Kafka message partition and offset, Can I find and access data directly like other DBMS?” ELK is not necessary on my service at that time. Just I need to search trace events and it is in Kafka topic message. All I need to know is where the data I want to find is My desktop and your desktop are too powerful, expensive to use only for web browser operation and AI question agent. Let’s use local disk and local application for indexing storage and search engine about Kafka message. This local-first concept is implemented as “Kaflow Search desktop tool” If you don’t need to keep data for a long time or permanently You don’t have to go through the hassle of installing and managing ELK or other Kafka ecosystem infrastructure Just keep and sync index with Kafka cluster server And find messages you want easily with just this tool installed It will save your operation time. When operating company services, The costs of installing server, managing additional ELK/Kafka Connect, ksqlDB instances didn’t come out of my own pocket so I felt free to spend my time and server resources to set additional data pipeline up. but if it is my own service that I afford, I can’t help saving my server cost. I can’t afford the additional infra costs from cloud service (like AWS) I’m not a rich developer, who just has a laptop to develop software. I’ll make the most of my laptop resources, until my service generates enough money, and then I can handle additional server costs. I’ll only use Kafka and this tool on my service, before I really need ELK and other Kafka ecosystem. I dedicate this rich client Kafka tool to developers who, like me, are poor. GitHub : https://github.com/whsoul/kaflow-search <If you have kafka cluster> => Download From release assets ( macOS, Windows ) - https://github.com/whsoul/kaflow-search/releases/tag/v0.1.3 From brew ( macOS ) - brew install --cask whsoul/tap/kaflow-search <If you don’t have kafka cluster> => Download demo From release assets ( macOS, Windows ) - https://github.com/whsoul/kaflow-search/releases/tag/demo-v0... From brew ( macOS ) - brew install --cask whsoul/tap/kaflow-search-demo Please try my tool ( on your macOS, Windows ) and feedback for me consumer ID, consumer Group do not remain on your cluster. It does only read operation, not managing function ( such as delete, create topic, repartition cluster topic). It normally only reads once messages that you select, and read local-first index for search your message. So feel free to install this. But it is not developer signed.. (Sorry because I’m poor developer..) Please follow this install guide ( https://github.com/whsoul/kaflow-search#install ) about untrusted developer warning when install and first launch.