Capillaries: Distributed data processing with Go and Cassandra

Hacker News

I started thinking about this approach after working on a large-scale project for a major financial company where our group developed a distributed in-house data processing solution. On a regular basis, it ingested a few gigabytes of financial data and, within a tight SLA time limit, produced a lot of enriched/aggregated/validated data for a number of customers. Sometimes, source data had errors, so operators with domain knowledge had to verify data validity at some checkpoints, immediately make corrections, and re-run parts of the workflow manually. The solution involved complex web service orchestration, custom database and was very demanding on the infrastructure availability. Capillaries is a built from scratch, open-source Go solution that does just that: ingests data and applies user-defined transforms - Go one-liner expressions, Python formulas, joins, aggregations, denormalization - using Cassandra for intermediate data storage and RabbitMQ for task scheduling. End users just have to provide: - source data in CSV files; - Capillaries script (JSON file) that defines the workflow and the transforms; - Python code that performs complex calculations (only if needed). The whole data processing pipeline can be split into separate runs that can be started independently and re-run by the user if needed. The goal is to build a platform that is tolerant to database and processing node failures, and allows users to focus on data transform logic and data quality control. “Getting started” Docker-based demo calculates ARK funds performance, using EOD holdings and transactions data acquired from public sources. There are also integration tests that use non-financial data. There is a test deploy tool that uses Openstack API for provisioning in the cloud.

Visit product View on Hacker NewsMay 16, 2023

Share card

Actual performance

70points

25comments

Made the leaderboard

Launch Intel predictions

Analyze your own launch →

Indie HackersFits the IH revenue-focused audience · Strong signals: started, para · Missing: supports, reddit linkedin, podcasting

89%89% predicted probability of success on Indie Hackers, based on ML models trained on real launch data.

best fitHighest predicted score across all platforms for this description.

Product HuntOn track for Day 1 leaderboard · Strong signals: user, dock, using · Missing: mac, agents, macos

87%87% predicted probability of success on Product Hunt, based on ML models trained on real launch data.

Hacker NewsStrong engagement from HN community · Strong signals: ide, pipe, io · Missing: https docs, excited, just released

68%68% predicted probability of success on Hacker News, based on ML models trained on real launch data.

nativeThis product was originally launched on this platform.

TrustMRRFits verified-revenue profile · Strong signals: users, para · Missing: mobile apps, ios, personal

61%61% predicted probability of success on TrustMRR, based on ML models trained on real launch data.

AppSumoMay struggle as an AppSumo deal · Strong signals: platform, users · Missing: plus, intuitive, reviews

31%31% predicted probability of success on AppSumo, based on ML models trained on real launch data.

Acquire.comPre-revenue stage for this audience · Missing: arr, mrr, revenue

12%12% predicted probability of success on Acquire.com, based on ML models trained on real launch data.

BetaListMay not resonate with beta-testers · Missing: web3, chat, crypto

0%0% predicted probability of success on BetaList, based on ML models trained on real launch data.

Correct prediction on native model

Similar products

TaskPipes – IFTTT for data processing67%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

TaskPipes – IFTTT for data processing

Hacker News4

IFTTT for data processing67%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

IFTTT for data processing

Hacker News10

Spade – UI for Data Processing70%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Spade – UI for Data Processing

Hacker News6

bsv – maximum performance data processing56%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

bsv – maximum performance data processing

Hacker News7

Pg_incremental – Incremental Data Processing in Postgres80%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Pg_incremental – Incremental Data Processing in Postgres

Hacker News6

BoomFilters – Probabilistic data structures for processing streams69%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

BoomFilters – Probabilistic data structures for processing streams

Hacker News70

Rasteret – 8x Faster Earth Observation Data Processing66%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Rasteret – 8x Faster Earth Observation Data Processing

Hacker News2

Que – Distributed Job Processing in Elixir with Mnesia82%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Que – Distributed Job Processing in Elixir with Mnesia

Hacker News12

Parseflow.io56%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

An automated data processing platform

Product Hunt+126

A Python toolkit for processing tabular data65%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

A Python toolkit for processing tabular data

Hacker News17

Browse all products Analyze your launch