Denormalized – Embeddable Stream Processing in Rust and DataFusion

Hacker News

tl;dr we built an embeddable stream processing engine in Rust using apache DataFusion, check us out at https://github.com/probably-nothing-labs/denormalized Hey HN, We’d like to showcase a very early version of our embeddable stream processing engine called Denormalized. The rise of DuckDB has abundantly made it clear that even for many workloads of Terabyte scale, a single node system outshines the distributed query engines of previous generation such as Spark, Snowflake etc in terms of both performance and cost. Now a lot of workloads DuckDB is used for were normally considered to be “big data” in the previous generation, but no more. In the context of streaming especially, this problem is more acute. A streaming system is designed to incrementally process large amounts of data over a period of time. Even on the upper end of scale, productionized use-cases of stream processing are rarely performing compute on more than tens of gigabytes of data at a given time. Even so, the standard stream processing solutions such as Flink involve spinning up a distributed JVM cluster to even compute against the simplest of event streams. To that end, we’re building Denormalized designed to be embeddable in your applications and scale up to hundreds of thousands of events per second with a Flink-like dataflow API. While we currently only support Rust, we have plans for Python and Typescript bindings soon. We’re built atop DataFusion and the Arrow ecosystems and currently support streaming joins as well as windowed aggregations on Kafka topics. Please check out out repo at: https://github.com/probably-nothing-labs/denormalized We’d love to hear your feedback.

Visit product View on Hacker NewsAug 15, 2024

Share card

Actual performance

125points

31comments

Made the leaderboard

Launch Intel predictions

Analyze your own launch →

Indie HackersFits the IH revenue-focused audience · Missing: supports, reddit linkedin, podcasting

77%77% predicted probability of success on Indie Hackers, based on ML models trained on real launch data.

best fitHighest predicted score across all platforms for this description.

Product HuntOn track for Day 1 leaderboard · Strong signals: context, single, using · Missing: mac, agents, macos

72%72% predicted probability of success on Product Hunt, based on ML models trained on real launch data.

Hacker NewsStrong engagement from HN community · Strong signals: ide, io · Missing: https docs, excited, just released

57%57% predicted probability of success on Hacker News, based on ML models trained on real launch data.

nativeThis product was originally launched on this platform.

AppSumoMay struggle as an AppSumo deal · Strong signals: soon · Missing: plus, platform, intuitive

46%46% predicted probability of success on AppSumo, based on ML models trained on real launch data.

TrustMRRLess likely to generate early MRR · Missing: mobile apps, ios, personal

31%31% predicted probability of success on TrustMRR, based on ML models trained on real launch data.

Acquire.comPre-revenue stage for this audience · Strong signals: arr · Missing: mrr, revenue, profit

14%14% predicted probability of success on Acquire.com, based on ML models trained on real launch data.

BetaListMay not resonate with beta-testers · Missing: web3, chat, crypto

0%0% predicted probability of success on BetaList, based on ML models trained on real launch data.

Correct prediction on native model

Similar products

Bytewax, JVM free stream processing Py+Rust51%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Bytewax, JVM free stream processing Py+Rust

Hacker News4

Fluvio – Distributed stream processing system written in Rust and WASM76%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Fluvio – Distributed stream processing system written in Rust and WASM

Hacker News58

Rust Stream API visualized and exposed53%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Rust Stream API visualized and exposed

Hacker News3

We built rescaling for distributed stateful stream processing69%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

We built rescaling for distributed stateful stream processing

Hacker News5

Jet – in-memory, fault-tolerant, distributed stream processing64%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Jet – in-memory, fault-tolerant, distributed stream processing

Hacker News129

Benthos, stream multiplexer and processing service60%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Benthos, stream multiplexer and processing service

Hacker News1

Upstream – Stream processing, for humans65%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Upstream – Stream processing, for humans

Hacker News5

Minibatch – Python stream processing61%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Minibatch – Python stream processing

Hacker News50

Bytewax – Python Stateful Stream Processing on Timely Dataflow61%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Bytewax – Python Stateful Stream Processing on Timely Dataflow

Hacker News21

Python Stream Processing with Denormalized83%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Python Stream Processing with Denormalized

Hacker News20

Browse all products Analyze your launch