Mycelial – Point to Point Cloudless ETL (100% in Rust)

Hacker News

Share card

Mycelial – Point to Point Cloudless ETL (100% in Rust)

Hey HN! We’re Michael and Joseph from Mycelial (https://www.mycelial.com/). Mycelial is a point-to-point ETL solution, which means you can connect any db, file directory, or Excel workbook to a Mycelial daemon and move that data anywhere you’d like without having to transit through the Cloud. Historically, ETL processes have been bottlenecked by the need to route data through central Cloud services (Segment, Fivetran, Airbyte), introducing latency, security concerns, and unnecessary compute costs. Mycelial is our answer to these challenges, a SaaS control plane (you can run your own control plane if you’d prefer or need to air-gap) that orchestrates direct, point-to-point ETL pipelines, bypassing the need for central Cloud transit entirely. All you need is a Mycelial Daemon (`myceliald`) running, and you can remotely orchestrate ETL pipelines that move your data any which way you want. Highlights: • `myceliald` is 100% Rust and the binary is only ~20mb. • `myceliald` has a constant memory footprint - <30mb! - even under significant load. • `myceliald` has connectors for many of the key data services you’re already running, such as: • Postgres • SQLite • MySQL • Kafka • Snowflake • Amazon Redshift, and • File streaming • Mycelial is 100% OSS and Apache 2 licensed. You are welcome to run your own Control Plane if you’d like! (See https://docs.mycelial.com/) • Mycelial is “real-time” - as source data changes, `myceliald` will sync your data on the cadence you choose. No connection? `myceliald` keeps track of what data has already been synced to which destinations and will catch up when connection is restored. • Mycelial gives you Apache NiFi-like speeds and securely transfers data directly between daemons. This ensures data integrity and speed without the traditional Cloud overhead. Use the compute you’ve got with no extra fees! With Mycelial, you can achieve unparalleled data transfer efficiency, enhanced security, and significant cost savings, all while maintaining full control over your data. We're excited to share the Mycelial Beta with the Hacker News community and would love to hear your thoughts, feedback, and questions. Your insights are invaluable as we continue to refine and enhance Mycelial. Join us on our journey to redefine ETL. Visit Mycelial (https://mycelial.com/) to get started, or check out our docs (https://docs.mycelial.com/) to get started!