I made an Open Source Personal Data Pipeline

Hacker News

Hi there! I've been working with data in one form or another, professionally, for about 5 years. I've been thinking about my own personal data and how it's used for at least twice that long. I've been sort of building something in my head for a while that solves my own problem and, in the beginning of this year, I found the opportunity to spend some time building it out. I'll leave the detailed explanation to the blog post but, in short, I built what amounts to an API crawler combined with a data processor to help you download your personal data from 3rd party services and work with it using relatively simple YAML recipes. It's using DuckDB under the hood, which is quite impressive at turning unstructured JSON into queryable form. The goal is making your personal cloud data available locally for backup, exploration, and combination. The tool works end-to-end for limited APIs and use cases right now. I'm working on solving a few of my own problems with it (events added to Obsidian daily notes, workout summaries across multiple trackers, mini-CRM linking events and emails to notes) and would be happy to help folks get this up and running. I have links to Discord and Substack (bottom of the post) if you want to lurk and see what happens and I welcome any and all contributions you are motivated to make! Thanks for checking it out!

Visit product View on Hacker NewsJul 31, 2024

Share card

Actual performance

5points

2comments

Did not reach leaderboard

Launch Intel predictions

Analyze your own launch →

Product HuntOn track for Day 1 leaderboard · Strong signals: email, using, notes · Missing: mac, agents, macos

88%88% predicted probability of success on Product Hunt, based on ML models trained on real launch data.

best fitHighest predicted score across all platforms for this description.

Indie HackersFits the IH revenue-focused audience · Missing: supports, reddit linkedin, podcasting

86%86% predicted probability of success on Indie Hackers, based on ML models trained on real launch data.

Hacker NewsStrong engagement from HN community · Strong signals: open source, pipe, io · Missing: https docs, excited, just released

74%74% predicted probability of success on Hacker News, based on ML models trained on real launch data.

nativeThis product was originally launched on this platform.

TrustMRRFits verified-revenue profile · Strong signals: personal · Missing: mobile apps, ios, entrepreneurs

54%54% predicted probability of success on TrustMRR, based on ML models trained on real launch data.

AppSumoMay struggle as an AppSumo deal · Missing: plus, platform, intuitive

32%32% predicted probability of success on AppSumo, based on ML models trained on real launch data.

Acquire.comPre-revenue stage for this audience · Missing: arr, mrr, revenue

14%14% predicted probability of success on Acquire.com, based on ML models trained on real launch data.

BetaListMay not resonate with beta-testers · Missing: web3, chat, crypto

0%0% predicted probability of success on BetaList, based on ML models trained on real launch data.

Incorrect prediction on native model

Similar products

I built an open-source data pipeline tool in Go64%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

I built an open-source data pipeline tool in Go

Hacker News200

TagSpaces – An open source personal data manager71%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

TagSpaces – An open source personal data manager

Hacker News361

SirixDB – Storing and Querying of Temporal Data (Java and Open Source)62%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

SirixDB – Storing and Querying of Temporal Data (Java and Open Source)

Hacker News13

Streamdal – an open-source tail -f for your data81%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Streamdal – an open-source tail -f for your data

Hacker News148

Open-Source Data Replication and Anonymization73%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Open-Source Data Replication and Anonymization

Hacker News24

Neosync – Open Source Data Replication and Anonymization73%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Neosync – Open Source Data Replication and Anonymization

Hacker News4

A poor man's data pipeline56%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

A poor man's data pipeline

Hacker News3

Goodreads Data Pipeline51%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Goodreads Data Pipeline

Hacker News213

Multiwoven – Open-Source Data Activation Platform – Ruby on Rails76%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Multiwoven – Open-Source Data Activation Platform – Ruby on Rails

Hacker News1

Open-source Data Anonymization tool nxs-data-anonymizer62%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Open-source Data Anonymization tool nxs-data-anonymizer

Hacker News4

Browse all products Analyze your launch