Neosync – Open Source Data Replication and Anonymization

Hacker News

Hey HN, we're Evis and Nick, we're excited to be launching Neosync on HN! Neosync is an open source data replication and anonymization project that helps developers create safe, anonymized test data and sync it across all environments for high-quality local, stage and CI testing. This is how it works: 1. You select a job type. Today we support data sync jobs (these sync data between two databases and run on a schedule you define) and a data generation job (this generates synthetic data from scratch and sends it to a destination). 2. Next you define your source database and a destination(s) database (you can connect multiple destinations). 3. Next we pull in the schema from the source DB and then you can decide how to you to want transform your data. We ship with 40+ transformers (email, first name, address, random int64, random string, random float64, etc). You can create your own custom transformations as well. We've designed our transformers to be as flexible as possible so you can use them across almost any data type. You could also use Neosync in passthrough mode which means that none of the data will be transformed and you can use it for data replication. 4. Lastly, you can defined subsets. This is a way to filter the data that gets sent to the destination. You can provide a custom SQL query or filters to do this. For example, you can filter the data by an id, customerType, column, date, etc. This is very flexible. And that's it! The job will run on the schedule you determine. We handle things like retries and backoffs and referential integrity between tables. We also ship with APIs, a CLI and Github action so that you can use Neosync to hydrate a CI database in your CI pipeline. As a developer, you can use the CLI to hydrate a local database from S3 or another source. We're working on releasing a Terraform provider shortly. Deployment is pretty straightforward. You can deploy Neosync using Docker Compose (we provide a script) or on Kubernetes using our helm chart. So what's next? Here's a brief overview: Real time mode (hook up Neosync to Kafka/SQS and anonymize and send the data to destinations in real time) and more connections (MongoDB, Snowflake, CSV). On the ML side, supporting use-cases like consistent data generation (providing a seed value), statistically consistent data and more. You can check out our roadmap in our Github project. Here's a brief loom demo: https://www.loom.com/share/66224a81074a464b92808ecdc82b7ddb?... We'd love your feedback and contributions. We strongly believe that your data should be yours and it should stay on your infrastructure and open source is the best way to bring that vision to life.

Visit product View on Hacker NewsDec 8, 2023

Share card

Actual performance

4points

1comments

Did not reach leaderboard

Launch Intel predictions

Analyze your own launch →

Product HuntOn track for Day 1 leaderboard · Strong signals: dock, email, using · Missing: mac, agents, macos

89%89% predicted probability of success on Product Hunt, based on ML models trained on real launch data.

best fitHighest predicted score across all platforms for this description.

Indie HackersFits the IH revenue-focused audience · Missing: supports, reddit linkedin, podcasting

78%78% predicted probability of success on Indie Hackers, based on ML models trained on real launch data.

Hacker NewsStrong engagement from HN community · Strong signals: excited, open source, ide · Missing: https docs, just released, exist

73%73% predicted probability of success on Hacker News, based on ML models trained on real launch data.

nativeThis product was originally launched on this platform.

TrustMRRLess likely to generate early MRR · Strong signals: way · Missing: mobile apps, ios, personal

38%38% predicted probability of success on TrustMRR, based on ML models trained on real launch data.

AppSumoMay struggle as an AppSumo deal · Missing: plus, platform, intuitive

21%21% predicted probability of success on AppSumo, based on ML models trained on real launch data.

Acquire.comPre-revenue stage for this audience · Missing: arr, mrr, revenue

14%14% predicted probability of success on Acquire.com, based on ML models trained on real launch data.

BetaListMay not resonate with beta-testers · Strong signals: real time · Missing: web3, chat, crypto

0%0% predicted probability of success on BetaList, based on ML models trained on real launch data.

Incorrect prediction on native model

Similar products

SirixDB – Storing and Querying of Temporal Data (Java and Open Source)62%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

SirixDB – Storing and Querying of Temporal Data (Java and Open Source)

Hacker News13

Streamdal – an open-source tail -f for your data81%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Streamdal – an open-source tail -f for your data

Hacker News148

Open-Source Data Replication and Anonymization73%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Open-Source Data Replication and Anonymization

Hacker News24

Multiwoven – Open-Source Data Activation Platform – Ruby on Rails76%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Multiwoven – Open-Source Data Activation Platform – Ruby on Rails

Hacker News1

Open-source Data Anonymization tool nxs-data-anonymizer62%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Open-source Data Anonymization tool nxs-data-anonymizer

Hacker News4

Open-source tool to find PII across data silos64%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Open-source tool to find PII across data silos

Hacker News3

Desbordante – an open-source data profiling tool63%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Desbordante – an open-source data profiling tool

Hacker News2

I made an open source tool for transfering data63%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

I made an open source tool for transfering data

Hacker News2

Dejavu 1.0 – An open-source data browser for Elasticsearch80%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Dejavu 1.0 – An open-source data browser for Elasticsearch

Hacker News9

Open-source API for IACS vessel classification data62%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Open-source API for IACS vessel classification data

Hacker News4

Browse all products Analyze your launch