Open-Source Data Anonymization for Developers

Hacker News

Share card

Open-Source Data Anonymization for Developers

Hey HN, we're Evis and Nick from Neosync ( https://www.github.com/nucleuscloud/neosync ). Since we last introduced Neosync on HN 4 months ago, we’ve made a lot of progress and we’re excited to be launching several new features. As a reminder, Neosync is an open source platform that helps developers anonymize production data, generate synthetic data, subset it and sync it across their environments for better testing, debugging and developer experience. We do all of this while handling referential integrity. Whether you have primary keys, foreign keys, unique constraints, circular dependencies (within a table and across tables), sequences and more, Neosync preserves those references. Our goal is to give every developer production-like, representative data for a better developer experience without any security and privacy issues. First, we’ve added new integrations. In addition to supporting Postgres and Mysql, we’re introducing first class support for DynamoDB, MongoDB and SQL Server. You can also sync to object storage like S3 and GCP Cloud storage. Next, we’ve completely revamped our transformers. Transformers are how you anonymize sensitive data and generate new data. We’ve added new Transformers that you can use out of the box or you can write your own custom one in javascript. We’ve added real time validation and the ability to combine transformers together to create your own anonymization scheme. We’ve also added in new features to make Neosync easier to use. For example, the ability to automatically map transformers to your schema. The ability to only append new records instead of a full refresh. And to stop jobs from running when the schema changes. We've also upgrade our AI Synthetic Data features. You can use any LLM to generate synthetic data and Neosync will handle the orchestration between your database and the LLM. Lastly, we’re also announcing Neosync Cloud. Our hosted platform that allows you to use Neosync without having to run any of the infrastructure yourself. All you have to do is connect your source and destination databases(s), configure your schema and you’re done. Of course, you can use Neosync Open Source on-prem and hundreds of companies do. Neosync is written in Go and Typescript and can be started locally with a single make command. We'd love any feedback you have and contributions are always welcome.