Data Catering – Data generation and testing tool

Hacker News

Hi everyone, I'm Peter. A Software Engineer who has been around the data space for the majority of my career. Data Catering is a tool I've been itching to try and implement as I've spent lots of time debugging data issues, generating manual data to test jobs/services or replicating production-like data flows, all driven by metadata. The usual way to solve this problem is to stream/load a subset of production data, mask it and store it in a pre-prod environment. This is okay but can increase the risk of personal data leaking as pre-prod environments may not have the same security measures in place (Optus data leak as an example https://en.wikipedia.org/wiki/2022_Optus_data_breach ), having network connections between prod and non-prod being opened, and it may not cover all data scenarios (different permutations/combinations of values). Another approach is to generate data yourself. There are a number of existing tools that can aid with data generation but lack the ability to clean up the generated data, generate both batch and event data, define relationships between datasets (for example, an account can have multiple transactions associated with it, or foreach account create event, the same account should exist in a CSV file), or validating data. With this in mind and following along the ideas of having a tool that can be run anywhere, ability to automatically discover, generate and validate data, and giving users the ability to customise, Data Catering was created. Main features include: - Metadata discovery - Batch or event data generation - Maintain referential integrity across any dataset - Create custom data generation scenarios - Clean up generated data - Validate data - Suggest data validations In terms of technical details, it is a Spark based application where users can use the Java/Scala API or YAML files to interface with it. It is a single docker image that can be run following the quick start found here ( https://data.catering/get-started/docker/ ). I believe there is room for improvement in the data testing area as we should look to be more proactive in detecting and resolving bugs related to data quality. Any feedback is appreciated!

Visit product View on Hacker NewsOct 9, 2023

Share card

Actual performance

1points

Did not reach leaderboard

Launch Intel predictions

Analyze your own launch →

Indie HackersFits the IH revenue-focused audience · Strong signals: created, started, ios · Missing: supports, reddit linkedin, podcasting

91%91% predicted probability of success on Indie Hackers, based on ML models trained on real launch data.

best fitHighest predicted score across all platforms for this description.

Product HuntOn track for Day 1 leaderboard · Strong signals: user, dock, single · Missing: mac, agents, macos

63%63% predicted probability of success on Product Hunt, based on ML models trained on real launch data.

Hacker NewsStrong engagement from HN community · Strong signals: exist, existing, ide · Missing: https docs, excited, just released

57%57% predicted probability of success on Hacker News, based on ML models trained on real launch data.

nativeThis product was originally launched on this platform.

AppSumoMay struggle as an AppSumo deal · Strong signals: interface, users · Missing: plus, platform, intuitive

41%41% predicted probability of success on AppSumo, based on ML models trained on real launch data.

TrustMRRLess likely to generate early MRR · Strong signals: ios, personal, users · Missing: mobile apps, entrepreneurs, apps

37%37% predicted probability of success on TrustMRR, based on ML models trained on real launch data.

Acquire.comPre-revenue stage for this audience · Strong signals: active · Missing: arr, mrr, revenue

10%10% predicted probability of success on Acquire.com, based on ML models trained on real launch data.

BetaListMay not resonate with beta-testers · Missing: web3, chat, crypto

0%0% predicted probability of success on BetaList, based on ML models trained on real launch data.

Incorrect prediction on native model

Similar products

Data Caterer – Data testing tool for any data source56%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Data Caterer – Data testing tool for any data source

Hacker News2

Yadget Synthetic Data Generation for Testing42%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Yadget Synthetic Data Generation for Testing

Hacker News2

I made a data agent for hypothesis testing25%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

I made a data agent for hypothesis testing

Hacker News3

Tool to sanitize data from Java heap dumps37%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Tool to sanitize data from Java heap dumps

Hacker News2

Data Caterer – Data generation and validation tool29%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Data Caterer – Data generation and validation tool

Hacker News1

Procedural world generation example: Prague with elevation data data65%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Procedural world generation example: Prague with elevation data data

Hacker News7

Fixture Generation Tool for Go48%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Fixture Generation Tool for Go

Hacker News1

GraphQL Centaur – GraphQL Generation tool for MongoDB Stucco72%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

GraphQL Centaur – GraphQL Generation tool for MongoDB Stucco

Hacker News28

Favicon Generation Tool48%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Favicon Generation Tool

Hacker News3

JourneoAI47%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Itinerary generation tool

Indie Hackers1$100/moai

Browse all products Analyze your launch