FiddleCube – Generate Q&A to test your LLM

Hacker News

Convert your vector embeddings into a set of questions and their ideal responses. Use this dataset to test your LLM and catch failures caused by prompt or RAG updates. Get started in 3 lines of code: ``` pip3 install fiddlecube ``` ``` from fiddlecube import FiddleCube fc = FiddleCube(api_key="<api-key>") dataset = fc.generate( [ "The cat did not want to be petted.", "The cat was not happy with the owner's behavior.", ], 10, ) dataset ``` Generate your API key: https://dashboard.fiddlecube.ai/api-key # Ideal QnA datasets for testing, eval and training LLMs Testing, evaluation or training LLMs requires an ideal QnA dataset aka the golden dataset. This dataset needs to be diverse, covering a wide range of queries with accurate responses. Creating such a dataset takes significant manual effort. As the prompt or RAG contexts are updated, which is nearly all the time for early applications, the dataset needs to be updated to match. # FiddleCube generates ideal QnA from vector embeddings - The questions cover the entire RAG knowledge corpus. - Complex reasoning, safety alignment and 5 other question types are generated. - Filtered for correctness, context relevance and style. - Auto-updated with prompt and RAG updates.

Visit product View on Hacker NewsJun 25, 2024

Share card

Actual performance

78points

18comments

Made the leaderboard

Launch Intel predictions

Analyze your own launch →

Indie HackersFits the IH revenue-focused audience · Strong signals: started · Missing: supports, reddit linkedin, podcasting

73%73% predicted probability of success on Indie Hackers, based on ML models trained on real launch data.

best fitHighest predicted score across all platforms for this description.

Product HuntOn track for Day 1 leaderboard · Strong signals: context, code · Missing: mac, agents, macos

72%72% predicted probability of success on Product Hunt, based on ML models trained on real launch data.

TrustMRRLess likely to generate early MRR · Missing: mobile apps, ios, personal

48%48% predicted probability of success on TrustMRR, based on ML models trained on real launch data.

Hacker NewsMay not resonate with HN audience · Strong signals: lua, ide, io · Missing: https docs, excited, just released

35%35% predicted probability of success on Hacker News, based on ML models trained on real launch data.

nativeThis product was originally launched on this platform.

AppSumoMay struggle as an AppSumo deal · Missing: plus, platform, intuitive

33%33% predicted probability of success on AppSumo, based on ML models trained on real launch data.

Acquire.comPre-revenue stage for this audience · Strong signals: training · Missing: arr, mrr, revenue

17%17% predicted probability of success on Acquire.com, based on ML models trained on real launch data.

BetaListMay not resonate with beta-testers · Missing: web3, chat, crypto

1%1% predicted probability of success on BetaList, based on ML models trained on real launch data.

Incorrect prediction on native model

Similar products

Redis-LLM – Redis module integrates LLM with Redis45%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Redis-LLM – Redis module integrates LLM with Redis

Hacker News2

LitLLM the Spiciest LLM Wrapper34%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

LitLLM the Spiciest LLM Wrapper

Hacker News1

LLM Reasonsers46%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

LLM Reasonsers

Hacker News2

Resilient LLM23%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Resilient LLM

Hacker News1

Hegelion – Force your LLM to argue with itself before answering58%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Hegelion – Force your LLM to argue with itself before answering

Hacker News1

I Stopped Hoping My LLM Would Cooperate49%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

I Stopped Hoping My LLM Would Cooperate

Hacker News3

Module for LLM Homeostasis (PoC)22%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Module for LLM Homeostasis (PoC)

Hacker News1

Doom Compiled into an LLM50%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Doom Compiled into an LLM

Hacker News2

The Smallest LLM46%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

The Smallest LLM

Hacker News3

LLM Hotkey39%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

TrustMRROther

Browse all products Analyze your launch