I made an app to help my gf cook experimentally

Hacker News

Share card

I made an app to help my gf cook experimentally

I like to cook, my gf does not - so usually I do the cooking. But sometimes, she wants to, except then we often have fairly random ingredients at home. She is someone who needs to follow a recipe, so I made an app where you can take a photo of the ingredients you have and chatGPT will come up with a reasonable looking recipe. I did also want to learn LLM integrations, so there was an ulterior motive ;) You can try it out for yourself, but you need to create an account (currently the requests to openAI cost me money). Stack is React + material UI, talking to FastAPI with the openAI python package as main LLM interface, all hosted on digitalocean. I did check out langchain, may decide to integrate that in the future. I am somewhat disappointed by the openAI API, you can supposedly specify JSON responses only by setting the relevant request parameter, but that does not always work(!) Additionally, the API will seemingly randomly decide to reject requests, both GPT-Vision and GPT3.5-turbo (and GPT4), for content violations. And these violations don't always set the relevant field in the response object, sometimes the response looks fine but the message string is some variation of "error: contentViolation" (never consistent). I tested this many times using the same exact inputs, both image and text. The API will definitely complain if the ingredients are too diverse, e.g. soy sauce and apples will usually raise a content violation. I am setting up prompt context (you are PantryBot, an AI recipe generator... etc.) using sets of system prompts as specified in their API docs, but who knows. Probably the most reliable part of the whole system is the ingredient recognition - it usually even translates German or other language packaging into English descriptions. The usual shortcomings of LLM outputs apply to the recipes, YMMV. I did also develop automatic illustrations for recipes - both method step-by-step illustrations and one of the final dish. This was extremely unreliable both by generating content violations but also just completely unrelated images, so it is disabled for now. One day I may decide to self host an LLM with some amount of additional fine tuning or RAG or whatever. Finally - I delayed posting this because I originally had the donation link pointing to buymeacoffee, but they disabled my account for fraud (after I had already had some donations...) and will not reply to the appeal. So now I'm using ko-fi, let's see how long that lasts. Looking forward to your feedback!