Roundtable – Estimating survey results in seconds

Hacker News

Share card

Roundtable – Estimating survey results in seconds

Recent academic work ([1], [2]) has suggested that LLMs can effectively simulate different Internet subpopulations. For example, you may ask ChatGPT to emulate being a high school teacher explaining Newton’s laws of physics. Building upon this, we created Roundtable, a platform that uses LLMs to predict how people will respond to any arbitrary survey question. To do so, we needed to first reduce bias arising from GPT’s training procedure. Because these models are primarily trained on Internet data, they can be heavily skewed towards the demographics of heavy Internet users (e.g., high-income, male). We addressed this by fine-tuning GPT on the GSS (General Social Survey) to ‘de-bias’ the model into emulating a more representative U.S. population. We allow users to ask any multiple-choice question and add conditioning questions and/or descriptions of their target population. Here are some examples: Simulation 1 (General Interest) Are you interested in buying an e-bike? Yes 28%, No 72% ([3]) Are you interested in buying an e-bike? conditioned on "Yes" to "Do you own a Tesla car?" Yes 40%, No 60% ([4]) Simulation 2 (reproducing the Stack Overflow Developer Survey; [5]) Where did you learn to code? conditioned on "Yes" to "Are you 45 years or older?" Books 55%, Online 45% ([6]) Where did you learn to code? conditioned on "No" to "Are you 45 years or older?" Books 26%, Online 74% ([7]) Simulation 3 (USA vs. Stack Overflow Developers vs. Hacker News Users) Do you code? Yes 24%, No 76% ([8]; USA) Do you code? Yes >99%, No 0% ([9]; Stack Overflow Developers) Do you code? Yes 83%, No 17% ([10]; Hacker News Users) — Of course, a natural question is whether we can trust these results. If you click ‘Investigate Results’, we report the most similar (in terms of cosine distance between LLM embeddings) GSS questions as a way of estimating how much extrapolation / interpolation is going on. This doesn’t quite address the accuracy of the subpopulations / conditioning questions (we are working on this), but we thought we are at a sufficiently advanced point to share what we’ve built with you all. Feedback would be greatly appreciated. ---- [1] https://arxiv.org/pdf/2209.06899.pdf [2] https://openreview.net/pdf?id=eYlLlvzngu [3] https://roundtable.ai/sandbox/e02e92a9ad20fdd517182788f4ae7e... [4] https://roundtable.ai/sandbox/6b4bf8740ad1945b08c0bf584c84c1... [5] https://survey.stackoverflow.co/2023/ [6] https://roundtable.ai/sandbox/d701556248385d05ce5d26ce7fc776... [7] https://roundtable.ai/sandbox/8bd80babad042cf60d500ca28c40f7... [8] https://roundtable.ai/sandbox/4a9d2fd6025459bd73b7798a8b2fdc... [9] https://roundtable.ai/sandbox/7e41ed16c01de48247bce02700c398... [10] https://roundtable.ai/sandbox/13aaa142e87337201601fb4b76d125...