I scraped 25M Shopify products to build a search engine

Hacker News

Hi HN! I built Agora as a side-project leading up to the holiday season. I wanted to find an easier way to find Christmas gifts, without needing to go store-by-store. My wife asked me for a a pair of red shoes for Christmas. I quickly typed it into Google and found a combination of ads from large retailers and links to a 1948 movie called 'Red Shoes'. I decided to build Agora to solve my own problem (and stay happily married). The product is a search engine that automatically crawls thousands of Shopify stores and makes them easily accessible with a search interface. There's a few additional features to enhance the buying experience including saving products, filters, reviews, and popular products. I've started with exclusively Shopify stores and plan to expand the crawler to other e-commerce platforms like BigCommerce, WooCommerce, Wix, etc. The technical challenge I've found is keeping the search speed and performance strong as the data set becomes larger. There's about 25 million products on Agora right now. I'll ramp this up carefully to make sure we don't compromise the search speed and user experience. I'd love any feedback!

Visit product View on Hacker NewsDec 13, 2023

Share card

Actual performance

317points

268comments

Made the leaderboard

Launch Intel predictions

Analyze your own launch →

Indie HackersFits the IH revenue-focused audience · Strong signals: started, wife, including · Missing: supports, reddit linkedin, podcasting

93%93% predicted probability of success on Indie Hackers, based on ML models trained on real launch data.

best fitHighest predicted score across all platforms for this description.

Product HuntOn track for Day 1 leaderboard · Strong signals: google, user · Missing: mac, agents, macos

78%78% predicted probability of success on Product Hunt, based on ML models trained on real launch data.

Hacker NewsStrong engagement from HN community · Strong signals: ide, io, including · Missing: https docs, excited, just released

59%59% predicted probability of success on Hacker News, based on ML models trained on real launch data.

nativeThis product was originally launched on this platform.

AppSumoMay struggle as an AppSumo deal · Strong signals: platform, reviews, exclusive · Missing: plus, intuitive, host

45%45% predicted probability of success on AppSumo, based on ML models trained on real launch data.

TrustMRRLess likely to generate early MRR · Strong signals: google, way · Missing: mobile apps, ios, personal

43%43% predicted probability of success on TrustMRR, based on ML models trained on real launch data.

Acquire.comPre-revenue stage for this audience · Strong signals: arr, shopify · Missing: mrr, revenue, profit

14%14% predicted probability of success on Acquire.com, based on ML models trained on real launch data.

BetaListMay not resonate with beta-testers · Missing: web3, chat, crypto

0%0% predicted probability of success on BetaList, based on ML models trained on real launch data.

Correct prediction on native model

Similar products

I scraped 200M Shopify products to build a search engine62%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

I scraped 200M Shopify products to build a search engine

Hacker News23

AISLER – Search Engine for electronic circuits53%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

AISLER – Search Engine for electronic circuits

Hacker News9

Threadbase – A T-Shirt Search Engine60%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Threadbase – A T-Shirt Search Engine

Hacker News4

PressPanda, a journalist search engine60%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

PressPanda, a journalist search engine

Hacker News6

Search engine for standalone lectures56%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Search engine for standalone lectures

Hacker News7

A reactive search engine45%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

A reactive search engine

Hacker News10

Vineyard - A search engine for Vine60%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Vineyard - A search engine for Vine

Hacker News3

Sentiment Search Engine65%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Sentiment Search Engine

Hacker News12

We made a search engine60%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

We made a search engine

Hacker News3

Healthforum – Drug Side Effect Search Engine51%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Healthforum – Drug Side Effect Search Engine

Hacker News2

Browse all products Analyze your launch