Crawlee – Web scraping and browser automation library for Node.js

Hacker News

Hey HN, This is Jan, founder of Apify, a web scraping and automation platform. Drawing on our team's years of experience, today we're launching Crawlee [1], the web scraping and browser automation library for Node.js that's designed for the fastest development and maximum reliability in production. For details, see the short video [2] or read the announcement blog post [3]. Main features: - Supports headless browsers with Playwright or Puppeteer - Supports raw HTTP crawling with Cheerio or JSDOM - Automated parallelization and scaling of crawlers for best performance - Avoids blocking using smart sessions, proxies, and browser fingerprints - Simple management and persistence of queues of URLs to crawl - Written completely in TypeScript for type safety and code autocompletion - Comprehensive documentation, code examples, and tutorials - Actively maintained and developed by Apify—we use it ourselves! - Lively community on Discord To get started, visit https://crawlee.dev or run the following command: npx crawlee create my-crawler If you have any questions or comments, our team will be happy to answer them here. [1] https://crawlee.dev/ [2] https://www.youtube.com/watch?v=g1Ll9OlFwEQ [3] https://blog.apify.com/announcing-crawlee-the-web-scraping-a...

Visit product View on Hacker NewsAug 23, 2022

Share card

Actual performance

282points

80comments

Made the leaderboard

Launch Intel predictions

Analyze your own launch →

Indie HackersFits the IH revenue-focused audience · Strong signals: supports, started, para · Missing: reddit linkedin, podcasting, created

90%90% predicted probability of success on Indie Hackers, based on ML models trained on real launch data.

best fitHighest predicted score across all platforms for this description.

Product HuntOn track for Day 1 leaderboard · Strong signals: using, code · Missing: mac, agents, macos

80%80% predicted probability of success on Product Hunt, based on ML models trained on real launch data.

Hacker NewsStrong engagement from HN community · Strong signals: ide, io · Missing: https docs, excited, just released

78%78% predicted probability of success on Hacker News, based on ML models trained on real launch data.

nativeThis product was originally launched on this platform.

AppSumoStrong fit for a featured deal · Strong signals: platform · Missing: plus, intuitive, reviews

58%58% predicted probability of success on AppSumo, based on ML models trained on real launch data.

TrustMRRLess likely to generate early MRR · Strong signals: video, para · Missing: mobile apps, ios, personal

46%46% predicted probability of success on TrustMRR, based on ML models trained on real launch data.

Acquire.comPre-revenue stage for this audience · Strong signals: active · Missing: arr, mrr, revenue

19%19% predicted probability of success on Acquire.com, based on ML models trained on real launch data.

BetaListMay not resonate with beta-testers · Strong signals: smart · Missing: web3, chat, crypto

2%2% predicted probability of success on BetaList, based on ML models trained on real launch data.

Correct prediction on native model

Similar products

Revenant – Node.js browser automation with PhantomJS43%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Revenant – Node.js browser automation with PhantomJS

Hacker News7

A Node.js script powered by Puppeteer for undetectable web scraping70%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

A Node.js script powered by Puppeteer for undetectable web scraping

Hacker News6

Web scraping focused HTTP client for Node.js64%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Web scraping focused HTTP client for Node.js

Hacker News3

BrowserAct50%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

AI-powered web scraping & browser automation — just describe

Indie Hackerscommitment-full-time

Yomuco – A simple web crawling library for Node.js41%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Yomuco – A simple web crawling library for Node.js

Hacker News23

Please critique my CouchDB ORM library for Node.js52%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Please critique my CouchDB ORM library for Node.js

Hacker News2

Node-Lepton – Node.js Library for Dropbox Lepton60%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Node-Lepton – Node.js Library for Dropbox Lepton

Hacker News4

Phobetor – browser automation library with Nightmare.js51%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Phobetor – browser automation library with Nightmare.js

Hacker News2

Yakuza – Scraping framework for Node.js48%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Yakuza – Scraping framework for Node.js

Hacker News3

CheerpX – x86 Node.js/python3/Ruby REPL in browser via WebAssembly JIT77%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

CheerpX – x86 Node.js/python3/Ruby REPL in browser via WebAssembly JIT

Hacker News88

Browse all products Analyze your launch