Crul – Query Any Webpage or API

Hacker News

Hi HN, we’re Carl and Nic, the creators of crul ( https://www.crul.com ), and we’ve been hard at work for the last year and a half building our dream of turning the web into a dataset. In a nutshell crul is a tool for querying and building web and api data feeds from anywhere to anywhere. With crul you can crawl and transform web pages into csv tables, explore and dynamically query APIs, filter and organize data, and push data sets to third party data lakes and analytics tools. Here’s a demo video, we’ve been told Nic sounds like John Mayer (lol) ( https://www.crul.com/demo-video ) We’ve personally struggled wrangling data from the web using puppeteer/playwright/selenium, jq or cobbling together python scripts, client libraries, and schedulers to consume APIs. The reality is that shit is hard, doesn’t scale (classic blocking for-loop or async saturation), and comes with thorny maintenance/security issues. The tools we love to hate. Crul’s value prop is simple: Query any Webpage or API for free. At its core, crul is based on the foundational linked nature of Web/API content. It consists of a purpose built map/expand/reduce engine for hierarchical Web/API content (kind of like postman but with a membership to Gold's Gym) with a familiar parser expression grammar that naturally gets the job done (and layered caching to make it quick to fix when it doesn’t on the first try). There’s a boatload of other features like domain policies, scheduler, checkpoints, templates, REST API, Web UI, vault, OAuth for third parties and 20+ stores to send your data to. Our goal is to open source crul as time and resources permit. At the end of the day it’s just the two of us trying to figure things out as we go! We’re just getting started. Crul is one bad mother#^@%*& and the web is finally yours! Download crul for free as a Mac OS desktop application or as a Docker image ( https://www.crul.com ) and let us know if you love it or hate it. ( https://forms.gle/5BXb5bLC1D5QG7i99 ) And come say hello to us on our slack channel - we’re a friendly bunch! ( https://crulinc.slack.com/ ) Nic and Carl ( https://www.crul.com/early-days )

Visit product View on Hacker NewsFeb 28, 2023

Share card

Actual performance

241points

79comments

Made the leaderboard

Launch Intel predictions

Analyze your own launch →

Indie HackersFits the IH revenue-focused audience · Strong signals: started · Missing: supports, reddit linkedin, podcasting

91%91% predicted probability of success on Indie Hackers, based on ML models trained on real launch data.

best fitHighest predicted score across all platforms for this description.

Product HuntOn track for Day 1 leaderboard · Strong signals: mac, slack, dock · Missing: agents, macos, agent

84%84% predicted probability of success on Product Hunt, based on ML models trained on real launch data.

Hacker NewsStrong engagement from HN community · Strong signals: open source, ide, io · Missing: https docs, excited, just released

78%78% predicted probability of success on Hacker News, based on ML models trained on real launch data.

nativeThis product was originally launched on this platform.

TrustMRRFits verified-revenue profile · Strong signals: personal, video · Missing: mobile apps, ios, entrepreneurs

51%51% predicted probability of success on TrustMRR, based on ML models trained on real launch data.

AppSumoMay struggle as an AppSumo deal · Strong signals: friendly · Missing: plus, platform, intuitive

45%45% predicted probability of success on AppSumo, based on ML models trained on real launch data.

Acquire.comPre-revenue stage for this audience · Missing: arr, mrr, revenue

16%16% predicted probability of success on Acquire.com, based on ML models trained on real launch data.

BetaListMay not resonate with beta-testers · Missing: web3, chat, crypto

0%0% predicted probability of success on BetaList, based on ML models trained on real launch data.

Correct prediction on native model

Similar products

A free API to query 3.5B subdomains52%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

A free API to query 3.5B subdomains

Hacker News4

Monglorious – Query MongoDB with Strings33%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Monglorious – Query MongoDB with Strings

Hacker News2

Query the national alcoholic beverage retailing monopoly of Finland43%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Query the national alcoholic beverage retailing monopoly of Finland

Hacker News2

VelocityPack QueryBoost38%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

MySQL / MariaDB Query Caching and Acceleration for cPanel

Indie Hackersemployees-10-plus

Firestore Query Observer API wrapper library41%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Firestore Query Observer API wrapper library

Hacker News1

Hauntify – A Helloween-I-fy any webpage Bookmarklet43%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Hauntify – A Helloween-I-fy any webpage Bookmarklet

Hacker News2

Query DB_To_FileMemoryDB Tool44%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Query DB_To_FileMemoryDB Tool

Hacker News2

Mixpanel Query Tool44%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Mixpanel Query Tool

Hacker News14

PyEQS – query Elasticsearch like a Django Queryset62%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

PyEQS – query Elasticsearch like a Django Queryset

Hacker News55

Query 1.6B rows in milliseconds, live42%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Query 1.6B rows in milliseconds, live

Hacker News271

Browse all products Analyze your launch