Partial content web crawling using HTTP/2 and Go

Hacker News

Hi, I wrote a low-level HTTP/2 web crawler in Go, which can scrape partial content to save traffic. Tl;dr e.g. the HTML of a YouTube video contains the video description, views, likes etc. in its first 600KB, the remaining 900KB are of no use for me, but I have to pay my proxies by the gigabyte. My crawler receives packet per packet, and if I got everything I needed I reset the request, and only pay-for-what-i-crawled. This is also potentially useful for large-scale crawling operations, where duplicates matter. You could compute a simHash on the fly, and reset on-the-fly before crawling the entire document (again).

Visit product View on Hacker NewsJan 26, 2026

Share card

Actual performance

5points

Did not reach leaderboard

Launch Intel predictions

Analyze your own launch →

Product HuntOn track for Day 1 leaderboard · Strong signals: using · Missing: mac, agents, macos

81%81% predicted probability of success on Product Hunt, based on ML models trained on real launch data.

best fitHighest predicted score across all platforms for this description.

Hacker NewsStrong engagement from HN community · Strong signals: ide, io · Missing: https docs, excited, just released

62%62% predicted probability of success on Hacker News, based on ML models trained on real launch data.

nativeThis product was originally launched on this platform.

Indie HackersFits the IH revenue-focused audience · Missing: supports, reddit linkedin, podcasting

59%59% predicted probability of success on Indie Hackers, based on ML models trained on real launch data.

TrustMRRFits verified-revenue profile · Strong signals: video · Missing: mobile apps, ios, personal

52%52% predicted probability of success on TrustMRR, based on ML models trained on real launch data.

AppSumoMay struggle as an AppSumo deal · Missing: plus, platform, intuitive

38%38% predicted probability of success on AppSumo, based on ML models trained on real launch data.

Acquire.comPre-revenue stage for this audience · Missing: arr, mrr, revenue

21%21% predicted probability of success on Acquire.com, based on ML models trained on real launch data.

BetaListMay not resonate with beta-testers · Missing: web3, chat, crypto

13%13% predicted probability of success on BetaList, based on ML models trained on real launch data.

Incorrect prediction on native model

Similar products

Nuster – A HTTP/2 web cache accelerator66%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Nuster – A HTTP/2 web cache accelerator

Hacker News8

Redux-rx-http, a pragmatic HTTP layer for redux using redux-observable47%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Redux-rx-http, a pragmatic HTTP layer for redux using redux-observable

Hacker News2

Handled 6.87M HTTP Req/sec with Rapidoid on Top of Java NIO56%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Handled 6.87M HTTP Req/sec with Rapidoid on Top of Java NIO

Hacker News7

Lint for HTTP63%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Lint for HTTP

Hacker News138

Zefner, Knight that will attack your HTTP's endpoint49%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Zefner, Knight that will attack your HTTP's endpoint

Hacker News1

Symfony HTTP Reponder: ADR Implemented67%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Symfony HTTP Reponder: ADR Implemented

Hacker News1

Of-Watchdog – OpenFaaS Watchdog for HTTP46%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Of-Watchdog – OpenFaaS Watchdog for HTTP

Hacker News1

Bhttp Binary HTTP (RFC 9292) for Go63%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Bhttp Binary HTTP (RFC 9292) for Go

Hacker News4

Oblivious HTTP for Go53%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Oblivious HTTP for Go

Hacker News10

Bowtie, a (hopefully) idiomatic HTTP middleware for Go51%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Bowtie, a (hopefully) idiomatic HTTP middleware for Go

Hacker News7

Browse all products Analyze your launch