Faking SIMD to Search and Sort Strings 5x Faster

Hacker News

I want to share a really dumb, but very practical project I have packaged this summer, to perform operations on strings much faster. I was using Python to work with a multi-terabyte newline-delimited file. Reading, splitting, and shuffling it was a nightmare. So, I wrapped a trivial hardware-friendly heuristic I've been using for the last few years into a CPython library. The part I enjoyed the most is implementing SIMD behavior without SIMD instructions... Using 64-bit words to work at 8-bit granularity. Unlike conventional SIMD, the code would remain the same for ~~almost~~ any hardware. Let this library be a reminder of how awesome bit-level hacks are! Feel free to use it when working with CommonCrawl or any other sizeable textual dataset.

Visit product View on Hacker NewsAug 26, 2023

Share card

Actual performance

29points

5comments

Made the leaderboard

Launch Intel predictions

Analyze your own launch →

Product HuntOn track for Day 1 leaderboard · Strong signals: new, using, code · Missing: mac, agents, macos

85%85% predicted probability of success on Product Hunt, based on ML models trained on real launch data.

best fitHighest predicted score across all platforms for this description.

Indie HackersFits the IH revenue-focused audience · Missing: supports, reddit linkedin, podcasting

54%54% predicted probability of success on Indie Hackers, based on ML models trained on real launch data.

Hacker NewsStrong engagement from HN community · Strong signals: io · Missing: https docs, excited, just released

53%53% predicted probability of success on Hacker News, based on ML models trained on real launch data.

nativeThis product was originally launched on this platform.

TrustMRRLess likely to generate early MRR · Missing: mobile apps, ios, personal

39%39% predicted probability of success on TrustMRR, based on ML models trained on real launch data.

AppSumoMay struggle as an AppSumo deal · Strong signals: friendly · Missing: plus, platform, intuitive

38%38% predicted probability of success on AppSumo, based on ML models trained on real launch data.

Acquire.comPre-revenue stage for this audience · Missing: arr, mrr, revenue

16%16% predicted probability of success on Acquire.com, based on ML models trained on real launch data.

BetaListMay not resonate with beta-testers · Missing: web3, chat, crypto

1%1% predicted probability of success on BetaList, based on ML models trained on real launch data.

Correct prediction on native model

Similar products

A 70x faster and 5x smaller XSS sanitizer than DOMPurify73%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

A 70x faster and 5x smaller XSS sanitizer than DOMPurify

Hacker News3

Faster than std:sort and pdqsort35%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Faster than std:sort and pdqsort

Hacker News1

TDD Screencast #2: Implementing Sort19%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

TDD Screencast #2: Implementing Sort

Hacker News3

Cluster-Sort27%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Cluster-Sort

Hacker News2

Search and sort npm by pagerank21%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Search and sort npm by pagerank

Hacker News1

QuadSort, Esoteric Fast Sort30%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

QuadSort, Esoteric Fast Sort

Hacker News78

TADA74%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

1:1 text-acoustic alignment for 5x faster speech generation

Product Hunt+127Open Source

TensorBase: 5x~10000x Faster Drop-In/Accelerator for ClickHouse in Rust76%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

TensorBase: 5x~10000x Faster Drop-In/Accelerator for ClickHouse in Rust

Hacker News19

YouTube Sort by Likes47%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

YouTube Sort by Likes

Hacker News1

Search, Sort, Filter Prices from US Hospitals39%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Search, Sort, Filter Prices from US Hospitals

Hacker News2

Browse all products Analyze your launch