Parallelized random access to bzip2 files from Python written in C++

Hacker News

The project can be found here: https://github.com/mxmlnkn/indexed_bzip2 The readme shows how to use it from Python, via ratarmount, and via its command-line tool ibzip2. There are really a lot of bzip2 implementations out there but few that have been parallelized, even fewer with random access, and, to my knowledge, none of which are both parallel and offer random seeking in constant time. This is why I started indexed_bzip2. It is a header-only C++ library with Python bindings so that I can use it as a backend from ratarmount. For that, it also supports working with Python file-like objects by using the Python API. For this project, I started with the super-permissively 0-BSD licensed and small bzip2 decoder from the toybox project and extracted what I needed from it. The architecture for the parallel decoding plus random seeking took me some iterations and contemplation to get right. It uses block-based caching and prefetching to achieve both of these things. "Right" means that it is modular enough that I can reason about it and be convinced that it is correct. Of course, there are also a lot of automated tests with all kinds of sanitizers to check for correctness because I am wary when it comes to reading user data. All in all, this project was a nice brainteaser with which I could grow and I think it is largely finished for now. The single-core bzip2 decoder might profit from further optimizations but it already outperforms the internal Python bz2 module for 2 cores and more. And it also outperforms the internal Python bz2 module when it comes to random access because it is in constant time with indexed_bzip2. All this comes at the cost of higher memory usage (for the block cache), though. Currently, I'm trying to reuse the infrastructure I built for this to implement parallel random access to gzip files for usage in ratarmount but I'm not sure when that will be finished.

Visit product View on Hacker NewsJun 25, 2022

Share card

Actual performance

2points

Did not reach leaderboard

Launch Intel predictions

Analyze your own launch →

Product HuntOn track for Day 1 leaderboard · Strong signals: user, single, using · Missing: mac, agents, macos

74%74% predicted probability of success on Product Hunt, based on ML models trained on real launch data.

best fitHighest predicted score across all platforms for this description.

Indie HackersFits the IH revenue-focused audience · Strong signals: supports, started, para · Missing: reddit linkedin, podcasting, created

53%53% predicted probability of success on Indie Hackers, based on ML models trained on real launch data.

Hacker NewsMay not resonate with HN audience · Strong signals: io · Missing: https docs, excited, just released

47%47% predicted probability of success on Hacker News, based on ML models trained on real launch data.

nativeThis product was originally launched on this platform.

TrustMRRLess likely to generate early MRR · Strong signals: para · Missing: mobile apps, ios, personal

39%39% predicted probability of success on TrustMRR, based on ML models trained on real launch data.

AppSumoMay struggle as an AppSumo deal · Strong signals: plus · Missing: platform, intuitive, reviews

36%36% predicted probability of success on AppSumo, based on ML models trained on real launch data.

Acquire.comPre-revenue stage for this audience · Strong signals: profit · Missing: arr, mrr, revenue

14%14% predicted probability of success on Acquire.com, based on ML models trained on real launch data.

BetaListMay not resonate with beta-testers · Missing: web3, chat, crypto

0%0% predicted probability of success on BetaList, based on ML models trained on real launch data.

Correct prediction on native model

Similar products

Gsubpy, an interpreter for subset of Python, written in Go55%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Gsubpy, an interpreter for subset of Python, written in Go

Hacker News65

Incremental-Backups-Tools, written in Python63%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Incremental-Backups-Tools, written in Python

Hacker News3

A poetry generator written in Python named poemy50%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

A poetry generator written in Python named poemy

Hacker News2

Venvrc – simple Python-venv wrapper written in Bash58%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Venvrc – simple Python-venv wrapper written in Bash

Hacker News1

Easyfile – Python Random File Access for Humans53%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Easyfile – Python Random File Access for Humans

Hacker News4

Bincopy – An alternative to SRecord written in Python60%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Bincopy – An alternative to SRecord written in Python

Hacker News2

Logparser – Alternative to GoAccess Written in Python60%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Logparser – Alternative to GoAccess Written in Python

Hacker News54

A read-only FUSE for Dropbox written in Python56%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

A read-only FUSE for Dropbox written in Python

Hacker News3

Thread-Parallel Decompression and Random Access to Gzip Files (Pragzip)59%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Thread-Parallel Decompression and Random Access to Gzip Files (Pragzip)

Hacker News15

Middleware framework written in Python with asyncio48%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Middleware framework written in Python with asyncio

Hacker News1

Browse all products Analyze your launch