I

I simulated Bigtable in BigQuery as a Type 2 SCD

Hacker News

I simulated Bigtable in BigQuery as a Type 2 SCD

I hit a wall recently where we needed to design a data store that can both (1) ingest schema-less updates at scale and (2) allow for large analytical queries. GCP native doesn't have something that does both of these out-of-the-box, so we kinda hacked our own solution by replicating updates from Bigtable into a Type 2 SCD table in BigQuery. i couldn't find many resources online on this sort of trick when I was building it out, so I wrote up this blog with the details. Hope it's helpful and curious if anyone has used or built something similar. Will stick around for comments, happy to answer any questions or feedback.

Visit product View on Hacker NewsMay 27, 2025

Actual performance

15points

Made the leaderboard

Launch Intel predictions

Analyze your own launch →

Product HuntOn track for Day 1 leaderboard · Missing: mac, agents, macos

51%51% predicted probability of success on Product Hunt, based on ML models trained on real launch data.

best fitHighest predicted score across all platforms for this description.

AppSumoMay struggle as an AppSumo deal · Missing: plus, platform, intuitive

45%45% predicted probability of success on AppSumo, based on ML models trained on real launch data.

Indie HackersIH features products with proven revenue · Missing: supports, reddit linkedin, podcasting

45%45% predicted probability of success on Indie Hackers, based on ML models trained on real launch data.

TrustMRRLess likely to generate early MRR · Missing: mobile apps, ios, personal

44%44% predicted probability of success on TrustMRR, based on ML models trained on real launch data.

Hacker NewsMay not resonate with HN audience · Strong signals: io · Missing: https docs, excited, just released

40%40% predicted probability of success on Hacker News, based on ML models trained on real launch data.

nativeThis product was originally launched on this platform.

Acquire.comPre-revenue stage for this audience · Missing: arr, mrr, revenue

18%18% predicted probability of success on Acquire.com, based on ML models trained on real launch data.

BetaListMay not resonate with beta-testers · Missing: web3, chat, crypto

0%0% predicted probability of success on BetaList, based on ML models trained on real launch data.

Incorrect prediction on native model

Similar products

GMP Deserialization Type Confusion Vulnerability [MyBB43%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

GMP Deserialization Type Confusion Vulnerability [MyBB

Co-Occurrence of Type in Pokémon45%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Co-Occurrence of Type in Pokémon

Type someone's name and find out how to pronounce it45%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Type someone's name and find out how to pronounce it

Hindley–Milner type inferencing in C57%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Hindley–Milner type inferencing in C

Hash any comparable type in Go55%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Hash any comparable type in Go

KeyVocab – drill vocab while you type45%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

KeyVocab – drill vocab while you type

Type45%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Type

a type-agnostic wrapper of leveldb for Go26%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

a type-agnostic wrapper of leveldb for Go

Miami Stone Restoration - Select Your Stone Type17%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

stone, restoration, marble, granit

Indie Hackerscommitment-full-time

Type 2 [.] Space34%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Towards type-2 civilization and beyond

Indie Hackers1ai

Browse all products Analyze your launch