SQLFrame – I ran PySpark without Spark on a SQL database

Hacker News

Recently I open-sourced SQLFrame, a DataFrame library that implements the PySpark DataFrame API but removes Spark as a dependency. It does this by generating the corresponding SQL for the DataFrame operations using SQLGlot. Since the output is SQL this also means that the PySpark DataFrame API can now be used directly against other databases without the Spark middleman. I built this because of two common problems I have faced in my career: 1. I prefer to write complex pipelines in PySpark but they can be hard to read for SQL-proficient co-workers. Therefore I find myself in a tradeoff between maintainability and accessibility. 2. I really enjoy using the PySpark DataFrame API but not every project requires Spark and therefore I'm not able to use the DataFrame library I am most proficient in. The library currently focuses on transformation pipelines (reading from and writing to tables) and data analysis as key use cases. It does offer some ability to read from files directly but they must be small although this can be improved over time if there is demand for it. SQLFrame currently supports BigQuery, DuckDB, and Postgres with Clickhouse, Redshift, Snowflake, Spark, and Trino in development or planned. You can use the "Standalone" session to test running against any engine supported by SQLGlot but there could be issues with more advanced functions that will be resolved once officially supported by SQLFrame. Blog post for more details: https://github.com/eakmanrq/sqlframe/blob/main/blogs/sqlfram... Would love to answer any questions or hear any feedback you may have!

Visit product View on Hacker NewsMay 21, 2024

Share card

Actual performance

11points

4comments

Made the leaderboard

Launch Intel predictions

Analyze your own launch →

Indie HackersFits the IH revenue-focused audience · Strong signals: supports · Missing: reddit linkedin, podcasting, created

81%81% predicted probability of success on Indie Hackers, based on ML models trained on real launch data.

best fitHighest predicted score across all platforms for this description.

Product HuntOn track for Day 1 leaderboard · Strong signals: using, open · Missing: mac, agents, macos

78%78% predicted probability of success on Product Hunt, based on ML models trained on real launch data.

Hacker NewsStrong engagement from HN community · Strong signals: pipe, clickhouse, io · Missing: https docs, excited, just released

63%63% predicted probability of success on Hacker News, based on ML models trained on real launch data.

nativeThis product was originally launched on this platform.

TrustMRRLess likely to generate early MRR · Missing: mobile apps, ios, personal

33%33% predicted probability of success on TrustMRR, based on ML models trained on real launch data.

AppSumoMay struggle as an AppSumo deal · Missing: plus, platform, intuitive

31%31% predicted probability of success on AppSumo, based on ML models trained on real launch data.

Acquire.comPre-revenue stage for this audience · Missing: arr, mrr, revenue

12%12% predicted probability of success on Acquire.com, based on ML models trained on real launch data.

BetaListMay not resonate with beta-testers · Missing: web3, chat, crypto

0%0% predicted probability of success on BetaList, based on ML models trained on real launch data.

Correct prediction on native model

Similar products

DatabaseFileRecovery SQL Database Recovery45%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Repirs and recovers corrupt SQL database

Indie Hackerscommitment-full-time

Resql – Lightweight SQL database with replication70%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Resql – Lightweight SQL database with replication

Hacker News3

An SQL database on top of a blockchain70%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

An SQL database on top of a blockchain

Hacker News11

SQL Playground for Database51%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

SQL Playground for Database

Hacker News1

A Schemaless Data Store Within Your SQL Database71%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

A Schemaless Data Store Within Your SQL Database

Hacker News1

A Schemaless Data Store Within Your SQL Database71%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

A Schemaless Data Store Within Your SQL Database

Hacker News2

A Schemaless Data Store Within Your SQL Database65%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

A Schemaless Data Store Within Your SQL Database

Hacker News45

Reactive Web UI for Your SQL Database64%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Reactive Web UI for Your SQL Database

Hacker News2

QueryTuner – SQL diagnostics across 5 DBs, no database connection60%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

QueryTuner – SQL diagnostics across 5 DBs, no database connection

Hacker News8

can we get rid of SQL?67%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

can we get rid of SQL?

Hacker News2

Browse all products Analyze your launch