SQLMesh – The Future of DataOps

Hacker News

Share card

SQLMesh – The Future of DataOps

Hey Show HN! I’m Toby and over the last few months, I’ve been working with a team of engineers from Airbnb, Apple, Google, and Netflix, to simplify developing data pipelines with SQLMesh ( https://github.com/TobikoData/sqlmesh ). We’re tired of fragile pipelines, untested SQL queries, and expensive staging environments for data. Software engineers have reaped the benefits of DevOps through unit tests, continuous integration, and continuous deployment for years. We felt like it was time for data teams to have the same confidence and efficiency in development as their peers. It’s time for DataOps! SQLMesh can be used through a CLI/notebook or in our open source web based IDE (in preview). SQLMesh builds efficient dev / staging environments through “Virtual Data Marts” using views, which allows you to seamlessly rollback or roll forward your changes! With a simple pointer swap you can promote your “staging” data into production. This means you get unlimited copy-on-write environments that make data exploration and preview of changes cheap, easy, safe. Some other key features are: Automatic DAG generation by semantically parsing and understanding SQL or Python scripts CI-Runnable Unit and Integration tests with optional conversion to DuckDB Change detection and reconciliation through column level lineage Native Airflow Integration Import an existing DBT project and run it on SQLMesh’s runtime (in preview) We’re just getting started on our journey to change the way data pipelines are built and deployed. We’re huge proponents of open source and hope that we can grow together with your feedback and contributions. Try out SQLMesh by following the quick start guide ( https://sqlmesh.readthedocs.io/en/stable/quick_start/ ). We’d love to chat and hear about your experiences and ideas in our Slack community ( https://join.slack.com/t/tobiko-data/shared_invite/zt-1ma66d... ).