Start Data Engineering

JANUARY 16, 2021

Introduction Setup Code Conditional logic to read from mock input Custom macro to test for equality Setup environment specific test Run ELT using dbt Conclusion Further reading Introduction With the recent advancements in data warehouses and tools like dbt most transformations(T of ELT) are being done directly in the data warehouse.

SQL 130

SQL Data Warehouse Coding IT 130

Data Engineering Podcast

JUNE 25, 2023

You can collect, transform, and route data across your entire stack with its event streaming, ETL, and reverse ETL pipelines. You can Implement RudderStack SDKs once, then automatically send events to your warehouse and 150+ business tools, and you’ll never have to worry about API changes again.

Data Engineering 225

Data Engineering Data Engineer Python Engineering 225

Christophe Blefari

MARCH 15, 2024

With yato you give a folder with SQL queries and it guesses the DAG and runs the queries in the right order. BigQuery supports DELETE to delete partitions in a SQL query. Arrow doing a lot of the data operation heavy lifting. Give a lot of insights on the market. this is more common sense but always works.

Metadata 272

Metadata Data Datasets Data Warehouse 272

Snowflake

NOVEMBER 26, 2024

Many of our customers — from Marriott to AT&T — start their journey with the Snowflake AI Data Cloud by migrating their data warehousing workloads to the platform. Today we’re focusing on customers who migrated from a cloud data warehouse to Snowflake and some of the benefits they saw.

Data Warehouse 97

Data Warehouse Cloud PostgreSQL Hadoop 97

Snowflake

NOVEMBER 2, 2023

Over the years, the technology landscape for data management has given rise to various architecture patterns, each thoughtfully designed to cater to specific use cases and requirements. These patterns include both centralized storage patterns like data warehouse , data lake and data lakehouse , and distributed patterns such as data mesh.

Data Lake 107

Data Lake Data Warehouse Cloud Unstructured Data 107

Data Engineering Podcast

FEBRUARY 18, 2024

Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). Data lakes are notoriously complex. Data lakes are notoriously complex. Your first 30 days are free! Your first 30 days are free!

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Towards Data Science

JUNE 6, 2023

It runs locally, has extensive SQL support and can run queries directly on Pandas data, Parquet, JSON data. The fact it’s insanely fast and does (mostly) all processing in memory make it a good choice for building my personal data warehouse. Extra points for its seamless integration with Python and R.

Data Warehouse 98

Data Warehouse Data SQL Data Analysis 98

Snowflake

JANUARY 28, 2025

Migrating from a traditional data warehouse to a cloud data platform is often complex, resource-intensive and costly. As part of this announcement, Snowflake is also announcing private preview support of a new end-to-end data migration experience for Amazon Redshift.

Data Warehouse 68

Data Warehouse Professional Services SQL Data 68

Data Engineering Podcast

JANUARY 1, 2022

In this episode Emily Riederer shares her work to create a controlled vocabulary for managing the semantic elements of the data managed by her team and encoding it in the schema definitions in her data warehouse. Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams.

Data Warehouse 100

Data Warehouse BI Data Workflow Data Engineering 100

databricks

AUGUST 10, 2023

At this year's Data+AI Summit, Databricks SQL continued to push the boundaries of what a data warehouse can be, leveraging AI across the.

SQL 98

SQL Data Warehouse Data 98

Snowflake

JANUARY 28, 2025

Migrating from a traditional data warehouse to a cloud data platform is often complex, resource-intensive and costly. As part of this announcement, Snowflake is also announcing private preview support of a new end-to-end data migration experience for Amazon Redshift.

Data Warehouse 71

Data Warehouse Professional Services SQL Coding 71

Data Engineering Podcast

MAY 1, 2022

Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams. Datafold built automated regression testing to help data and analytics engineers deal with data quality in their pull requests. No more shipping and praying, you can now know exactly what will change in your database!

Data Warehouse 100

Data Warehouse Data Integration Cloud Google Cloud 100

Data Engineering Podcast

OCTOBER 28, 2021

He also explains why he started Decodable to address that limitation and the work that he and his team have done to let data engineers build streaming pipelines entirely in SQL. Start trusting your data with Monte Carlo today! Hightouch is the easiest way to sync data into the platforms that your business teams rely on.

Data Pipeline 100

Data Pipeline SQL Data Warehouse Data Lake 100

Christophe Blefari

JUNE 21, 2024

Snowflake was founded in 2012 around its data warehouse product, which is still its core offering, and Databricks was founded in 2013 from academia with Spark co-creator researchers, becoming Apache Spark in 2014. you could write the same pipeline in Java, in Scala, in Python, in SQL, etc.—with 3) Spark 4.0

Metadata 147

Metadata Data Warehouse BI MySQL 147

databricks

FEBRUARY 12, 2023

Oracle is a well-known technology for hosting Enterprise Data Warehouse solutions. However, many customers like Optum and the U.S. Citizenship and Immigration Services.

Coding 122

Coding SQL Data Warehouse Technology 122

Cloudera

JULY 11, 2023

Apache Impala is a distributed C++ backed SQL engine that integrates with Kudu to serve BI results over millions of rows meeting sub-second service-level agreements. Cloudera offers Apache Kudu to run in Real Time DataMart Clusters , and Apache Impala to run in Kubernetes in the Cloudera Data Warehouse form factor.

Data Warehouse 70

Data Warehouse Pharmaceutical BI Banking 70

Cloudera

APRIL 3, 2023

In this blog, we will share with you in detail how Cloudera integrates core compute engines including Apache Hive and Apache Impala in Cloudera Data Warehouse with Iceberg. We will publish follow up blogs for other data services. Try Cloudera Data Warehouse (CDW) by signing up for a 60 day trial , or test drive CDP.

Data Warehouse 82

Data Warehouse Java Metadata Data 82

Engineering at Meta

JANUARY 22, 2025

These stages propagate through various systems including function-based systems that load, process, and propagate data through stacks of function calls in different programming languages (e.g., For simplicity, we will demonstrate these for the web, the data warehouse, and AI, per the diagram below. Hack, C++, Python, etc.)

Data Warehouse 116

Data Warehouse SQL Programming Language Data 116

Seattle Data Guy

MAY 26, 2024

Many data engineers and analysts don’t realize how valuable the knowledge they have is. They’ve spent hours upon hours learning SQL, Python, how to properly analyze data, build data warehouses, and understand the differences between eight different ETL solutions.

Consulting 130

Consulting Engineering Data Warehouse SQL 130

databricks

MAY 3, 2023

Caching is an essential technique for improving the performance of data warehouse systems by avoiding the need to recompute or fetch the same.

SQL 105

SQL Data Warehouse Systems Data 105

Monte Carlo

FEBRUARY 11, 2025

Think of your data warehouse like a well-organized library. Thats where data warehouse schemas come in. A data warehouse schema is a blueprint for how your data is structured and linkedusually with fact tables (for measurable data) and dimension tables (for descriptive attributes).

Data Warehouse 52

Data Warehouse Electronics Retail Data 52

Data Engineering Podcast

FEBRUARY 5, 2023

Materialize’s PostgreSQL-compatible interface lets users leverage the tools they already use, with unsurpassed simplicity enabled by full ANSI SQL support. Sign up now for early access to Materialize and get started with the power of streaming data with the same simplicity and low implementation cost as batch cloud data warehouses.

Data Engineering 147

Data Engineering Data Engineer Engineering PostgreSQL 147

Data Engineering Podcast

SEPTEMBER 3, 2023

Summary Cloud data warehouses and the introduction of the ELT paradigm has led to the creation of multiple options for flexible data integration, with a roughly equal distribution of commercial and open source options. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products.

Data Integration 162

Data Integration BI SQL Python 162

Sync Computing

NOVEMBER 7, 2024

Data volume and velocity, governance, structure, and regulatory requirements have all evolved and continue to. Despite these limitations, data warehouses, introduced in the late 1980s based on ideas developed even earlier, remain in widespread use today for certain business intelligence and data analysis applications.

Data Lake 52

Data Lake Data Warehouse Business Intelligence Unstructured Data 52

Data Engineering Podcast

JANUARY 8, 2023

In this episode Ori Rafael explains how they are automating the creation and scheduling of orchestration flows and their related transforations in a unified SQL interface. Trusted by the data teams at Fox, JetBlue, and PagerDuty, Monte Carlo solves the costly problem of broken data pipelines. Missing data?

PostgreSQL 130

PostgreSQL Data Lake Data Warehouse BI 130

Cloudera

JANUARY 19, 2024

As described in our recent blog post , an SQL AI Assistant has been integrated into Hue with the capability to leverage the power of large language models (LLMs) for a number of SQL tasks. This is a real game-changer for data analysts on all levels and will make SQL development faster, easier, and less error-prone.

SQL 72

SQL Data Warehouse AWS Accessible 72

Data Engineering Podcast

JANUARY 21, 2024

A substantial amount of the data that is being managed in these systems is related to customers and their interactions with an organization. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

Data Lake 147

Data Lake High Quality Data NoSQL Data Warehouse 147

Seattle Data Guy

FEBRUARY 10, 2024

Photo by Tiger Lily Data warehouses and data lakes play a crucial role for many businesses. It gives businesses access to the data from all of their various systems. As well as often integrating data so that end-users can answer business critical questions.

Data Lake 162

Data Lake Data Warehouse Data Accessible 162

Striim

NOVEMBER 19, 2024

SQL2Fabric Mirroring is a new fully managed service offered by Striim to mirror on premise SQL Databases. It’s a collaborative service between Striim and Microsoft based on Fabric Open Mirroring that enables real-time data replication from on-premise SQL Server databases to Azure Fabric OneLake.

SQL 52

SQL Database Data Warehouse Data Pipeline 52

Christophe Blefari

SEPTEMBER 15, 2023

How to reduce warehouse costs? — Hugo propose 7 hacks to optimise data warehouse cost. Dimensional data modeling with dbt — A great 6-steps process to create a simple dim-fact model with dbt. teej/titan — Titan is a Python library to manage data warehouse infrastructure. Crazy amounts.

Data Warehouse 130

Data Warehouse Data SQL Python 130

Data Engineering Podcast

JANUARY 15, 2023

With a PostgreSQL-compatible interface, you can now work with real-time data using ANSI SQL including the ability to perform multi-way complex joins, which support stream-to-stream, stream-to-table, table-to-table, and more, all in standard SQL.

Coding 130

Coding Building PostgreSQL Data Lake 130

Data Engineering Podcast

DECEMBER 28, 2022

Datafold built automated regression testing to help data and analytics engineers deal with data quality in their pull requests. Datafold shows how a change in SQL code affects your data, both on a statistical level and down to individual rows and values before it gets merged to production. Pricing for SQLake is simple.

Data Lake 130

Data Lake Data Warehouse Data Pipeline MongoDB 130

Maxime Beauchemin

JANUARY 7, 2018

In the context of a SQL ELT-type approach which has become common nowadays, it is likely to be simply overwriting a portion of a table (partition). This means that ideally the logic in source control describes how to build the full state of the data warehouse throughout all time periods.

Data Process 183

Data Process Data Engineering Data Engineer Process 183

Data Engineering Podcast

DECEMBER 25, 2022

Datafold built automated regression testing to help data and analytics engineers deal with data quality in their pull requests. Datafold shows how a change in SQL code affects your data, both on a statistical level and down to individual rows and values before it gets merged to production. Pricing for SQLake is simple.

Machine Learning 130

Machine Learning Systems Data Lake Data Warehouse 130

Towards Data Science

JUNE 1, 2023

Supercharge your BigQuery experience with these 6 best practices Continue reading on Towards Data Science »

Data Warehouse 98

Data Warehouse Data Science Data SQL 98

Striim

NOVEMBER 19, 2024

SQL2Fabric Mirroring is a new fully managed service offered by Striim to mirror on premise SQL Databases. Its a collaborative service between Striim and Microsoft based on Fabric Open Mirroring that enables real-time data replication from on-premise SQL Server databases to Azure Fabric OneLake.

SQL 52

SQL Database Data Warehouse Data Pipeline 52

Cloudera

DECEMBER 21, 2023

Now imagine you had a personal assistant who knew everything about your data sets and was an expert in SQL, sitting alongside you every step of the way to help you quickly problem solve, write optimized code, explain queries, and much more. Well imagine it no longer, as Cloudera’s SQL AI Assistant is exactly that!

SQL 67

SQL Data Warehouse Business Analyst Data Schemas 67