Data Engineering Podcast

DECEMBER 3, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Systems 130

Systems Designing Data Lake SQL 130

Data Engineering Podcast

MARCH 17, 2024

Summary A significant portion of data workflows involve storing and processing information in database engines. Validating that the information is stored and processed correctly can be complex and time-consuming, especially when the source and destination speak different dialects of SQL. Data lakes are notoriously complex.

Database 147

Database Data Lake High Quality Data Data Workflow 147

Data Engineering Podcast

MARCH 31, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data.

Project 130

Project Data Lake High Quality Data Data Workflow 130

Data Engineering Podcast

MARCH 24, 2024

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

Data Lake 162

Data Lake High Quality Data Hadoop Data Pipeline 162

Data Engineering Podcast

APRIL 28, 2024

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

Building 147

Building Data Lake High Quality Data Machine Learning 147

Data Engineering Podcast

MAY 5, 2024

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

Building 147

Building Data Lake High Quality Data Machine Learning 147

Data Engineering Podcast

NOVEMBER 26, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Data Engineering Podcast

JANUARY 7, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data.

Data Process 147

Data Process Process Data Lake High Quality Data 147

Data Engineering Podcast

FEBRUARY 11, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data.

Data Lake 147

Data Lake High Quality Data Government Data Pipeline 147

Data Engineering Podcast

DECEMBER 31, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data.

Designing 130

Designing Data Lake High Quality Data SQL 130

Data Engineering Podcast

JANUARY 21, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data.

Data Lake 147

Data Lake High Quality Data NoSQL Data Warehouse 147

Data Engineering Podcast

MARCH 10, 2024

Summary Data lakehouse architectures are gaining popularity due to the flexibility and cost effectiveness that they offer. The link that bridges the gap between data lake and warehouse capabilities is the catalog. Data lakes are notoriously complex. What is involved in integrating Nessie into a given data stack?

Data Lake 147

Data Lake High Quality Data Architecture Data Pipeline 147

Data Engineering Podcast

NOVEMBER 19, 2023

RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. With Materialize, you can!

Project 147

Project Data Lake High Quality Data SQL 147

Data Engineering Podcast

NOVEMBER 5, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

PostgreSQL 147

PostgreSQL Data Lake High Quality Data SQL 147

Data Engineering Podcast

DECEMBER 17, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Data Lake 130

Data Lake High Quality Data SQL Architecture 130

Data Engineering Podcast

MARCH 3, 2024

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

Programming 162

Programming Data Lake High Quality Data Data Pipeline 162

Data Engineering Podcast

FEBRUARY 25, 2024

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

Database 162

Database Technology Data Lake High Quality Data 162

Data Engineering Podcast

NOVEMBER 12, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Software Engineer 147

Software Engineer Software Engineering Engineering Data Lake 147

Data Engineering Podcast

DECEMBER 10, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management You shouldn't have to throw away the database to build with fast-changing data. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products.

Data Lake 130

Data Lake High Quality Data SQL Architecture 130

Data Engineering Podcast

MAY 26, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. SQL Server version upgrade) Section 2: Types of Migrations for Infrastructure Focus Storage migration: Moving data between systems (HDD to SSD, SAN to NAS, etc.)

Systems 130

Systems Data Lake High Quality Data Google Cloud 130

Data Engineering Podcast

DECEMBER 18, 2022

In this episode Abe Gong brings his experiences with the Great Expectations project and community to discuss the technical and organizational considerations involved in implementing these constraints to your data workflows. Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams.

Metadata 130

Metadata Business Intelligence Data Lake BI 130

François Nguyen

MARCH 22, 2021

TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. With this 3rd platform generation, you have more real time data analytics and a cost reduction because it is easier to manage this infrastructure in the cloud thanks to managed services.

Technology 130

Technology Architecture Google Cloud Metadata 130

Striim

SEPTEMBER 11, 2024

This method is advantageous when dealing with structured data that requires pre-processing before storage. Conversely, in an ELT-based architecture, data is initially loaded into storage systems such as data lakes in its raw form. Transformation occurs post-loading, allowing for flexible and scalable processing.

Data Pipeline 98

Data Pipeline Designing Data Lake Data Warehouse 98

Knowledge Hut

SEPTEMBER 25, 2023

Skill Requirements for Azure Data Engineer Job Description Here are some important skill requirements that you may find in a job description for Azure Data Engineers: 1. Azure Data Engineers work with these and other solutions. They guarantee that the data is efficiently cleaned, converted, and loaded.

Data Engineer 52

Data Engineer Data Engineering Engineering Data Lake 52

Monte Carlo

MAY 14, 2024

It was designed to support high-volume data exchange and compatibility across different system versions, which is essential for streaming architectures such as Apache Kafka. This development was crucial for enabling both batch and streaming data workflows in dynamic environments, ensuring consistency and durability in big data processing.

Data Lake 52

Data Lake Metadata Hadoop Data Governance 52

Data Engineering Podcast

JANUARY 1, 2022

In the same way that application performance monitoring ensures reliable software and keeps application downtime at bay, Monte Carlo solves the costly problem of broken data pipelines. Start trusting your data with Monte Carlo today! The data you’re looking for is already in your data warehouse and BI tools.

Data Warehouse 100

Data Warehouse SQL Hadoop Data Lake 100

Knowledge Hut

MARCH 28, 2024

Implemented and managed data storage solutions using Azure services like Azure SQL Database , Azure Data Lake Storage, and Azure Cosmos DB. Education & Skills Required Proficiency in SQL, Python, or other programming languages. Develop predictive models and data-driven solutions to address business challenges.

Data Engineer 52

Data Engineer Data Engineering Engineering Data Warehouse 52

Data Engineering Podcast

NOVEMBER 27, 2021

In the same way that application performance monitoring ensures reliable software and keeps application downtime at bay, Monte Carlo solves the costly problem of broken data pipelines. Start trusting your data with Monte Carlo today! The data you’re looking for is already in your data warehouse and BI tools.

Data Warehouse 100

Data Warehouse Data Lake BI Business Intelligence 100

Data Engineering Podcast

JULY 3, 2022

In order to quickly identify if and how two data systems are out of sync Gleb Mezhanskiy and Simon Eskildsen partnered to create the open source data-diff utility. The Ascend Data Automation Cloud provides a unified platform for data ingestion, transformation, orchestration, and observability.

Data Integration 100

Data Integration MongoDB Scala MySQL 100

Knowledge Hut

NOVEMBER 2, 2023

Azure Data Ingestion Pipeline Create an Azure Data Factory data ingestion pipeline to extract data from a source (e.g., CSV, SQL Server), transform it, and load it into a target storage (e.g., Azure SQL Database, Azure Data Lake Storage). A strong understanding of data sourcing with SQL.

Data Engineer 52

Data Engineer Data Engineering Project Coding 52

Knowledge Hut

NOVEMBER 17, 2023

They enhance data pipelines, transform data, and guarantee the accuracy, integrity, and compliance of the data. Their job entails Azure data engineer skills like using big data, databases, data lakes, and analytics to help firms make efficient data-driven decisions.

Data Engineer 52

Data Engineer Data Engineering Engineering Data Lake 52

Data Engineering Podcast

SEPTEMBER 11, 2022

RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control.

Data Pipeline 130

Data Pipeline Building MongoDB Scala 130

Data Engineering Weekly

JANUARY 15, 2023

SiliconANGLE theCUBE: Analyst Predictions 2023 - The Future of Data Management By far one of the best analyses of trends in Data Management. The author writes an exciting article about switching to Github actions for dbt to save $65,000 [link] Sponsored: Upsolver - Write a SQL Query, Get a Data-in-Motion Pipeline!

Data Engineer 52

Data Engineer Data Engineering Engineering Metadata 52

Monte Carlo

JUNE 14, 2023

The modern data stack era , roughly 2017 to present data, saw the widespread adoption of cloud computing and modern data repositories that decoupled storage from compute such as data warehouses, data lakes, and data lakehouses. Zero ETL is a bit of a misnomer.

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

Knowledge Hut

DECEMBER 28, 2023

This certification, often referred to as the Azure Data Engineer Associate certification, validates the competency of individuals in implementing Azure data solutions. It’s a testament to their ability to create scalable, efficient and secure data pipelines. What is the Azure Data Engineer Certification?

Certification 52

Certification Data Engineer Data Engineering Engineering 52

Ripple Engineering

MAY 27, 2021

Our data analysts used to schedule queries on BigQuery for transformation workflows and test the transformed data manually. We did not have a single tool that would automate the building, compiling, testing and documenting of SQL models, so we had no way to scale the process. SQL Models A model is a single.sql file.

Building 52

Building Raw Data SQL Data 52

DataKitchen

JANUARY 3, 2022

If the IT or data engineering team can’t respond with an enabling data platform in the required time frame, the business analyst does the necessary data work themselves. This ad hoc data engineering work often means coping with numerous data tables and diverse data sets using Alteryx, SQL, Excel or similar tools. .

Business Analyst 98

Business Analyst Data Lake Consulting Data Analytics 98