Data Engineer, Data Pipeline and SQL - Data Engineering Digest

Seamless SQL And Python Transformations For Data Engineers And Analysts With SQLMesh

Data Engineering Podcast

JUNE 25, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. Rudderstack]([link] RudderStack provides all your customer data pipelines in one platform.

Data Engineering

Data Engineering Data Engineer Python Engineering

Snowflake’s New Python API Empowers Data Engineers to Build Modern Data Pipelines with Ease

Snowflake

APRIL 17, 2024

In today’s data-driven world, developer productivity is essential for organizations to build effective and reliable products, accelerate time to value, and fuel ongoing innovation. While the Python API connector remains available for specific SQL use cases, the new API is designed to be your go-to solution.

Data Pipeline

Data Pipeline Python Data Engineering Data Engineer

Tackling Real Time Streaming Data With SQL Using RisingWave

Data Engineering Podcast

FEBRUARY 4, 2024

Summary Stream processing systems have long been built with a code-first design, adding SQL as a layer on top of the existing framework. RisingWave is a database engine that was created specifically for stream processing, with S3 as the storage layer. Can you describe what RisingWave is and the story behind it?

SQL

SQL Data Lake High Quality Data Data Pipeline

Webinars

Apache Airflow®: The Ultimate Guide to DAG Writing

MORE WEBINARS

How to improve at SQL as a data engineer

Start Data Engineering

OCTOBER 22, 2021

SQL skills 2.1. Data modeling 2.1.1. Data storage 2.2. Data transformation 2.2.1. Data pipeline 2.4. Data analytics 3. Introduction SQL is the bread and butter of data engineering. Introduction 2. Gathering requirements 2.1.2. Exploration 2.1.3. Modeling 2.1.4. Practice 4.

SQL

SQL Data Engineering Data Engineer Engineering

Streaming Data Pipelines Made SQL With Decodable

Data Engineering Podcast

OCTOBER 28, 2021

He also explains why he started Decodable to address that limitation and the work that he and his team have done to let data engineers build streaming pipelines entirely in SQL. The data you’re looking for is already in your data warehouse and BI tools. No more scripts, just SQL.

Data Pipeline

Data Pipeline SQL Data Warehouse Data Lake

Using Trino And Iceberg As The Foundation Of Your Data Lakehouse

Data Engineering Podcast

FEBRUARY 18, 2024

Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). Data lakes are notoriously complex. Visit: dataengineeringpodcast.com/data-council today. Your first 30 days are free!

Data Lake

Data Lake High Quality Data Data Warehouse Google Cloud

Ready-to-go sample data pipelines with Dataflow

Netflix Tech

DECEMBER 3, 2022

by Jasmine Omeke , Obi-Ike Nwoke , Olek Gorajek Intro This post is for all data practitioners, who are interested in learning about bootstrapping, standardization and automation of batch data pipelines at Netflix. You may remember Dataflow from the post we wrote last year titled Data pipeline asset management with Dataflow.

Data Pipeline

Data Pipeline Scala Metadata Food

Unpacking The Seven Principles Of Modern Data Pipelines

Data Engineering Podcast

AUGUST 13, 2023

Summary Data pipelines are the core of every data product, ML model, and business intelligence dashboard. The folks at Rivery distilled the seven principles of modern data pipelines that will help you stay out of trouble and be productive with your data. Rudderstack : ![Rudderstack]([link]

Data Pipeline

Data Pipeline BI SQL Business Intelligence

Troubleshooting Kafka In Production

Data Engineering Podcast

DECEMBER 24, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Kafka

Kafka Data Lake High Quality Data SQL

How to learn data engineering

Christophe Blefari

JANUARY 20, 2024

Learn data engineering, all the references ( credits ) This is a special edition of the Data News. But right now I'm in holidays finishing a hiking week in Corsica 🥾 So I wrote this special edition about: how to learn data engineering in 2024. Who are the data engineers?

Data Engineering

Data Engineering Data Engineer Engineering Hadoop

10 Skills to Ace Your Data Engineering Interviews

Start Data Engineering

OCTOBER 11, 2021

Leetcode: data structures and algorithms 4. Data modeling 4.1 Data warehousing 4.2 Data pipelines 6. Introduction Skills 1. Distributed system fundamentals 7. Event streaming 8. System design 9. Business questions 10. Cloud computing 11.

Data Engineering

Data Engineering Data Engineer Engineering Cloud Computing

Our First Netflix Data Engineering Summit

Netflix Tech

DECEMBER 14, 2023

Engineers from across the company came together to share best practices on everything from Data Processing Patterns to Building Reliable Data Pipelines. The result was a series of talks which we are now sharing with the rest of the Data Engineering community! In this video, Sr.

Data Engineering

Data Engineering Data Engineer Engineering Metadata

Data Engineering Weekly #175

Data Engineering Weekly

JUNE 10, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. As I have shared , its impact on data engineering is exciting.

Data Engineering

Data Engineering Data Engineer Engineering Kafka

How Data Engineering Teams Power Machine Learning With Feature Platforms

Data Engineering Podcast

JULY 2, 2023

In this episode Razi Raziuddin shares how data engineering teams can support the machine learning workflow through the development and support of systems that empower data scientists and ML engineers to build and maintain their own features. How is this distinct from other forms of data pipeline development and delivery?

Machine Learning

Machine Learning Data Engineering Data Engineer Engineering

Cloudera Data Engineering 2021 Year End Review

Cloudera

DECEMBER 21, 2021

Since the release of Cloudera Data Engineering (CDE) more than a year ago , our number one goal was operationalizing Spark pipelines at scale with first class tooling designed to streamline automation and observability. Data pipelines are composed of multiple steps with dependencies and triggers. Happy New Year.

Data Engineering

Data Engineering Data Engineer Engineering Data Pipeline

Building Data Pipelines That Run From Source To Analysis And Activation With Hevo Data

Data Engineering Podcast

SEPTEMBER 11, 2022

Building reliable data pipelines is a complex and costly undertaking with many layered requirements. In order to reduce the amount of time and effort required to build pipelines that power critical insights Manish Jethani co-founded Hevo Data. Data stacks are becoming more and more complex.

Data Pipeline

Data Pipeline Building MongoDB MySQL

Making Email Better With AI At Shortwave

Data Engineering Podcast

APRIL 21, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Go to dataengineeringpodcast.com/dagster today to get started.

Data Lake

Data Lake High Quality Data Data Pipeline Machine Learning

Simplified End-to-End Development for Production-Ready Data Pipelines, Applications, and ML Models

Snowflake

JUNE 4, 2024

Streamlined development across SQL and Python Snowflake now offers data teams a suite of intuitive tools designed to simplify development and accelerate workflows. This suite extends seamlessly across Snowflake’s offerings, including Snowpark, Native Apps, Streamlit and more, for building anything with your data.

Data Pipeline

Data Pipeline Python SQL Government

Building Databricks Data Pipelines 101

Confessions of a Data Guy

MARCH 29, 2024

Have you ever wondered at a high level what it’s like to build production-level data pipelines on Databricks? The post Building Databricks Data Pipelines 101 appeared first on Confessions of a Data Guy. What does it look like, what tools do you use?

Data Pipeline

Data Pipeline Building Data IT

Scale Your Spatial Analysis By Building It In SQL With Syntax Extensions

Data Engineering Podcast

FEBRUARY 6, 2022

In order to make geospatial analytics more maintainable and scalable there has been an increase in the number of database engines that provide extensions to their SQL syntax that supports manipulation of spatial data. Once you’re up and running, your smart data pipelines are resilient to data drift.

SQL

SQL Building IT Google Cloud

Data Engineer vs Data Analyst: Key Differences and Similarities

Knowledge Hut

MAY 3, 2023

With companies increasingly relying on data-driven insights to make informed decisions, there has never been a greater need for skilled specialists who can manage and evaluate vast amounts of data. The roles of data analyst and data engineer have emerged as two of the most in-demand professions in today's job market.

Data Engineering

Data Engineering Data Engineer Engineering Data Cleanse

Cloud Native Data Orchestration For Machine Learning And Data Engineering With Flyte

Data Engineering Podcast

MAY 22, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Machine Learning

Machine Learning Data Engineering Data Engineer Cloud

Ship Smarter Not Harder With Declarative And Collaborative Data Orchestration On Dagster+

Data Engineering Podcast

MARCH 24, 2024

In this episode Pete Hunt, CEO of Dagster labs, outlines these new capabilities, how they reduce the burden on data teams, and the increased collaboration that they enable across teams and business units. Data lakes are notoriously complex. Go to dataengineeringpodcast.com/dagster today to get started. Your first 30 days are free!

Data Lake

Data Lake High Quality Data Hadoop Data Pipeline

Introducing Self-Service, No-Code Airflow Authoring UI in Cloudera Data Engineering

Cloudera

OCTOBER 19, 2021

Airflow has been adopted by many Cloudera Data Platform (CDP) customers in the public cloud as the next generation orchestration service to setup and operationalize complex data pipelines. We started out by interviewing customers to understand where the most friction exists in their pipeline development workflows today.

Coding

Coding Data Engineering Data Engineer Engineering

Establish A Single Source Of Truth For Your Data Consumers With A Semantic Layer

Data Engineering Podcast

APRIL 7, 2024

Summary Maintaining a single source of truth for your data is the biggest challenge in data engineering. Different roles and tasks in the business need their own ways to access and analyze the data in the organization. Dagster offers a new approach to building and running data platforms and data pipelines.

Data Lake

Data Lake High Quality Data BI Data Workflow

Find Out About The Technology Behind The Latest PFAD In Analytical Database Development

Data Engineering Podcast

FEBRUARY 25, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Go to dataengineeringpodcast.com/dagster today to get started.

Database

Database Technology Data Lake High Quality Data

When And How To Conduct An AI Program

Data Engineering Podcast

MARCH 3, 2024

Colleen Tartow has worked across all stages of the data lifecycle, and in this episode she shares her hard-earned wisdom about how to conduct an AI program for your organization. Data lakes are notoriously complex. Visit dataengineeringpodcast.com/data-council and use code dataengpod20 to register today!

Programming

Programming Data Lake High Quality Data Data Pipeline

The Chaos Data Engineering Manifesto

Towards Data Science

FEBRUARY 24, 2023

The Chaos Data-Engineering Manifesto Another lesson we can learn from software engineers: break stuff to make it more reliable. While this idea isn’t completely foreign to data engineering, it can certainly be described as an extremely uncommon practice. Data is different. It’s terrifying. I’m afraid so.

Data Engineering

Data Engineering Data Engineer Engineering Software Engineer

Data Engineering Weekly #179

Data Engineering Weekly

JULY 7, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. The blog explains how we can programmatically measure the real cost of Azure databricks SQL warehouse instances.

Data Engineering

Data Engineering Data Engineer Engineering Data Lake

Designing A Non-Relational Database Engine

Data Engineering Podcast

APRIL 14, 2024

In this episode Oren Eini, CEO and creator of RavenDB, explores the nuances of relational vs. non-relational engines, and the strategies for designing a non-relational database. Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex.

Non-relational Database

Non-relational Database Relational Database Database Designing

A Reflection On Learning A Lot More Than 97 Things Every Data Engineer Should Know

Data Engineering Podcast

JANUARY 30, 2022

Summary The Data Engineering Podcast has been going for five years now and has included conversations and interviews with a huge number of guests, covering a broad range of topics. In this episode he shares some reflections on producing the podcast, compiling the book, and relevant trends in the ecosystem of data engineering.

Data Engineering

Data Engineering Data Engineer Engineering Data Pipeline

Build Your Second Brain One Piece At A Time

Data Engineering Podcast

APRIL 28, 2024

In this episode he explains the data collection and preparation process, the collection of model types and sizes that work together to power the experience, and how to incorporate it into your workflow to act as a second brain. Data lakes are notoriously complex. Go to dataengineeringpodcast.com/dagster today to get started.

Building

Building Data Lake High Quality Data Machine Learning

Data Sharing Across Business And Platform Boundaries

Data Engineering Podcast

FEBRUARY 11, 2024

In this episode Andrew Jefferson explains the complexities of building a robust system for data sharing, the techno-social considerations, and how the Bobsled platform that he is building aims to simplify the process. Dagster offers a new approach to building and running data platforms and data pipelines.

Data Lake

Data Lake High Quality Data Government Data Pipeline

Data Engineering Weekly #198

Data Engineering Weekly

NOVEMBER 24, 2024

Editor’s Note: Launching Data & Gen-AI courses in 2025 I can’t believe DEW will reach almost its 200th edition soon. What I started as a fun hobby has become one of the top-rated newsletters in the data engineering industry. The blog narrates a few examples of Pipe Syntax in comparison with the SQL queries.

Data Engineering

Data Engineering Data Engineer Engineering Insurance

Reconciling The Data In Your Databases With Datafold

Data Engineering Podcast

MARCH 17, 2024

Validating that the information is stored and processed correctly can be complex and time-consuming, especially when the source and destination speak different dialects of SQL. Data lakes are notoriously complex. Visit dataengineeringpodcast.com/data-council and use code dataengpod20 to register today!

Database

Database Data Lake High Quality Data Data Workflow

Data Engineering Weekly #173

Data Engineering Weekly

MAY 26, 2024

[link] Tweeq: Tweeq Data Platform: Journey and Lessons Learned: Clickhouse, dbt, Dagster, and Superset Tweeq writes about its journey of building a data platform with cloud-agnostic open-source solutions and some integration challenges. and why one prefer dataframe over sql.

Data Engineering

Data Engineering Data Engineer Engineering Google Cloud

10 DataOps Principles for Overcoming Data Engineer Burnout

DataKitchen

NOVEMBER 18, 2021

Add all these facts together, and it paints a picture that something is amiss in the data world. . Yet, among all this, one area that hasn’t been studied is the data engineering role. We thought it would be interesting to look at how data engineers are doing under these circumstances. Blaming and finger-pointing.

Data Engineering

Data Engineering Data Engineer Engineering Government

Barking Up The Wrong GPTree: Building Better AI With A Cognitive Approach

Data Engineering Podcast

MAY 5, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Go to dataengineeringpodcast.com/dagster today to get started.

Building

Building Data Lake High Quality Data Machine Learning

An Exploration Of The Data Engineering Requirements For Bioinformatics

Data Engineering Podcast

SEPTEMBER 19, 2021

In this episode Jillian Rowe shares her experience of working in the field and supporting teams of scientists and analysts with the data infrastructure that they need to get their work done. This is a fascinating exploration of the collaboration between data professionals and scientists. If this resonates with you, you’re not alone.

Data Engineering

Data Engineering Data Engineer Engineering Data Lake

Data Scientist vs Data Engineer: Differences and Why You Need Both

AltexSoft

OCTOBER 30, 2021

If you’re an executive who has a hard time understanding the underlying processes of data science and get confused with terminology, keep reading. We will try to answer your questions and explain how two critical data jobs are different and where they overlap. Data science vs data engineering.

Data Engineering

Data Engineering Data Engineer Engineering Machine Learning

Enhancing The Abilities Of Software Engineers With Generative AI At Tabnine

Data Engineering Podcast

NOVEMBER 12, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Software Engineer

Software Engineer Software Engineering Engineering Data Lake

Top 12 Data Engineering Project Ideas [With Source Code]

Knowledge Hut

JUNE 26, 2023

Welcome to the world of data engineering, where the power of big data unfolds. If you're aspiring to be a data engineer and seeking to showcase your skills or gain hands-on experience, you've landed in the right spot. What are Data Engineering Projects?

Data Engineering

Data Engineering Data Engineer Coding Project

Pushing The Limits Of Scalability And User Experience For Data Processing WIth Jignesh Patel

Data Engineering Podcast

JANUARY 7, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. What are the open questions today in technical scalability of data engines? What are the open questions today in technical scalability of data engines?

Data Process

Data Process Process Data Lake High Quality Data

Version Your Data Lakehouse Like Your Software With Nessie

Data Engineering Podcast

MARCH 10, 2024

In this episode Alex Merced explains how the branching and merging functionality in Nessie allows you to use the same versioning semantics for your data lakehouse that you are used to from Git. Data lakes are notoriously complex. Visit dataengineeringpodcast.com/data-council and use code dataengpod20 to register today!

Data Lake

Data Lake High Quality Data Data Pipeline Architecture

Seamless SQL And Python Transformations For Data Engineers And Analysts With SQLMesh

Snowflake’s New Python API Empowers Data Engineers to Build Modern Data Pipelines with Ease

Tackling Real Time Streaming Data With SQL Using RisingWave

Webinars

How to improve at SQL as a data engineer

Streaming Data Pipelines Made SQL With Decodable

Using Trino And Iceberg As The Foundation Of Your Data Lakehouse

Ready-to-go sample data pipelines with Dataflow

Unpacking The Seven Principles Of Modern Data Pipelines

Troubleshooting Kafka In Production

How to learn data engineering

10 Skills to Ace Your Data Engineering Interviews

Our First Netflix Data Engineering Summit

Data Engineering Weekly #175

How Data Engineering Teams Power Machine Learning With Feature Platforms

Cloudera Data Engineering 2021 Year End Review

Building Data Pipelines That Run From Source To Analysis And Activation With Hevo Data

Making Email Better With AI At Shortwave

Simplified End-to-End Development for Production-Ready Data Pipelines, Applications, and ML Models

Building Databricks Data Pipelines 101

Scale Your Spatial Analysis By Building It In SQL With Syntax Extensions

Data Engineer vs Data Analyst: Key Differences and Similarities

Cloud Native Data Orchestration For Machine Learning And Data Engineering With Flyte

Ship Smarter Not Harder With Declarative And Collaborative Data Orchestration On Dagster+

Introducing Self-Service, No-Code Airflow Authoring UI in Cloudera Data Engineering

Establish A Single Source Of Truth For Your Data Consumers With A Semantic Layer

Find Out About The Technology Behind The Latest PFAD In Analytical Database Development

When And How To Conduct An AI Program

The Chaos Data Engineering Manifesto

Data Engineering Weekly #179

Designing A Non-Relational Database Engine

A Reflection On Learning A Lot More Than 97 Things Every Data Engineer Should Know

Build Your Second Brain One Piece At A Time

Data Sharing Across Business And Platform Boundaries

Data Engineering Weekly #198

Reconciling The Data In Your Databases With Datafold

Data Engineering Weekly #173

10 DataOps Principles for Overcoming Data Engineer Burnout

Barking Up The Wrong GPTree: Building Better AI With A Cognitive Approach

An Exploration Of The Data Engineering Requirements For Bioinformatics

Data Scientist vs Data Engineer: Differences and Why You Need Both

Enhancing The Abilities Of Software Engineers With Generative AI At Tabnine

Top 12 Data Engineering Project Ideas [With Source Code]

Pushing The Limits Of Scalability And User Experience For Data Processing WIth Jignesh Patel

Version Your Data Lakehouse Like Your Software With Nessie

Stay Connected