Data Engineering Podcast

MARCH 10, 2023

In this episode Priyendra Deshwal explains how NetSpring is designed to empower your product and data teams to build and explore insights around your products in a streamlined and maintainable workflow. As a listener to the Data Engineering Podcast you can get a special discount of 20% off your ticket by using the promo code dataengpod20.

Data Warehouse 130

Data Warehouse Data Lake Machine Learning Data Science 130

Data Engineering Weekly

JULY 21, 2024

Editor’s Note: A New Series on Data Engineering Tools Evaluation There are plenty of data tools and vendors in the industry. Data Engineering Weekly is launching a new series on software evaluation focused on data engineering to better guide data engineering leaders in evaluating data tools.

Data Engineer 117

Data Engineer Data Engineering Engineering Generalist 117

Seattle Data Guy

JANUARY 12, 2023

A data warehouse consultant plays an important role in companies looking to become data-driven. They help companies design and deploy centralized data sets that are easy to use and reliable. But in order to understand why you need a data warehouse consultant we should take a step back.

Consulting 130

Consulting Data Warehouse Data Designing 130

Data Engineering Podcast

FEBRUARY 5, 2023

In that time there have been a number of generational shifts in how data engineering is done. Sign up now for early access to Materialize and get started with the power of streaming data with the same simplicity and low implementation cost as batch cloud data warehouses.

Data Engineer 147

Data Engineer Data Engineering Engineering PostgreSQL 147

Data Engineering Podcast

JUNE 25, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. Support Data Engineering Podcast RudderStack also supports real-time use cases.

Data Engineer 225

Data Engineer Data Engineering Python Engineering 225

Towards Data Science

JANUARY 28, 2024

How I organize everything in my new data warehouse Continue reading on Towards Data Science »

Data Warehouse 64

Data Warehouse Designing Data Science Data 64

Start Data Engineering

MAY 11, 2022

Data lake structure 5. Loading user purchase data into the data warehouse 5.2 Loading classified movie review data into the data warehouse 5.3 Prerequisite 4.2 AWS Infrastructure costs 4.3 Code walkthrough 5.1 Generating user behavior metric 5.4. Checking results 6. Tear down infra 7. Next steps 9.

Data Engineer 246

Data Engineer Data Engineering Project Data Lake 246

Analytics Vidhya

APRIL 3, 2023

Real-time dashboards such as GCP provide strong data visualization and actionable information for decision-makers. Nevertheless, setting up a streaming data pipeline to power such dashboards may […] The post Data Engineering for Streaming Data on GCP appeared first on Analytics Vidhya.

Data Engineer 251

Data Engineer Data Engineering Engineering Data 251

Data Engineering Podcast

JUNE 4, 2023

Summary A significant portion of the time spent by data engineering teams is on managing the workflows and operations of their pipelines. Agile Data Engine is a platform designed to handle the infrastructure side of the DataOps equation, as well as providing the insights that you need to manage the human side of the workflow.

Data Engineer 130

Data Engineer Data Engineering Engineering Data Lake 130

Jesse Anderson

DECEMBER 12, 2022

Big data projects were given to data scientists and data warehouse teams, where the projects subsequently failed. As clearly evident as that sounds now, my writing about needing data engineering went heavily against the grain of everything that was written at the time. Some people blamed the technologies.

Data Engineer 130

Data Engineer Data Engineering Engineering Hadoop 130

Data Engineering Weekly

AUGUST 25, 2024

Try For Free → Conference Alert: Data Engineering for AI/ML This is a virtual conference at the intersection of Data and AI. Instead, you will see engineers, product managers, and founders talking about the data engineering practices of AI/ML. It is not a conference for the hype.

Data Engineer 70

Data Engineer Data Engineering Engineering Database-centric 70

Netflix Tech

JUNE 20, 2024

A summary of sessions at the first Data Engineering Open Forum at Netflix on April 18th, 2024 The Data Engineering Open Forum at Netflix on April 18th, 2024. At Netflix, we aspire to entertain the world, and our data engineering teams play a crucial role in this mission by enabling data-driven decision-making at scale.

Data Engineer 81

Data Engineer Data Engineering Engineering Data Warehouse 81

Data Engineering Podcast

MAY 13, 2021

Summary There is a lot of attention on the database market and cloud data warehouses. While they provide a measure of convenience, they also require you to sacrifice a certain amount of control over your data. Firebolt is the fastest cloud data warehouse. Visit dataengineeringpodcast.com/firebolt to get started.

PostgreSQL 100

PostgreSQL Data Warehouse Building MySQL 100

Seattle Data Guy

AUGUST 2, 2023

A few months ago, I uploaded a video where I discussed data warehouses, data lakes, and transactional databases. However, the world of data management is evolving rapidly, especially with the resurgence of AI and machine learning.

Data Lake 130

Data Lake Data Warehouse Data Machine Learning 130

Data Engineering Weekly

JUNE 10, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. Databricks and Snowflake offer a data warehouse on top of cloud providers like AWS, Google Cloud, and Azure.

Data Engineer 103

Data Engineer Data Engineering Engineering Kafka 103

Simon Späti

MARCH 9, 2021

This post focuses on practical data pipelines with examples from web-scraping real-estates, uploading them to S3 with MinIO, Spark and Delta Lake, adding some Data Science magic with Jupyter Notebooks, ingesting into Data Warehouse Apache Druid, visualising dashboards with Superset and managing everything with Dagster.

Data Engineer 189

Data Engineer Data Engineering Engineering Project 189

Hepta Analytics

FEBRUARY 24, 2022

Examples of such systems are: ATM Shopping cart MoMo ticket systems OLAP on the other hand is used for data analysis. Data is aggregated from multiple sources and used by the business team to make decisions. Data structure and schema is defined in advance. The main goal is to become the single source of truth in an organization.

Data Warehouse 52

Data Warehouse Data Engineer Data Engineering Engineering 52

Data Engineering Weekly

OCTOBER 6, 2024

The key benefits are Improved data quality, Enhanced data governance Increased security Cost efficiency [link] HomeToGo: How HomeToGo improved our Superset Monitoring Framework Apache Superset is the most popular open-source BI tool in the industry. “text-to-SQL” and “text-to-insight.”

Data Engineer 62

Data Engineer Data Engineering Engineering Data Warehouse 62

Start Data Engineering

OCTOBER 12, 2021

Introduction Responsibilities of a data engineer 1. Move data between systems 2. Manage data warehouse 3. Schedule, execute, and monitor data pipelines 4. Serve data to the end-users 5. Data strategy for the company 6.

Data Engineer 130

Data Engineer Data Engineering Engineering Data Warehouse 130

KDnuggets

OCTOBER 30, 2023

A comparative overview of data warehouses, data lakes, and data marts to help you make informed decisions on data storage solutions for your data architecture.

Data Lake 134

Data Lake Data Warehouse Data Storage Data 134

Towards Data Science

MAY 8, 2023

Data Warehouse Dimensional Modeling Design 101 Continue reading on Towards Data Science »

Data Warehouse 65

Data Warehouse Data Engineer Data Engineering Building 65

Data Engineering Podcast

JANUARY 1, 2022

In this episode Emily Riederer shares her work to create a controlled vocabulary for managing the semantic elements of the data managed by her team and encoding it in the schema definitions in her data warehouse. star/snowflake schema, data vault, etc.)

Data Warehouse 100

Data Warehouse BI Data Workflow Data Engineer 100

Monte Carlo

NOVEMBER 22, 2024

Your data engineering pipeline started simple: a few CSV exports, some Python scripts, and manual updates every week. You’re left wondering if there’s a breaking point where your DIY data solution won’t cut it anymore—and honestly, you might be there already. It means you’re scaling!

Data Engineer 52

Data Engineer Data Engineering Building Engineering 52

Acceldata

FEBRUARY 16, 2023

Why Data Engineers Must Manage Data Pipelines, Not Data Warehouses

Data Warehouse 52

Data Warehouse Data Pipeline Data Engineer Data Engineering 52

Cloudera

JULY 11, 2023

Apache Impala is a distributed C++ backed SQL engine that integrates with Kudu to serve BI results over millions of rows meeting sub-second service-level agreements. Cloudera offers Apache Kudu to run in Real Time DataMart Clusters , and Apache Impala to run in Kubernetes in the Cloudera Data Warehouse form factor.

Data Warehouse 69

Data Warehouse Pharmaceutical BI Banking 69

Data Engineering Podcast

FEBRUARY 8, 2021

In this episode Zeeshan Qureshi and Michelle Ark share their experiences using DBT to manage the data warehouse for Shopify. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. What kinds of data sources are you working with?

Data Warehouse 100

Data Warehouse Building BI SQL 100

Snowflake

NOVEMBER 2, 2023

Over the years, the technology landscape for data management has given rise to various architecture patterns, each thoughtfully designed to cater to specific use cases and requirements. These patterns include both centralized storage patterns like data warehouse , data lake and data lakehouse , and distributed patterns such as data mesh.

Data Lake 95

Data Lake Data Warehouse Cloud Unstructured Data 95

Data Engineering Weekly

MAY 26, 2024

The idea will infact open flexibility in programming data over the SQL-based data warehouse systems. A closer look at the OBTs Star Schema vs OBT (One Big Table) data modeling is another ongoing debate in data engineering practitioners. and why one prefer dataframe over sql.

Data Engineer 76

Data Engineer Data Engineering Engineering Google Cloud 76

Towards Data Science

JULY 26, 2024

Learn from Software Engineers and Discover the Joy of ‘Worse is Better’ Thinking source: unsplash.com Recently, I have had the fortune of speaking to a number of data engineers and data architects about the problems they face with data in their businesses. That was it. So, What the Heck is Radical Simplicity?

Data Engineer 57

Data Engineer Data Engineering Engineering Data Warehouse 57

Data Engineering Podcast

AUGUST 31, 2020

Summary Data warehouse technology has been around for decades and has gone through several generational shifts in that time. The current trends in data warehousing are oriented around cloud native architectures that take advantage of dynamic scaling and the separation of compute and storage.

Data Warehouse 100

Data Warehouse Cloud Building Data Lake 100

Towards Data Science

DECEMBER 4, 2023

A Glossary with Use Cases for First-Timers in Data Engineering An happy Data Engineer at work Are you a data engineering rookie interested in knowing more about modern data infrastructures? In this guide Data Engineering meets Formula 1. Data models are built around business needs.

Data Engineer 81

Data Engineer Data Engineering Engineering Data Lake 81

Data Engineering Weekly

SEPTEMBER 22, 2024

I’m fairly confident that a framework that supports a deterministic way of building a data pipeline and a conversational way of building business logic is the future of data pipeline engineering. link] Sponsored: IMPACT - Have you seen the agenda for IMPACT: The Data Observability Summit yet?

Data Engineer 70

Data Engineer Data Engineering Engineering Data Lake 70

Data Engineering Weekly

SEPTEMBER 2, 2024

Try For Free → Conference Alert: Data Engineering for AI/ML This is a virtual conference at the intersection of Data and AI. Instead, you will see engineers, product managers, and founders talking about the data engineering practices of AI/ML. It is not a conference for the hype.

Data Engineer 67

Data Engineer Data Engineering Engineering Entertainment 67

Monte Carlo

FEBRUARY 6, 2023

So, you’re planning a cloud data warehouse migration. But be warned, a warehouse migration isn’t for the faint of heart. As you probably already know if you’re reading this, a data warehouse migration is the process of moving data from one warehouse to another. A worthy quest to be sure.

Data Warehouse 52

Data Warehouse AWS Data Validation Data 52

Seattle Data Guy

DECEMBER 19, 2022

Whether its helping increase revenue by finding new customers or reducing costs, all of it starts with data.

Data Lake 130

Data Lake Data Warehouse Database Data Storage 130

Monte Carlo

OCTOBER 31, 2024

One job that has become increasingly popular across enterprise data teams is the role of the AI data engineer. Demand for AI data engineers has grown rapidly in data-driven organizations. But what does an AI data engineer do? Table of Contents What Does an AI Data Engineer Do?

Data Engineer 52

Data Engineer Data Engineering Engineering Unstructured Data 52

Towards Data Science

JUNE 6, 2023

The fact it’s insanely fast and does (mostly) all processing in memory make it a good choice for building my personal data warehouse. Fitbit activity data The first collection of files I looked at was activity data. Extra points for its seamless integration with Python and R.

Data Warehouse 88

Data Warehouse Data SQL Data Analysis 88