Cloudera

NOVEMBER 21, 2024

As advanced analytics and AI continue to drive enterprise strategy, leaders are tasked with building flexible, resilient data pipelines that accelerate trusted insights. A New Level of Productivity with Remote Access The new Cloudera Data Engineering 1.23 Why Cloudera Data Engineering?

Pipeline-centric 66

Pipeline-centric Data Engineer Data Engineering Engineering 66

Data Engineering Weekly

JUNE 24, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. Should we term this as an AI product curve of disappointment ?

Data Engineer 109

Data Engineer Data Engineering Engineering Google Cloud 109

Data Engineering Weekly

JUNE 10, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. As I have shared , its impact on data engineering is exciting.

Data Engineer 103

Data Engineer Data Engineering Engineering Kafka 103

Confessions of a Data Guy

JULY 24, 2024

We all keep hearing about Arrow this and Arrow that … seems every new tool built today for Data Engineering seems to be at least partly based on Arrow’s in-memory format. So, […] The post PyArrow vs Polars (vs DuckDB) for Data Pipelines. appeared first on Confessions of a Data Guy.

Data Pipeline 162

Data Pipeline Data Engineer Data Engineering Data 162

Data Engineering Weekly

JUNE 30, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. It is a good reminder to the data industry that we need to solve the fundamentals of data engineering to utilize AI better.

Data Engineer 102

Data Engineer Data Engineering Engineering Unstructured Data 102

Confessions of a Data Guy

SEPTEMBER 9, 2023

In the vast world of data, it’s not just about gathering and analyzing information anymore; it’s also about ensuring that data pipelines, processes, and platforms run seamlessly and efficiently.

Data Engineer 162

Data Engineer Data Engineering Engineering Data Pipeline 162

Data Engineering Podcast

JUNE 25, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. Rudderstack]([link] RudderStack provides all your customer data pipelines in one platform.

Data Engineer 225

Data Engineer Data Engineering Python Engineering 225

Netflix Tech

DECEMBER 14, 2023

Engineers from across the company came together to share best practices on everything from Data Processing Patterns to Building Reliable Data Pipelines. The result was a series of talks which we are now sharing with the rest of the Data Engineering community! In this video, Sr.

Data Engineer 105

Data Engineer Data Engineering Engineering Metadata 105

Data Engineering Weekly

AUGUST 25, 2024

Take Astro (the fully managed Airflow solution) for a test drive today and unlock a suite of features designed to simplify, optimize, and scale your data pipelines. Try For Free → Conference Alert: Data Engineering for AI/ML This is a virtual conference at the intersection of Data and AI.

Data Engineer 70

Data Engineer Data Engineering Engineering Database-centric 70

Monte Carlo

NOVEMBER 22, 2024

Your data engineering pipeline started simple: a few CSV exports, some Python scripts, and manual updates every week. You’re left wondering if there’s a breaking point where your DIY data solution won’t cut it anymore—and honestly, you might be there already. It means you’re scaling!

Data Engineer 52

Data Engineer Data Engineering Building Engineering 52

Towards Data Science

NOVEMBER 20, 2024

It is worth mentioning that this article comes from an Electrical and Electronic Engineer graduate who went all the way and spent almost 8 years in academia learning about the Energy sector (and when I say all the way, I mean from a bachelor degree to a PhD and postdoc). Similarly, data engineering positions have seen a 98% increase.

Data Engineer 61

Data Engineer Data Engineering Engineering Electronics 61

Data Engineering Weekly

NOVEMBER 3, 2024

Data Engineering Weekly readers get 15% discount by registering the following link, [link] Gustavo Akashi: Building data pipelines effortlessly with a DAG Builder for Apache Airflow Every code-first data workflow grew into a UI-based or Yaml-based workflow.

Data Engineer 62

Data Engineer Data Engineering Engineering Pipeline-centric 62

Data Engineering Weekly

SEPTEMBER 2, 2024

Take Astro (the fully managed Airflow solution) for a test drive today and unlock a suite of features designed to simplify, optimize, and scale your data pipelines. Try For Free → Conference Alert: Data Engineering for AI/ML This is a virtual conference at the intersection of Data and AI.

Data Engineer 67

Data Engineer Data Engineering Engineering Entertainment 67

Analytics Vidhya

FEBRUARY 6, 2023

Introduction The demand for data to feed machine learning models, data science research, and time-sensitive insights is higher than ever thus, processing the data becomes complex. To make these processes efficient, data pipelines are necessary. appeared first on Analytics Vidhya.

Amazon Web Services 218

Amazon Web Services Data Pipeline Machine Learning Data Science 218

Seattle Data Guy

FEBRUARY 11, 2023

Apache Airflow is a very popular tool that data engineers rely on. Why do data engineers like Airflow? What are… Read more The post What Is Apache Airflow – Data Engineering Consulting appeared first on Seattle Data Guy. Also, what does Apache Airflow event do? What is a DAG?

Consulting 147

Consulting Data Engineer Data Engineering Engineering 147

Data Engineering Podcast

JULY 2, 2023

In this episode Razi Raziuddin shares how data engineering teams can support the machine learning workflow through the development and support of systems that empower data scientists and ML engineers to build and maintain their own features. How is this distinct from other forms of data pipeline development and delivery?

Machine Learning 130

Machine Learning Data Engineer Data Engineering Engineering 130

Ascend.io

OCTOBER 8, 2024

Data engineering is the backbone of any data-driven organization, responsible for building and maintaining the infrastructure that supports data collection, storage, and analysis. Traditionally, data engineers have focused on the technical aspects of data management, ensuring data pipelines run smoothly and efficiently.

Data Engineer 52

Data Engineer Data Engineering Engineering Data Architecture 52

Data Engineering Weekly

SEPTEMBER 22, 2024

Uber: QueryGPT – Natural Language to SQL Using Generative AI Companies are increasingly adopting the natural language interface to interact with the enterprise data. link] Sponsored: IMPACT - Have you seen the agenda for IMPACT: The Data Observability Summit yet?

Data Engineer 70

Data Engineer Data Engineering Engineering Data Lake 70

Data Engineering Weekly

OCTOBER 20, 2024

link] Zach Wilson: How GenAI will impact data engineering An excellent overview of the potential impact of GenAI in data engineering workflow that can change the pipeline authoring, shipping pipeline, and maintaining the pipeline. I presume the typescript-to-js conversion helps here.

Data Engineer 52

Data Engineer Data Engineering Engineering Manufacturing 52

Data Engineering Weekly

AUGUST 4, 2024

Take Astro (the fully managed Airflow solution) for a test drive today and unlock a suite of features designed to simplify, optimize, and scale your data pipelines. link] Jason Liu & Eugene Yan: 10 Ways to Be Data Illiterate (and How to Avoid Them) If you ask any executives in an organization, they will say they are data-driven.

Data Engineer 80

Data Engineer Data Engineering Engineering Data 80

Knowledge Hut

JULY 5, 2024

Large enterprises heavily rely on data for informed decision-making, and this reliance is where data engineers step in. Data engineers like myself play a pivotal role in assessing infrastructure and taking relevant actions. Looking ahead, the future of data engineering appears promising.

Data Engineer 52

Data Engineer Data Engineering Engineering Data Cleanse 52

Monte Carlo

OCTOBER 31, 2024

One job that has become increasingly popular across enterprise data teams is the role of the AI data engineer. Demand for AI data engineers has grown rapidly in data-driven organizations. But what does an AI data engineer do? Table of Contents What Does an AI Data Engineer Do?

Data Engineer 52

Data Engineer Data Engineering Engineering Unstructured Data 52

Start Data Engineering

OCTOBER 12, 2021

Introduction Responsibilities of a data engineer 1. Move data between systems 2. Manage data warehouse 3. Schedule, execute, and monitor data pipelines 4. Serve data to the end-users 5. Data strategy for the company 6.

Data Engineer 130

Data Engineer Data Engineering Engineering Data Warehouse 130

Simon Späti

MARCH 9, 2021

This post focuses on practical data pipelines with examples from web-scraping real-estates, uploading them to S3 with MinIO, Spark and Delta Lake, adding some Data Science magic with Jupyter Notebooks, ingesting into Data Warehouse Apache Druid, visualising dashboards with Superset and managing everything with Dagster.

Data Engineer 189

Data Engineer Data Engineering Engineering Project 189

Data Engineering Weekly

NOVEMBER 11, 2024

dbt’s Jinja templating and non-deterministic macros Elimination of YAML Plan-Apply workflow [link] Astrafy: Dynamic Data Pipelines with Airflow Datasets and Pub/Sub Airflow is increasing support for data-aware scheduling as it takes a more prominent role in the upcoming releases.

Data Engineer 57

Data Engineer Data Engineering Engineering Datasets 57

Data Engineering Podcast

JUNE 4, 2023

Summary A significant portion of the time spent by data engineering teams is on managing the workflows and operations of their pipelines. Agile Data Engine is a platform designed to handle the infrastructure side of the DataOps equation, as well as providing the insights that you need to manage the human side of the workflow.

Data Engineer 130

Data Engineer Data Engineering Engineering Data Lake 130

Data Engineering Weekly

SEPTEMBER 15, 2024

During his keynote, he’ll share his insights into how data and pattern recognition shape the world, build trust, and—sometimes—even help us predict the future. The blog narrates how the lack of neutral guidance or standards leads to ambiguity and the formation of a central team to bring trust in data.

Data Engineer 67

Data Engineer Data Engineering Engineering Data Lake 67

Data Engineering Weekly

OCTOBER 6, 2024

The learning goes back to the fundamentals of pipeline design principles. Regularly review if pipelines are still required. Minimize the data used in pipelines, aka do incremental data pipeline design. Optimize pipeline schedules. Filter data effectively to make sure the query uses partition pruning.

Data Engineer 62

Data Engineer Data Engineering Engineering Data Warehouse 62

Start Data Engineering

FEBRUARY 22, 2024

Data Pipeline Logging Best Practices 3.1. Metadata: Information about pipeline runs, & data flowing through your pipeline 3.2. Introduction 2. Setup & Logging architecture 3. Obtain visibility into the code’s execution sequence using text logs 3.3. Understand resource usage by tracking Metrics 3.4.

Metadata 130

Metadata Data Engineer Data Engineering Engineering 130

Netflix Tech

JUNE 20, 2024

A summary of sessions at the first Data Engineering Open Forum at Netflix on April 18th, 2024 The Data Engineering Open Forum at Netflix on April 18th, 2024. At Netflix, we aspire to entertain the world, and our data engineering teams play a crucial role in this mission by enabling data-driven decision-making at scale.

Data Engineer 81

Data Engineer Data Engineering Engineering Data Warehouse 81

Data Engineering Weekly

MAY 26, 2024

[link] Tweeq: Tweeq Data Platform: Journey and Lessons Learned: Clickhouse, dbt, Dagster, and Superset Tweeq writes about its journey of building a data platform with cloud-agnostic open-source solutions and some integration challenges. It is refreshing to see an open stack after the Hadoop era.

Data Engineer 76

Data Engineer Data Engineering Engineering Google Cloud 76

Cloudera

JULY 13, 2021

After the launch of CDP Data Engineering (CDE) on AWS a few months ago, we are thrilled to announce that CDE, the only cloud-native service purpose built for enterprise data engineers, is now available on Microsoft Azure. . Prerequisites for deploying CDP Data Engineering on Azure can be found here.

Data Engineer 125

Data Engineer Data Engineering Engineering Pipeline-centric 125

Data Engineering Podcast

AUGUST 13, 2023

Summary Data pipelines are the core of every data product, ML model, and business intelligence dashboard. The folks at Rivery distilled the seven principles of modern data pipelines that will help you stay out of trouble and be productive with your data.

Data Pipeline 130

Data Pipeline BI SQL Business Intelligence 130

Start Data Engineering

OCTOBER 11, 2021

Data modeling 4.1 Data warehousing 4.2 Data pipelines 6. Probabilistic data structures (optional) Interview prep, the TL;DR version Conclusion Introduction Are you a student, analyst, engineer, or someone preparing for a data engineering interview and overwhelmed by all the tools and concepts?

Data Engineer 130

Data Engineer Data Engineering Engineering Cloud Computing 130

Data Engineering Weekly

JULY 7, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more.

Data Engineer 98

Data Engineer Data Engineering Engineering Data Lake 98

Data Engineering Weekly

JUNE 16, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more.

Data Engineer 104

Data Engineer Data Engineering Engineering Kafka 104

Netflix Tech

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty At Netflix, our Membership and Finance Data Engineering team harnesses diverse data related to plans, pricing, membership life cycle, and revenue to fuel analytics, power various dashboards, and make data-informed decisions.

Data Engineer 107

Data Engineer Data Engineering Engineering Metadata 107