Data Engineering Weekly

MARCH 3, 2024

Editor’s Note: Chennai, India Meetup - March-08 Update We are thankful to Ideas2IT to host our first Data Hero’s meetup. There will be food, networking, and real-world talks around data engineering. The article concludes with a look at data contracts as a concrete example of these principles in practice.

Data Engineering 67

Data Engineering Data Engineer Engineering Pipeline-centric 67

Ascend.io

MAY 31, 2023

In the modern world of data engineering, two concepts often find themselves in a semantic tug-of-war: data pipeline and ETL. Fast forward to the present day, and we now have data pipelines. Data Ingestion Data ingestion is the first step of both ETL and data pipelines.

Data Pipeline 52

Data Pipeline ETL Tools Pipeline-centric Data Warehouse 52

Ascend.io

AUGUST 29, 2023

A star-studded baseball team is analogous to an optimized “end-to-end data pipeline” — both require strategy, precision, and skill to achieve success. Just as every play and position in baseball is key to a win, each component of a data pipeline is integral to effective data management.

Data Pipeline 52

Data Pipeline Pipeline-centric Database-centric Data Ingestion 52

Data Engineering Weekly

JULY 2, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork, and SQL grunt work out of building complete customer profiles, so you can quickly ship actionable, enriched data to every downstream team. So, let's shape the future of Data Engineering together.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Data Engineering Podcast

APRIL 7, 2019

Summary How much time do you spend maintaining your data pipeline? This was a fascinating conversation with someone who has spent his entire career working on simplifying complex data problems. How does the data-centric approach of DataCoral differ from the way that other platforms think about processing information?

Data Pipeline 100

Data Pipeline Pipeline-centric Database-centric AWS 100

Knowledge Hut

NOVEMBER 2, 2023

For those aspiring to build a career within the Azure ecosystem, navigating the choices between Azure Data Engineers and Azure DevOps Engineers can be quite challenging. Azure Data Engineers and Azure DevOps Engineers are two critical components of the Azure ecosystem for different but interconnected reasons.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Knowledge Hut

JUNE 20, 2023

Let's find out the differences between a data scientist and a machine learning engineer below to make an informative decision. Data Engineer vs Machine Learning Engineer While there are similarities between a data engineer and a machine learning engineer, both play a key role in the technological world.

Machine Learning 52

Machine Learning Data Engineering Data Engineer Engineering 52

Data Engineering Weekly

APRIL 16, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make collecting data from every application, website, and SaaS platform easy, then activating it in your warehouse and business tools. 🍒 We'll send exclusive The Data Stack Show swag just for participating!

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Data Engineering Podcast

MAY 7, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. However, that's also something we're re-thinking with our warehouse-centric strategy.

Pipeline-centric 130

Pipeline-centric Data Lake Machine Learning Data Warehouse 130

DareData

JULY 29, 2020

One paper suggests that there is a need for a re-orientation of the healthcare industry to be more "patient-centric". Furthermore, clean and accessible data, along with data driven automations, can assist medical professionals in taking this patient-centric approach by freeing them from some time-consuming processes.

Data Pipeline 52

Data Pipeline Healthcare Medical Pipeline-centric 52

Knowledge Hut

NOVEMBER 17, 2023

The demand for data-related professions, including data engineering, has indeed been on the rise due to the increasing importance of data-driven decision-making in various industries. Becoming an Azure Data Engineer in this data-centric landscape is a promising career choice.

Data Engineering 52

Data Engineering Data Engineer Engineering Scala 52

Ascend.io

SEPTEMBER 14, 2023

The rise of data-intensive operations has positioned data engineering at the core of today’s organizations. As the demand to efficiently collect, process, and store data increases, data engineers have started to rely on Python to meet this escalating demand. Why Python for Data Engineering?

Data Engineering 52

Data Engineering Data Engineer Python Engineering 52

Christophe Blefari

SEPTEMBER 13, 2024

NVidia released Eagle a vision-centric multimodal LLM — Look at the example in the Github repo, given an image and a user input the LLM is able to answer things like "Describe the image in detail" or "Which car in the picture is more aerodynamic" based on a drawing.

Pipeline-centric 100

Pipeline-centric Data Python Data Science 100

Data Engineering Weekly

JANUARY 8, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Pipelines for data in motion can quickly turn into DAG hell.

Data Engineering 40

Data Engineering Data Engineer Engineering Pipeline-centric 40

Grouparoo

OCTOBER 6, 2021

It was a fun experience and I think we made a good choice by picking 97 Things Every Data Engineer Should Know. This provided a nice overview of the breadth of topics that are relevant to data engineering including data warehouses/lakes, pipelines, metadata, security, compliance, quality, and working with other teams.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Ascend.io

MAY 7, 2024

to bring its cutting-edge automation platform that revolutionizes modern data engineering. This platform, combined with Wizeline’s deep AI expertise, provides a powerful foundation and offering for smarter, faster, and more secure data operations. “Our collaboration with Ascend.io ” The partnership enables Ascend.io

Pipeline-centric 52

Pipeline-centric Data Cleanse Data Security Data Pipeline 52

Ascend.io

JUNE 8, 2023

Treating data as a product is more than a concept; it’s a paradigm shift that can significantly elevate the value that business intelligence and data-centric decision-making have on the business. Data pipelines Data integrity Data lineage Data stewardship Data catalog Data product costing Let’s review each one in detail.

Pipeline-centric 52

Pipeline-centric Database-centric Data Ingestion Data Pipeline 52

Dataquest

JANUARY 25, 2017

Data scientists have become extremely sought after , and for good reason — a skilled data scientist can add incredible value to a business. But what about data engineers? A data scientist is only as good as the data they have access to. A data engineer transforms data into a useful format for analysis.

Data Engineering 52

Data Engineering Data Engineer Pipeline-centric Database-centric 52

Data Engineering Podcast

AUGUST 13, 2022

Summary Data engineers have typically left the process of data labeling to data scientists or other roles because of its nature as a manual and process heavy undertaking, focusing instead on building automation and repeatable systems. Data stacks are becoming more and more complex.

Machine Learning 100

Machine Learning Pipeline-centric Database-centric MongoDB 100

DataKitchen

FEBRUARY 21, 2023

On 24 January 2023, Gartner released the article “ 5 Ways to Enhance Your Data Engineering Practices.” Data team morale is consistent with DataKitchen’s own research. We surveyed 600 data engineers , including 100 managers, to understand how they are faring and feeling about the work that they are doing.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

DataKitchen

NOVEMBER 5, 2024

By storing data in its native state in cloud storage solutions such as AWS S3, Google Cloud Storage, or Azure ADLS, the Bronze layer preserves the full fidelity of the data. We have also seen a fourth layer, the Platinum layer , in companies’ proposals that extend the Data pipeline to OneLake and Microsoft Fabric.

Architecture 79

Architecture Raw Data Pipeline-centric Data Ingestion 79

Monte Carlo

MARCH 28, 2024

As well as answering that question, we’ll be delving into how they differ from data scientists and data engineers. We’ll also be covering what they do day to day, what sort of salaries they command, and how individuals might go about pivoting to become a DataOps engineer. What does a DataOps engineer do?

Engineering 75

Engineering Pipeline-centric BI Google Cloud 75

Monte Carlo

MAY 30, 2024

Generative AI has the potential to transform your business and your data engineering team, but only when it’s done right. So how can your data team actually drive value with your LLM or GenAI initiative? But when it’s done right, RAG can add an incredible amount of value to AI-powered data products. What is Fine Tuning?

Pipeline-centric 52

Pipeline-centric Database-centric Datasets Data Pipeline 52

Christophe Blefari

APRIL 8, 2023

I still firmly believe that this is not the role of a data engineer. A data engineer should still be a software engineer working with data, empowering others with tooling and apps. Data modeling should not be a required data engineer skill. Enters the analytics engineer.

Pipeline-centric 130

Pipeline-centric Database-centric Algorithm Data 130

Christophe Blefari

APRIL 8, 2023

I still firmly believe that this is not the role of a data engineer. A data engineer should still be a software engineer working with data, empowering others with tooling and apps. Data modeling should not be a required data engineer skill. Enters the analytics engineer.

Pipeline-centric 130

Pipeline-centric Database-centric Algorithm Data 130

Cloudera

DECEMBER 13, 2023

To enable LGIM to better utilize its wealth of data, LGIM required a centralized platform that made internal data discovery easy for all teams and could securely integrate external partners and third-party outsourced data pipelines.

Pipeline-centric 87

Pipeline-centric Professional Services Datasets BI 87

Ascend.io

OCTOBER 20, 2023

The Nuances of Snowflake Costing Snowflake’s pricing strategy is an exemplification of its user-centric approach: pay for what you use. The Predictability of Pipelines In stark contrast to ad-hoc queries, pipelines are where cost optimization efforts can yield significant dividends.

Pipeline-centric 52

Pipeline-centric IT Data Pipeline Bytes 52

Pipeline Data Engineering

MARCH 2, 2022

Data engineering salon. News and interesting reads about the world of data. The Unbundling of Airflow Gorkem Yurtseven, Co-Founder, Features and Labels A diverse set of tools is unbundling Airflow and this diversity is causing substantial fragmentation in modern data stack. What the hell happened?

Pipeline-centric 52

Pipeline-centric BI Consulting Data Warehouse 52

Snowflake

JUNE 28, 2023

At Snowflake, we’re helping data scientists, data engineers, and application developers build faster and more efficiently in the Data Cloud. The end-to-end process requires collaboration across many data, engineering and business teams in order to capitalize on the value that can be captured from AI-powered insights.

Pipeline-centric 59

Pipeline-centric Programming Language Python Government 59

François Nguyen

MARCH 22, 2021

TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. The next problem will be the diversity of these mini data platforms (because of the configuration) and you even go deeper in problems with managing different technologies or version.

Technology 130

Technology Architecture Google Cloud Metadata 130

Snowflake

JUNE 28, 2023

Since its launch two years ago, Snowpark has been empowering data scientists, data engineers, and application developers to streamline their architectures, accelerate development, and boost the performance of data engineering and ML/AI workloads on Snowflake.

Python 52

Python Accessible Accessibility Pipeline-centric 52

Ascend.io

JUNE 21, 2023

Cost Savings : Insights derived from data products can directly lead to cost savings in the business. Risk Reduction : Data-centric decision-making can also contribute to risk reduction. In the case of data products, these are networks of data pipelines, which makes them an integral part of your modern operational machinery.

Pipeline-centric 52

Pipeline-centric Data Pipeline Metadata Data 52

Monte Carlo

JULY 15, 2024

For example, as a data owner in a retail company, your analysis of customer purchasing patterns could inform product development and marketing strategies. Career advancement: As organizations become more data-centric, your role as a data owner offers opportunities for career growth.

Pipeline-centric 52

Pipeline-centric Government Data Governance Retail 52

Towards Data Science

MAY 19, 2023

Business users are unable to find and access data assets critical to their workflows. Data engineers spend countless hours troubleshooting broken pipelines. The data team is constantly burning out and has a high employee turnover. Stakeholders fail to see the ROI behind expensive data initiatives.

Pipeline-centric 104

Pipeline-centric Data Software Engineer Software Engineering 104

Monte Carlo

NOVEMBER 14, 2023

This is the world that data orchestration tools aim to create. Data orchestration tools minimize manual intervention by automating the movement of data within data pipelines. According to one Redditor on r/dataengineering, “Seems like 99/100 data engineering jobs mention Airflow.”

Pipeline-centric 40

Pipeline-centric Google Cloud Python Data Workflow 40

Databand.ai

JULY 19, 2023

This helps to enhance data quality, facilitate data governance, and enable regulatory compliance. Notable Data Lineage Tools and Solutions Let’s take a look at several notable data lineage tools that can improve the quality and efficiency of your data pipeline.

Pipeline-centric 52

Pipeline-centric Data Governance Metadata Government 52

DataKitchen

AUGUST 27, 2021

It implemented hundreds of schema and data set changes per week without introducing errors. Arguably the most agile and effective data analytics capability in the pharmaceutical industry was accomplished cost-effectively, with a data engineering team of seven and another 10-12 data analysts.

Pharmaceutical 98

Pharmaceutical Pipeline-centric Data Analytics Data Lake 98