Cloudera

JULY 13, 2021

After the launch of CDP Data Engineering (CDE) on AWS a few months ago, we are thrilled to announce that CDE, the only cloud-native service purpose built for enterprise data engineers, is now available on Microsoft Azure. . Prerequisites for deploying CDP Data Engineering on Azure can be found here.

Data Engineer 125

Data Engineer Data Engineering Engineering Pipeline-centric 125

Ascend.io

MAY 31, 2023

In the modern world of data engineering, two concepts often find themselves in a semantic tug-of-war: data pipeline and ETL. Fast forward to the present day, and we now have data pipelines. Data Ingestion Data ingestion is the first step of both ETL and data pipelines.

Data Pipeline 52

Data Pipeline ETL Tools Pipeline-centric Data Warehouse 52

Ascend.io

AUGUST 29, 2023

A star-studded baseball team is analogous to an optimized “end-to-end data pipeline” — both require strategy, precision, and skill to achieve success. Just as every play and position in baseball is key to a win, each component of a data pipeline is integral to effective data management.

Data Pipeline 52

Data Pipeline Pipeline-centric Database-centric Data Ingestion 52

Data Engineering Podcast

APRIL 7, 2019

Summary How much time do you spend maintaining your data pipeline? This was a fascinating conversation with someone who has spent his entire career working on simplifying complex data problems. How does the data-centric approach of DataCoral differ from the way that other platforms think about processing information?

Data Pipeline 100

Data Pipeline Pipeline-centric Database-centric AWS 100

Monte Carlo

MARCH 28, 2024

In recent years, we’ve seen all sorts of new job titles emerge that would have been inscrutable just a decade or two ago – cloud architect, data reliability engineer , data product manager , director of hybrid working, and yes, DataOps engineer. So what exactly IS a DataOps engineer? It depends!

Engineering 75

Engineering Pipeline-centric BI Google Cloud 75

Knowledge Hut

JUNE 20, 2023

A novice data scientist prepared to start a rewarding journey may need clarification on the differences between a data scientist and a machine learning engineer. Many people are learning data science for the first time and need help comprehending the two job positions.

Machine Learning 52

Machine Learning Data Engineer Data Engineering Engineering 52

DareData

JULY 29, 2020

One paper suggests that there is a need for a re-orientation of the healthcare industry to be more "patient-centric". Furthermore, clean and accessible data, along with data driven automations, can assist medical professionals in taking this patient-centric approach by freeing them from some time-consuming processes.

Healthcare 52

Healthcare Data Pipeline Medical Pipeline-centric 52

Data Engineering Weekly

JULY 2, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork, and SQL grunt work out of building complete customer profiles, so you can quickly ship actionable, enriched data to every downstream team. So, let's shape the future of Data Engineering together. Should you?

Data Engineer 52

Data Engineer Data Engineering Engineering Pipeline-centric 52

Knowledge Hut

NOVEMBER 2, 2023

For those aspiring to build a career within the Azure ecosystem, navigating the choices between Azure Data Engineers and Azure DevOps Engineers can be quite challenging. Azure Data Engineers and Azure DevOps Engineers are two critical components of the Azure ecosystem for different but interconnected reasons.

Data Engineer 52

Data Engineer Data Engineering Engineering Pipeline-centric 52

Data Engineering Podcast

MAY 7, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. However, that's also something we're re-thinking with our warehouse-centric strategy.

Pipeline-centric 130

Pipeline-centric Data Lake Machine Learning Data Warehouse 130

Christophe Blefari

SEPTEMBER 13, 2024

NVidia released Eagle a vision-centric multimodal LLM — Look at the example in the Github repo, given an image and a user input the LLM is able to answer things like "Describe the image in detail" or "Which car in the picture is more aerodynamic" based on a drawing.

Pipeline-centric 100

Pipeline-centric Data Python Data Science 100

Snowflake

APRIL 8, 2024

The list of Top 10 semi-finalists is a perfect example: we have use cases for cybersecurity, gen AI, food safety, restaurant chain pricing, quantitative trading analytics, geospatial data, sales pipeline measurement, marketing tech and healthcare. Our sincere thanks go out to everyone who participated in this year’s competition.

Pipeline-centric 128

Pipeline-centric Food Healthcare Unstructured Data 128

Data Engineering Weekly

APRIL 16, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make collecting data from every application, website, and SaaS platform easy, then activating it in your warehouse and business tools. 🍒 We'll send exclusive The Data Stack Show swag just for participating!

Data Engineer 52

Data Engineer Data Engineering Engineering Pipeline-centric 52

Knowledge Hut

NOVEMBER 17, 2023

The demand for data-related professions, including data engineering, has indeed been on the rise due to the increasing importance of data-driven decision-making in various industries. Becoming an Azure Data Engineer in this data-centric landscape is a promising career choice.

Data Engineer 52

Data Engineer Data Engineering Engineering Scala 52

Ascend.io

JUNE 8, 2023

Treating data as a product is more than a concept; it’s a paradigm shift that can significantly elevate the value that business intelligence and data-centric decision-making have on the business. Data pipelines Data integrity Data lineage Data stewardship Data catalog Data product costing Let’s review each one in detail.

Pipeline-centric 52

Pipeline-centric Database-centric Data Ingestion Data Pipeline 52

Ascend.io

MAY 7, 2024

to bring its cutting-edge automation platform that revolutionizes modern data engineering. This platform, combined with Wizeline’s deep AI expertise, provides a powerful foundation and offering for smarter, faster, and more secure data operations. “Our collaboration with Ascend.io ” The partnership enables Ascend.io

Pipeline-centric 52

Pipeline-centric Data Cleanse Data Security Data Pipeline 52

Ascend.io

SEPTEMBER 14, 2023

The rise of data-intensive operations has positioned data engineering at the core of today’s organizations. As the demand to efficiently collect, process, and store data increases, data engineers have started to rely on Python to meet this escalating demand. Why Python for Data Engineering?

Data Engineer 52

Data Engineer Data Engineering Python Engineering 52

Grouparoo

OCTOBER 6, 2021

It was a fun experience and I think we made a good choice by picking 97 Things Every Data Engineer Should Know. This provided a nice overview of the breadth of topics that are relevant to data engineering including data warehouses/lakes, pipelines, metadata, security, compliance, quality, and working with other teams.

Data Engineer 52

Data Engineer Data Engineering Engineering Pipeline-centric 52

Monte Carlo

MAY 30, 2024

Generative AI has the potential to transform your business and your data engineering team, but only when it’s done right. So how can your data team actually drive value with your LLM or GenAI initiative? But when it’s done right, RAG can add an incredible amount of value to AI-powered data products. What is Fine Tuning?

Pipeline-centric 52

Pipeline-centric Database-centric Datasets Data Pipeline 52

Data Engineering Weekly

JANUARY 8, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Pipelines for data in motion can quickly turn into DAG hell.

Data Engineer 40

Data Engineer Data Engineering Engineering Pipeline-centric 40

Ascend.io

OCTOBER 20, 2023

The Nuances of Snowflake Costing Snowflake’s pricing strategy is an exemplification of its user-centric approach: pay for what you use. The Predictability of Pipelines In stark contrast to ad-hoc queries, pipelines are where cost optimization efforts can yield significant dividends.

Pipeline-centric 52

Pipeline-centric IT Data Pipeline Bytes 52

Data Engineering Podcast

AUGUST 13, 2022

Summary Data engineers have typically left the process of data labeling to data scientists or other roles because of its nature as a manual and process heavy undertaking, focusing instead on building automation and repeatable systems. Data stacks are becoming more and more complex.

Machine Learning 100

Machine Learning Pipeline-centric Database-centric MongoDB 100

Dataquest

JANUARY 25, 2017

But what about data engineers? A data scientist is only as good as the data they have access to. Most companies store their data in variety of formats across databases and text files. This is where data engineers come in — they build pipelines that transform that data into formats that data scientists can use.

Data Engineer 52

Data Engineer Data Engineering Pipeline-centric Database-centric 52

DataKitchen

NOVEMBER 5, 2024

By storing data in its native state in cloud storage solutions such as AWS S3, Google Cloud Storage, or Azure ADLS, the Bronze layer preserves the full fidelity of the data. We have also seen a fourth layer, the Platinum layer , in companies’ proposals that extend the Data pipeline to OneLake and Microsoft Fabric.

Architecture 79

Architecture Raw Data Pipeline-centric Data Ingestion 79

Christophe Blefari

APRIL 8, 2023

The only normalisation I did was back at the engineering school while learning SQL with Normal Forms. Actually what I cared was physical storage, data formats, logical partitioning or indexing. I still firmly believe that this is not the role of a data engineer. Data modeling should not be a required data engineer skill.

Pipeline-centric 130

Pipeline-centric Database-centric Algorithm Data 130

Christophe Blefari

APRIL 8, 2023

The only normalisation I did was back at the engineering school while learning SQL with Normal Forms. Actually what I cared was physical storage, data formats, logical partitioning or indexing. I still firmly believe that this is not the role of a data engineer. Data modeling should not be a required data engineer skill.

Pipeline-centric 130

Pipeline-centric Database-centric Algorithm Data 130

Cloudera

DECEMBER 13, 2023

To enable LGIM to better utilize its wealth of data, LGIM required a centralized platform that made internal data discovery easy for all teams and could securely integrate external partners and third-party outsourced data pipelines.

Pipeline-centric 87

Pipeline-centric Professional Services Datasets BI 87

DataKitchen

FEBRUARY 21, 2023

On 24 January 2023, Gartner released the article “ 5 Ways to Enhance Your Data Engineering Practices.” Data team morale is consistent with DataKitchen’s own research. We surveyed 600 data engineers , including 100 managers, to understand how they are faring and feeling about the work that they are doing.

Data Engineer 52

Data Engineer Data Engineering Engineering Pipeline-centric 52

Ascend.io

JUNE 21, 2023

Cost Savings : Insights derived from data products can directly lead to cost savings in the business. Risk Reduction : Data-centric decision-making can also contribute to risk reduction. In the case of data products, these are networks of data pipelines, which makes them an integral part of your modern operational machinery.

Pipeline-centric 52

Pipeline-centric Data Pipeline Metadata Data 52

Snowflake

JUNE 28, 2023

At Snowflake, we’re helping data scientists, data engineers, and application developers build faster and more efficiently in the Data Cloud. The end-to-end process requires collaboration across many data, engineering and business teams in order to capitalize on the value that can be captured from AI-powered insights.

Pipeline-centric 59

Pipeline-centric Programming Language Python Government 59

Netflix Tech

JANUARY 10, 2024

The Netflix video processing pipeline went live with the launch of our streaming service in 2007. By integrating with studio content systems, we enabled the pipeline to leverage rich metadata from the creative side and create more engaging member experiences like interactive storytelling.

Process 93

Process Pipeline-centric Media Metadata 93

Pipeline Data Engineering

MARCH 2, 2022

Data engineering salon. News and interesting reads about the world of data. The Unbundling of Airflow Gorkem Yurtseven, Co-Founder, Features and Labels A diverse set of tools is unbundling Airflow and this diversity is causing substantial fragmentation in modern data stack. What the hell happened?

Pipeline-centric 52

Pipeline-centric BI Consulting Data Warehouse 52

Ripple Engineering

JANUARY 29, 2024

  A lack of a centralized system makes building a single source of high-quality data difficult. The key aspect of any business-centric team in delivering products and features is to make critical decisions on ensuring low latency, high throughput, cost-effective storage, and highly efficient infrastructure.

Database-centric 52

Database-centric Pipeline-centric NoSQL High Quality Data 52

François Nguyen

MARCH 22, 2021

TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. The next problem will be the diversity of these mini data platforms (because of the configuration) and you even go deeper in problems with managing different technologies or version.

Technology 130

Technology Architecture Google Cloud Metadata 130

Snowflake

JUNE 28, 2023

Since its launch two years ago, Snowpark has been empowering data scientists, data engineers, and application developers to streamline their architectures, accelerate development, and boost the performance of data engineering and ML/AI workloads on Snowflake.

Python 52

Python Accessible Accessibility Pipeline-centric 52

ProjectPro

DECEMBER 16, 2021

Are you a newbie in the data science domain ready to embark on a rewarding journey but are confused between the roles of a Machine Learning Engineer vs Data Scientist? Read this article to understand the significant differences and similarities between a machine learning engineer and a data scientist.

Machine Learning 52

Machine Learning Engineering Pipeline-centric Database-centric 52

The Modern Data Company

DECEMBER 29, 2022

This means moving beyond product-centric thinking to a data-driven customer experience model that’s consistent across all channels. Next, the wealth management industry is also shifting away from a product focus to a client-centric model. DataOS is the world’s first operating system.

Banking 52

Banking Pipeline-centric Building Finance 52