Data Engineering Podcast

AUGUST 13, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Machine Learning 100

Machine Learning Pipeline-centric Database-centric MongoDB 100

Christophe Blefari

APRIL 8, 2023

At the same time Maxime Beauchemin wrote a post about Entity-Centric data modeling. In the recent years dbt simplified and revolutionised the tooling to create data models. This week I discovered SQLMesh , a all-in-one data pipelines tool. I hope he will fill the gaps. dbt, as of today, is the leading framework.

Pipeline-centric 130

Pipeline-centric Database-centric Algorithm Data 130

Christophe Blefari

APRIL 8, 2023

At the same time Maxime Beauchemin wrote a post about Entity-Centric data modeling. In the recent years dbt simplified and revolutionised the tooling to create data models. This week I discovered SQLMesh , a all-in-one data pipelines tool. I hope he will fill the gaps. dbt, as of today, is the leading framework.

Pipeline-centric 130

Pipeline-centric Database-centric Algorithm Data 130

Knowledge Hut

DECEMBER 26, 2023

Data Engineering is typically a software engineering role that focuses deeply on data – namely, data workflows, data pipelines, and the ETL (Extract, Transform, Load) process. What is the role of a Data Engineer? Data Engineers are skilled professionals who lay the foundation of databases and architecture.

Data Engineering 98

Data Engineering Data Engineer Engineering Hadoop 98

Ascend.io

AUGUST 29, 2023

A star-studded baseball team is analogous to an optimized “end-to-end data pipeline” — both require strategy, precision, and skill to achieve success. Just as every play and position in baseball is key to a win, each component of a data pipeline is integral to effective data management.

Data Pipeline 52

Data Pipeline Pipeline-centric Database-centric Data Ingestion 52

Ascend.io

MAY 31, 2023

In the modern world of data engineering, two concepts often find themselves in a semantic tug-of-war: data pipeline and ETL. Fast forward to the present day, and we now have data pipelines. Data Ingestion Data ingestion is the first step of both ETL and data pipelines.

Data Pipeline 52

Data Pipeline ETL Tools Pipeline-centric Data Warehouse 52

François Nguyen

MARCH 22, 2021

TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. With this 3rd platform generation, you have more real time data analytics and a cost reduction because it is easier to manage this infrastructure in the cloud thanks to managed services.

Technology 130

Technology Architecture Google Cloud Metadata 130

Data Engineering Weekly

AUGUST 25, 2024

Take Astro (the fully managed Airflow solution) for a test drive today and unlock a suite of features designed to simplify, optimize, and scale your data pipelines. Try For Free → Conference Alert: Data Engineering for AI/ML This is a virtual conference at the intersection of Data and AI.

Data Engineering 70

Data Engineering Data Engineer Engineering Database-centric 70

Dataquest

JANUARY 25, 2017

But what about data engineers? A data scientist is only as good as the data they have access to. Most companies store their data in variety of formats across databases and text files. This is where data engineers come in — they build pipelines that transform that data into formats that data scientists can use.

Data Engineering 52

Data Engineering Data Engineer Pipeline-centric Database-centric 52

Monte Carlo

MAY 30, 2024

Retrieval augmented generation (RAG) is an architecture framework introduced by Meta in 2020 that connects your large language model (LLM) to a curated, dynamic database. Data retrieval: Based on the query, the RAG system searches the database to find relevant data. A RAG flow in Databricks can be visualized like this.

Pipeline-centric 52

Pipeline-centric Database-centric Datasets Data Pipeline 52

DataKitchen

NOVEMBER 4, 2021

Your IT organization may have a permanent data lake, but data analytics teams need the ability to rapidly create insight from data. The DataKitchen Platform serves as a process hub that builds temporary analytic databases for daily and weekly ad hoc analytics work. Figure 3: Example process hub for biologic launch.

Process 98

Process Data Process Pharmaceutical Data Lake 98

Cloudera

NOVEMBER 15, 2021

In terms of representation, data can be broadly classified into two types: structured and unstructured. Structured data can be defined as data that can be stored in relational databases, and unstructured data as everything else. There are many implications of large unstructured data for engineering.

Unstructured Data 90

Unstructured Data Pipeline-centric Database-centric Entertainment 90

Data Engineering Weekly

MARCH 3, 2024

Here is the agenda, 1) Data Application Lifecycle Management - Harish Kumar( Paypal) Hear from the team in PayPal on how they build the data product lifecycle management (DPLM) systems. 3) DataOPS at AstraZeneca The AstraZeneca team talks about data ops best practices internally established and what worked and what didn’t work!!!

Data Engineering 67

Data Engineering Data Engineer Engineering Pipeline-centric 67

Data Engineering Weekly

JULY 2, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork, and SQL grunt work out of building complete customer profiles, so you can quickly ship actionable, enriched data to every downstream team. The author made an excellent attempt to describe what is vector databases to a 5-year-old.

Data Engineering 52

Data Engineering Data Engineer Engineering Database-centric 52

Knowledge Hut

JUNE 20, 2023

The generalist position would suit a data scientist looking for a transition into a data engineer. Pipeline-Centric Engineer: These data engineers prefer to serve in distributed systems and more challenging projects of data science with a midsize data analytics team.

Machine Learning 52

Machine Learning Data Engineering Data Engineer Engineering 52

Ascend.io

JUNE 8, 2023

Treating data as a product is more than a concept; it’s a paradigm shift that can significantly elevate the value that business intelligence and data-centric decision-making have on the business. Data pipelines Data integrity Data lineage Data stewardship Data catalog Data product costing Let’s review each one in detail.

Pipeline-centric 52

Pipeline-centric Database-centric Data Ingestion Data Pipeline 52

Knowledge Hut

FEBRUARY 27, 2023

Engineers work with Data Scientists to help make the most of the data they collect and have deep knowledge of distributed systems and computer science. In large organizations, data engineers concentrate on analytical databases, operate data warehouses that span multiple databases, and are responsible for developing table schemas.

Data Engineering 52

Data Engineering Data Engineer Database-centric Pipeline-centric 52

Edureka

JULY 3, 2024

ADF connects to various data sources, including on-premises systems, cloud services, and SaaS applications. It then gathers and relocates information to a centralized hub in the cloud using the Copy Activity within data pipelines. Transform and Enhance the Data: Once centralized, data undergoes transformation and enrichment.

Pipeline-centric 52

Pipeline-centric Data Lake Database-centric Data Pipeline 52

Grouparoo

OCTOBER 6, 2021

Themes I was drawn to the articles that speak to a theme in the data world that I am passionate about: how data pipelines and data team practices are evolving to be more like traditional product development. 7 Be Intentional About the Batching Model in Your Data Pipelines Different batching models.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

ThoughtSpot

OCTOBER 18, 2024

It aims to explain how we transformed our development practices with a data-centric approach and offers recommendations to help your teams address similar challenges in your software development lifecycle. This approach ensured comprehensive data extraction while handling various edge cases and log formats.

Building 52

Building Process Pipeline-centric Database-centric 52

Ripple Engineering

JANUARY 29, 2024

For Ripple's product capabilities, the Payments team of Ripple, for example, ingests millions of transactional records into databases and performs analytics to generate invoices, reports, and other related payment operations.    A lack of a centralized system makes building a single source of high-quality data difficult.

Database-centric 52

Database-centric Pipeline-centric NoSQL High Quality Data 52

Knowledge Hut

NOVEMBER 2, 2023

An Azure Data Engineer is a professional responsible for designing, implementing, and managing data solutions using Microsoft's Azure cloud platform. They work with various Azure services and tools to build scalable, efficient, and reliable data pipelines, data storage solutions, and data processing systems.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Netflix Tech

JANUARY 10, 2024

To achieve this, Cosmos was developed as a computing platform for workflow-driven, media-centric microservices. The results are saved to a database so they can be reused. It fetches the complexity data from CAS and runs the optimization algorithm to create encoding recipes. 264, AV1, etc.).

Process 95

Process Pipeline-centric Media Metadata 95

Ascend.io

SEPTEMBER 14, 2023

Data engineers can find one for almost any need, from data extraction to complex transformations, ensuring that they’re not reinventing the wheel by writing code that’s already been written. Exceptional at data retrieval and manipulation within RDBMS. It's specialized for database querying.

Data Engineering 52

Data Engineering Data Engineer Python Engineering 52

Knowledge Hut

NOVEMBER 17, 2023

The demand for data-related professions, including data engineering, has indeed been on the rise due to the increasing importance of data-driven decision-making in various industries. Becoming an Azure Data Engineer in this data-centric landscape is a promising career choice.

Data Engineering 52

Data Engineering Data Engineer Engineering Scala 52

Databand.ai

JULY 19, 2023

Learn more in our detailed guide to data lineage visualization (coming soon) Integration with Multiple Data Sources Data lineage tools are designed to integrate with a wide range of data sources, including databases, data warehouses, and cloud-based data platforms.

Pipeline-centric 52

Pipeline-centric Data Governance Metadata Government 52

Knowledge Hut

APRIL 19, 2023

Owing to the vitality of application software, businesses are actively seeking professionals with excellent technical expertise and a consumer-centric mindset to develop more practical application software systems that enhance customer experience. Database An automated data-keeping system is what a database management system (DBMS) is.

Database-centric 52

Database-centric Entertainment Education Pipeline-centric 52

Knowledge Hut

DECEMBER 26, 2023

What i s Data Science ? Data science is the process of analyzing data to extract meaningful insights. The data from which these insights are extracted can come from various sources, including databases, business transactions, sensors, and more.

Data Science 52

Data Science Database-centric Pipeline-centric Data Mining 52

Monte Carlo

JULY 28, 2022

But perhaps one of the most common reasons for data quality challenges are software feature updates and other changes made upstream by software engineers. These are particularly frustrating, because while they are breaking data pipelines constantly, it’s not their fault. He suggested : “Private vs. public methods.

Software Engineer 52

Software Engineer Software Engineering Pipeline-centric Database-centric 52

RandomTrees

NOVEMBER 29, 2023

The choice of tools and adopting the right methodologies becomes the drive for any business and in this the selection on the database technologies is very crucial. Having said this, the need to understand all the primary components of a data warehouse and then designing the data vault components is described.

Data Warehouse 52

Data Warehouse Data Lake Database-centric Data Cleanse 52

ProjectPro

JUNE 1, 2017

Datos IO has extended its on-premise and public cloud data protection to RDBMS and Hadoop distributions. Its RecoverX distributed database backup product of latest version v2.0 Cloudera is more inclined on becoming a product centric business with 23% of its revenue coming from services past year in comparison to 31% for Hortonworks.

Hadoop 52

Hadoop Medical Pipeline-centric Database-centric 52

Knowledge Hut

JANUARY 30, 2024

Whether you're a seasoned data scientist or just stepping into the world of data, come with me as we unravel the secrets of data extraction and learn how it empowers us to unleash the full potential of data. What is data extraction? Patterns, trends, relationships, and knowledge discovered from the data.

ETL Tools 52

ETL Tools Database-centric Data Mining Raw Data 52

Knowledge Hut

SEPTEMBER 26, 2023

It offers a wide range of services, including computing, storage, databases, machine learning, and analytics, making it a versatile choice for businesses looking to harness the power of the cloud. This cloud-centric approach ensures scalability, flexibility, and cost-efficiency for your data workloads.

Data Lake 52

Data Lake Database-centric Machine Learning Pipeline-centric 52

Monte Carlo

JANUARY 23, 2024

Gen AI can whip up serviceable code in moments — making it much faster to build and test data pipelines. Today’s LLMs can already process enormous amounts of unstructured data, automating much of the monotonous work of data science. But what does that mean for the roles of data engineers and data scientists going forward?

Database-centric 52

Database-centric Pipeline-centric Metadata Unstructured Data 52

Knowledge Hut

JULY 3, 2023

Real-time Data ingestion performs the utilization of data from various origins, does the data cleaning, validation, and preprocessing operations and at the end store it in the required format, either structured or unstructured. As real-time insights gain popularity, real-time data ingestion remains vital for companies worldwide.

Data Ingestion 52

Data Ingestion Google Cloud Pipeline-centric Media 52

DataKitchen

FEBRUARY 21, 2023

This is the equivalent advice for data teams provided to software teams in the classic “Mythical Man Month.” Brooks law (for data): “ Adding data engineer personpower to a late data project makes it later.” Shouldn’t Marcus consider upgrading his technology? A better ETL tool? Pick some other hot tool?

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

ProjectPro

DECEMBER 16, 2021

They need to know everything about the data and apply various mathematical and statistical tools to identify the most significant features using feature selection, feature engineering , feature transformation, etc.

Machine Learning 52

Machine Learning Engineering Pipeline-centric Database-centric 52