Data Engineering Podcast

APRIL 28, 2024

In order to simplify the integration of AI capabilities into developer workflows Tsavo Knott helped create Pieces, a powerful collection of tools that complements the tools that developers already use.

Building 147

Building Data Lake High Quality Data Machine Learning 147

Edureka

JULY 18, 2024

The secret sauce is data collection. Data is everywhere these days, but how exactly is it collected? This article breaks it down for you with thorough explanations of the different types of data collection methods and best practices to gather information. What Is Data Collection?

Data Collection 40

Data Collection Media Data Science Government 40

Striim

MARCH 21, 2025

Take a streaming-first approach to data integration The first, and most important decision is to take a streaming first approach to integration. This means that at least the initial collection of all data should be continuous and real-time.

Process 52

Process Data Warehouse Kafka Data Pipeline 52

AltexSoft

JUNE 26, 2023

While today’s world abounds with data, gathering valuable information presents a lot of organizational and technical challenges, which we are going to address in this article. We’ll particularly explore data collection approaches and tools for analytics and machine learning projects. What is data collection?

Data Collection 52

Data Collection Machine Learning Unstructured Data Non-relational Database 52

Ascend.io

JULY 28, 2023

Data pipelines are integral to business operations, regardless of whether they are meticulously built in-house or assembled using various tools. As companies become more data-driven, the scope and complexity of data pipelines inevitably expand. Ready to fortify your data management practice?

Data Pipeline 52

Data Pipeline Architecture Lambda Architecture Data Architecture 52

Ascend.io

MAY 31, 2023

In the modern world of data engineering, two concepts often find themselves in a semantic tug-of-war: data pipeline and ETL. Fast forward to the present day, and we now have data pipelines. Data Ingestion Data ingestion is the first step of both ETL and data pipelines.

Data Pipeline 52

Data Pipeline ETL Tools Pipeline-centric Data Warehouse 52

Ascend.io

JANUARY 2, 2024

But let’s be honest, creating effective, robust, and reliable data pipelines, the ones that feed your company’s reporting and analytics, is no walk in the park. From building the connectors to ensuring that data lands smoothly in your reporting warehouse, each step requires a nuanced understanding and strategic approach.

Data Pipeline 52

Data Pipeline Building Raw Data Data Warehouse 52

Data Engineering Podcast

NOVEMBER 20, 2022

Especially once they realize 90% of all major data sources like Google Analytics, Salesforce, Adwords, Facebook, Spreadsheets, etc., Hevo Data is a highly reliable and intuitive data pipeline platform used by data engineers from 40+ countries to set up and run low-latency ELT pipelines with zero maintenance.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Data Engineering Weekly

MARCH 2, 2025

[link] Netflix: Cloud Efficiency at Netflix Data is the Key Optimization starts with collecting data and asking the right questions. Netflix writes an excellent article describing its approach to cloud efficiency, starting with data collection to questioning the business process.

Data Engineering 62

Data Engineering Data Engineer Engineering Datasets 62

Data Engineering Podcast

NOVEMBER 6, 2022

Especially once they realize 90% of all major data sources like Google Analytics, Salesforce, Adwords, Facebook, Spreadsheets, etc., Hevo Data is a highly reliable and intuitive data pipeline platform used by data engineers from 40+ countries to set up and run low-latency ELT pipelines with zero maintenance.

Building 130

Building IT Metadata MongoDB 130

Cloudera

APRIL 9, 2021

This blog series follows the manufacturing and operations data lifecycle stages of an electric car manufacturer – typically experienced in large, data-driven manufacturing companies. The first blog introduced a mock vehicle manufacturing company, The Electric Car Company (ECC) and focused on Data Collection.

Machine Learning 98

Machine Learning Manufacturing Data Collection Data Science 98

Cloudera

JANUARY 20, 2021

We have simplified this journey into five discrete steps with a common sixth step speaking to data security and governance. The six steps are: Data Collection – data ingestion and monitoring at the edge (whether the edge be industrial sensors or people in a brick and mortar retail store). Data Collection Challenge.

Manufacturing 110

Manufacturing Data Warehouse Kafka Retail 110

Data Engineering Podcast

FEBRUARY 24, 2020

Are you spending too much time maintaining your data pipeline? Snowplow empowers your business with a real-time event data pipeline running in your own cloud account without the hassle of maintenance. What are some of the ways that compliance or data quality issues can arise from these projects?

IT 100

IT Data Lake Data Pipeline Media 100

Cloudera

JUNE 2, 2022

Companies have not treated the collection, distribution, and tracking of data throughout their data estate as a first-class problem requiring a first-class solution. Instead they built or purchased tools for data collection that are confined with a class of sources and destinations.

Systems 104

Systems Data Lake Google Cloud Cloud 104

Cloudera

APRIL 15, 2019

While Cloudera Flow Management has been eagerly awaited by our Cloudera customers for use on their existing Cloudera platform clusters, Cloudera Edge Management has generated equal buzz across the industry for the possibilities that it brings to enterprises in their IoT initiatives around edge management and edge data collection.

Management 17

Management Data Ingestion Data Collection Government 17

Data Engineering Podcast

FEBRUARY 25, 2018

Preamble Hello and welcome to the Data Engineering Podcast, the show about modern data infrastructure When you’re ready to launch your next project you’ll need somewhere to deploy it. What have you found to be the most difficult aspects of data collection, and do you have any tooling to simplify the implementation for user?

Kafka 100

Kafka AWS Data Data Engineering 100

Confluent

JUNE 13, 2019

You might think that data collection in astronomy consists of a lone astronomer pointing a telescope at a single object in a static sky. While that may be true in some cases (I collected the data for my Ph.D. thesis this way), the field of astronomy is rapidly changing into a data-intensive science with real-time needs.

Kafka 102

Kafka Bytes Python Data Pipeline 102

Knowledge Hut

JUNE 26, 2023

From exploratory data analysis (EDA) and data cleansing to data modeling and visualization, the greatest data engineering projects demonstrate the whole data process from start to finish. Data pipeline best practices should be shown in these initiatives. Which queries do you have?

Data Engineering 98

Data Engineering Data Engineer Coding Project 98

Towards Data Science

DECEMBER 4, 2023

We won’t be alone in this data collection; thankfully, there are data integration tools available in the market that can be adopted to configure and maintain ingestion pipelines in one place (e.g. Data Warehouse & Data Transformation We’ll have numerous pipelines dedicated to data transformation and normalisation.

Data Engineering 98

Data Engineering Data Engineer Engineering Data Lake 98

Data Engineering Podcast

SEPTEMBER 19, 2021

This brings with it a unique set of challenges for data collection, data management, and analytical capabilities. In this episode Jillian Rowe shares her experience of working in the field and supporting teams of scientists and analysts with the data infrastructure that they need to get their work done.

Data Engineering 100

Data Engineering Data Engineer Engineering Data Lake 100

Cloudera

AUGUST 30, 2022

A data mesh supports distributed, domain-specific data consumers and views data as a product, with each domain handling its own data pipelines. Towards Data Science ). Solutions that support MDAs are purpose-built for data collection, processing, and sharing.

Data Architecture 79

Data Architecture Architecture Data Lake NoSQL 79

Cloudera

DECEMBER 16, 2022

We have been investing in development for years to deliver common security, governance, and metadata management across the entire data layer with capabilities to mask data, provide fine grained access, and deliver a single data catalog to view all data across the enterprise. 5-Integrated open data collection.

Database 92

Database Cloud Systems Management 92

Knowledge Hut

DECEMBER 26, 2023

Data Engineering is typically a software engineering role that focuses deeply on data – namely, data workflows, data pipelines, and the ETL (Extract, Transform, Load) process. However, as we progressed, data became complicated, more unstructured, or, in most cases, semi-structured.

Data Engineering 98

Data Engineering Data Engineer Engineering Hadoop 98

Hevo

MARCH 31, 2023

Data engineers are the foundation for any data-driven initiative in organizations. However, the rapid increase in data collection within organizations is clogging data engineers with several challenges. Streamlining the entire data flow at the pace of collecting data is a significant challenge for data engineers.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Pipeline 52

Hevo

MARCH 15, 2023

As organizations accumulate more data, analysts face challenges in effectively utilizing the data collected by companies. Since big data comes in different forms and sizes, companies fail to create robust data pipelines to move data as soon as it arrives.

Cloud 52

Cloud Big Data Data Pipeline Data Collection 52

Snowflake

OCTOBER 25, 2023

While behavioral data is important, it’s rarely the only type of data needed to properly train an AI model for marketing purposes. If your behavioral data is siloed, organizations may be forced to build data pipelines to support AI model training on a comprehensive corpus of required data.

Government 110

Government Accessible Accessibility Technology 110

Databand.ai

JULY 10, 2023

An observability platform is a comprehensive solution that allows data engineers to monitor, analyze, and optimize their data pipelines. By providing a holistic view of the data pipeline, observability platforms help teams rapidly identify and address issues or bottlenecks.

Data Pipeline 52

Data Pipeline Algorithm Data Engineering Data Engineer 52

dbt Developer Hub

APRIL 24, 2023

Alteryx is a visual data transformation platform with a user-friendly interface and drag-and-drop tools. Nonetheless, Alteryx may have difficulties to cope with the complexity increase within an organization’s data pipeline, and it can become a suboptimal tool when companies start dealing with large and complex data transformations.

BI 83

BI Data Workflow SQL Data Pipeline 83

WeCloudData

OCTOBER 19, 2021

With a significant weekly readership and the rapid transition to digital content, the client first created a data pipeline which could collect and store the millions of rows of clickstream data their users generated on a daily basis. Automate article recommendation generation through Databricks built-in job scheduler.

Consulting 52

Consulting Systems NoSQL Raw Data 52

WeCloudData

OCTOBER 19, 2021

With a significant weekly readership and the rapid transition to digital content, the client first created a data pipeline which could collect and store the millions of rows of clickstream data their users generated on a daily basis. Automate article recommendation generation through Databricks built-in job scheduler.

Consulting 52

Consulting Systems NoSQL Raw Data 52

Monte Carlo

MARCH 18, 2024

Data quality audits are meant to ensure the data fueling your business decisions is high-quality. If your data quality is lacking or inaccurate across certain points of your data pipeline, you can pinpoint, triage, and resolve those inaccuracies quickly and efficiently.

Datasets 52

Datasets Data Pipeline BI Data 52

Ascend.io

JANUARY 10, 2024

This continuous adaptation ensures that your data management stays effective and compliant with current standards. Let’s dive into what this involves and how you can make it actionable in your own setting: Data Ingestion: First things first: getting the data into the system. Actionable tip?

Data Pipeline 52

Data Pipeline Government Data Governance Raw Data 52

Monte Carlo

MARCH 24, 2023

How Do You Maintain Data Integrity? Data integrity issues can arise at multiple points across the data pipeline. We often refer to these issues as data freshness or stale data. For example: The source system could provide corrupt data or rows with excessive NULLs. What Is Data Validity?

Data Validation 52

Data Validation Data Integration Data Cleanse Data Pipeline 52

Snowflake

FEBRUARY 15, 2023

Nothing was wrong with their data pipelines and it was unlikely that croissants had fallen out of style, so the team dug deeper. In this post we’ll discuss strategies to turn your workers into data assets. Why is this data being collected? Breakfast sausage. The stick doesn’t have to be punitive, though.

Food 52

Food Data Collection Data Data Warehouse 52

ProjectPro

FEBRUARY 6, 2023

Programming Knowledge Although they are not required to be master coders like data or software engineers, analytics engineers must still be proficient in Python coding. The majority of data pipeline technologies use Python, and it is necessary when creating your own pipeline.

Data Analytics 52

Data Analytics Engineering IT Computer Science 52

Knowledge Hut

MAY 3, 2024

Apache Kafka Stream Kafka is actually a message broker with a really good performance so that all your data can flow through it before being redistributed to applications. Kafka works as a data pipeline. Kafka Streams is a client library for processing and analyzing data stored in Kafka.

Kafka 98

Kafka Scala Java Amazon Web Services 98

DareData

JULY 5, 2023

Users: Who are users that will interact with your data and what's their technical proficiency? Data Sources: How different are your data sources? Latency: What is the minimum expected latency between data collection and analytics? And what is their format?

Data Lake 52

Data Lake Building Raw Data ETL Tools 52