Data Engineering Podcast

NOVEMBER 20, 2022

Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days or even weeks. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

Data Lake 130

Data Lake Data Ingestion MongoDB MySQL 130

Ascend.io

DECEMBER 19, 2022

We hope the real-time demonstrations of Ascend automating data pipelines were a real treat—a long with the special edition T-Shirt designed specifically for the show (picture of our founder and CEO rocking the t-shirt below). Thank you to the hundreds of AWS re:Invent attendees who stopped by our booth!

Data Ingestion 52

Data Ingestion Data Pipeline Metadata AWS 52

Data Engineering Podcast

NOVEMBER 6, 2022

Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools. Sign up free at dataengineeringpodcast.com/rudder Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

MongoDB 130

MongoDB MySQL Scala Machine Learning 130

Monte Carlo

FEBRUARY 20, 2024

At the heart of every data-driven decision is a deceptively simple question: How do you get the right data to the right place at the right time? The growing field of data ingestion tools offers a range of answers, each with implications to ponder. Fivetran Image courtesy of Fivetran.

Data Ingestion 52

Data Ingestion Google Cloud Kafka AWS 52

Striim

OCTOBER 11, 2024

A well-executed data pipeline can make or break your company’s ability to leverage real-time insights and stay competitive. Thriving in today’s world requires building modern data pipelines that make moving data and extracting valuable insights quick and simple. What is a Data Pipeline?

Data Pipeline 52

Data Pipeline MongoDB Unstructured Data Data Lake 52

Data Engineering Podcast

OCTOBER 30, 2022

Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days or even weeks. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

Engineering 100

Engineering MongoDB MySQL Scala 100

Cloudera

APRIL 15, 2019

While Cloudera Flow Management has been eagerly awaited by our Cloudera customers for use on their existing Cloudera platform clusters, Cloudera Edge Management has generated equal buzz across the industry for the possibilities that it brings to enterprises in their IoT initiatives around edge management and edge data collection.

Management 17

Management Data Ingestion Data Collection Government 17

Cloudera

JANUARY 20, 2021

We have simplified this journey into five discrete steps with a common sixth step speaking to data security and governance. The six steps are: Data Collection – data ingestion and monitoring at the edge (whether the edge be industrial sensors or people in a brick and mortar retail store). Conclusion.

Manufacturing 110

Manufacturing Data Warehouse Kafka Retail 110

Data Engineering Podcast

JUNE 5, 2022

Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days or even weeks. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

Data Security 100

Data Security Metadata MongoDB MySQL 100

Data Engineering Podcast

NOVEMBER 13, 2022

Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days or even weeks. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

MongoDB 100

MongoDB MySQL Google Cloud Scala 100

Cloudera

APRIL 9, 2021

Before we dive into our ECC parts demand forecasting use case, let’s look at some of the common ML challenges that are shared across industries where modern, data-driven businesses rely on predictive capabilities to drive their strategic decisions – in addition to historical and real-time analytics. . Additional Resources.

Machine Learning 98

Machine Learning Manufacturing Data Collection Data Science 98

Monte Carlo

MAY 26, 2023

A better measurement is the data downtime formula (above), which more comprehensively measures the amount of time the data was inaccurate, missing, or otherwise erroneous. A proactive approach for measuring data freshness is to create service level agreements or SLAs for specific data pipelines.

Data Pipeline 98

Data Pipeline Data Data Warehouse Machine Learning 98

Towards Data Science

JULY 8, 2024

Leveraging TensorFlow Transform for scaling data pipelines for production environments Photo by Suzanne D. Williams on Unsplash Data pre-processing is one of the major steps in any Machine Learning pipeline. I have used Colab for this demo, as it is much easier (and faster) to configure the environment.

Data Preparation 70

Data Preparation Datasets Metadata Data Ingestion 70

Snowflake

APRIL 24, 2023

However, this has resulted in dated systems that cause workflow inefficiencies, and data and technology silos that add to cost and complexity. Data management becomes increasingly manual, creating elongated data pipelines, delayed analytics, and greater potential for error.

Management 52

Management Portfolio Data Ingestion Data Warehouse 52

Snowflake

SEPTEMBER 11, 2023

Set up the demo environment. The intention of Dynamic Tables is to apply incremental transformations on near real-time data ingestion that Snowflake now supports with Snowpipe Streaming. Dynamic Tables do not replace Streams & Tasks but rather offer an alternative to how you manage your data pipelines within Snowflake.

SQL 95

SQL Data Raw Data Architecture 95

Rockset

JUNE 21, 2022

Rockset efficiently organizes data in a Converged Index ™, which is optimized for real-time data ingestion and low-latency analytical queries. Rockset’s ingest rollups enable developers to pre-aggregate real-time data using SQL without the need for complex real-time data pipelines.

Kafka 52

Kafka Data Warehouse BI Analytics Application 52

Ascend.io

FEBRUARY 12, 2023

Stakeholders have grown frustrated with how long it takes to build data pipelines. Business users are questioning the accuracy and data reliability of the data pipelines and often have shifted back to operating on hunches rather than facts.

Business Analyst 52

Business Analyst Data Lake Data Pipeline Data 52

Ascend.io

FEBRUARY 12, 2023

Stakeholders have grown frustrated with how long it takes to build data pipelines. Business users are questioning the accuracy and data reliability of the data pipelines and often have shifted back to operating on hunches rather than facts.

Business Analyst 52

Business Analyst Data Lake Data Pipeline Data 52

Ascend.io

FEBRUARY 12, 2023

Stakeholders have grown frustrated with how long it takes to build data pipelines. Business users are questioning the accuracy and data reliability of the data pipelines and often have shifted back to operating on hunches rather than facts.

Business Analyst 52

Business Analyst Data Lake Data Pipeline Data 52

AltexSoft

MARCH 30, 2023

Databricks architecture Databricks provides an ecosystem of tools and services covering the entire analytics process — from data ingestion to training and deploying machine learning models. Besides that, it’s fully compatible with various data ingestion and ETL tools. Let’s see what exactly Databricks has to offer.

Scala 64

Scala Data Lake Machine Learning BI 64

ProjectPro

JANUARY 19, 2022

Along with this, you will learn how to perform data analysis using GraphX and Neo4j. Apache Zeppelin Demo Big Data Project for Data Analysis : This project is best for beginners exploring big data tools. It will introduce you to Apache Zeppelin and guide you to write Spark, Hive, and Pig code in notebooks.

Data Engineer 52

Data Engineer Data Engineering Engineering Amazon Web Services 52

Cloudera

OCTOBER 15, 2020

Provide a collection Name under Destination (in this example, we named it ‘solr-nifi-demo’). nifi-solr-demo. nifi-solr-demo. solr-nifi-demo-collection. In this post, we demonstrated how Cloudera Data Platform components can collaborate with each other, while still being resource isolated and managed separately.

AWS 118

AWS Data Cloud Accessibility 118

Christophe Blefari

OCTOBER 14, 2023

Then Marc Lamberti gave a huge update about Airflow but done differently — It wasn't about slides with a list of new features but rather about how you can write, in 2023, a data pipeline with Airflow. He also demo a event-based DAG parsing that instantaneously display DAGs in the UI.

Python 130

Python Datasets Data Banking 130

Snowflake

APRIL 17, 2025

Snowflake experts, customers and partners will share strategic insights and practical tips for building a solid and collaboration-ready data foundation for AI. The events will also feature demos of key use cases and best practices. Watch demos to see real-world AI in action. Accelerate Public Sector is Thursday, April 24.

Healthcare 75

Healthcare AWS Education Government 75