Remove Architecture Remove Data Warehouse Remove Lambda Architecture
article thumbnail

Exploring Processing Patterns For Streaming Data Integration In Your Data Lake

Data Engineering Podcast

Datafold also helps automate regression testing of ETL code with its Data Diff feature that instantly shows how a change in ETL or BI code affects the produced data, both on a statistical level and down to individual rows and values. Can you start by giving an overview of the state of the market for data lakes today?

Data Lake 100
article thumbnail

An Exploration Of The Expectations, Ecosystem, and Realities Of Real-Time Data Applications

Data Engineering Podcast

Select Star’s data discovery platform solves that out of the box, with an automated catalog that includes lineage from where the data originated, all the way to which dashboards rely on it and who is viewing them every day. Can you describe what is driving the adoption of real-time analytics? When is Rockset the wrong choice?

Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

article thumbnail

Maintaining Your Data Lake At Scale With Spark

Data Engineering Podcast

This conversation was useful for getting a better idea of the challenges that exist in large scale data analytics, and the current state of the tradeoffs between data lakes and data warehouses in the cloud. Coming up this fall is the combined events of Graphorum and the Data Architecture Summit.

Data Lake 100
article thumbnail

An Overview of Real Time Data Warehousing on Cloudera

Cloudera

Users today are asking ever more from their data warehouse. As an example of this, in this post we look at Real Time Data Warehousing (RTDW), which is a category of use cases customers are building on Cloudera and which is becoming more and more common amongst our customers. What is Real Time Data Warehousing?

article thumbnail

StreamNative Brings Streaming Data To The Cloud Native Landscape With Pulsar

Data Engineering Podcast

You monitor your website to make sure that you’re the first to know when something goes wrong, but what about your data? Tidy Data is the DataOps monitoring platform that you’ve been missing. You monitor your website to make sure that you’re the first to know when something goes wrong, but what about your data?

Cloud 100
article thumbnail

Building A Data Lake For The Database Administrator At Upsolver

Data Engineering Podcast

How does the introduction of a universal SQL layer change the staffing requirements for building and maintaining a data lake? What are the advantages of a data lake over a data warehouse if everything is being managed via SQL anyway?

Data Lake 100
article thumbnail

8 Essential Data Pipeline Design Patterns You Should Know

Monte Carlo

Whether it’s customer transactions, IoT sensor readings, or just an endless stream of social media hot takes, you need a reliable way to get that data from point A to point B while doing something clever with it along the way. That’s where data pipeline design patterns come in. Lambda Architecture Pattern 4.