Data Engineering Podcast

NOVEMBER 26, 2023

In this episode Tobias Macey shares his thoughts on the challenges that he is facing as he prepares to build the next set of architectural layers for his data platform to enable a larger audience to start accessing the data being managed by his team. With Materialize, you can! Want to see Starburst in action? Want to see Starburst in action?

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Scott Logic

APRIL 1, 2025

During the InferESG project we made a pivotal decision to create an alternative architecture, one that sits parallel to the agentic framework used for the conversational part of the system. We found that these two architectures can work harmoniously together, each addressing different aspects of the ESG analysis challenge.

Architecture 52

Architecture Generalist Project Systems 52

Striim

APRIL 22, 2025

But complexity stands in the way: incompatible platforms, brittle pipelines, fragmented architectures, and the growing pressure of data privacy and compliance risks make it challenging for teams to deliver trusted, real-time data to models and applications. Define the must-have characteristics of a data streaming architecture.

Architecture 52

Architecture Data Architecture Building Datasets 52

Precisely

JANUARY 9, 2025

In the beginning, there was a data warehouse The data warehouse (DW) was an approach to data architecture and structured data management that really hit its stride in the early 1990s. Data marts soon evolved as a core part of a DW architecture to eliminate this noise. financial reporting, customer analytics, supply chain management).

Data Integration 119

Data Integration Hadoop Data Warehouse Data Lake 119

Snowflake

APRIL 2, 2025

Data storage has been evolving, from databases to data warehouses and expansive data lakes, with each architecture responding to different business and data needs. Traditional databases excelled at structured data and transactional workloads but struggled with performance at scale as data volumes grew.

Data Lake 97

Data Lake Cloud Storage Metadata Data Warehouse 97

Yelp Engineering

JANUARY 21, 2025

At Yelp, we encountered challenges that prompted us to enhance the training time of our ad-revenue generating models, which use a Wide and Deep Neural Network architecture for predicting ad click-through rates (pCTR). These models handle large tabular datasets with small parameter spaces, requiring innovative data solutions.

Datasets 104

Datasets Architecture Data Solutions Data 104

Start Data Engineering

SEPTEMBER 18, 2024

Data flow architecture 3. Define SLAs so stakeholders know what to expect 3.1.4. Define checks to ensure the output dataset is usable 3.2. Identify what tool to use to process data 3.3.

Project 240

Project Building Datasets Architecture 240

Jesse Anderson

FEBRUARY 11, 2025

We delve into the architecture of Kuzu, an embedded graph database designed for high performance, exploring its unique storage format, query planning techniques like Sideway Information Passing (SIP), and the rationale behind its schema-based approach.

Computer Science 130

Computer Science Database Design Software Engineering Software Engineer 130

The Pragmatic Engineer

OCTOBER 10, 2023

We originally built M3 to support Uber’s scale on a microservices/container (cloud native) architecture. It was that very architecture that created the scale, reliability, performance and cost-efficiency challenges. None of the existing vendors had built and optimized for a cloud-native architecture.

Software Engineering 327

Software Engineering Software Engineer Architecture Media 327

Data Engineering Podcast

FEBRUARY 18, 2024

And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data. What are the technical/architectural/UX challenges that have hindered the progression of lakehouses? Want to see Starburst in action? Want to see Starburst in action?

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

databricks

MAY 1, 2024

Modern data stacks provide the necessary flexibility and efficiency for analytics and AI. Learn how the Databricks Data Intelligence Platform makes use of them.

Data Architecture 125

Data Architecture Architecture Data 125

Data Engineering Weekly

DECEMBER 28, 2024

The debate around table formats and Lakehouse architectures continues, but the focus is on unifying data ecosystems to enable AI-driven insights. The shift towards intelligent data platforms will continue, with enterprises seeking to seamlessly integrate structured and unstructured data, ensuring quality, governance, and trustworthiness.

Unstructured Data 113

Unstructured Data Metadata Data Government 113

Towards Data Science

JANUARY 30, 2025

Methodology I used LeNet-5 as my model architecture. Experiment with how model architecture impacts the performance of data pruning strategies. Then using one of the strategies below I pruned the training dataset of MNIST and trained a model. Testing was done against the full testset. Full code and results available here onGitHub.

Database-centric 106

Database-centric Datasets Data Architecture 106

Data Engineering Podcast

DECEMBER 24, 2023

You should be able to keep the familiarity of SQL and the proven architecture of cloud warehouses, but swap the decades-old batch computation model for an efficient incremental engine to get complex queries that are always up-to-date. With Materialize, you can! Want to see Starburst in action? Want to see Starburst in action?

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Precisely

JANUARY 30, 2025

Metadata maturity is key to a strong foundation Metadata management maturity is growing in importance, especially with the rise of new data management architectures like data mesh (which takes a decentralized approach) and data fabric (which takes a unified approach).

Data Governance 103

Data Governance Government Metadata Data 103

Cloudera

DECEMBER 4, 2024

High-level architecture The diagram above shows a high-level architecture of Cloudera AI Inference service in context: KServe and Knative handle model and application orchestration, respectively. We will dive deeper into the architecture in our next post, so please stay tuned. We also outlined many of its capabilities.

Architecture 73

Architecture Machine Learning Deep Learning BI 73

Data Engineering Weekly

FEBRUARY 18, 2025

Architecture Difference The first difference is the Data Model. The fourth difference is the Lakehouse Architecture. Fluss embraces the Lakehouse Architecture. On the other hand, Fluss is a Kappa Architecture ; it stores one copy of data and presents it as a stream or a table, depending on the use case.

Kafka 74

Kafka Lambda Architecture SQL Architecture 74

Cloudera

JANUARY 7, 2025

Leveraging Clouderas hybrid architecture, the organization optimized operational efficiency for diverse workloads, providing secure and compliant operations across jurisdictions while improving response times for public health initiatives. This transition streamlined data analytics workflows to accommodate significant growth in data volumes.

Cloud 82

Cloud Government AWS Unstructured Data 82

Snowflake

APRIL 5, 2025

Faster and high-quality inference with a Mixture of Experts Architecture (MoE) Llama 4 are the first models from Meta to use a MoE architecture a single token activates only a fraction of the total parameters. Within Snowflake, Llama 4 Maverick and Llama 4 Scout can be integrated with gen AI applications.

Architecture 74

Architecture SQL Data Pipeline Accessible 74

Snowflake

FEBRUARY 18, 2025

Cortex AI makes available functions to process unstructured data, create vector embeddings and run vector search, deploy foundational LLMs, build retrieval-augmented generation (RAG) architectures or chat with structured data in one unified architecture. This unified stack allows our customers to spend their time on driving AI ROI.

Unstructured Data 87

Unstructured Data Insurance Structured Data Government 87

Snowflake

MARCH 6, 2025

As data volumes grow and AI automation expands, cost efficiency in processing with LLMs depends on both system architecture and model flexibility. This isn't incremental improvement it's a transformational efficiency gain that saves thousands of labor hours and dramatically accelerates response times.

Unstructured Data 97

Unstructured Data Medical Media Data Workflow 97

Edureka

APRIL 22, 2025

Its multi-cluster shared data architecture is one of its primary features. Because of the architecture’s ability to abstract infrastructure complexity, users can focus solely on data workflows. This blog post explains the main distinctions between these two in terms of architecture, features, integration, and use cases.

BI 52

BI Pipeline-centric Data Lake Google Cloud 52

Zalando Engineering

MARCH 6, 2025

At Zalando, our event-driven architecture for Price and Stock updates became a bottleneck, introducing delays and scaling challenges. You'll learn about the caching strategies, low-latency optimizations, and architectural decisions that enabled us to deliver this performance. If so, a simple request"Where do I get Product data?"could

Algorithm 63

Algorithm Architecture Transportation Data Ingestion 63

phData: Data Engineering

NOVEMBER 7, 2024

However, with the introduction of the Transformer architecture—initially successful in Natural Language Processing (NLP)—the landscape has shifted. In this blog, we will explore the underlying architecture of transformer-based large vision models, offering insights into how they function. What Are Large Vision Models?

Architecture 52

Architecture Project Datasets Utilities 52

Cloudera

DECEMBER 17, 2024

Traditional data architectures struggle to handle these workloads, and without a robust, scalable hybrid data platform, the risk of falling behind is real. As more data is processed, carriers increasingly need to adopt hybrid cloud architectures to balance different workload demands.

Building 62

Building Telecommunication Data Architecture Architecture 62

ThoughtSpot

APRIL 22, 2025

Thats where ThoughtSpots architecture comes in. Analytics architecture: ThoughtSpots AI-native architecture and advanced semantic model augment LLM reasoning, using search tokens to improve text-to-SQL accuracy and provide greater transparency.

BI 59

BI Datasets Business Intelligence Raw Data 59

Precisely

DECEMBER 9, 2024

To name a few: privacy and security considerations compliance demands interest in emerging data management architectures like data mesh and data fabric increased AI adoption The findings show that data governance is the most-cited data challenge inhibiting progress toward AI initiatives (62%). This is likely driven by various factors.

Data Governance 64

Data Governance Government Data Integration Architecture 64

Pinterest Engineering

APRIL 4, 2025

Technical Design LLM as Relevance Model Model Architecture We use a cross-encoder language model to predict a Pins relevance to a query, along with Pin text, as shown in Figure 1. Figure 1: The cross-encoder architecture in the relevance teacher model. The task is formulated as a multiclass classification problem.

Machine Learning 56

Machine Learning Metadata Architecture Datasets 56

Data Engineering Podcast

JUNE 23, 2024

Summary Data lakehouse architectures have been gaining significant adoption. To accelerate adoption in the enterprise Microsoft has created the Fabric platform, based on their OneLake architecture. To accelerate adoption in the enterprise Microsoft has created the Fabric platform, based on their OneLake architecture.

Data Lake 162

Data Lake High Quality Data Hadoop Machine Learning 162

Christophe Blefari

JANUARY 11, 2025

The future of data querying with Natural Language — What are all the architecture block needed to make natural language query working with data (esp. when you have a semantic layer). Hard data integration problems — As always Max describes the best way the reality.

Data 130

Data Data Warehouse Coding Programming Language 130

Monte Carlo

NOVEMBER 21, 2024

They’re basically architectural blueprints for moving and processing your data. Lambda Architecture Pattern 4. Kappa Architecture Pattern 5. Lambda Architecture Pattern Here’s where things get interesting. That’s where data pipeline design patterns come in. Batch Processing Pattern 2.

Data Pipeline 52

Data Pipeline Designing Lambda Architecture Kafka 52

Data Engineering Podcast

FEBRUARY 25, 2024

And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data. Can you start by describing the FDAP stack and how the components combine to provide a foundational architecture for database engines? Want to see Starburst in action?

Database 162

Database Technology Data Lake High Quality Data 162

Snowflake

NOVEMBER 11, 2024

Data and AI architecture matter “Before focusing on AI/ML use cases such as hyper personalization and fraud prevention, it is important that the data and data architecture are organized and structured in a way which meets the requirements and standards of the local regulators around the world.

Banking 98

Banking Finance Retail Food 98