Precisely

JANUARY 9, 2025

The simple idea was, hey how can we get more value from the transactional data in our operational systems spanning finance, sales, customer relationship management, and other siloed functions. But those end users werent always clear on which data they should use for which reports, as the data definitions were often unclear or conflicting.

Data Integration 119

Data Integration Hadoop Data Warehouse Data Lake 119

Data Engineering Podcast

SEPTEMBER 11, 2022

Summary Data engineering systems are complex and interconnected with myriad and often opaque chains of dependencies. Prefect is the modern Dataflow Automation platform for the modern data stack, empowering data practitioners to build, run and monitor robust pipelines at scale. Can you describe what Schemata is and the story behind it?

Systems 100

Systems Metadata Building MongoDB 100

Knowledge Hut

NOVEMBER 20, 2023

The DevOps lifecycle phases are in order from left to right, with each phase building upon the last. It is about automating the process of building, testing, deploying, and maintaining applications to reduce time-to-market for new features and functionality. Operate - This version is now convenient for users to utilize.

Utilities 97

Utilities Programming Coding Designing 97

Data Engineering Podcast

FEBRUARY 18, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. To start, can you share your definition of what constitutes a "Data Lakehouse"? Your first 30 days are free!

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Netflix Tech

JANUARY 6, 2025

Part 2: Navigating Ambiguity By: VarunKhaitan With special thanks to my stunning colleagues: Mallika Rao , Esmir Mesic , HugoMarques Building on the foundation laid in Part 1 , where we explored the what behind the challenges of title launch observability at Netflix, this post shifts focus to the how.

Metadata 87

Metadata Algorithm Systems Building 87

Data Engineering Podcast

NOVEMBER 6, 2022

In this episode Nick King discusses how you can be intentional about data creation in your applications and services to reduce the friction and errors involved in building data products and ML applications. Can you share your definition of "behavioral data" and how it is differentiated from other sources/types of data?

Building 130

Building IT Metadata MongoDB 130

Snowflake

APRIL 2, 2025

The data warehouse solved for performance and scale but, much like the databases that preceded it, relied on proprietary formats to build vertically integrated systems. Tune into our webinar Data Engineering Connect: Building Pipelines for Open Lakehouse on April 29, featuring two virtual demos and a hands-on lab.

Data Lake 88

Data Lake Cloud Storage Metadata Data Warehouse 88

Knowledge Hut

JANUARY 22, 2024

When it comes to managing data, a database management system (DBMS) is a vital tool. Database management systems (DBMS) use entities to represent and manage data. In a database management system (DBMS), an entity is a piece of data tracked and stored by the system. But what is an entity? What is Entity Set in DBMS?

MongoDB 98

MongoDB Database Data Mining Relational Database 98

The Pragmatic Engineer

OCTOBER 10, 2023

He’s solved interesting engineering challenges along the way, too – like building observability for Amazon’s EC2 offering, and being one of the first engineers on Uber’s observability platform. I wrote code for drivers on Windows, and started to put a basic observability system in place.

Software Engineer 310

Software Engineer Software Engineering Architecture Media 310

Data Engineering Podcast

MARCH 27, 2022

In this episode Balaji Ganesan shares how his experiences building and maintaining Ranger in previous roles helped him understand the needs of organizations and engineers as they define and evolve their data governance policies and practices. How have the design and goals of the system changed or evolved since you started working on it?

Data Governance 130

Data Governance Government Cloud Building 130

Cloudera

NOVEMBER 13, 2024

This acquisition delivers access to trusted data so organizations can build reliable AI models and applications by combining data from anywhere in their environment. Additionally, multiple copies of the same data locked in proprietary systems contribute to version control issues, redundancies, staleness, and management headaches.

Metadata 90

Metadata Management Data Governance Government 90

Knowledge Hut

MAY 6, 2024

Applying systems thinking views a system as a set of interconnected and interdependent components defined by its limits and more than the sum of their parts (subsystems). When one component of a system is altered, the effects frequently spread across the entire system. are the main objectives of systems thinking.

Systems 98

Systems Transportation Certification Healthcare 98

The Pragmatic Engineer

SEPTEMBER 26, 2023

This means more repositories are needed, which are fast enough to build and work with, but which increase fragmentation. Executing a build is much slower while on a call. Plus, a CPU and memory-intensive build can impact the quality of the video call, and make the local environment much less responsive.  Larger codebases.

Cloud 297

Cloud Software Engineer Software Engineering Cloud Computing 297

The Pragmatic Engineer

APRIL 19, 2023

I still remember being in a meeting where a Very Respected Engineer was explaining how they are building a project, and they said something along the lines of "and, of course, idempotency is non-negotiable." I was sceptical that any system would automatically reject resumes, because I never saw this as a hiring manager.

Software Engineer 333

Software Engineer Software Engineering Engineering Media 333

Data Engineering Podcast

SEPTEMBER 3, 2021

In this episode Lior Gavish, Lior Solomon, and Atul Gupte share their view of what it means to have a data platform, discuss their experiences building them at various companies, and provide advice on how to treat them like a software product. The data you’re looking for is already in your data warehouse and BI tools.

Designing 100

Designing Building SQL BI 100

Data Engineering Weekly

JANUARY 15, 2025

The answer lies in unstructured data processing—a field that powers modern artificial intelligence (AI) systems. To address these challenges, AI Data Engineers have emerged as key players, designing scalable data workflows that fuel the next generation of AI systems. How does a self-driving car understand a chaotic street scene?

Data Engineer 62

Data Engineer Data Engineering Unstructured Data Engineering 62

Data Engineering Podcast

OCTOBER 2, 2021

He also discusses the technical implementation that allows for such extreme performance and how the data model contributes to the scalability of the system. If you need to deal with massive data, at high velocities, in milliseconds, then Aerospike is definitely worth learning about.

Building 100

Building BI Data Architecture Architecture 100

Monte Carlo

FEBRUARY 23, 2024

And this renewed focus on data quality is bringing much needed visibility into the health of technical systems. In addition, Snow f lake Cortex enables organizations to quickly analyze data and build AI applications directly within Snowflake. GenAI will definitely make us more productive. The opportunities are just boundless.”

Systems 64

Systems Building Retail Data Lake 64

Snowflake

OCTOBER 2, 2023

To help customers overcome these challenges, RudderStack and Snowflake recently launched Profiles , a new product that allows every data team to build a customer 360 directly in their Snowflake Data Cloud environment. Now teams can leverage their existing data engineering tools and workflows to build their customer 360.

Cloud 115

Cloud Building Insurance Data Engineer 115

The Pragmatic Engineer

AUGUST 10, 2023

Ayhan visualized this data and observed a definite fall in all metrics: page views, visits, questions asked, votes. Q&A activity is definitely down: the company is aware of this metric taking a dive, and said they’re actively working to address it. Booking.com says a systems migration is the reason for the delay.

Retail 188

Retail Utilities Software Engineer Software Engineering 188

Netflix Tech

FEBRUARY 14, 2025

It requires a state-of-the-art system that can track and process these impressions while maintaining a detailed history of each profiles exposure. In this multi-part blog series, we take you behind the scenes of our system that processes billions of impressions daily. Impressions onhomepage Why do we need impression history?

Kafka 66

Kafka Datasets Metadata Utilities 66

Engineering at Meta

FEBRUARY 4, 2025

Here we explore initial system designs we considered, an overview of the current architecture, and some important principles Meta takes into account in making data accessible and easy to understand. We also considered caching data logs in an online system capable of supporting a range of indexed per-user queries. What are data logs?

Accessible 80

Accessible Accessibility Raw Data Data Warehouse 80

Data Engineering Podcast

MARCH 3, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. What are the skills and systems that need to be in place to effectively execute on an AI program? "AI" Your first 30 days are free!

Programming 162

Programming Data Lake High Quality Data Machine Learning 162

Snowflake

NOVEMBER 11, 2024

What if you could streamline your efforts while still building an architecture that best fits your business and technology needs? At BUILD 2024, we announced several enhancements and innovations designed to help you build and manage your data architecture on your terms. Here’s a closer look.

Data Architecture 105

Data Architecture Architecture Data Lake Kafka 105

Precisely

OCTOBER 31, 2024

With data volumes skyrocketing, and complexities increasing in variety and platforms, traditional centralized data management systems often struggle to keep up. As data management grows increasingly complex, you need modern solutions that allow you to integrate and access your data seamlessly.

Data Architecture 69

Data Architecture Architecture Metadata Government 69

Snowflake

MARCH 12, 2025

For example, ticketing, merchandise, fantasy engagement and game viewership data often reside in separate systems (or with separate entities), making it a challenge to bring together a cohesive view of each fan. Spaulding Ridge: Turning fan 360 from vision to reality Building a fan 360 requires a comprehensive approach.

Media 63

Media Cloud Programming Data Collection 63

Data Engineering Podcast

DECEMBER 21, 2021

Summary One of the perennial challenges of data analytics is having a consistent set of definitions, along with a flexible and performant API endpoint for querying them. a framework for building analytics APIs to power your applications and BI dashboards Interview Introduction How did you get involved in the area of data management?

Data Analytics 130

Data Analytics BI Computer Science SQL 130

AltexSoft

MAY 12, 2022

The approach finds application in security systems for user authentication. Systems like Audio Analytic ‘listen’ to the events inside and outside your car, enabling the vehicle to make adjustments in order to increase a driver’s safety. Building an app for snore and teeth grinding detection. Music recognition.

Machine Learning 122

Machine Learning Building Deep Learning Healthcare 122

Towards Data Science

JULY 13, 2023

Building a maintainable and modular LLM application stack with Hamilton in 13 minutes LLM Applications are dataflows, use a tool specifically designed to express them LLM stacks. Hamilton is great for describing any type of dataflow , which is exactly what you’re doing when building an LLM powered application. Image from pixabay.

Building 97

Building Database-centric Database Coding 97

Data Engineering Podcast

MARCH 10, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Go to dataengineeringpodcast.com/dagster today to get started. Your first 30 days are free! Data lakes are notoriously complex.

Data Lake 147

Data Lake High Quality Data Architecture Machine Learning 147

Data Engineering Weekly

FEBRUARY 18, 2025

Jark is a key figure in the Apache Flink community, known for his work in building Flink SQL from the ground up and creating Flink CDC and Fluss. Fluss is an analytical Kafka that builds on top of Lakehouse and integrates seamlessly with Flink to reduce costs, achieve better performance, and unlock new use cases for real-time analytics.

Kafka 73

Kafka Lambda Architecture SQL Architecture 73

Data Engineering Podcast

JULY 2, 2023

In this episode Razi Raziuddin shares how data engineering teams can support the machine learning workflow through the development and support of systems that empower data scientists and ML engineers to build and maintain their own features. What is the overall lifecycle of a feature, from definition to deployment and maintenance?

Machine Learning 130

Machine Learning Data Engineer Data Engineering Engineering 130

Zalando Engineering

MAY 13, 2024

As a design system evolves alongside with the brand it represents, there are often multiple occasions when a need to introduce variations arises. The previous article on this blog gives a wider overview of the Zalando Design System. The previous article on this blog gives a wider overview of the Zalando Design System.

Designing 60

Designing Systems Architecture Coding 60

Cloudera

AUGUST 16, 2021

Apache Nifi is a powerful tool to build data movement pipelines using a visual flow designer. Ultimately these challenges force NiFi teams to spend a lot of time on managing the cluster infrastructure instead of building new data flows which slows down use case adoption. The need for a cloud-native Apache NiFi service. and later).

Cloud 122

Cloud Unstructured Data Utilities Metadata 122

Netflix Tech

DECEMBER 3, 2022

Thanks to the Netflix internal lineage system (built by Girish Lingappa ) Dataflow migration can then help you identify downstream usage of the table in question. This is one way to build trust with our internal user base. The main workflow definition file holds the logic of a single run, in this case one day-worth of data.

Data Pipeline 144

Data Pipeline Scala Metadata Food 144

Ascend.io

NOVEMBER 14, 2024

By focusing on these attributes, data engineers can build pipelines that not only meet current demands but are also prepared for future challenges. Each section will provide actionable insights and practical tips to help you build pipelines that are robust, efficient, and ready for whatever the future holds.

Data Pipeline 52

Data Pipeline Amazon Web Services Data Integration Data 52

DoorDash Engineering

FEBRUARY 27, 2024

We reviewed the architecture of our global search at DoorDash in early 2022 and concluded that our rapid growth meant within three years we wouldn’t be able to scale the system efficiently, particularly as global search shifted from store-only to a hybrid item-and-store search experience. latency reduction and a 75% hardware cost decrease.

Engineering 130

Engineering Systems Designing Architecture 130