Data Engineering Podcast

FEBRUARY 18, 2024

Join in with the event for the global data community, Data Council Austin. Don't miss out on their only event this year! What are the pain points that are still prevalent in lakehouse architectures as compared to warehouse or vertically integrated systems? Don't miss out on our only event this year!

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Knowledge Hut

MARCH 22, 2024

I have comprehensively analyzed the area of physical security, particularly the ongoing discussion surrounding fail safe vs fail-safe secure electric strike locking systems. On the other hand, fail-secure systems focus on maintaining continuous security, keeping doors locked even in difficult conditions to protect assets.

Systems 105

Systems Electronics Hospitality Architecture 105

Data Engineering Podcast

DECEMBER 25, 2022

Recognizing this shortcoming and the capabilities that could be unlocked by a robust solution Rishabh Poddar helped to create Opaque Systems as an outgrowth of his PhD studies. Can you describe what you are building at Opaque Systems and the story behind it? Sign up free at dataengineeringpodcast.com/rudder Build Data Pipelines.

Machine Learning 130

Machine Learning Systems Data Lake Data Warehouse 130

Confluent

DECEMBER 2, 2022

With over 4,700 stores, learn how Walmart used Kafka to build an event-driven architecture for real-time inventory management, providing a seamless omnichannel experience.

Kafka 119

Kafka Systems Architecture Building 119

Engineering at Meta

SEPTEMBER 7, 2023

We’re introducing Arcadia, Meta’s unified system that simulates the compute, memory, and network performance of AI training clusters. We need a systemized source of truth that can simulate various performance factors across compute, storage, and network collectively. For instance, the AI Research SuperCluster for AI research.

Systems 112

Systems Architecture Designing Utilities 112

Engineering at Meta

AUGUST 9, 2023

Explore is one of the largest recommendation systems on Instagram. Using more advanced machine learning models, like Two Towers neural networks, we’ve been able to make the Explore recommendation system even more scalable and flexible. locally popular media), which further contributes to system scalability.

Systems 98

Systems Media Algorithm Machine Learning 98

Pinterest Engineering

JANUARY 18, 2024

In particular, our machine learning powered ads ranking systems are trying to understand users’ engagement and conversion intent and promote the right ads to the right user at the right time. Specifically, such discrepancies unfold into the following scenarios: Bug-free scenario : Our ads ranking system is working bug-free.

Systems 94

Systems Machine Learning Data Mining Algorithm 94

Confluent

MAY 29, 2024

The dual-write problem can arise in any distributed system. Fortunately, it has solutions in event sourcing & the transactional outbox & listen-to-yourself patterns.

Systems 94

Systems IT Kafka 94

The Pragmatic Engineer

APRIL 11, 2024

From Sella’s status page : “Following the installation of an update to the operating system and related firmware which led to an unstable situation. Still, I’m puzzled by how long the system has been down. If it was an update to Oracle, or to the operating system, then why not roll back the update?

Banking 221

Banking Utilities Database Engineering 221

databricks

SEPTEMBER 12, 2023

As the lakehouse becomes increasingly mission-critical to data-forward organizations, so too grows the risk that unexpected events, outages, and security incidents may derail.

Systems 110

Systems Data 110

Data Engineering Podcast

DECEMBER 24, 2023

Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. In the event of these different cluster errors, what are the strategies for mitigating and recovering from those failures? Operating it at scale, however, is notoriously challenging.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Cloudera

NOVEMBER 1, 2024

Over the last few months, Cloudera has been traversing the globe hosting our EVOLVE24 event series. The speakers covered everything from different leadership approaches to overcoming systemic barriers and driving organizational transformation and inclusion. This was one of the key points raised during the discussion. “As

Technology 97

Technology Recruitment Banking AWS 97

The Pragmatic Engineer

JUNE 1, 2023

Juraj included system monitoring parts which monitor the server’s capacity he runs the app on: The monitoring page on the Rides app And it doesn’t end here. Juraj created a systems design explainer on how he built this project, and the technologies used: The systems design diagram for the Rides application The app uses: Node.js

Education 363

Education Project PostgreSQL Software Engineer 363

Precisely

DECEMBER 16, 2024

Modern IT environments require comprehensive data for successful AIOps, that includes incorporating data from legacy systems like IBM i and IBM Z into ITOps platforms. AIOps presents enormous promise, but many organizations face hurdles in its implementation: Complex ecosystems made of multiple, fragmented systems that lack interoperability.

Systems 52

Systems IT Machine Learning Insurance 52

The Pragmatic Engineer

NOVEMBER 21, 2023

If you had a continuous deployment system up and running around 2010, you were ahead of the pack: but today it’s considered strange if your team would not have this for things like web applications.  We dabbled in network engineering, database management, and system administration. and hand-rolled C -code.

Engineering 302

Engineering Bytes Cloud Computing AWS 302

Data Engineering Podcast

MARCH 3, 2024

Join us at the top event for the global data community, Data Council Austin. Don't miss out on our only event this year! What are the skills and systems that need to be in place to effectively execute on an AI program? "AI" Visit dataengineeringpodcast.com/data-council and use code dataengpod20 to register today!

Programming 162

Programming Data Lake High Quality Data Data Pipeline 162

Netflix Tech

JUNE 1, 2023

In this blog post, we’ll discuss the methods we used to ensure a successful launch, including: How we tested the system Netflix technologies involved Best practices we developed Realistic Test Traffic Netflix traffic ebbs and flows throughout the day in a sinusoidal pattern. We used Elasticsearch dashboards to analyze results.

Algorithm 140

Algorithm Metadata Kafka Systems 140

Data Engineering Podcast

MAY 21, 2023

The batch world has been the default for years because of the complexities of running a reliable streaming system at scale. Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools.

Data Lake 162

Data Lake Machine Learning Kafka Data Warehouse 162

Data Engineering Podcast

FEBRUARY 25, 2024

Over the decades of research and development into building these software systems there are a number of common components that are shared across implementations. Join us at the top event for the global data community, Data Council Austin. Don't miss out on our only event this year! Don't miss out on our only event this year!

Database 162

Database Technology Data Lake High Quality Data 162

Netflix Tech

NOVEMBER 12, 2024

By: Rajiv Shringi , Oleksii Tkachuk , Kartik Sathyanarayanan Introduction In our previous blog post, we introduced Netflix’s TimeSeries Abstraction , a distributed service designed to store and query large volumes of temporal event data with low millisecond latencies. Today, we’re excited to present the Distributed Counter Abstraction.

Datasets 101

Datasets Computer Science Systems Kafka 101

The Pragmatic Engineer

OCTOBER 17, 2024

We recently covered how CockroachDB joins the trend of moving from open source to proprietary and why Oxide decided to keep using it with self-support , regardless Web hosting:  Netlify : chosen thanks to their super smooth preview system with SSR support. This was one section from last week’s The Pulse.

Cloud 278

Cloud AWS Metadata Cloud Computing 278

The Pragmatic Engineer

OCTOBER 10, 2024

Wordpress is the most popular content management system (CMS),  estimated  to power around 43% of all websites; a staggering number! “ With the current events, it is clear that Automattic will start to enforce commercial trademark usage  in some cases  – which they have all right to do.

Consulting 306

Consulting AWS Engineering Software Engineer 306

Data Engineering Podcast

MAY 28, 2023

Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools. Have you found it more practical to hire outside talent to build out the first data systems, or grow that talent internally?

Data Lake 162

Data Lake Machine Learning Data Warehouse Education 162

The Pragmatic Engineer

NOVEMBER 9, 2023

From to this lawsuit, we get an inside look at how events unfolded inside Frank. She asked the Director of Engineering if he could help take a known set of FAFSA application data and use it to artificially augment a much larger set of anonymous data tht her systems had collected over time. non-existent customers.

Software Engineer 227

Software Engineer Software Engineering Engineering Coding 227

Data Engineering Podcast

MAY 7, 2023

The challenge is that most companies have a multitude of systems that contain fragments of the customer's interactions and stitching that together is complex and time consuming. Segment created the Unify product to reduce the burden of building a comprehensive view of customers and synchronizing it to all of the systems that need it.

Pipeline-centric 130

Pipeline-centric Data Lake Machine Learning Data Warehouse 130

Cloudera

JUNE 17, 2024

In past years, Cloudera employees have hosted events, both in-person and virtually, to celebrate. Last year, our team in Cork, Ireland joined those efforts and hosted its first on-site Pride event. The event proved to be a resounding success, but don’t just take our word for it.

Systems 112

Systems Building IT 112

Data Engineering Podcast

MARCH 17, 2024

Join us at the top event for the global data community, Data Council Austin. Don't miss out on our only event this year! Data Council Logo]([link] Join us at the top event for the global data community, Data Council Austin. Don't miss out on our only event this year! Data Council : ![Data

Database 147

Database Data Lake High Quality Data Data Workflow 147

Data Engineering Podcast

MARCH 10, 2024

Join us at the top event for the global data community, Data Council Austin. Don't miss out on our only event this year! Data Council Logo]([link] Join us at the top event for the global data community, Data Council Austin. Don't miss out on our only event this year! Promo Code: dataengpod20 Dagster : ![Dagster

Data Lake 147

Data Lake High Quality Data Architecture Data Pipeline 147

Data Engineering Podcast

APRIL 16, 2023

As the capabilities of these systems has improved and become more accessible, the target of what self-serve means changes. Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools.

Business Intelligence 147

Business Intelligence Building Data Lake BI 147

DoorDash Engineering

MARCH 27, 2024

Real-time event processing is a critical component of a distributed system’s scalability. At DoorDash, we rely on message queue systems based on Kafka to handle billions of real-time events. One of the challenges we face, however, is how to properly validate the system before going live.

Kafka 103

Kafka Architecture Algorithm Systems 103

Snowflake

JUNE 10, 2024

Leveraging Snowflake’s Query History, Event Tables , Alerts and Notifications as the telemetry foundation, Snowflake Trail provides enhanced visibility into data quality, pipelines and applications. Events are all within Snowflake with no need for additional data transfer. Explore logs in your event table easily within Snowsight.

Python 107

Python Java Hadoop Coding 107

Robinhood

JUNE 6, 2024

Because some of these risks and uncertainties cannot be predicted or quantified and some are beyond our control, you should not rely on our forward-looking statements as predictions of future events. The post Robinhood to Acquire Bitstamp appeared first on Robinhood Newsroom.

Retail 129

Retail Systems Process Management 129

Data Engineering Podcast

JUNE 11, 2023

Summary Data engineering is all about building workflows, pipelines, systems, and interfaces to provide stable and reliable data. Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools. Sponsored By: Rudderstack : ![Rudderstack]([link]

Project 130

Project Building Data Lake Machine Learning 130

Data Engineering Podcast

MARCH 18, 2023

He also explains why data security is distinct from application security and some methods for reducing the challenge of working across different data systems. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Join in with the event for the global data community, Data Council Austin.

Data Security 130

Data Security Machine Learning Data Science Metadata 130

Netflix Tech

SEPTEMBER 10, 2024

On Titus , our multi-tenant compute platform, a "noisy neighbor" refers to a container or system service that heavily utilizes the server's resources, causing performance degradation in adjacent containers. During this event, we generate a timestamp and store it in an eBPF hash map using the process ID as the key.

Utilities 100

Utilities Coding Programming Java 100

The Pragmatic Engineer

SEPTEMBER 28, 2023

Willem Spruijt is a software engineer whom I worked on the same team with at Uber in Amsterdam, building payments systems. For example: A recently joined senior staff engineer decided to propose a project to do a re-architecture of a system. A roundup of recent events in tech, with commentary.

Software Engineer 253

Software Engineer Software Engineering Engineering Building 253

Data Engineering Podcast

NOVEMBER 6, 2022

He also describes the considerations involved in bringing behavioral data into your systems, and the ways that he and the rest of the Snowplow team are working to make that an easy addition to your platforms. What technical systems are required to generate and collect those interactions?

Building 130

Building IT Metadata MongoDB 130