Data Engineering Podcast

DECEMBER 3, 2023

Data transfer systems are a critical component of data enablement, and building them to support large volumes of information is a complex endeavor. With Datafold, you can seamlessly plan, translate, and validate data across systems, massively accelerating your migration project. When is DoubleCloud Data Transfer the wrong choice?

Systems 130

Systems Designing Data Lake SQL 130

Engineering at Meta

JANUARY 22, 2025

Data lineage is an instrumental part of Metas Privacy Aware Infrastructure (PAI) initiative, a suite of technologies that efficiently protect user privacy. It is a critical and powerful tool for scalable discovery of relevant data and data flows, which supports privacy controls across Metas systems.

Data Warehouse 109

Data Warehouse SQL Programming Language Data 109

Precisely

JANUARY 9, 2025

Disclaimer: Throughout this post, I discuss a variety of complex technologies but avoid trying to explain how these technologies work. The goal of this post is to understand how data integrity best practices have been embraced time and time again, no matter the technology underpinning. But simply moving the data wasnt enough.

Data Integration 119

Data Integration Hadoop Data Warehouse Data Lake 119

Knowledge Hut

APRIL 25, 2024

Big data in information technology is used to improve operations, provide better customer service, develop customized marketing campaigns, and take other actions to increase revenue and profits. In the world of technology, things are always changing. In this blog post, we will discuss such technologies.

Big Data 98

Big Data Technology Hadoop NoSQL 98

Data Engineering Weekly

DECEMBER 28, 2024

Investment in an Agent Management System (AMS) is crucial, as it offers a framework for scaling, monitoring, and refining AI agents. AI engineers, in particular, will find their skills in high demand as they navigate managing and optimizing agents to ensure reliability within enterprise systems.

Unstructured Data 112

Unstructured Data Metadata Data Government 112

The Pragmatic Engineer

NOVEMBER 21, 2023

If you had a continuous deployment system up and running around 2010, you were ahead of the pack: but today it’s considered strange if your team would not have this for things like web applications.  Joshua is currently VP of Product & Strategy at VMware, a cloud computing and virtualization technology company.

Engineering 325

Engineering Bytes Cloud Computing AWS 325

databricks

NOVEMBER 12, 2024

Monolithic to Modular The proof of concept (POC) of any new technology often starts with large, monolithic units that are difficult to characterize.

Systems 122

Systems Engineering Technology Data 122

Knowledge Hut

MARCH 22, 2024

I have comprehensively analyzed the area of physical security, particularly the ongoing discussion surrounding fail safe vs fail-safe secure electric strike locking systems. On the other hand, fail-secure systems focus on maintaining continuous security, keeping doors locked even in difficult conditions to protect assets.

Systems 105

Systems Electronics Hospitality Architecture 105

The Pragmatic Engineer

JUNE 1, 2023

Juraj included system monitoring parts which monitor the server’s capacity he runs the app on: The monitoring page on the Rides app And it doesn’t end here. Juraj created a systems design explainer on how he built this project, and the technologies used: The systems design diagram for the Rides application The app uses: Node.js

Education 363

Education Project PostgreSQL Software Engineering 363

Snowflake

JANUARY 8, 2025

Though AI is (still) the hottest technology topic, its not the overriding issue for enterprise security in 2025. Understanding AI as an attack vector Last year, we published an AI security framework that identifies 20 attack vectors against large language models and generative AI systems.

Data Lake 105

Data Lake Data Security Machine Learning Technology 105

Knowledge Hut

MAY 6, 2024

Applying systems thinking views a system as a set of interconnected and interdependent components defined by its limits and more than the sum of their parts (subsystems). When one component of a system is altered, the effects frequently spread across the entire system. are the main objectives of systems thinking.

Systems 98

Systems Transportation Certification Healthcare 98

Analytics Vidhya

MARCH 5, 2023

Introduction Microsoft Azure HDInsight(or Microsoft HDFS) is a cloud-based Hadoop Distributed File System version. A distributed file system runs on commodity hardware and manages massive data collections. It is a fully managed cloud-based environment for analyzing and processing enormous volumes of data.

Hadoop 254

Hadoop Data Collection Cloud Technology 254

Data Engineering Podcast

MAY 18, 2024

Summary The purpose of business intelligence systems is to allow anyone in the business to access and decode data to help them make informed decisions. Contact Info Ryan LinkedIn Paul LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

Building 278

Building Data Lake High Quality Data Business Intelligence 278

Engineering at Meta

MARCH 4, 2025

QLC technology addresses these challenges by forming a middle tier between HDDs and TLC SSDs. QLC flash as a technology has been around since 2009. Pure Storages DFMs can allow scaling up to 600TB with the same NAND package technology. As discussed above, our QLC systems are very high in density.

Bytes 101

Bytes Media Data Technology 101

Snowflake

NOVEMBER 6, 2024

But as technology speeds forward, organizations of all sizes are realizing that generative AI isn’t just aspirational: It’s accessible and applicable now. Alberta Health Services ER doctors automate note-taking to treat 15% more patients The integrated health system of Alberta, Canada’s third-most-populous province, with 4.5

Hospitality 97

Hospitality Medical Government Software Engineering 97

Snowflake

DECEMBER 4, 2024

In the enterprise technology space, both the greatest certainties and the most significant potential surprises come from one area: the rapidly advancing field of artificial intelligence. But businesses will continue to hesitate to put in front of customers a technology that may display bias or provide inaccurate responses.

Unstructured Data 97

Unstructured Data Data Lake Deep Learning Structured Data 97

The Pragmatic Engineer

SEPTEMBER 19, 2024

In the early 90’s, DOS programs like the ones my company made had its own Text UI screen rendering system. This rendering system was easy for me to understand, even on day one. Our rendering system was very memory inefficient, but that could be fixed. By doing so, I got to see every screen of the system.

Recruitment 260

Recruitment Java Coding Project 260

Snowflake

NOVEMBER 26, 2024

A consolidated data system to accommodate a big(ger) WHOOP When a company experiences exponential growth over a short period, it’s easy for its data foundation to feel a bit like it was built on the fly. This blog post is the second in a three-part series on migrations. They watch costs skyrocket while performance degrades.

Data Warehouse 105

Data Warehouse Cloud PostgreSQL Hadoop 105

Data Engineering Podcast

FEBRUARY 18, 2024

What are the pain points that are still prevalent in lakehouse architectures as compared to warehouse or vertically integrated systems? Contact Info LinkedIn dain on GitHub Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Engineering at Meta

FEBRUARY 4, 2025

Here we explore initial system designs we considered, an overview of the current architecture, and some important principles Meta takes into account in making data accessible and easy to understand. 2019: Users can view their activity off Meta-technologies and clear their history. What are data logs?

Accessibility 80

Accessibility Accessible Raw Data Data Warehouse 80

Snowflake

MARCH 12, 2025

For example, ticketing, merchandise, fantasy engagement and game viewership data often reside in separate systems (or with separate entities), making it a challenge to bring together a cohesive view of each fan. Sports entity data teams are often mighty but small making complex technology solutions unrealistic to leverage.

Media 70

Media Cloud Programming Data Collection 70

Jesse Anderson

OCTOBER 5, 2023

Data quality problems are solved by more than just technology. Data products need to be exposed with the right technologies for the various use cases. In another post , I introduced a concept from Justin Coffey about how much systems can be changed from their original design. Flink is an excellent piece of technology.

Kafka 195

Kafka Finance Cloud Designing 195

Snowflake

FEBRUARY 3, 2025

The MIT report identifies three common challenges: Data silos and fragmentation: Disconnected systems prevent organizations from accessing the full value of their data. Underdeveloped AI governance: Without strong governance frameworks, businesses struggle with trust, security and compliance in their AI systems.

Government 93

Government Data Governance Accessibility Accessible 93

The Pragmatic Engineer

SEPTEMBER 22, 2023

There are people whose company pays them to maintain Node for their own, company systems — such as IBM paying to maintain Node so that it stays compatible with IBM AIX — a proprietary Unix operating system designed to run on IBM® Power® servers.

Programming Language 281

Programming Language Project Coding Engineering 281

Precisely

NOVEMBER 18, 2024

A long-term approach to your data strategy is key to success as business environments and technologies continue to evolve. The rapid pace of technological change has made data-driven initiatives more crucial than ever within modern business strategies. Overall, AI success truly depends on a business outcome-driven approach. “We

Data Analytics 72

Data Analytics Data Governance Data Integration Government 72

Netflix Tech

JANUARY 6, 2025

In the dynamic world of technology, its tempting to leap into problem-solving mode. In this case, the main stakeholders are: - Title Launch Operators Role: Responsible for setting up the title and its metadata into our systems. How do we ensure every title launches seamlessly and remains discoverable by the right audience?

Metadata 86

Metadata Algorithm Systems Building 86

Data Engineering Podcast

DECEMBER 24, 2023

Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. Contact Info LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Closing Announcements Thank you for listening!

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Data Engineering Podcast

APRIL 21, 2024

Summary Generative AI has rapidly transformed everything in the technology sector. How do you manage the personalization of the AI functionality in your system for each user/team? Contact Info LinkedIn Blog Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

Data Lake 182

Data Lake High Quality Data Data Pipeline Machine Learning 182

Data Engineering Weekly

JANUARY 15, 2025

The answer lies in unstructured data processing—a field that powers modern artificial intelligence (AI) systems. To address these challenges, AI Data Engineers have emerged as key players, designing scalable data workflows that fuel the next generation of AI systems. How does a self-driving car understand a chaotic street scene?

Data Engineer 62

Data Engineer Data Engineering Unstructured Data Engineering 62

Data Engineering Podcast

SEPTEMBER 10, 2023

Summary Data systems are inherently complex and often require integration of multiple technologies. In this episode Nick Schrock, creator of Dagster, shares his perspective on the state of data orchestration technology and its application to help inform its implementation in your environment.

BI 208

BI SQL Machine Learning Data 208

dbt Developer Hub

JANUARY 22, 2025

Ever since dbt Labs acquired SDF Labs last week , I've been head-down diving into their technology and making sense of it all. For the first time, SDF provides the technology necessary to make this possible. At Level 2, the system produces a complete Logical Plan. Data errors can only be captured by a Level 3 system.

SQL 78

SQL Database Coding Technology 78

The Pragmatic Engineer

OCTOBER 10, 2023

I wrote code for drivers on Windows, and started to put a basic observability system in place. EC2 had no observability system back then: people would spin up EC2 instances but have no idea whether or not they worked. With my team, we built the basics of what is now called AWS Systems Manager. We needed $11M to get started.

Software Engineering 293

Software Engineering Software Engineer Architecture Media 293

Netflix Tech

FEBRUARY 14, 2025

It requires a state-of-the-art system that can track and process these impressions while maintaining a detailed history of each profiles exposure. This nuanced integration of data and technology empowers us to offer bespoke content recommendations.

Kafka 66

Kafka Datasets Metadata Utilities 66

The Pragmatic Engineer

APRIL 19, 2023

I was sceptical that any system would automatically reject resumes, because I never saw this as a hiring manager. I've observed several hype cycles in tech when people get involved in new areas in technologies, when they lack understanding of how things down. " that hundreds of sites took over on the internet.

Software Engineering 319

Software Engineering Software Engineer Engineering Media 319