Data Engineering Podcast

MAY 26, 2024

Summary Any software system that survives long enough will require some form of migration or evolution. When that system is responsible for the data layer the process becomes more challenging. As you have gone through successive migration projects, how has that influenced the ways that you think about architecting data systems?

Systems 130

Systems Data Lake High Quality Data Google Cloud 130

Precisely

JANUARY 9, 2025

Disclaimer: Throughout this post, I discuss a variety of complex technologies but avoid trying to explain how these technologies work. The goal of this post is to understand how data integrity best practices have been embraced time and time again, no matter the technology underpinning. Can AIs responses be trusted?

Data Integration 119

Data Integration Hadoop Data Warehouse Data Lake 119

Engineering at Meta

JANUARY 22, 2025

Data lineage is an instrumental part of Metas Privacy Aware Infrastructure (PAI) initiative, a suite of technologies that efficiently protect user privacy. It is a critical and powerful tool for scalable discovery of relevant data and data flows, which supports privacy controls across Metas systems.

Data Warehouse 109

Data Warehouse SQL Programming Language Data 109

Data Engineering Podcast

FEBRUARY 25, 2024

Over the decades of research and development into building these software systems there are a number of common components that are shared across implementations. What are some of the other tools/technologies that can benefit from some or all of the pieces of the FDAP stack? Closing Announcements Thank you for listening!

Database 162

Database Technology Data Lake High Quality Data 162

The Pragmatic Engineer

NOVEMBER 21, 2023

If you had a continuous deployment system up and running around 2010, you were ahead of the pack: but today it’s considered strange if your team would not have this for things like web applications. Joshua is currently VP of Product & Strategy at VMware, a cloud computing and virtualization technology company.

Engineering 345

Engineering Bytes Cloud Computing AWS 345

Data Engineering Weekly

DECEMBER 28, 2024

Investment in an Agent Management System (AMS) is crucial, as it offers a framework for scaling, monitoring, and refining AI agents. AI engineers, in particular, will find their skills in high demand as they navigate managing and optimizing agents to ensure reliability within enterprise systems.

Unstructured Data 113

Unstructured Data Metadata Data Government 113

RandomTrees

NOVEMBER 25, 2024

The scope of telecom services is growing in size and complexity, owing to technologies such as 5G, the Internet of Things (IoT), and cloud technology. And one technology that has potential to transform the telecom sector is Generative AI , or GAI, which lies in the focus of creating new things, be it content, ideas or solutions.

Telecommunication 52

Telecommunication IT Unstructured Data Data Mining 52

Snowflake

APRIL 22, 2025

We're talking about potential headaches around data quality, the amount of data your systems can handle, the risk of bias getting amplified and those privacy concerns the ones about proprietary business information or customer personal data possibly leaking out in the outputs. These essential steps introduce some real challenges.

IT 59

IT Unstructured Data Government Cloud 59

phData: Data Engineering

NOVEMBER 8, 2024

The world we live in today presents larger datasets, more complex data, and diverse needs, all of which call for efficient, scalable data systems. These systems are built on open standards and offer immense analytical and transactional processing flexibility. In this blog, we will discuss: What is the Open Table format (OTF)?

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Data Engineering Podcast

DECEMBER 3, 2023

Data transfer systems are a critical component of data enablement, and building them to support large volumes of information is a complex endeavor. With Datafold, you can seamlessly plan, translate, and validate data across systems, massively accelerating your migration project. When is DoubleCloud Data Transfer the wrong choice?

Systems 130

Systems Designing Data Lake SQL 130

The Pragmatic Engineer

JUNE 1, 2023

Juraj included system monitoring parts which monitor the server’s capacity he runs the app on: The monitoring page on the Rides app And it doesn’t end here. Juraj included system monitoring parts which monitor the server’s capacity he runs the app on: The monitoring page on the Rides app And it doesn’t end here.

Education 364

Education Project PostgreSQL Software Engineering 364

Snowflake

JANUARY 8, 2025

Though AI is (still) the hottest technology topic, its not the overriding issue for enterprise security in 2025. Understanding AI as an attack vector Last year, we published an AI security framework that identifies 20 attack vectors against large language models and generative AI systems.

Data Lake 105

Data Lake Data Security Machine Learning Technology 105

The Pragmatic Engineer

SEPTEMBER 19, 2024

In the early 90’s, DOS programs like the ones my company made had its own Text UI screen rendering system. This rendering system was easy for me to understand, even on day one. Our rendering system was very memory inefficient, but that could be fixed. By doing so, I got to see every screen of the system.

Recruitment 297

Recruitment Java Coding Project 297

Snowflake

NOVEMBER 6, 2024

But as technology speeds forward, organizations of all sizes are realizing that generative AI isn’t just aspirational: It’s accessible and applicable now. Alberta Health Services ER doctors automate note-taking to treat 15% more patients The integrated health system of Alberta, Canada’s third-most-populous province, with 4.5

Hospitality 97

Hospitality Medical Government Software Engineering 97

LinkedIn Engineering

OCTOBER 5, 2023

The magic in search happens because of the technology that powers it. And one of the key technologies behind LinkedIn's search and recommendation features is embedding based retrieval (EBR). LinkedIn is on the forefront of leveraging EBR technology to revolutionize the way we approach search and recommendation systems.

IT 133

IT Metadata Designing Cloud 133

Monte Carlo

MARCH 28, 2025

And its not sufficient to simply build these data + AI applications – as in any other technological discipline, you have to do it reliably, too. And its not sufficient to simply build these data + AI applications – as in any other technological discipline, you have to do it reliably, too. The commodities just keep coming.

Coding 52

Coding Systems Data Pipeline ETL Tools 52

Analytics Vidhya

MARCH 5, 2023

Introduction Microsoft Azure HDInsight(or Microsoft HDFS) is a cloud-based Hadoop Distributed File System version. A distributed file system runs on commodity hardware and manages massive data collections. It is a fully managed cloud-based environment for analyzing and processing enormous volumes of data.

Hadoop 254

Hadoop Data Collection Cloud Technology 254

Snowflake

DECEMBER 4, 2024

In the enterprise technology space, both the greatest certainties and the most significant potential surprises come from one area: the rapidly advancing field of artificial intelligence. But businesses will continue to hesitate to put in front of customers a technology that may display bias or provide inaccurate responses.

Unstructured Data 97

Unstructured Data Data Lake Deep Learning Structured Data 97

Snowflake

AUGUST 1, 2024

Contextual AI provides an all-in-one RAG solution A typical RAG system combines a vector database, embedding models and other components that must be engineered and integrated together. technology seamlessly integrates retrieval and generation to deliver unmatched accuracy for RAG workloads without any additional, error-prone system tuning.

Cloud 116

Cloud IT Technology Government 116

The Pragmatic Engineer

SEPTEMBER 22, 2023

There are people whose company pays them to maintain Node for their own, company systems — such as IBM paying to maintain Node so that it stays compatible with IBM AIX — a proprietary Unix operating system designed to run on IBM® Power® servers. Many developers are giving Bun a go. But how is Bun so fast?

Programming Language 316

Programming Language Project Coding Engineering 316

Snowflake

NOVEMBER 26, 2024

A consolidated data system to accommodate a big(ger) WHOOP When a company experiences exponential growth over a short period, it’s easy for its data foundation to feel a bit like it was built on the fly. That’s why we’ve collected these migration success stories to help you get started on your migration to Snowflake.

Data Warehouse 105

Data Warehouse Cloud PostgreSQL Hadoop 105

Snowflake

APRIL 13, 2025

Managing and utilizing data effectively is crucial for organizational success in today's fast-paced technological landscape. As the Snowflake CTO at Deloitte, I have seen the powerful impact of these technologies, especially when leveraging the combined experience of the Deloitte and Snowflake alliance. What is agentic AI?

Data Management 80

Data Management Management Consulting Unstructured Data 80

Edureka

APRIL 2, 2025

What is Steganography? It is the practice of concealing information within ordinary files or media, making the hidden data undetectable to anyone unaware of its presence. From ancient techniques like invisible ink to modern digital methods that embed messages in images, audio, or network traffic, it has evolved significantly.

IT 40

IT Metadata Algorithm Coding 40

Snowflake

APRIL 2, 2025

The data warehouse solved for performance and scale but, much like the databases that preceded it, relied on proprietary formats to build vertically integrated systems. As technology continues to evolve, Snowflake continues to prioritize its customers by supporting open source initiatives. Now you dont have to choose. What is Iceberg?

Data Lake 97

Data Lake Cloud Storage Metadata Data Warehouse 97

The Pragmatic Engineer

OCTOBER 10, 2023

We don’t frequently have CEOs on The Pragmatic Engineer: in fact, today is the first issue when we’re talking with a CEO – partially – about their CEO job. I’ve always wondered what it would be like to go from a developer, to eventually become the CEO of a large and growing company.

Software Engineer 327

Software Engineer Software Engineering Architecture Media 327

Snowflake

MARCH 12, 2025

According to a new report by MIT Technology Review Insights , done in partnership with Snowflake, more than half of those surveyed indicated that data quality is a top priority. Anomalos automated AI technology detects upstream data quality issues in a customers data tables, views and pipelines.

Unstructured Data 95

Unstructured Data High Quality Data Banking Machine Learning 95

Snowflake

MARCH 14, 2024

Changes to SAP’s warehousing strategy over time mean many customers may need to patch together and manage multiple technologies to extract data in a useful format. Glue provides a simple, direct way for organizations with SAP systems to quickly and securely ingest SAP data into Snowflake. Try SNP Glue Connector for SAP: Get it now!

IT 121

IT Data Ingestion Data AWS 121

Snowflake

JANUARY 21, 2025

Data and technology (yes, AI) can now deeply impact the relevance of advertising creative, but that data needs to be secured and democratized across all levels and all departments within the agency landscape. Now, more than ever, the siloes of art and science need to be realigned.

Media 98

Media Unstructured Data Data Retail 98

The Pragmatic Engineer

APRIL 19, 2023

When I think back on the software engineers I looked up to, they all shared this trait where they never took anything at face value. They regularly questioned statements that did not make sense to them, no matter how small the topic was: even if it involved admitting they did not understand a concept. This person explained it. Or does it?

Software Engineering 343

Software Engineering Software Engineer Engineering Media 343

Snowflake

NOVEMBER 11, 2024

One of the most impactful, yet underdiscussed, areas is the potential of autonomous finance, where systems not only automate payments but manage accounts and financial processes with minimal human intervention. Those requirements can be fulfilled by leveraging cloud infrastructure and services.

Banking 98

Banking Finance Retail Food 98

Engineering at Meta

APRIL 28, 2025

Meta’s vast and diverse systems make it particularly challenging to comprehend its structure, meaning, and context at scale. To address these challenges, we made substantial investments in advanced data understanding technologies, as part of our Privacy Aware Infrastructure (PAI).

Metadata 64

Metadata Data Utilities Data Warehouse 64

Data Engineering Podcast

FEBRUARY 18, 2024

What are the pain points that are still prevalent in lakehouse architectures as compared to warehouse or vertically integrated systems? Contact Info LinkedIn dain on GitHub Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Engineering at Meta

FEBRUARY 4, 2025

Here we explore initial system designs we considered, an overview of the current architecture, and some important principles Meta takes into account in making data accessible and easy to understand. 2019: Users can view their activity off Meta-technologies and clear their history. What are data logs?

Accessible 80

Accessible Accessibility Raw Data Data Warehouse 80

The Pragmatic Engineer

MAY 11, 2023

A quick summary of these technologies: Prometheus : a time series database. A very popular open-source solution for systems and services monitoring. The internet has been speculating the past few days on which crypto company spent $65M on Datadog in 2022. I confirmed it was Coinbase, and here are the details of what happened.

AWS 347

AWS Software Engineering Software Engineer Google Cloud 347

Netflix Tech

JANUARY 6, 2025

In the dynamic world of technology, its tempting to leap into problem-solving mode. In this case, the main stakeholders are: - Title Launch Operators Role: Responsible for setting up the title and its metadata into our systems. How do we ensure every title launches seamlessly and remains discoverable by the right audience?

Metadata 87

Metadata Algorithm Systems Building 87

Edureka

FEBRUARY 12, 2025

Applications: BERT powers various real-world applications, including search engines, voice assistants, and advanced text classification systems. Its ability to understand nuanced language has revolutionized NLP tasks, making it a cornerstone of modern AI systems. Optimizing for the task by adjusting hyperparameters like learning rate.

IT 40

IT Banking Datasets Architecture 40