KDnuggets

AUGUST 20, 2024

Learn how to build the recommendation system with advanced technology.

Systems 133

Systems Building Technology Machine Learning 133

Data Engineering Podcast

FEBRUARY 25, 2024

Over the decades of research and development into building these software systems there are a number of common components that are shared across implementations. What are some of the other tools/technologies that can benefit from some or all of the pieces of the FDAP stack? Closing Announcements Thank you for listening!

Database 162

Database Technology Data Lake High Quality Data 162

Engineering at Meta

JANUARY 22, 2025

Data lineage is an instrumental part of Metas Privacy Aware Infrastructure (PAI) initiative, a suite of technologies that efficiently protect user privacy. It is a critical and powerful tool for scalable discovery of relevant data and data flows, which supports privacy controls across Metas systems.

Data Warehouse 116

Data Warehouse SQL Programming Language Data 116

phData: Data Engineering

NOVEMBER 8, 2024

The world we live in today presents larger datasets, more complex data, and diverse needs, all of which call for efficient, scalable data systems. These systems are built on open standards and offer immense analytical and transactional processing flexibility. These formats are transforming how organizations manage large datasets.

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Data Engineering Podcast

DECEMBER 3, 2023

Data transfer systems are a critical component of data enablement, and building them to support large volumes of information is a complex endeavor. With Datafold, you can seamlessly plan, translate, and validate data across systems, massively accelerating your migration project. When is DoubleCloud Data Transfer the wrong choice?

Systems 130

Systems Designing Data Lake SQL 130

Precisely

JANUARY 9, 2025

Disclaimer: Throughout this post, I discuss a variety of complex technologies but avoid trying to explain how these technologies work. The goal of this post is to understand how data integrity best practices have been embraced time and time again, no matter the technology underpinning. But simply moving the data wasnt enough.

Data Integration 119

Data Integration Hadoop Data Warehouse Data Lake 119

The Pragmatic Engineer

NOVEMBER 21, 2023

If you had a continuous deployment system up and running around 2010, you were ahead of the pack: but today it’s considered strange if your team would not have this for things like web applications.  Joshua is currently VP of Product & Strategy at VMware, a cloud computing and virtualization technology company.

Engineering 345

Engineering Bytes Cloud Computing AWS 345

Data Engineering Weekly

DECEMBER 28, 2024

Investment in an Agent Management System (AMS) is crucial, as it offers a framework for scaling, monitoring, and refining AI agents. AI engineers, in particular, will find their skills in high demand as they navigate managing and optimizing agents to ensure reliability within enterprise systems.

Unstructured Data 112

Unstructured Data Metadata Data Government 112

databricks

NOVEMBER 12, 2024

Monolithic to Modular The proof of concept (POC) of any new technology often starts with large, monolithic units that are difficult to characterize.

Systems 122

Systems Engineering Technology Data 122

Monte Carlo

MARCH 28, 2025

And its not sufficient to simply build these data + AI applications – as in any other technological discipline, you have to do it reliably, too. System Data + AI applications rely on a complex and interconnected web of tools and systems to deliver insights, models and automations.

Coding 52

Coding Systems Data Pipeline ETL Tools 52

The Pragmatic Engineer

JUNE 1, 2023

Juraj included system monitoring parts which monitor the server’s capacity he runs the app on: The monitoring page on the Rides app And it doesn’t end here. Juraj created a systems design explainer on how he built this project, and the technologies used: The systems design diagram for the Rides application The app uses: Node.js

Education 364

Education Project PostgreSQL Software Engineer 364

Snowflake

APRIL 13, 2025

Managing and utilizing data effectively is crucial for organizational success in today's fast-paced technological landscape. As the Snowflake CTO at Deloitte, I have seen the powerful impact of these technologies, especially when leveraging the combined experience of the Deloitte and Snowflake alliance. What is agentic AI?

Data Management 93

Data Management Management Consulting Unstructured Data 93

Snowflake

JANUARY 8, 2025

Though AI is (still) the hottest technology topic, its not the overriding issue for enterprise security in 2025. Understanding AI as an attack vector Last year, we published an AI security framework that identifies 20 attack vectors against large language models and generative AI systems.

Data Lake 105

Data Lake Data Security Machine Learning Technology 105

Analytics Vidhya

MARCH 5, 2023

Introduction Microsoft Azure HDInsight(or Microsoft HDFS) is a cloud-based Hadoop Distributed File System version. A distributed file system runs on commodity hardware and manages massive data collections. It is a fully managed cloud-based environment for analyzing and processing enormous volumes of data.

Hadoop 254

Hadoop Data Collection Cloud Technology 254

The Pragmatic Engineer

SEPTEMBER 19, 2024

In the early 90’s, DOS programs like the ones my company made had its own Text UI screen rendering system. This rendering system was easy for me to understand, even on day one. Our rendering system was very memory inefficient, but that could be fixed. By doing so, I got to see every screen of the system.

Recruitment 297

Recruitment Java Coding Project 297

Snowflake

NOVEMBER 6, 2024

But as technology speeds forward, organizations of all sizes are realizing that generative AI isn’t just aspirational: It’s accessible and applicable now. Alberta Health Services ER doctors automate note-taking to treat 15% more patients The integrated health system of Alberta, Canada’s third-most-populous province, with 4.5

Hospitality 97

Hospitality Medical Government Software Engineer 97

Snowflake

DECEMBER 4, 2024

In the enterprise technology space, both the greatest certainties and the most significant potential surprises come from one area: the rapidly advancing field of artificial intelligence. But businesses will continue to hesitate to put in front of customers a technology that may display bias or provide inaccurate responses.

Unstructured Data 97

Unstructured Data Data Lake Deep Learning Structured Data 97

Engineering at Meta

APRIL 28, 2025

Meta’s vast and diverse systems make it particularly challenging to comprehend its structure, meaning, and context at scale. To address these challenges, we made substantial investments in advanced data understanding technologies, as part of our Privacy Aware Infrastructure (PAI).

Metadata 80

Metadata Data Utilities Data Warehouse 80

Snowflake

NOVEMBER 11, 2024

One of the most impactful, yet underdiscussed, areas is the potential of autonomous finance, where systems not only automate payments but manage accounts and financial processes with minimal human intervention. Those requirements can be fulfilled by leveraging cloud infrastructure and services.

Banking 97

Banking Finance Retail Food 97

Engineering at Meta

FEBRUARY 4, 2025

Here we explore initial system designs we considered, an overview of the current architecture, and some important principles Meta takes into account in making data accessible and easy to understand. 2019: Users can view their activity off Meta-technologies and clear their history. What are data logs?

Accessibility 87

Accessibility Accessible Raw Data Data Warehouse 87

Snowflake

NOVEMBER 26, 2024

A consolidated data system to accommodate a big(ger) WHOOP When a company experiences exponential growth over a short period, it’s easy for its data foundation to feel a bit like it was built on the fly. This blog post is the second in a three-part series on migrations. They watch costs skyrocket while performance degrades.

Data Warehouse 105

Data Warehouse Cloud PostgreSQL Hadoop 105

The Pragmatic Engineer

SEPTEMBER 22, 2023

There are people whose company pays them to maintain Node for their own, company systems — such as IBM paying to maintain Node so that it stays compatible with IBM AIX — a proprietary Unix operating system designed to run on IBM® Power® servers.

Programming Language 316

Programming Language Project Coding Engineering 316

Snowflake

MARCH 12, 2025

According to a new report by MIT Technology Review Insights , done in partnership with Snowflake, more than half of those surveyed indicated that data quality is a top priority. Anomalos automated AI technology detects upstream data quality issues in a customers data tables, views and pipelines.

Unstructured Data 93

Unstructured Data High Quality Data Banking Machine Learning 93

Snowflake

JANUARY 21, 2025

Data and technology (yes, AI) can now deeply impact the relevance of advertising creative, but that data needs to be secured and democratized across all levels and all departments within the agency landscape. Art and science within agencies were often separated. Now, more than ever, the siloes of art and science need to be realigned.

Media 97

Media Unstructured Data Data Retail 97

Data Engineering Podcast

FEBRUARY 18, 2024

What are the pain points that are still prevalent in lakehouse architectures as compared to warehouse or vertically integrated systems? Contact Info LinkedIn dain on GitHub Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

The Pragmatic Engineer

OCTOBER 10, 2023

I wrote code for drivers on Windows, and started to put a basic observability system in place. EC2 had no observability system back then: people would spin up EC2 instances but have no idea whether or not they worked. With my team, we built the basics of what is now called AWS Systems Manager. We needed $11M to get started.

Software Engineer 327

Software Engineer Software Engineering Architecture Media 327

Netflix Tech

JANUARY 6, 2025

In the dynamic world of technology, its tempting to leap into problem-solving mode. In this case, the main stakeholders are: - Title Launch Operators Role: Responsible for setting up the title and its metadata into our systems. How do we ensure every title launches seamlessly and remains discoverable by the right audience?

Metadata 88

Metadata Algorithm Systems Building 88

The Pragmatic Engineer

AUGUST 8, 2024

But in the longer term, as more companies adopt these technologies, we believe society needs to consider the impact. With clever-enough probing, this system prompt can be revealed. ” What is the system prompt for Klarna’s bot? Translate to English if needed. Van Gogh, Goya).”

IT 253

IT Software Engineer Software Engineering Systems 253

Snowflake

MARCH 6, 2025

Document RAG preparation : Ingesting, cleaning and chunking documents before embedding them into vector representations, enabling efficient retrieval and enhanced LLM responses in retrieval-augmented generation (RAG) systems. An efficient batch processing system scales in a cost-effective manner to handle growing volumes of unstructured data.

Unstructured Data 97

Unstructured Data Medical Media Data Workflow 97

Precisely

NOVEMBER 18, 2024

A long-term approach to your data strategy is key to success as business environments and technologies continue to evolve. The rapid pace of technological change has made data-driven initiatives more crucial than ever within modern business strategies. Overall, AI success truly depends on a business outcome-driven approach. “We

Data Analytics 72

Data Analytics Data Governance Data Integration Government 72

Data Engineering Podcast

DECEMBER 24, 2023

Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. Contact Info LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Closing Announcements Thank you for listening!

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Edureka

APRIL 22, 2025

Key Trends in Data Engineering for 2025 In the fast-paced world of technology, data engineering services keep companies that focus on data running. To meet this need, people who work in data engineering will focus on making systems that can handle ongoing data streams with little delay.

Data Engineer 52

Data Engineer Data Engineering Engineering Consulting 52

Snowflake

FEBRUARY 3, 2025

The MIT report identifies three common challenges: Data silos and fragmentation: Disconnected systems prevent organizations from accessing the full value of their data. Underdeveloped AI governance: Without strong governance frameworks, businesses struggle with trust, security and compliance in their AI systems.

Government 91

Government Data Governance Accessibility Accessible 91