phData: Data Engineering

NOVEMBER 8, 2024

The world we live in today presents larger datasets, more complex data, and diverse needs, all of which call for efficient, scalable data systems. These systems are built on open standards and offer immense analytical and transactional processing flexibility. These formats are transforming how organizations manage large datasets.

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Edureka

APRIL 10, 2025

When you hear the term System Hacking, it might bring to mind shadowy figures behind computer screens and high-stakes cyber heists. In this blog, we’ll explore the definition, purpose, process, and methods of prevention related to system hacking, offering a detailed overview to help demystify the concept.

Systems 40

Systems Education Accessibility Accessible 40

Monte Carlo

MARCH 28, 2025

Table of Contents Understanding How Data + AI Can Break Data System Code Model Data + AI observability must cover inputs and outputs it is all or nothing Understanding How Data + AI Can Break Data + AI applications are complex. But code takes on new weight in the data + AI system.

Coding 52

Coding Systems Data Pipeline ETL Tools 52

Start Data Engineering

JANUARY 20, 2025

Introduction 2. Guide the interviewer through the process 2.1. Requirements gathering] Make sure you clearly understand the requirements & business use case 2.2. Understand source data] Know what you have to work with 2.3. Model your data] Define data models for historical analytics 2.4.

Designing 130

Designing Systems Data Engineering Data Engineer 130

KDnuggets

APRIL 2, 2025

Let's discover what small language models (SLMs) are, how they can be used in RAG systems and applications, and when to use them over their large language counterparts.

Systems 97

Systems 97

Striim

JANUARY 30, 2025

Systems must be capable of handling high-velocity data without bottlenecks. However, leveraging AI agents like Striims Sherlock and Sentinel, which enable encryption and masking for PII, can help ensure that data is safe even in the event a breach occurs. As you can see, theres a lot to consider in adopting real-time AI.

Systems 52

Systems Management Hospitality Healthcare 52

Engineering at Meta

JANUARY 22, 2025

It is a critical and powerful tool for scalable discovery of relevant data and data flows, which supports privacy controls across Metas systems. It enhances the traceability of data flows within systems, ultimately empowering developers to swiftly implement privacy controls and create innovative products. Hack, C++, Python, etc.)

Data Warehouse 109

Data Warehouse SQL Programming Language Data 109

KDnuggets

MARCH 7, 2025

Explore how AI agents are transforming industries, from chatbots to autonomous vehicles, and learn what data scientists need to know to implement them effectively.

Systems 108

Systems Data 108

Simon Späti

MARCH 28, 2025

The database landscape has reached 394 ranked systems across multiple categoriesrelational, document, key-value, graph, search engine, time series, and the rapidly emerging vector databases. And perhaps most importantlydo we really need separate systems for these workloads?

Technology 130

Technology PostgreSQL MySQL Database 130

The Pragmatic Engineer

NOVEMBER 21, 2023

If you had a continuous deployment system up and running around 2010, you were ahead of the pack: but today it’s considered strange if your team would not have this for things like web applications.  We dabbled in network engineering, database management, and system administration. and hand-rolled C -code.

Engineering 338

Engineering Bytes Cloud Computing AWS 338

Netflix Tech

NOVEMBER 12, 2024

Failures in a distributed system are a given, and having the ability to safely retry requests enhances the reliability of the service. Implementing idempotency would likely require using an external system for such keys, which can further degrade performance or cause race conditions.

Datasets 100

Datasets Computer Science Systems Kafka 100

The Pragmatic Engineer

SEPTEMBER 19, 2024

In the early 90’s, DOS programs like the ones my company made had its own Text UI screen rendering system. This rendering system was easy for me to understand, even on day one. Our rendering system was very memory inefficient, but that could be fixed. By doing so, I got to see every screen of the system.

Recruitment 279

Recruitment Java Coding Project 279

Data Engineering Weekly

DECEMBER 28, 2024

Investment in an Agent Management System (AMS) is crucial, as it offers a framework for scaling, monitoring, and refining AI agents. AI engineers, in particular, will find their skills in high demand as they navigate managing and optimizing agents to ensure reliability within enterprise systems.

Unstructured Data 112

Unstructured Data Metadata Data Government 112

Christophe Blefari

JANUARY 11, 2025

AI companies are aiming for the moon—AGI—promising it will arrive once OpenAI develops a system capable of generating at least $100 billion in profits. Meaning: a YAML configuration system for ingestion and transformations, and now, visualisation with BI-as-code. Meanwhile, the AI landscape remains unpredictable.

Data 130

Data Data Warehouse Coding Programming Language 130

The Pragmatic Engineer

MARCH 12, 2024

Tools and approaches at our disposal, which didn’t exist in 1975, or were not widespread in 1995, include: Git – the now-dominant version control system used by much of the industry, with exceptions for projects with very large assets, like video games Code reviews : these became common in parallel with version control.

Engineering 317

Engineering Programming Language Hospitality Programming 317

Jesse Anderson

FEBRUARY 11, 2025

Semih is a researcher and entrepreneur with a background in distributed systems and databases. He then pursued his doctoral studies at Stanford University, delving into the complexities of database systems.

Computer Science 130

Computer Science Database Design Software Engineering Software Engineer 130

Snowflake

JANUARY 27, 2025

Additionally, the infrastructure supporting our systems was unreliable under heavy load, requiring manual retries and frustrating developers with lost productivity. SnowCI Historically, our CI system was powered by sprawling, disjointed Jenkins scripts that had grown over the years.

Programming Language 93

Programming Language Coding Cloud Systems 93

Snowflake

MARCH 18, 2025

DeepTempo defends against increasingly sophisticated and relentless attackers through deep learning that is proven to find advanced attacks that earlier generation systems miss while reducing costs thanks to low false positives and our partnership with Snowflake. Aging rules-based systems are failing to detect attacks effectively.

Deep Learning 92

Deep Learning Banking Government Systems 92

Precisely

JANUARY 9, 2025

The simple idea was, hey how can we get more value from the transactional data in our operational systems spanning finance, sales, customer relationship management, and other siloed functions. There was no easy way to consolidate and analyze this data to more effectively manage our business. But simply moving the data wasnt enough.

Data Integration 119

Data Integration Hadoop Data Warehouse Data Lake 119

The Pragmatic Engineer

JUNE 1, 2023

Juraj included system monitoring parts which monitor the server’s capacity he runs the app on: The monitoring page on the Rides app And it doesn’t end here. Juraj created a systems design explainer on how he built this project, and the technologies used: The systems design diagram for the Rides application The app uses: Node.js

Education 363

Education Project PostgreSQL Software Engineering 363

Snowflake

JANUARY 8, 2025

Understanding AI as an attack vector Last year, we published an AI security framework that identifies 20 attack vectors against large language models and generative AI systems. In particular, the ability of AI systems to summarize security incidents is going to be a great advance.

Data Lake 105

Data Lake Data Security Machine Learning Technology 105

Seattle Data Guy

JANUARY 18, 2025

Because they can preserve the visual layout of documents and are compatible with a wide range of devices and operating systems, PDFs are used for everything from business forms and educational material to creative designs. PDF files are one of the most popular file formats today.

IT 130

IT Education Data Designing 130

Simon Späti

FEBRUARY 26, 2025

Traditionally, answering this question would require expensive GIS (Geographic Information Systems) software or complex database setups. ” This seemingly simple question requires analyzing competitor locations, population density, traffic patterns, and demographicsall spatial data.

Database 130

Database Data Engineering Data Engineer Accessibility 130

The Pragmatic Engineer

FEBRUARY 29, 2024

But first, a few current cases of systems whose developers didn’t: In Sweden, card payments are down at a leading supermarket chain. Airline Avianca printed tickets dated as 3/1 instead of 2/29, thanks to their system not accounting for the leap day. The system was almost fully restored before noon.”

Software Engineering 263

Software Engineering Software Engineer Banking Engineering 263

Jesse Anderson

OCTOBER 29, 2024

He sees logs as a treasure trove of insights and believes effective log analysis is critical in today’s complex systems. We discussed his early experiences with distributed systems, including his work on creating graphs and entity resolution. Lastly, we go in-depth into Scanner.dev, covering what it is and how it works.

Data Engineering 100

Data Engineering Data Engineer Systems Engineering 100

Simon Späti

MARCH 11, 2025

Therefore, you’ve probably come across terms like OLAP (Online Analytical Processing) systems, data warehouses, and, more recently, real-time analytical databases. But data volumes grow, analytical demands become more complex, and Postgres stops being enough.

Database 130

Database Data Warehouse Data Engineering Data Engineer 130

The Pragmatic Engineer

AUGUST 8, 2024

When an “out of bounds” topic comes up, the chatbot hands over to a human agent: My first attempt to get the chatbot to talk about non-shopping related topics led to a swift handoff to a human agent Poking at the system prompt A system prompt is a way to provide context and instructions before passing on the user input.

IT 236

IT Software Engineer Software Engineering Systems 236

The Pragmatic Engineer

APRIL 11, 2024

From Sella’s status page : “Following the installation of an update to the operating system and related firmware which led to an unstable situation. Still, I’m puzzled by how long the system has been down. If it was an update to Oracle, or to the operating system, then why not roll back the update?

Banking 255

Banking Utilities Database Engineering 255

Analytics Vidhya

MARCH 5, 2023

Introduction Microsoft Azure HDInsight(or Microsoft HDFS) is a cloud-based Hadoop Distributed File System version. A distributed file system runs on commodity hardware and manages massive data collections. It is a fully managed cloud-based environment for analyzing and processing enormous volumes of data.

Hadoop 254

Hadoop Data Collection Cloud Technology 254

Analytics Vidhya

FEBRUARY 27, 2023

Introduction The Hadoop Distributed File System (HDFS) is a Java-based file system that is Distributed, Scalable, and Portable. Still, it does include shell commands and Java Application Programming Interface (API) functions that are similar to other file systems.

Hadoop 240

Hadoop Java Data Storage Programming 240

Netflix Tech

JANUARY 6, 2025

In this case, the main stakeholders are: - Title Launch Operators Role: Responsible for setting up the title and its metadata into our systems. In this context, were focused on developing systems that ensure successful title launches, build trust between content creators and our brand, and reduce engineering operational overhead.

Metadata 87

Metadata Algorithm Systems Building 87