Edureka

APRIL 10, 2025

When you hear the term System Hacking, it might bring to mind shadowy figures behind computer screens and high-stakes cyber heists. In this blog, we’ll explore the definition, purpose, process, and methods of prevention related to system hacking, offering a detailed overview to help demystify the concept.

Systems 40

Systems Education Banking Accessibility 40

Christophe Blefari

JANUARY 11, 2025

Over the past four weeks, I took a break from blogging and LinkedIn to focus on building nao. AI companies are aiming for the moon—AGI—promising it will arrive once OpenAI develops a system capable of generating at least $100 billion in profits. Meanwhile, the AI landscape remains unpredictable.

Data 130

Data Data Warehouse Coding Programming Language 130

Striim

JANUARY 30, 2025

Systems must be capable of handling high-velocity data without bottlenecks. However, leveraging AI agents like Striims Sherlock and Sentinel, which enable encryption and masking for PII, can help ensure that data is safe even in the event a breach occurs. As you can see, theres a lot to consider in adopting real-time AI.

Systems 52

Systems Management Hospitality Healthcare 52

The Pragmatic Engineer

MAY 12, 2023

For a realtime alerting system! I have since talked with engineers on the OpsGenie team who said that it felt that Atlassian rushed the OpsGenie integration - after buying the company - onto their unified internal stack, ignoring warnings that an outage in their identity system would take OpsGenie down. Yes: 2 for weeks!

Systems 284

Systems Management Engineering IT 284

Snowflake

NOVEMBER 26, 2024

This blog post is the second in a three-part series on migrations. A consolidated data system to accommodate a big(ger) WHOOP When a company experiences exponential growth over a short period, it’s easy for its data foundation to feel a bit like it was built on the fly.

Data Warehouse 104

Data Warehouse Cloud PostgreSQL Hadoop 104

databricks

NOVEMBER 12, 2024

Monolithic to Modular The proof of concept (POC) of any new technology often starts with large, monolithic units that are difficult to characterize.

Systems 122

Systems Engineering Technology Data 122

databricks

DECEMBER 21, 2023

We are excited to announce that Gartner has recognized Databricks as a Leader for a third consecutive year in the 2023 Gartner® Magic.

Database 145

Database Systems Cloud Management 145

Data Engineering Weekly

MAY 4, 2025

The blog highlights how emerging AI tools automate otherwise cognitively intensive manual tasks to bring reliability in software engineering. link] Whatnot: Evolving Feed Ranking at Whatnot Whatnot describes their transition from a batch prediction system to an online inference framework for ranking, which is shown in their "For You Feed."

Data Engineering 67

Data Engineering Data Engineer Engineering Java 67

Data Engineering Weekly

OCTOBER 27, 2024

The blog is an excellent summary of the existing unstructured data landscape. It is exciting to read probably the first blog on building a vector search infrastructure at scale. The blog from Meta discusses how it designed a privacy-preserving storage.

Data Engineering 57

Data Engineering Data Engineer Engineering Unstructured Data 57

Cloudera

NOVEMBER 13, 2024

Additionally, multiple copies of the same data locked in proprietary systems contribute to version control issues, redundancies, staleness, and management headaches. The post Octopai Acquisition Enhances Metadata Management to Trust Data Across Entire Data Estate appeared first on Cloudera Blog.

Metadata 90

Metadata Management Data Governance Government 90

Jesse Anderson

MAY 5, 2025

Finally, Shane outlines how observability is crucial for emerging AI/ML workflows like RAG pipelines, discussing the monitoring of vector databases (like Pinecone), unstructured data, and the entire AI system lifecycle, concluding with a look at Monte Carlo’s exciting roadmap, including AI-powered troubleshooting agents.

Unstructured Data 130

Unstructured Data Finance Metadata Architecture 130

Data Engineering Weekly

NOVEMBER 24, 2024

The blog highlights how moving from 6-character base-64 to 20-digit base-2 file distribution brings more distribution in S3 and reduces request failures. The blog is a good summary of how to use Snowflake QUERY_TAG to measure and monitor query performance. The blog post made me curious to understand DataFusion's internals.

Data Engineering 57

Data Engineering Data Engineer Engineering Insurance 57

Engineering at Meta

APRIL 28, 2025

Meta’s vast and diverse systems make it particularly challenging to comprehend its structure, meaning, and context at scale. We discovered that a flexible and incremental approach was necessary to onboard the wide variety of systems and languages used in building Metas products. We believe that privacy drives product innovation.

Metadata 79

Metadata Data Utilities Data Warehouse 79

Data Engineering Weekly

APRIL 20, 2025

The blog took out the last edition’s recommendation on AI and summarized the current state of AI adoption in enterprises. The simplistic model expressed in the blog made it easy for me to reason about the transactional system design. Kafka is probably the most reliable data infrastructure in the modern data era.

Data Engineering 67

Data Engineering Data Engineer Engineering Kafka 67

Netflix Tech

MAY 4, 2023

Behind the scenes, a myriad of systems and services are involved in orchestrating the product experience. These backend systems are consistently being evolved and optimized to meet and exceed customer and product expectations. This blog series will examine the tools, techniques, and strategies we have utilized to achieve this goal.

Utilities 139

Utilities Systems Architecture Coding 139

The Pragmatic Engineer

OCTOBER 18, 2024

Corporate conflict recap Automattic is the creator of open source WordPress content management system (CMS), and WordPress powers an incredible 43% of webpages and 65% of CMSes. This event is shameful and unprecedented in the history of open source on the web. Automattic raised $980M in venture funding and was valued at $7.5B

Engineering 357

Engineering Government Project AWS 357

Cloudera

DECEMBER 4, 2024

System metrics, such as inference latency and throughput, are available as Prometheus metrics. Users can manage all of their models and applications on the Cloudera AI Inference service with common CI/CD systems using Cloudera service accounts, also known as machine users.

Architecture 73

Architecture Machine Learning BI Deep Learning 73

databricks

NOVEMBER 13, 2024

As organizations integrate AI agent systems into. Generative AI has become a powerful reality, transforming industries by enhancing customer experiences and automating decisions.

Systems 86

Systems 86

Edureka

APRIL 22, 2025

This blog will explore the significant advancements, challenges, and opportunities impacting data engineering in 2025, highlighting the increasing importance for companies to stay updated. In 2025, this blog will discuss the most important data engineering trends, problems, and opportunities that companies should be aware of.

Data Engineering 52

Data Engineering Data Engineer Engineering Consulting 52

Data Engineering Weekly

MARCH 2, 2025

I found the blog to be a fresh take on the skill in demand by layoff datasets. DeepSeek continues to impact the Data and AI landscape with its recent open-source tools, such as Fire-Flyer File System (3FS) and smallpond. The blog provides an excellent analysis of smallpond compared to Spark and Daft.

Data Engineering 62

Data Engineering Data Engineer Engineering Datasets 62

Jesse Anderson

FEBRUARY 11, 2025

Semih is a researcher and entrepreneur with a background in distributed systems and databases. He then pursued his doctoral studies at Stanford University, delving into the complexities of database systems.

Computer Science 130

Computer Science Database Design Software Engineer Software Engineering 130

databricks

FEBRUARY 29, 2024

Special thanks to Phillip Jones, Senior Product Manager, and Harshal Brahmbhatt, Systems Engineer from Cloudflare for their contributions to this blog. Organizations across.

Engineering 140

Engineering Systems Management 140

databricks

DECEMBER 23, 2024

We are excited to announce that Gartner has recognized Databricks as a Leader for a fourth consecutive year in the 2024 Gartner Magic.

Database 111

Database Systems Cloud Management 111

DataKitchen

DECEMBER 9, 2024

This blog dives into the remarkable journey of a data team that achieved unparalleled efficiency using DataOps principles and software that transformed their analytics and data teams into a hyper-efficient powerhouse. A software system where processes can be developed and shared is required. Here is another example.

Pharmaceutical 65

Pharmaceutical Data Lake Cloud Storage Project 65

Pinterest Engineering

APRIL 4, 2025

In this blog, we will go through the technical design and share some offline and online results for our LLM-based search relevance pipeline. Pin Text Representations Pins on Pinterest are rich multimedia entities that feature images, videos, and other contents, often linked to external webpages or blogs.

Machine Learning 56

Machine Learning Metadata Architecture Datasets 56

Pinterest Engineering

NOVEMBER 18, 2024

In this blog post, we’ll explore what CDC is, why it’s important, and our journey of implementing Generic CDC solutions for all online databases at Pinterest. This is crucial for applications that require up-to-date information, such as fraud detection systems or recommendation engines. What is Change Data Capture?

Kafka 50

Kafka MySQL Database Software Engineer 50

Netflix Tech

FEBRUARY 14, 2025

It requires a state-of-the-art system that can track and process these impressions while maintaining a detailed history of each profiles exposure. In this multi-part blog series, we take you behind the scenes of our system that processes billions of impressions daily.

Kafka 66

Kafka Datasets Metadata Utilities 66

Picnic Engineering

APRIL 7, 2025

A little over a year ago, we shared a blog post about our journey to enhance customers meal planning experience with personalized recipe recommendations. We explained how a system that learns from your tastes and habits could solve this issue, ultimately making the daily task of choosing meals both effortless and inspiring.

Datasets 52

Datasets Systems Architecture Machine Learning 52

Data Engineering Weekly

FEBRUARY 9, 2025

I found the product blog from QuantumBlack gives a view of data quality in unstructured data. link] Pinterest: Advancements in Embedding-Based Retrieval at Pinterest Homefeed Pinterest writes about its embedding-based retrieval system enhancements for Homefeed personalization and engagement.

Data Engineering 57

Data Engineering Data Engineer Engineering Unstructured Data 57

Jesse Anderson

OCTOBER 29, 2024

He sees logs as a treasure trove of insights and believes effective log analysis is critical in today’s complex systems. We discussed his early experiences with distributed systems, including his work on creating graphs and entity resolution. Lastly, we go in-depth into Scanner.dev, covering what it is and how it works.

Data Engineering 100

Data Engineering Data Engineer Systems Engineering 100

Data Engineering Weekly

FEBRUARY 23, 2025

It covers nine categories: storage systems, data lake platforms, processing, integration, orchestration, infrastructure, ML/AI, metadata management, and analytics. I found the blog to be a comprehensive roadmap for data engineering in 2025. I wonder if these systems expand more capabilities that eventually fall on their own weight.

Data Engineering 62

Data Engineering Data Engineer Engineering Kafka 62

Data Engineering Weekly

MAY 11, 2025

[link] Alibaba: A Comprehensive Analysis and Practical Implementation of the New Features in the MCP Specification When I delved further into learning about the MCP specification, Alibaba's blog was a handy guide to understanding the protocol spec's evolution over the last four months.

Data Engineering 52

Data Engineering Data Engineer Engineering Data 52

Data Engineering Weekly

NOVEMBER 3, 2024

Foundation Capital: A System of Agents brings Service-as-Software to life software is no longer simply a tool for organizing work; software becomes the worker itself, capable of understanding, executing, and improving upon traditionally human-delivered services. It's good to know about Dapr and restate.dev.

Data Engineering 62

Data Engineering Data Engineer Engineering Pipeline-centric 62

Netflix Tech

DECEMBER 19, 2024

This followed a previous blog on the same topic. In this context, having a single configuration system to manage a ML project holistically gives users increased project coherence and decreased projectrisk. Configuration in Metaboost Ease of configuration and templatizing are core values of Metaboost.

Machine Learning 82

Machine Learning Project Data Warehouse Coding 82

Pinterest Engineering

APRIL 7, 2025

In recent years, while managing Pinterests EC2 infrastructure, particularly for our essential online storage systems, we identified a significant challenge: the lack of clear insights into EC2s network performance and its direct impact on our applications reliability and performance. 4xl with up to 12.5

AWS 57

AWS Bytes Database Data Ingestion 57

Netflix Tech

JUNE 1, 2023

In this blog post, we’ll discuss the methods we used to ensure a successful launch, including: How we tested the system Netflix technologies involved Best practices we developed Realistic Test Traffic Netflix traffic ebbs and flows throughout the day in a sinusoidal pattern. Basic with ads was launched worldwide on November 3rd.

Algorithm 140

Algorithm Kafka Metadata Systems 140

Netflix Tech

JANUARY 2, 2025

However, due to the absence of a control group in these countries, we adopt a synthetic control framework ( blog post ) to estimate the counterfactual scenario. We further break down performance by various dimensions, e.g. content type, genre, etc to help us pinpoint specific areas where the ASR system may encounter difficulties.

Engineering 77

Engineering Entertainment Designing Technology 77