Snowflake

NOVEMBER 26, 2024

This blog post is the second in a three-part series on migrations. A consolidated data system to accommodate a big(ger) WHOOP When a company experiences exponential growth over a short period, it’s easy for its data foundation to feel a bit like it was built on the fly. million in cost savings annually.

Data Warehouse 105

Data Warehouse Cloud PostgreSQL Hadoop 105

Cloudera

NOVEMBER 13, 2024

Cloudera, together with Octopai, will make it easier for organizations to better understand, access, and leverage all their data in their entire data estate – including data outside of Cloudera – to power the most robust data, analytics and AI applications.

Metadata 90

Metadata Management Data Governance Government 90

Data Engineering Podcast

NOVEMBER 20, 2022

Summary The majority of blog posts and presentations about data engineering and analytics assume that the consumers of those efforts are internal business users accessing an environment controlled by the business. Go to dataengineeringpodcast.com/hevodata and sign up for a free 14-day trial that also comes with 24×7 support.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Striim

JANUARY 30, 2025

Todays organizations have access to more data than ever before, and consequently are faced with the challenge of determining how to transform this tremendous stream of real-time information into actionable insights. Encryption, access controls, and regulatory compliance (HIPAA, GDPR, etc.) patient records or geolocation data).

Systems 52

Systems Management Hospitality Healthcare 52

The Pragmatic Engineer

OCTOBER 18, 2024

Corporate conflict recap Automattic is the creator of open source WordPress content management system (CMS), and WordPress powers an incredible 43% of webpages and 65% of CMSes. This event is shameful and unprecedented in the history of open source on the web. Automattic raised $980M in venture funding and was valued at $7.5B

Engineering 357

Engineering Government Project AWS 357

Data Engineering Weekly

OCTOBER 27, 2024

The blog is an excellent summary of the existing unstructured data landscape. It is exciting to read probably the first blog on building a vector search infrastructure at scale. The blog from Meta discusses how it designed a privacy-preserving storage. What are you waiting for? Register for IMPACT today!

Data Engineering 58

Data Engineering Data Engineer Engineering Unstructured Data 58

Snowflake

NOVEMBER 11, 2024

Optimize performance and cost with a broader range of model options Cortex AI provides easy access to industry-leading models via LLM functions or REST APIs, enabling you to focus on driving generative AI innovations. To learn more about these new features and related updates check out our Cortex Analyst blog post.

Unstructured Data 98

Unstructured Data SQL AWS Healthcare 98

The Pragmatic Engineer

OCTOBER 10, 2024

Wordpress is the most popular content management system (CMS),  estimated  to power around 43% of all websites; a staggering number! We are talking about a competitor to WP Engine (Automattic) with no concrete knowledge of WP Engine’s true revenue, and demanding full access to detailed revenue reports. 25 Sep: Block.

Consulting 347

Consulting AWS Engineering Software Engineer 347

Pinterest Engineering

JANUARY 18, 2024

In particular, our machine learning powered ads ranking systems are trying to understand users’ engagement and conversion intent and promote the right ads to the right user at the right time. Specifically, such discrepancies unfold into the following scenarios: Bug-free scenario : Our ads ranking system is working bug-free.

Systems 98

Systems Machine Learning Data Mining Algorithm 98

Netflix Tech

DECEMBER 19, 2024

This followed a previous blog on the same topic. For convenience, they support the dot-syntax (when possible) for accessing keys, making it easy to access values in a nested configuration. You can access Configs of any past runs easily through the Client API. The standard dictionary subscript notation is also available.

Machine Learning 81

Machine Learning Project Data Warehouse Coding 81

Cloudera

MAY 23, 2024

A “Knowledge Management System” (KMS) allows businesses to collate this information in one place, but not necessarily to search through it accurately. The interface allows for accurate, business-wide, querying that is quick and easy to scale with access to data sets provided through Cloudera’s platform.

Systems 80

Systems Building Management Data Lake 80

Edureka

APRIL 22, 2025

This blog will explore the significant advancements, challenges, and opportunities impacting data engineering in 2025, highlighting the increasing importance for companies to stay updated. In 2025, this blog will discuss the most important data engineering trends, problems, and opportunities that companies should be aware of.

Data Engineering 52

Data Engineering Data Engineer Engineering Consulting 52

Cloudera

DECEMBER 4, 2024

Services like Hugging Face and the ONNX Model Zoo made it easy to access a wide range of pre-trained models. System metrics, such as inference latency and throughput, are available as Prometheus metrics. Inference request and response payloads ship asynchronously to Apache Iceberg tables.

Architecture 73

Architecture Machine Learning Deep Learning BI 73

Cloudera

OCTOBER 23, 2024

By maintaining operational metadata within the table itself, Iceberg tables enable interoperability with many different systems and engines. It is a critical feature for delivering unified access to data in distributed, multi-engine architectures.

Metadata 84

Metadata BI Data Lake Business Intelligence 84

Engineering at Meta

APRIL 28, 2025

Meta’s vast and diverse systems make it particularly challenging to comprehend its structure, meaning, and context at scale. We discovered that a flexible and incremental approach was necessary to onboard the wide variety of systems and languages used in building Metas products. We believe that privacy drives product innovation.

Metadata 67

Metadata Data Utilities Data Warehouse 67

Cloudera

NOVEMBER 13, 2024

Several LLMs are publicly available through APIs from OpenAI , Anthropic , AWS , and others, which give developers instant access to industry-leading models that are capable of performing most generalized tasks.

Datasets 87

Datasets Machine Learning Coding Data Preparation 87

Data Engineering Weekly

FEBRUARY 9, 2025

I found the product blog from QuantumBlack gives a view of data quality in unstructured data. link] Pinterest: Advancements in Embedding-Based Retrieval at Pinterest Homefeed Pinterest writes about its embedding-based retrieval system enhancements for Homefeed personalization and engagement.

Data Engineering 58

Data Engineering Data Engineer Engineering Unstructured Data 58

Data Engineering Weekly

SEPTEMBER 26, 2024

Privacy and access management within data infrastructure is not just a best practice; it's a necessity. Robust privacy and access management protocols are crucial for GDPR compliance, protecting sensitive information, and maintaining user trust. For example, GDPR and HIPAA require strict access controls to protect sensitive data.

Systems 68

Systems Engineering Data Warehouse Architecture 68

Pinterest Engineering

NOVEMBER 18, 2024

In this blog post, we’ll explore what CDC is, why it’s important, and our journey of implementing Generic CDC solutions for all online databases at Pinterest. This is crucial for applications that require up-to-date information, such as fraud detection systems or recommendation engines. What is Change Data Capture?

Kafka 50

Kafka MySQL Database Software Engineer 50

Data Engineering Weekly

FEBRUARY 23, 2025

It covers nine categories: storage systems, data lake platforms, processing, integration, orchestration, infrastructure, ML/AI, metadata management, and analytics. I found the blog to be a comprehensive roadmap for data engineering in 2025. I wonder if these systems expand more capabilities that eventually fall on their own weight.

Data Engineering 63

Data Engineering Data Engineer Engineering Kafka 63

Data Engineering Weekly

MARCH 2, 2025

I found the blog to be a fresh take on the skill in demand by layoff datasets. DeepSeek continues to impact the Data and AI landscape with its recent open-source tools, such as Fire-Flyer File System (3FS) and smallpond. The blog provides an excellent analysis of smallpond compared to Spark and Daft.

Data Engineering 63

Data Engineering Data Engineer Engineering Datasets 63

Monte Carlo

APRIL 8, 2025

In practical terms, this means creating a system where everyone in your organization understands what data they’re handling and how to treat it appropriately, with safeguards if someone accidentally tries to mishandle sensitive information. And most importantlywho really needs access to this data? Want even tighter security?

PostgreSQL 52

PostgreSQL Medical Database Data 52

Netflix Tech

JANUARY 2, 2025

However, due to the absence of a control group in these countries, we adopt a synthetic control framework ( blog post ) to estimate the counterfactual scenario. To facilitate easier access to incrementality results, we have developed an interactive tool powered by this framework.

Engineering 76

Engineering Entertainment Designing Technology 76

Picnic Engineering

APRIL 7, 2025

A little over a year ago, we shared a blog post about our journey to enhance customers meal planning experience with personalized recipe recommendations. We explained how a system that learns from your tastes and habits could solve this issue, ultimately making the daily task of choosing meals both effortless and inspiring.

Datasets 52

Datasets Systems Architecture Machine Learning 52

DataKitchen

DECEMBER 9, 2024

This blog dives into the remarkable journey of a data team that achieved unparalleled efficiency using DataOps principles and software that transformed their analytics and data teams into a hyper-efficient powerhouse. A software system where processes can be developed and shared is required. Here is another example.

Pharmaceutical 64

Pharmaceutical Data Lake Cloud Storage Project 64

Pinterest Engineering

APRIL 7, 2025

In recent years, while managing Pinterests EC2 infrastructure, particularly for our essential online storage systems, we identified a significant challenge: the lack of clear insights into EC2s network performance and its direct impact on our applications reliability and performance. 4xl with up to 12.5

AWS 57

AWS Bytes Database Data Ingestion 57

Data Engineering Weekly

NOVEMBER 3, 2024

Foundation Capital: A System of Agents brings Service-as-Software to life software is no longer simply a tool for organizing work; software becomes the worker itself, capable of understanding, executing, and improving upon traditionally human-delivered services. What are you waiting for? Register for IMPACT today!

Data Engineering 63

Data Engineering Data Engineer Engineering Pipeline-centric 63

Edureka

APRIL 22, 2025

This blog will examine the model’s core principles, key components, and the shifting responsibilities across different AWS services. Identity and Access Management Customers create and manage IAM users, roles, policies, and integrate MFA, SSO, and federated identities.

Amazon Web Services 40

Amazon Web Services AWS Cloud Data Governance 40

Netflix Tech

FEBRUARY 14, 2025

It requires a state-of-the-art system that can track and process these impressions while maintaining a detailed history of each profiles exposure. In this multi-part blog series, we take you behind the scenes of our system that processes billions of impressions daily.

Kafka 66

Kafka Datasets Metadata Utilities 66

Data Engineering Podcast

APRIL 21, 2024

How do you manage the personalization of the AI functionality in your system for each user/team? Contact Info LinkedIn Blog Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? When is Shortwave the wrong choice? What do you have planned for the future of Shortwave?

Data Lake 182

Data Lake High Quality Data Machine Learning Data Pipeline 182

Data Engineering Weekly

MARCH 23, 2025

I honestly don’t have a solid answer, but this blog is an excellent overview of upskilling. The author emphasizes the importance of mastering state management, understanding "local first" data processing (prioritizing single-node solutions before distributed systems), and leveraging an asset graph approach for data pipelines.

Data Engineering 63

Data Engineering Data Engineer Engineering Data 63

Ascend.io

NOVEMBER 14, 2024

In this blog post, we’ll explore key strategies for future-proofing your data pipelines. APIs facilitate communication between different systems, allowing data to flow seamlessly across platforms. Encrypting data both at rest and in transit ensures that sensitive information remains protected from unauthorized access.

Data Pipeline 52

Data Pipeline Amazon Web Services Data Integration Data 52

Netflix Tech

JUNE 1, 2023

In this blog post, we’ll discuss the methods we used to ensure a successful launch, including: How we tested the system Netflix technologies involved Best practices we developed Realistic Test Traffic Netflix traffic ebbs and flows throughout the day in a sinusoidal pattern. Basic with ads was launched worldwide on November 3rd.

Algorithm 139

Algorithm Kafka Metadata Systems 139

Snowflake

JUNE 5, 2024

At the same time, organizations must ensure the right people have access to the right content, while also protecting sensitive and/or Personally Identifiable Information (PII) and fulfilling a growing list of regulatory requirements.

Government 116

Government Accessible Accessibility Cloud 116

Picnic Engineering

APRIL 14, 2025

(Written by Kirill Voloshin & Abdullah Abusamrah ) In our previous blog posts , we have covered our server-driven UI framework called Picnic Page Platform. This blog post explores how weve further evolved our framework to support more complex flows that interact with our back-end systems, persist data andmore.

Java 52

Java Retail SQL Database 52

Edureka

APRIL 22, 2025

We’ll explain everything in this blog post in the most straightforward manner possible—no complicated terms, just the features, advantages, and reasons why moving to Lightning might revolutionize your company. This gives you access to many ready-to-use components. You can access better reports and dashboards.

Designing 52

Designing Accessible Accessibility Coding 52

phData: Data Engineering

NOVEMBER 1, 2024

This blog is a collection of those insights, but for the full trendbook, we recommend downloading the PDF. It serves as a vital protective measure, ensuring proper data access while managing risks like data breaches and unauthorized use. They can identify where risks are and what to avoid.

Government 52

Government Data Governance Finance Metadata 52