phData: Data Engineering

NOVEMBER 8, 2024

The world we live in today presents larger datasets, more complex data, and diverse needs, all of which call for efficient, scalable data systems. These systems are built on open standards and offer immense analytical and transactional processing flexibility. In this blog, we will discuss: What is the Open Table format (OTF)?

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Knowledge Hut

FEBRUARY 1, 2024

Not only could this recommendation system save time browsing through lists of movies, it can also give more personalized results so users don’t feel overwhelmed by too many options. What are Movie Recommendation Systems? Recommender systems have two main categories: content-based & collaborative filtering.

Systems 98

Systems Entertainment Algorithm Datasets 98

Data Engineering Podcast

NOVEMBER 20, 2022

In this episode Ian Schweer shares his experiences at Riot Games supporting player-focused features such as machine learning models and recommeder systems that are deployed as part of the game binary. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Data Engineering Podcast

AUGUST 24, 2020

The key to those solutions is a robust and flexible metadata management system. LinkedIn has gone through several iterations on the most maintainable and scalable approach to metadata, leading them to their current work on DataHub. What were you using at LinkedIn for metadata management prior to the introduction of DataHub?

Metadata 100

Metadata Management Kafka Data Engineer 100

Data Engineering Podcast

NOVEMBER 6, 2022

He also describes the considerations involved in bringing behavioral data into your systems, and the ways that he and the rest of the Snowplow team are working to make that an easy addition to your platforms. Atlan is the metadata hub for your data ecosystem.

Building 130

Building IT Metadata MongoDB 130

Data Engineering Podcast

OCTOBER 15, 2021

Summary The binding element of all data work is the metadata graph that is generated by all of the workflows that produce the assets used by teams across the organization. It was also designed to be able to work for small scale systems that are just starting to develop in complexity. No more scripts, just SQL.

Metadata 100

Metadata BI Data Warehouse Government 100

The Pragmatic Engineer

OCTOBER 17, 2024

Results are stored in git and their database, together with benchmarking metadata. We recently covered how CockroachDB joins the trend of moving from open source to proprietary and why Oxide decided to keep using it with self-support , regardless Web hosting:  Netlify : chosen thanks to their super smooth preview system with SSR support.

Cloud 279

Cloud AWS Metadata Cloud Computing 279

Netflix Tech

MARCH 7, 2024

The Machine Learning Platform (MLP) team at Netflix provides an entire ecosystem of tools around Metaflow , an open source machine learning infrastructure framework we started, to empower data scientists and machine learning practitioners to build and manage a variety of ML systems.

Systems 93

Systems Media Machine Learning Data Warehouse 93

Data Engineering Podcast

APRIL 22, 2018

For this reason metadata management systems are built to track the journey of your business data to aid in analysis, presentation, and compliance. These systems are frequently cumbersome and difficult to maintain, so Octopai was founded to alleviate that burden. What is involved in deploying your metadata collection agents?

Business Intelligence 113

Business Intelligence Metadata Management Data Governance 113

Towards Data Science

FEBRUARY 6, 2024

ERP and CRM systems are designed and built to fulfil a broad range of business processes and functions. Accessing Operational Data I used to connect to views in transactional databases or APIs offered by operational systems to request the raw data. Does it sound familiar?

Systems 78

Systems Raw Data Metadata Data Cleanse 78

Data Engineering Podcast

DECEMBER 19, 2021

The team at Metaphor are building a fully connected metadata layer to provide both technical and social intelligence about your data. Connect your warehouse to Hightouch, paste a SQL query, and use their visual mapper to specify how data should appear in your SaaS systems. No more scripts, just SQL.

Systems 100

Systems Building Metadata Data Warehouse 100

Data Engineering Podcast

SEPTEMBER 11, 2022

Summary Data engineering systems are complex and interconnected with myriad and often opaque chains of dependencies. Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities.

Systems 100

Systems Metadata Building MongoDB 100

Data Engineering Podcast

JULY 31, 2022

Because of its centrality to your data systems it is valuable for debugging, governance, understanding context, and myriad other purposes. Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities.

IT 100

IT Metadata MongoDB MySQL 100

LinkedIn Engineering

OCTOBER 5, 2023

LinkedIn is on the forefront of leveraging EBR technology to revolutionize the way we approach search and recommendation systems. Figure 1 - Example of an Embeddings Graph Embedding based retrieval (EBR) is a method that is used at the early stages of a recommendation or search system.

IT 133

IT Metadata Designing Cloud 133

Netflix Tech

SEPTEMBER 29, 2022

Timestone: Netflix’s High-Throughput, Low-Latency Priority Queueing System with Built-in Support for Non-Parallelizable Workloads by Kostas Christidis Introduction Timestone is a high-throughput, low-latency priority queueing system we built in-house to support the needs of Cosmos , our media encoding platform. Over the past 2.5

Systems 87

Systems Metadata Media Kafka 87

Data Engineering Podcast

SEPTEMBER 4, 2022

Monte Carlo is in the vanguard of companies who have been enabling data teams to observe and understand their complex data systems. Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities.

IT 100

IT Metadata MongoDB MySQL 100

Data Engineering Podcast

FEBRUARY 20, 2022

He also explains how he has architected the systems that ingest, process, and distribute the data that he is responsible for and the requirements that are introduced when collaborating with researchers, domain experts, and machine learning developers. You can observe your pipelines with built in metadata search and column level lineage.

Systems 100

Systems Software Engineer Software Engineering Data Warehouse 100

Cloudera

SEPTEMBER 15, 2022

In this blog post, we will talk about a single Ozone cluster with the capabilities of both Hadoop Core File System (HCFS) and Object Store (like Amazon S3). A unified storage architecture that can store both files and objects and provide a flexible, scalable, and high-performance system.

Systems 85

Systems Hadoop Metadata Telecommunication 85

LinkedIn Engineering

DECEMBER 6, 2022

With a reasonably sizable footprint of servers in data centers, LinkedIn is responsible for ensuring that these hosts are always on an operating system (OS) version deemed the ���latest and greatest��� for all intents and purposes. The existing solution had challenges with its maintainability, scalability, and high availability.

Systems 55

Systems Metadata Datasets Designing 55

Knowledge Hut

APRIL 25, 2024

Intrusion detection systems (IDS) are designed to identify suspicious and malicious activity through network traffic. So, let's get to know the meaning of an intrusion detection system and how it works. What Is a n Intrusion Detection System? Why Are Intrusion Detection Systems Important? and how it works.

Systems 52

Systems Recruitment Machine Learning Certification 52

Monte Carlo

APRIL 20, 2024

To do this, you need data monitors that drill “ deep ” into the data using both machine learning and user-defined rules, as well as metadata monitors to scale “ broadly ” across every production table in your environment and to be fully integrated across your stack. It can also identify cases when metrics deviate from their normal patterns.

IT 52

IT Metadata Data Warehouse Machine Learning 52

Data Engineering Podcast

APRIL 24, 2022

WhyLogs is a powerful library for flexibly instrumenting all of your data systems to understand the entire lifecycle of your data from source to productionized model. You have full control over your data and their plugin system lets you integrate with all of your other data tools, including data warehouses and SaaS platforms.

Machine Learning 100

Machine Learning Systems Data Lake Java 100

Cloudera

JUNE 7, 2022

Common security, governance, metadata, replication, and automation enable CDP to operate as an integrated system. Only a fraction of data created is actually stored and managed, with analysts estimating it to be between 4 – 6 ZB in 2020. Clearly, hybrid data presents a massive opportunity and a tough challenge.

IT 110

IT Unstructured Data Data Architecture Government 110

Data Engineering Podcast

AUGUST 28, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. And don’t forget to thank them for their continued support of this show! Email hosts@dataengineeringpodcast.com ) with your story.

Data Engineer 100

Data Engineer Data Engineering MongoDB Metadata 100

Data Engineering Podcast

AUGUST 13, 2022

The biggest challenge with modern data systems is understanding what data you have, where it is located, and who is using it. Summary Data is useless if it isn’t being used, and you can’t use it if you don’t know where it is. __init__ covers the Python language, its community, and the innovative ways it is being used.

Metadata 100

Metadata MongoDB Scala MySQL 100

Data Engineering Podcast

JUNE 16, 2024

What are the other systems that feed into and rely on the Trino/Iceberg service? what kinds of questions are you answering with table metadata what use case/team does that support comparative utility of iceberg REST catalog What are the shortcomings of Trino and Iceberg? Email hosts@dataengineeringpodcast.com with your story.

Data Lake 147

Data Lake High Quality Data Metadata Machine Learning 147

Cloudera

OCTOBER 23, 2024

In August, we wrote about how in a future where distributed data architectures are inevitable, unifying and managing operational and business metadata is critical to successfully maximizing the value of data, analytics, and AI. It is a critical feature for delivering unified access to data in distributed, multi-engine architectures.

Metadata 82

Metadata BI Data Lake Business Intelligence 82

dbt Developer Hub

OCTOBER 3, 2024

These formats are changing the way data is stored and metadata accessed. Iceberg Data Catalog - an open-source metadata management system that tracks the schema, partition, and versions of Iceberg tables. Storage systems should just work.” “We They are groundbreaking in many ways. But I have to be honest: I don’t care.

Metadata 86

Metadata Data Lake Data Storage Accessible 86

ThoughtSpot

NOVEMBER 5, 2024

ThoughtSpot prioritizes the high availability and minimal downtime of our systems to ensure a seamless user experience. In the realm of modern analytics platforms, where rapid and efficient processing of large datasets is essential, swift metadata access and management are critical for optimal system performance. What is Atlas?

Metadata 52

Metadata PostgreSQL Java Database 52

Edureka

JULY 16, 2024

In today’s digital era, where time is key and efficiency paramount, the CLI (Command Line Interface) is increasingly used in software development and systems administration. Command Line Interface (CLI) is a way to interact with software and/or operating systems via text. Table of Content What is CLI? What is the Salesforce CLI?

IT 52

IT Cloud Computing Coding Project 52

Cloudera

DECEMBER 16, 2022

We are pleased to announce that Cloudera has been named a Leader in the 2022 Gartner ® Magic Quadrant for Cloud Database Management Systems. Many of our customers use multiple solutions—but want to consolidate data security, governance, lineage, and metadata management, so that they don’t have to work with multiple vendors.

Database 90

Database Cloud Systems Management 90

Data Engineering Podcast

JULY 5, 2021

SaaSGlue is a managed service that lets you connect all of your systems, across clouds and physical infrastructure, and spanning all of your programming languages. This is a fascinating platform with an endless set of use cases and a great team of people behind it. And don’t forget to thank them for their continued support of this show!

Systems 100

Systems Management Data Warehouse Programming Language 100

Booking.com Engineering

DECEMBER 2, 2022

We take advantage of this feature in our ad bidding systems, maintaining consistent data views from our Account Specialists’ spreadsheets, to our Data Scientists’ notebooks, to our bidding system’s in-memory data. Managing Workflows Data, delivered with quality at the required moment, is at the heart of our system performance.

Systems 52

Systems Cloud MySQL Relational Database 52

Monte Carlo

AUGUST 29, 2022

To do this, you need data monitors that drill “ deep ” into the data using both machine learning and user-defined rules, as well as metadata monitors to scale “ broadly ” across every production table in your environment and to be fully integrated across your stack. It can also identify cases when metrics deviate from their normal patterns.

IT 52

IT Metadata Data Warehouse BI 52

Cloudera

NOVEMBER 15, 2024

All of this data is essential for investigations and threat hunting, but existing systems often struggle to manage it efficiently. Here’s how Cloudera makes it possible: One unified system: Cloudera’s open data lakehouse consolidates all critical log data into one system.

Metadata 82

Metadata Unstructured Data Data Lake Government 82

Data Engineering Weekly

JUNE 16, 2024

S3 Express One Zone In Transactional Data Systems S3 Express One Zone, with low latency and write ops certainty, is promising. Replication-based systems remain more economical at low to medium throughputs, especially with significant cross-AZ data transfer discounts. link] Picnic: Open-sourcing dbt-score: lint model metadata with ease!

Data Engineer 104

Data Engineer Data Engineering Engineering Kafka 104

Pinterest Engineering

JANUARY 10, 2023

Liang Ma | Software Engineer, Core Eng; Wei Zhu | Software Engineer, Observability In early 2020, during a critical iOS out of memory incident (we have a blogpost for that), we realized that we didn’t have much visibility of how the app is running or a good system to look up for monitoring and troubleshooting.

Systems 57

Systems Building Software Engineer Software Engineering 57