Cloudera

FEBRUARY 8, 2021

To achieve this, a data engineering framework must be used to allow the building of all of the piping and plumbing needed to move, manipulate, and manage data of the different vehicle parts in the data lifecycle. Building a Pipeline Using Cloudera Data Engineering. STEP 4: Generate data visualization dashboards and reports. Conclusion.

Data Pipeline 122

Data Pipeline Building Manufacturing Data Warehouse 122

Data Engineering Podcast

JUNE 16, 2024

For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst is an end-to-end data lakehouse platform built on Trino, the query engine Apache Iceberg was designed for, with complete support for all table formats including Apache Iceberg, Hive, and Delta Lake. Want to see Starburst in action?

Data Lake 147

Data Lake High Quality Data Metadata Machine Learning 147

Snowflake

NOVEMBER 1, 2023

And because the Notebook is natively integrated into Snowflake’s role-based access controls (RBAC), it’s easy to securely share and collaborate on your code and results without compromising on any enterprise data. What’s Next? Check out the Snowpark ML demo from Snowday to see the latest launches in action.

Building 111

Building Python SQL Programming Language 111

Netflix Tech

SEPTEMBER 28, 2021

The Revenue and Growth Tools (RGT) team decided to set off on a journey to build tools in an abstract manner to have solutions readily available within our organization. The traditional approach of bundling dependencies at build time lacks the flexibility we need to empower our stakeholders.

Building 97

Building Metadata Designing Coding 97

Snowflake

OCTOBER 2, 2023

To help customers overcome these challenges, RudderStack and Snowflake recently launched Profiles , a new product that allows every data team to build a customer 360 directly in their Snowflake Data Cloud environment. Now teams can leverage their existing data engineering tools and workflows to build their customer 360.

Cloud 108

Cloud Building Insurance Data Engineering 108

François Nguyen

MARCH 22, 2021

How do we build data products ? How to build Data Products or never call me Data Pipeline any more You have this interesting schema in her second article on Data Mesh by Zhamak Dehghani : “Data mesh introduces the concept of data product as its architectural quantum. .” What you have to code is this workflow !

Technology 130

Technology Architecture Google Cloud Metadata 130

Data Engineering Podcast

NOVEMBER 9, 2020

Summary A data catalog is a critical piece of infrastructure for any organization who wants to build analytics products, whether internal or external. While there are a number of platforms available for building that catalog, many of them are either difficult to deploy and integrate, or expensive to use at scale.

Building 100

Building PostgreSQL BI Metadata 100

Data Engineering Podcast

FEBRUARY 5, 2023

Sign up now for early access to Materialize and get started with the power of streaming data with the same simplicity and low implementation cost as batch cloud data warehouses.

Data Engineering 147

Data Engineering Data Engineer Engineering PostgreSQL 147

Ascend.io

JULY 11, 2024

Metadata is the information that provides context and meaning to data, ensuring it’s easily discoverable, organized, and actionable. This is what managing data without metadata feels like. This is what managing data without metadata feels like. Effective metadata management is no longer a luxury—it’s a necessity.

Metadata 52

Metadata IT Government High Quality Data 52

Cloudera

AUGUST 24, 2022

CML empowers organizations to build and deploy machine learning and AI capabilities for business at scale, efficiently and securely, anywhere they want. A Docker repository available for the user and also accessible for CML (e.g. Step 2: Prepare Dockerfile and build image. Prerequisites. docker.io). Instructions.

Machine Learning 88

Machine Learning Building Metadata Programming Language 88

Data Engineering Podcast

FEBRUARY 19, 2023

It's supposed to make building smarter, faster, and more flexible data infrastructures a breeze. By bringing all the layers of the data stack together, TimeXtender helps you build data solutions up to 10 times faster and saves you 70-80% on costs. Acryl]([link] The modern data stack needs a reimagined metadata management platform.

IT 147

IT Data Lake Metadata Data Warehouse 147

Data Engineering Podcast

JANUARY 23, 2022

He also provides some excellent insights into the factors that play into the build vs. buy decision at different organizational sizes. You can observe your pipelines with built in metadata search and column level lineage. You can observe your pipelines with built in metadata search and column level lineage.

Building 100

Building Management Data Pipeline Metadata 100

Data Engineering Podcast

FEBRUARY 13, 2022

To simplify the work of managing the full flow of your customer data and keep you in full control the team at Rudderstack created their eponymous open source platform that allows you to work with first and third party data, as well as build and manage reverse ETL workflows.

Building 100

Building Hadoop Data Pipeline Metadata 100

Data Engineering Podcast

OCTOBER 30, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Business Intelligence 130

Business Intelligence Metadata MongoDB MySQL 130

Data Engineering Podcast

NOVEMBER 26, 2023

Summary Building a data platform that is enjoyable and accessible for all of its end users is a substantial challenge. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Towards Data Science

FEBRUARY 9, 2024

How to build a modern, scalable data platform to power your analytics and data science projects (updated) Table of Contents: What’s changed? Orchestration I mentioned modularity as a core concept of building a modern data platform in my 2021 article, but I failed to emphasize the importance of data orchestration.

Building 76

Building Transportation Data Lake Metadata 76

Data Engineering Podcast

NOVEMBER 20, 2022

Summary The majority of blog posts and presentations about data engineering and analytics assume that the consumers of those efforts are internal business users accessing an environment controlled by the business. Atlan is the metadata hub for your data ecosystem.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

ThoughtSpot

OCTOBER 9, 2023

ThoughtSpot continually invests in this area, giving users the confidence to build the correct Answers needed for their analysis—and ensuring they can trust the data they are shown. How ThoughtSpot builds trust with data catalog connectors For many, the data catalog is still the primary home for metadata enrichment and governance.

Metadata 105

Metadata Data Government Data Governance 105

Cloudera

NOVEMBER 13, 2020

Unlike traditional planners that need to consider accessing a table via a variety of types of index, Impala’s planner always starts with a full table scan and then applies pruning techniques to reduce the data scanned. Metadata Caching. See the performance results below for an example of how metadata caching helps reduce latency.

Metadata 144

Metadata Coding SQL Database 144

Data Engineering Weekly

JUNE 16, 2024

One of the big benefits of having a Hive Meastore catalog is that it enables many query engines to build executing engines on top of it, which creates a strong ecosystem. Along the same line, NVIDIA open sources nmotran4 give developers a free, scalable way to generate synthetic data that can help build powerful LLMs.

Data Engineering 105

Data Engineering Data Engineer Engineering Kafka 105

Snowflake

OCTOBER 7, 2024

When scaling data science and ML workloads, organizations frequently encounter challenges in building large, robust production ML pipelines. Standard Snowflake RBAC is used to control access to the Feature Store and the objects within. Today we are announcing the general availability of the Snowflake Feature Store.

Management 82

Management Building Datasets Government 82

Netflix Tech

JUNE 1, 2023

To do this, we devised a novel way to simulate the projected traffic weeks ahead of launch by building upon the traffic migration framework described here. It also included metadata about ads, such as ad placement and impression-tracking events. As we were gearing up for launch, we wanted to ensure it would go as smoothly as possible.

Algorithm 140

Algorithm Metadata Kafka Systems 140

Netflix Tech

DECEMBER 3, 2022

This is one way to build trust with our internal user base. Obviously not all tools are made with the same use case in mind, so we are planning to add more code samples for other (than classical batch ETL) data processing purposes, e.g. Machine Learning model building and scoring.

Data Pipeline 144

Data Pipeline Scala Metadata Food 144

Rockset

MARCH 18, 2024

Building a real-time, contextual and trustworthy knowledge base for AI applications revolves around RAG pipelines. What are the challenges building RAG pipelines? When you are building applications for consistent, real-time performance at scale you will want to use a streaming-first architecture.

Cloud 64

Cloud Building Metadata Kafka 64

Snowflake

JANUARY 23, 2024

We have been making it easier and faster to build and manage ML models with Snowpark ML , the Python library and underlying infrastructure for end-to-end ML workflows in Snowflake. Many developers and enterprises looking to use machine learning (ML) to generate insights from data get bogged down by operational complexity.

Machine Learning 123

Machine Learning Metadata Python Telecommunication 123

Netflix Tech

FEBRUARY 13, 2023

Our goal in building a media-focused ML infrastructure is to reduce the time from ideation to productization for our media ML practitioners. We accomplish this by paving the path to: Accessing and processing media data (e.g. Step 1 We download a video file and produce shot boundary metadata.

Media 119

Media Machine Learning Metadata Algorithm 119

Pinterest Engineering

JULY 15, 2024

David Chang; Staff Software Engineer | Develocity, formerly known as Gradle Enterprise, is a powerful tool that speeds up local and CI build time, helps troubleshoot your builds, and analyzes your data. At Pinterest, we have a dedicated team, Mobile Builds, and we ensure that developers can build fast and often.

Building 65

Building Metadata Software Engineer Software Engineering 65

Precisely

OCTOBER 31, 2024

Data fabric is a unified approach to data management, creating a consistent way to manage, access, and share data across distributed environments. As data management grows increasingly complex, you need modern solutions that allow you to integrate and access your data seamlessly.

Data Architecture 69

Data Architecture Architecture Metadata Government 69

Snowflake

MAY 21, 2024

This will enable our joint customers to experience bidirectional data access between Snowflake and Microsoft Fabric, with a single copy of data with OneLake in Fabric. Organizations using both platforms will be able to do so more cost-effectively, rather than building pipelines or maintaining copies of data in each platform.

Metadata 126

Metadata Cloud Accessible Accessibility 126

Snowflake

JULY 25, 2024

Solving the challenges of building high-quality RAG applications From the beginning, Snowflake’s mission has been to empower customers to extract more value from their data. It supports “fuzzy” search — the service takes in natural language queries and returns the most relevant text results, along with associated metadata.

Unstructured Data 115

Unstructured Data Metadata Government SQL 115

Data Engineering Podcast

DECEMBER 4, 2022

The typical conception of how it is accessed is through a web or desktop application running on a powerful laptop. Zing Data is building a mobile native platform for business intelligence. In this episode Sabin Thomas shares his experiences building the platform and the interesting ways that it is being used.

Business Intelligence 100

Business Intelligence Metadata BI MongoDB 100

Data Engineering Podcast

MARCH 18, 2023

Data Council attendees are amazing founders, data scientists, lead engineers, CTOs, heads of data, investors and community organizers who are all working together to build the future of data. Visit: dataengineeringpodcast.com/data-council today RudderStack makes it easy for data teams to build a customer data platform on their own warehouse.

Data Security 130

Data Security Machine Learning Data Science Metadata 130

dbt Developer Hub

OCTOBER 3, 2024

These formats are changing the way data is stored and metadata accessed. Iceberg Data Catalog - an open-source metadata management system that tracks the schema, partition, and versions of Iceberg tables. The metadata management and performance make them very meaningful and should be paid attention to. What is Iceberg? ​

Metadata 86

Metadata Data Lake Data Storage Accessible 86

Snowflake

JUNE 21, 2024

Thousands of customers have worked with Snowflake to cost-effectively build a secure data foundation as they look to solve a growing variety of business problems with more data. Zero Ingest with Zero Silos : Iceberg data already managed in a data lake can be accessed directly by Snowflake via an Iceberg catalog integration.

Data Lake 117

Data Lake BI Business Intelligence Metadata 117

Data Engineering Podcast

MAY 18, 2021

In order to eliminate the wasted effort of building custom integrations every time you want to combine lineage information across systems Julien Le Dem introduced the OpenLineage specification. High-growth startups use Molecula’s feature store because of its unprecedented speed, cost savings, and simplified access to all enterprise data.

Metadata 100

Metadata Kafka Data Warehouse Hadoop 100

Data Engineering Podcast

AUGUST 28, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Engineering 100

Data Engineering Data Engineer MongoDB Metadata 100

Data Engineering Podcast

JULY 31, 2022

In this episode Ernie Ostic shares the approach that he and his team at Manta are taking to build a complete view of data lineage across the various data systems in your organization and the useful applications of that information in the work of every data stakeholder. Atlan is the metadata hub for your data ecosystem.

IT 100

IT Metadata MongoDB MySQL 100