Snowflake

OCTOBER 7, 2024

When scaling data science and ML workloads, organizations frequently encounter challenges in building large, robust production ML pipelines. Standard Snowflake RBAC is used to control access to the Feature Store and the objects within. Today we are announcing the general availability of the Snowflake Feature Store.

Management 78

Management Building Datasets Government 78

Pinterest Engineering

JULY 15, 2024

David Chang; Staff Software Engineer | Develocity, formerly known as Gradle Enterprise, is a powerful tool that speeds up local and CI build time, helps troubleshoot your builds, and analyzes your data. At Pinterest, we have a dedicated team, Mobile Builds, and we ensure that developers can build fast and often.

Building 64

Building Metadata Software Engineer Software Engineering 64

Cloudera

OCTOBER 23, 2024

In August, we wrote about how in a future where distributed data architectures are inevitable, unifying and managing operational and business metadata is critical to successfully maximizing the value of data, analytics, and AI. It is a critical feature for delivering unified access to data in distributed, multi-engine architectures.

Metadata 83

Metadata BI Data Lake Business Intelligence 83

Rockset

MARCH 18, 2024

Building a real-time, contextual and trustworthy knowledge base for AI applications revolves around RAG pipelines. What are the challenges building RAG pipelines? When you are building applications for consistent, real-time performance at scale you will want to use a streaming-first architecture.

Cloud 64

Cloud Building Metadata Kafka 64

Data Engineering Podcast

MARCH 27, 2022

In this episode Balaji Ganesan shares how his experiences building and maintaining Ranger in previous roles helped him understand the needs of organizations and engineers as they define and evolve their data governance policies and practices. Acryl]([link] The modern data stack needs a reimagined metadata management platform.

Data Governance 130

Data Governance Government Cloud Building 130

Towards Data Science

FEBRUARY 9, 2024

How to build a modern, scalable data platform to power your analytics and data science projects (updated) Table of Contents: What’s changed? Orchestration I mentioned modularity as a core concept of building a modern data platform in my 2021 article, but I failed to emphasize the importance of data orchestration.

Building 78

Building Transportation Data Lake Metadata 78

Data Engineering Podcast

JUNE 16, 2024

For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst is an end-to-end data lakehouse platform built on Trino, the query engine Apache Iceberg was designed for, with complete support for all table formats including Apache Iceberg, Hive, and Delta Lake. Want to see Starburst in action?

Data Lake 147

Data Lake High Quality Data Metadata Machine Learning 147

Snowflake

NOVEMBER 1, 2023

And because the Notebook is natively integrated into Snowflake’s role-based access controls (RBAC), it’s easy to securely share and collaborate on your code and results without compromising on any enterprise data. What’s Next? Check out the Snowpark ML demo from Snowday to see the latest launches in action.

Building 108

Building Python SQL Programming Language 108

dbt Developer Hub

OCTOBER 3, 2024

These formats are changing the way data is stored and metadata accessed. Iceberg Data Catalog - an open-source metadata management system that tracks the schema, partition, and versions of Iceberg tables. The metadata management and performance make them very meaningful and should be paid attention to. What is Iceberg? ​

Metadata 86

Metadata Data Lake Data Storage Accessible 86

Snowflake

OCTOBER 2, 2023

To help customers overcome these challenges, RudderStack and Snowflake recently launched Profiles , a new product that allows every data team to build a customer 360 directly in their Snowflake Data Cloud environment. Now teams can leverage their existing data engineering tools and workflows to build their customer 360.

Cloud 105

Cloud Building Insurance Data Engineering 105

Precisely

OCTOBER 31, 2024

Data fabric is a unified approach to data management, creating a consistent way to manage, access, and share data across distributed environments. As data management grows increasingly complex, you need modern solutions that allow you to integrate and access your data seamlessly.

Data Architecture 69

Data Architecture Architecture Metadata Government 69

ThoughtSpot

NOVEMBER 5, 2024

In the realm of modern analytics platforms, where rapid and efficient processing of large datasets is essential, swift metadata access and management are critical for optimal system performance. Any delays in metadata retrieval can negatively impact user experience, resulting in decreased productivity and satisfaction.

Metadata 52

Metadata PostgreSQL Java Database 52

Confluent

MAY 7, 2019

Although it is the simplest way to subscribe to and access events from Kafka, behind the scenes, Kafka consumers handle tricky distributed systems challenges like data consistency, failover and load balancing. Every developer who uses Apache Kafka ® has used a Kafka consumer at least once. Data processing requirements.

Kafka 111

Kafka Accessibility Accessible Metadata 111

Cloudera

AUGUST 24, 2022

CML empowers organizations to build and deploy machine learning and AI capabilities for business at scale, efficiently and securely, anywhere they want. A Docker repository available for the user and also accessible for CML (e.g. Step 2: Prepare Dockerfile and build image. Prerequisites. docker.io). Instructions.

Machine Learning 83

Machine Learning Building Metadata Programming Language 83

Data Council

JANUARY 21, 2021

Storing Cold Metadata with Alki (Dropbox) Dropbox shared insights into Alki , the petabyte-scale metadata store it designed for infrequently accessed metadata (“cold data”). Here's our January 2021 roundup of links from across the web that could be relevant to you: 1.

Metadata 52

Metadata Cloud AWS Database 52

Data Engineering Weekly

JUNE 16, 2024

One of the big benefits of having a Hive Meastore catalog is that it enables many query engines to build executing engines on top of it, which creates a strong ecosystem. Along the same line, NVIDIA open sources nmotran4 give developers a free, scalable way to generate synthetic data that can help build powerful LLMs.

Data Engineering 104

Data Engineering Data Engineer Engineering Kafka 104

Netflix Tech

SEPTEMBER 28, 2021

The Revenue and Growth Tools (RGT) team decided to set off on a journey to build tools in an abstract manner to have solutions readily available within our organization. The traditional approach of bundling dependencies at build time lacks the flexibility we need to empower our stakeholders.

Building 97

Building Metadata Designing Coding 97

ThoughtSpot

OCTOBER 9, 2023

ThoughtSpot continually invests in this area, giving users the confidence to build the correct Answers needed for their analysis—and ensuring they can trust the data they are shown. How ThoughtSpot builds trust with data catalog connectors For many, the data catalog is still the primary home for metadata enrichment and governance.

Metadata 105

Metadata Data Government Data Governance 105

Data Engineering Podcast

NOVEMBER 9, 2020

Summary A data catalog is a critical piece of infrastructure for any organization who wants to build analytics products, whether internal or external. While there are a number of platforms available for building that catalog, many of them are either difficult to deploy and integrate, or expensive to use at scale.

Building 100

Building PostgreSQL BI Metadata 100

Data Engineering Podcast

FEBRUARY 13, 2022

To simplify the work of managing the full flow of your customer data and keep you in full control the team at Rudderstack created their eponymous open source platform that allows you to work with first and third party data, as well as build and manage reverse ETL workflows.

Building 100

Building Hadoop Data Pipeline Metadata 100

Data Engineering Podcast

JANUARY 23, 2022

He also provides some excellent insights into the factors that play into the build vs. buy decision at different organizational sizes. You can observe your pipelines with built in metadata search and column level lineage. You can observe your pipelines with built in metadata search and column level lineage.

Building 100

Building Management Data Pipeline Metadata 100

Cloudera

NOVEMBER 15, 2024

By leveraging Apache Iceberg, an open table format designed for high-performance analytics on massive volumes of data, cyber defenders can access all of their data and conduct investigations with greater speed and efficiency. Whether they need to query data from today or from years past, the system scales up or down to meet their needs.

Metadata 77

Metadata Unstructured Data Data Lake Government 77

Data Engineering Podcast

FEBRUARY 5, 2023

Sign up now for early access to Materialize and get started with the power of streaming data with the same simplicity and low implementation cost as batch cloud data warehouses.

Data Engineering 147

Data Engineering Data Engineer Engineering PostgreSQL 147

Snowflake

NOVEMBER 12, 2024

Managing application state and metadata Use Hybrid Tables as the system of record for application configuration, user profiles, workflow state and other metadata that needs to be accessed with high concurrency. Build your own to-do list application in five minutes.

Food 71

Food Metadata Education AWS 71

Data Engineering Podcast

NOVEMBER 26, 2023

Summary Building a data platform that is enjoyable and accessible for all of its end users is a substantial challenge. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Netflix Tech

SEPTEMBER 18, 2024

Second, developers had to constantly re-learn new data modeling practices and common yet critical data access patterns. To overcome these challenges, we developed a holistic approach that builds upon our Data Gateway Platform. It also serves as central configuration of access patterns such as consistency or latency targets.

Bytes 99

Bytes Metadata Database Data 99

Netflix Tech

FEBRUARY 16, 2021

The original SKU catalog is a logic-heavy client library packaged with complex metadata configuration files and consumed by various services. Operational Efficiency: The majority of the changes require metadata configuration files and library code changes, usually taking days of testing and service release to adopt the updates.

Building 86

Building Management Metadata Entertainment 86

Databand.ai

AUGUST 30, 2023

Building a Winning Data Quality Strategy: Step by Step Eitan Chazbani August 30, 2023 What Is a Data Quality Strategy? This includes defining roles and responsibilities related to managing datasets and setting guidelines for metadata management. This starts with building a strong business case for your data quality strategy.

Building 52

Building Data Cleanse Data Governance Datasets 52

Snowflake

JUNE 11, 2024

Many enterprises stitch together a complex mix of various MLOps tools to build an end-to-end ML pipeline. Snowflake ML is the integrated set of capabilities that enable developers to securely build, deploy and manage ML data, features and models at scale on a single platform.

Management 91

Management Government Metadata Python 91

Snowflake

JULY 25, 2024

Solving the challenges of building high-quality RAG applications From the beginning, Snowflake’s mission has been to empower customers to extract more value from their data. It supports “fuzzy” search — the service takes in natural language queries and returns the most relevant text results, along with associated metadata.

Unstructured Data 111

Unstructured Data Metadata Government SQL 111

Data Engineering Podcast

OCTOBER 30, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Business Intelligence 130

Business Intelligence Metadata MongoDB MySQL 130

Ascend.io

JULY 23, 2024

We will explore some frameworks necessary for enabling adoption of these technologies and provide insights into building a more productive data engineering environment. AI Implementation Framework #1- Levels of Access The first framework focuses on defining levels of access for AI and automation tools.

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Data Engineering Podcast

FEBRUARY 19, 2023

It's supposed to make building smarter, faster, and more flexible data infrastructures a breeze. By bringing all the layers of the data stack together, TimeXtender helps you build data solutions up to 10 times faster and saves you 70-80% on costs. Acryl]([link] The modern data stack needs a reimagined metadata management platform.

IT 147

IT Data Lake Metadata Data Warehouse 147

Precisely

JULY 18, 2024

Key Takeaways Data Fabric is a modern data architecture that facilitates seamless data access, sharing, and management across an organization. Data management recommendations and data products emerge dynamically from the fabric through automation, activation, and AI/ML analysis of metadata.

Data Management 52

Data Management Management Metadata Database-centric 52

François Nguyen

MARCH 22, 2021

How do we build data products ? How to build Data Products or never call me Data Pipeline any more You have this interesting schema in her second article on Data Mesh by Zhamak Dehghani : “Data mesh introduces the concept of data product as its architectural quantum. .” What you have to code is this workflow !

Technology 130

Technology Architecture Google Cloud Metadata 130

Data Engineering Podcast

NOVEMBER 20, 2022

Summary The majority of blog posts and presentations about data engineering and analytics assume that the consumers of those efforts are internal business users accessing an environment controlled by the business. Atlan is the metadata hub for your data ecosystem.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Data Engineering Podcast

DECEMBER 4, 2022

The typical conception of how it is accessed is through a web or desktop application running on a powerful laptop. Zing Data is building a mobile native platform for business intelligence. In this episode Sabin Thomas shares his experiences building the platform and the interesting ways that it is being used.

Business Intelligence 100

Business Intelligence Metadata BI MongoDB 100