Precisely

OCTOBER 31, 2024

Key Takeaways: Data mesh is a decentralized approach to data management, designed to shift creation and ownership of data products to domain-specific teams. Data fabric is a unified approach to data management, creating a consistent way to manage, access, and share data across distributed environments.

Data Architecture 69

Data Architecture Architecture Metadata Government 69

Engineering at Meta

MAY 22, 2024

In recent years, Meta’s data management systems have evolved into a composable architecture that creates interoperability, promotes reusability, and improves engineering efficiency. Data is at the core of every product and service at Meta. Data is at the core of every product and service at Meta.

Data Management 91

Data Management Management Data SQL 91

Pinterest Engineering

JULY 25, 2023

Soam Acharya | Data Engineering Oversight; Keith Regier | Data Privacy Engineering Manager Background Businesses collect many different types of data. Each dataset needs to be securely stored with minimal access granted to ensure they are used appropriately and can easily be located and disposed of when necessary.

Big Data 98

Big Data Accessibility Accessible Hadoop 98

Engineering at Meta

FEBRUARY 20, 2024

This new convergence helps Meta and the larger community build data management systems that are unified, more efficient, and composable. Meta’s Data Infrastructure teams have been rethinking how data management systems are designed.

Data Management 100

Data Management Bytes Management Datasets 100

Data Engineering Podcast

APRIL 23, 2023

In this episode DeVaris Brown discusses the types of applications that are possible when teams don't have to manage the complex infrastructure necessary to support continuous data flows. What are the shifts that have made them more accessible to a wider variety of teams? Who are the target customers for Meroxa?

Data Lake 277

Data Lake Kafka Machine Learning Data Warehouse 277

Data Engineering Podcast

FEBRUARY 18, 2024

In this episode Dain Sundstrom, CTO of Starburst, explains how the combination of the Trino query engine and the Iceberg table format offer the ease of use and execution speed of data warehouses with the infinite storage and scalability of data lakes. What do you have planned for the future of Trino/Starburst?

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Snowflake

APRIL 7, 2025

We chose Snowflake for its robust, scalable and secure data infrastructure, perfectly suited for handling complex regulatory and quality data efficiently. Its real-time analytics and data-sharing capabilities enable us to deliver seamless AI-driven insights while prioritizing safety.

Food 90

Food Data Transparency Software Engineering Software Engineer 90

Snowflake

NOVEMBER 11, 2024

With Hybrid Tables’ fast, high-concurrency point operations, you can store application and workflow state directly in Snowflake, serve data without reverse ETL and build lightweight transactional apps while maintaining a single governance and security model for both transactional and analytical data — all on one platform.

Data Architecture 105

Data Architecture Architecture Data Lake Kafka 105

Data Engineering Podcast

DECEMBER 24, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Precisely

NOVEMBER 18, 2024

Ultimately, they are trying to serve data in their marketplace and make it accessible to business and data consumers,” Yoğurtçu says. However, they require a strong data foundation to be effective. ” The panel agreed that a holistic approach to data management is key, and data governance is the place to start.

Data Analytics 72

Data Analytics Data Governance Data Integration Government 72

Snowflake

NOVEMBER 11, 2024

This is not surprising when you consider all the benefits, such as reducing complexity [and] costs and enabling zero-copy data access (ideal for centralizing data governance). Commercially, we heard AI use cases around treasury services, fraud detection and risk analytics.

Banking 98

Banking Finance Retail Food 98

Precisely

JANUARY 30, 2025

Integrate data governance and data quality practices to create a seamless user experience and build trust in your data. When planning your data governance approach, start small, iterate purposefully, and foster data literacy to drive meaningful business outcomes.

Data Governance 103

Data Governance Government Metadata Data 103

Snowflake

FEBRUARY 13, 2025

Furthermore, most vendors require valuable time and resources for cluster spin-up and spin-down, disruptive upgrades, code refactoring or even migrations to new editions to access features such as serverless capabilities and performance improvements.

Management 93

Management Government Cloud Unstructured Data 93

Cloudera

OCTOBER 23, 2024

In August, we wrote about how in a future where distributed data architectures are inevitable, unifying and managing operational and business metadata is critical to successfully maximizing the value of data, analytics, and AI.

Metadata 84

Metadata BI Data Lake Business Intelligence 84

phData: Data Engineering

NOVEMBER 1, 2024

Data governance plays a critical role in the successful implementation of Generative AI (GenAI) and large language models (LLM), with 86.7% It serves as a vital protective measure, ensuring proper data access while managing risks like data breaches and unauthorized use.

Government 52

Government Data Governance Finance Metadata 52

Data Engineering Podcast

JUNE 23, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Data lakes in various forms have been gaining significant popularity as a unified interface to an organization's analytics. Closing Announcements Thank you for listening!

Data Lake 162

Data Lake High Quality Data Hadoop Machine Learning 162

Edureka

APRIL 22, 2025

It lets you describe data more complexly and make predictions. AI-powered data engineering solutions make it easier to streamline the data management process, which helps businesses find useful insights with little to no manual work. This will help make better analytics predictions and improve data management.

Data Engineering 52

Data Engineering Data Engineer Engineering Consulting 52

Data Engineering Podcast

APRIL 7, 2024

Summary Maintaining a single source of truth for your data is the biggest challenge in data engineering. Different roles and tasks in the business need their own ways to access and analyze the data in the organization. Can you start by outlining the technical elements of what it means to have a "semantic layer"?

Data Lake 162

Data Lake High Quality Data BI Data Workflow 162

Data Engineering Podcast

FEBRUARY 11, 2024

In this episode Andrew Jefferson explains the complexities of building a robust system for data sharing, the techno-social considerations, and how the Bobsled platform that he is building aims to simplify the process. What is the current state of the ecosystem for data sharing protocols/practices/platforms? tabular, image, etc.)

Data Lake 147

Data Lake High Quality Data Government Machine Learning 147

Data Engineering Podcast

JUNE 16, 2024

In this episode Kevin Liu shares some of the interesting features that they have built by combining those technologies, as well as the challenges that they face in supporting the myriad workloads that are thrown at this layer of their data platform. Can you describe what role Trino and Iceberg play in Stripe's data architecture?

Data Lake 147

Data Lake High Quality Data Metadata Machine Learning 147

Data Engineering Podcast

APRIL 21, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Can you describe what Shortwave is and the story behind it? What is the core problem that you are addressing with Shortwave?

Data Lake 182

Data Lake High Quality Data Machine Learning Data Pipeline 182

Data Engineering Podcast

JUNE 2, 2024

In this episode she shares the practical steps to implementing a data governance practice in your organization, and the pitfalls to avoid. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Are there any trends that concern you?

Data Governance 147

Data Governance Government Data Lake High Quality Data 147

Snowflake

APRIL 22, 2025

Even these early adopters, the ones who report great overall success, have hit some snags with their data platforms. At the data platform level, we found: 55% of organizations are hampered by time-consuming data management tasks such as labeling. 51% say data preparation is too hard.

IT 59

IT Unstructured Data Government Cloud 59

Ascend.io

DECEMBER 4, 2024

Are your tools simple to implement and accessible to users with diverse skill sets? Develop a roadmap for how each tool will integrate into your data stack, and involve IT from the beginning to ensure technical success. Further Exloration: What is data automation? Keep Learning: Dive Deeper Into DataOps 4.

Data Pipeline 52

Data Pipeline Metadata Data Workflow Data 52

Data Engineering Podcast

NOVEMBER 26, 2023

Summary Building a data platform that is enjoyable and accessible for all of its end users is a substantial challenge. One of the core complexities that needs to be addressed is the fractal set of integrations that need to be managed across the individual components. Closing Announcements Thank you for listening!

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Data Engineering Podcast

APRIL 16, 2023

Summary Business intellingence has been chasing the promise of self-serve data for decades. As the capabilities of these systems has improved and become more accessible, the target of what self-serve means changes. As the capabilities of these systems has improved and become more accessible, the target of what self-serve means changes.

Business Intelligence 147

Business Intelligence Building Data Lake BI 147

Snowflake

JANUARY 15, 2024

Leading companies around the world rely on Informatica data management solutions to manage and integrate data across various platforms from virtually any data source and on any cloud. Enterprise Data Integrator is fueled by Informatica Superpipe for Snowflake, which enables up to 3.5x

Data Integration 122

Data Integration BI Cloud Data 122

Pinterest Engineering

OCTOBER 2, 2024

As a data serving platform, SDS is responsible for various aspects of the data lifecycle management, including: Online Query Serving , which offers multiple data models such as graph, table, document, etc. Clients cannot easily use different interfaces to access data.

Data Management 64

Data Management Management Metadata SQL 64

Snowflake

SEPTEMBER 13, 2023

This helps them identify changes in customer behavior, allowing for smarter inventory management and marketing decisions, all leading to better customer experiences through AI-driven insights. Once the process is complete, the data share automatically becomes available to the Snowflake target account as secure views.

Finance 129

Finance AWS Accessibility Accessible 129

Data Engineering Podcast

DECEMBER 28, 2020

Summary One of the core responsibilities of data engineers is to manage the security of the information that they process. The team at Satori has a background in cybersecurity and they are using the lessons that they learned in that field to address the challenge of access control and auditing for data governance.

Data Governance 130

Data Governance Government Transportation Data 130

Data Engineering Weekly

APRIL 24, 2025

. - Data Infrastructure: A Robust data infrastructure is necessary to support the volume, velocity, and variety of data required by sophisticated AI models. Integration and Accessibility: The ease of integrating and accessing data within the organization significantly accelerates AI adoption and effectiveness.

Government 52

Government Data Governance High Quality Data Machine Learning 52

Data Engineering Podcast

JUNE 30, 2024

He highlights the role of data teams in modern organizations and how Synq is empowering them to achieve this. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Can you describe what Synq is and the story behind it?

Pipeline-centric 147

Pipeline-centric Engineering Data Lake High Quality Data 147

Data Engineering Podcast

FEBRUARY 4, 2024

In this episode Yingjun Wu explains how it is architected to power analytical workflows on continuous data flows, and the challenges of making it responsive and scalable. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

SQL 173

SQL Data Lake High Quality Data Machine Learning 173

Data Engineering Podcast

JANUARY 21, 2024

In this episode Tasso Argyros, CEO of ActionIQ, gives a summary of the major epochs in database technologies and how he is applying the capabilities of cloud data warehouses to the challenge of building more comprehensive experiences for end-users through a modern customer data platform (CDP).

Data Lake 147

Data Lake High Quality Data NoSQL Data Warehouse 147

Data Engineering Podcast

MAY 26, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Can you start by sharing some of your experiences with data migration projects? Closing Announcements Thank you for listening! Don't forget to check out our other shows.

Systems 130

Systems Data Lake High Quality Data Google Cloud 130

Data Engineering Podcast

DECEMBER 21, 2020

Summary Data governance is a term that encompasses a wide range of responsibilities, both technical and process oriented. One of the more complex aspects is that of access control to the data assets that an organization is responsible for managing. What is data governance? BigQuery, Snowflake, RedShift, etc.)?

Data Governance 130

Data Governance Government Data Lake Banking 130

Data Engineering Podcast

FEBRUARY 5, 2023

Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Sign up now for early access to Materialize and get started with the power of streaming data with the same simplicity and low implementation cost as batch cloud data warehouses.

Data Engineering 147

Data Engineering Data Engineer Engineering PostgreSQL 147