databricks

SEPTEMBER 30, 2024

This blog explores how new technologies such as Databricks Data Intelligence Platform can pave the way for more effective and efficient multi-omics data management.

Data Management 118

Data Management Management Data Technology 118

Engineering at Meta

MAY 22, 2024

In recent years, Meta’s data management systems have evolved into a composable architecture that creates interoperability, promotes reusability, and improves engineering efficiency. Data is at the core of every product and service at Meta. Data is at the core of every product and service at Meta.

Data Management 98

Data Management Management Data SQL 98

KDnuggets

NOVEMBER 13, 2023

This week, we delve into the vital world of Databases, SQL, Data Management, and Statistical Concepts in Data Science. Welcome back to Week 2 of KDnuggets’ "Back to Basics" series.

Database 139

Database SQL Data Management Management 139

Precisely

OCTOBER 23, 2023

When most people think of master data management, they first think of customers and products. But master data encompasses so much more than data about customers and products. Challenges of Master Data Management A decade ago, master data management (MDM) was a much simpler proposition than it is today.

Data Management 109

Data Management Management Data Data Integration 109

Snowflake

FEBRUARY 13, 2025

. "Serverless computing" has enabled customers to use cloud capabilities without provisioning, deploying and managing either hardware or software resources. Snowflake has embraced serverless since our founding in 2012, with customers providing their code to load, manage and query data and us taking care of the rest.

Management 85

Management Government Cloud Unstructured Data 85

KDnuggets

DECEMBER 12, 2023

This article provides an overview of two new data preparation techniques that enable data democratization while minimizing transformation burdens.

Data Management 151

Data Management Management Data Preparation Data 151

Engineering at Meta

FEBRUARY 20, 2024

This new convergence helps Meta and the larger community build data management systems that are unified, more efficient, and composable. Meta’s Data Infrastructure teams have been rethinking how data management systems are designed.

Data Management 107

Data Management Bytes Management Datasets 107

KDnuggets

AUGUST 31, 2023

These are my predictions based on my personal experiences, recent research and reports from leading platforms.

Data Management 142

Data Management Management Data Data Engineering 142

ArcGIS

FEBRUARY 3, 2025

Streamline data management, reduce duplication, enhance data quality, and provide a single source of truth for address and road data.

Data Management 78

Data Management Management Data Transportation 78

databricks

JUNE 28, 2024

Introduction Financial institutions face a demanding environment with complex regulatory examinations and a pressing need for flexible and comprehensive risk management solutions.

Data Management 98

Data Management Management Data 98

databricks

FEBRUARY 22, 2024

For many organizations, log data that security professionals need for effective. In today's environment, proactive cybersecurity is crucial to any public sector agency.

Data Management 122

Data Management Management Data 122

Data Engineering Podcast

MAY 12, 2024

Summary Building a data platform is a substrantial engineering endeavor. Once it is running, the next challenge is figuring out how to address release management for all of the different component parts. Support Data Engineering Podcast Summary Building a data platform is a substrantial engineering endeavor.

Management 147

Management Data Lake High Quality Data Machine Learning 147

ArcGIS

APRIL 21, 2025

Learn how to save geoprocessing tool outputs to the memory workspace, and about some updates in ArcGIS Pro 3.5!

Data Management 104

Data Management Management Data 104

Snowflake

NOVEMBER 2, 2023

In the ever-evolving world of data management, Snowflake is at the forefront of empowering our customers to make informed decisions about data while ensuring cost efficiency and control. Admins know that managing and optimizing platform costs can be a complex and time-consuming task.

Management 124

Management Retail SQL AWS 124

Precisely

JANUARY 9, 2025

In the beginning, there was a data warehouse The data warehouse (DW) was an approach to data architecture and structured data management that really hit its stride in the early 1990s. There was no easy way to consolidate and analyze this data to more effectively manage our business.

Data Integration 118

Data Integration Hadoop Data Warehouse Data Lake 118

KDnuggets

MAY 14, 2025

In this article, you'll master 10 essential Linux file system commands. This guide provides helpful examples to make working with files easier.

66

66

Data Engineering Podcast

APRIL 23, 2023

In this episode DeVaris Brown discusses the types of applications that are possible when teams don't have to manage the complex infrastructure necessary to support continuous data flows. Can you describe what Meroxa is and the story behind it? How have the focus and goals of the platform and company evolved over the past 2 years?

Data Lake 277

Data Lake Kafka Machine Learning Data Warehouse 277

Precisely

OCTOBER 31, 2024

Key Takeaways: Data mesh is a decentralized approach to data management, designed to shift creation and ownership of data products to domain-specific teams. Data fabric is a unified approach to data management, creating a consistent way to manage, access, and share data across distributed environments.

Data Architecture 68

Data Architecture Architecture Metadata Government 68

Monte Carlo

APRIL 29, 2025

Product data management fixes that. Product data management (PDM) is the practice of organizing, storing, and managing all the data related to a product in one central system. Product data management systems bring all of that information together into one structured, searchable place.

Data Management 52

Data Management Management Manufacturing Data Pipeline 52

RandomTrees

DECEMBER 4, 2024

As one of the most important sectors of the global economy, the food and beverage (F&B) industry works in highly volatile conditions and ensures its success by reducing waste and managing inventories. Managing production and consumption, meeting deadlines, cutting waste, and being environmentally friendly are always a challenge.

Food 52

Food Management Manufacturing Machine Learning 52

Data Engineering Podcast

DECEMBER 29, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you're ready to build your next pipeline, or want to test out the projects you hear about on the show, you'll need somewhere to deploy it, so check out our friends at Linode.

Management 130

Management Metadata Business Intelligence Data Lake 130

Data Engineering Podcast

FEBRUARY 18, 2024

In this episode Dain Sundstrom, CTO of Starburst, explains how the combination of the Trino query engine and the Iceberg table format offer the ease of use and execution speed of data warehouses with the infinite storage and scalability of data lakes. What do you have planned for the future of Trino/Starburst?

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Data Engineering Podcast

JUNE 25, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. What do you see as the potential benefits of integration with e.g. data-diff?

Data Engineering 225

Data Engineering Data Engineer Python Engineering 225

Precisely

JANUARY 30, 2025

Integrate data governance and data quality practices to create a seamless user experience and build trust in your data. When planning your data governance approach, start small, iterate purposefully, and foster data literacy to drive meaningful business outcomes.

Data Governance 102

Data Governance Government Metadata Data 102

Data Engineering Podcast

DECEMBER 24, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Precisely

NOVEMBER 18, 2024

Anandarajan notes that complexity is a top factor in the decline of data confidence, citing the report finding that about 42% of organizations find their data environments becoming increasingly complex. With the surge of new tools, platforms, and data types, managing these systems effectively is an ongoing challenge.

Data Analytics 71

Data Analytics Data Governance Data Integration Government 71

Data Engineering Podcast

SEPTEMBER 10, 2023

This offers a single location for managing visibility and error handling so that data platform engineers can manage complexity. In this episode Nick Schrock, creator of Dagster, shares his perspective on the state of data orchestration technology and its application to help inform its implementation in your environment.

BI 208

BI SQL Machine Learning Data 208

Snowflake

NOVEMBER 19, 2024

How will my data stay secure and governed? Managed Apache Spark environments — such as Databricks, Amazon EMR, and certain Cloudera deployments — can present teams with a plethora of pain points, which may include complexity, unpredictable costs, security concerns, or performance issues. What will costs look like?

Data Governance 98

Data Governance Government Healthcare Building 98

Engineering at Meta

APRIL 28, 2025

Managing and understanding large-scale data ecosystems is a significant challenge for many organizations, requiring innovative solutions to efficiently safeguard user data. To address these challenges, we made substantial investments in advanced data understanding technologies, as part of our Privacy Aware Infrastructure (PAI).

Metadata 79

Metadata Data Utilities Data Warehouse 79

Data Engineering Podcast

APRIL 21, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. How do you manage the personalization of the AI functionality in your system for each user/team?

Data Lake 182

Data Lake High Quality Data Machine Learning Data Pipeline 182

Data Engineering Podcast

JUNE 2, 2024

In this episode she shares the practical steps to implementing a data governance practice in your organization, and the pitfalls to avoid. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Are there any trends that concern you?

Data Governance 147

Data Governance Government Data Lake High Quality Data 147

Data Engineering Podcast

JUNE 23, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Data lakes in various forms have been gaining significant popularity as a unified interface to an organization's analytics. Closing Announcements Thank you for listening!

Data Lake 162

Data Lake High Quality Data Hadoop Machine Learning 162

Data Engineering Podcast

OCTOBER 29, 2023

In this episode Tanya Bragin shares her experiences as a product manager for two major vendors and the lessons that she has learned about how teams should approach the process of tool selection. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles.

Database 189

Database SQL BI Machine Learning 189

Data Engineering Weekly

JANUARY 8, 2025

What if your data lake could do more than just store information—what if it could think like a database? As data lakehouses evolve, they transform how enterprises manage, store, and analyze their data. Hudi, with its robust community and technical innovation, is well-positioned to lead this charge.

Data Lake 62

Data Lake Datasets Retail Data Ingestion 62

Edureka

APRIL 22, 2025

Data is more than simply numbers as we approach 2025; it serves as the foundation for business decision-making in all sectors. However, data alone is insufficient. To remain competitive in the current digital environment, businesses must effectively gather, handle, and manage it. Data engineering can help with it.

Data Engineering 52

Data Engineering Data Engineer Engineering Consulting 52

Data Engineering Podcast

MAY 28, 2023

Summary Building a data team is hard in any circumstance, but at a startup it can be even more challenging. The requirements are fluid, you probably don't have a lot of existing data talent to manage the hiring and onboarding, and there is a need to move fast. When is it more practical to outsource the data work?

Data Lake 162

Data Lake Machine Learning Data Warehouse Education 162

Data Engineering Podcast

JUNE 30, 2024

Petr shares his journey from being an engineer to founding Synq, emphasizing the importance of treating data systems with the same rigor as engineering systems. He discusses the challenges and solutions in data reliability, including the need for transparency and ownership in data systems.

Pipeline-centric 147

Pipeline-centric Engineering Data Lake High Quality Data 147