Data Engineering Weekly

JULY 7, 2024

Learn More → Notion: Building and scaling Notion’s data lake Notion writes about scaling the data lake by bringing critical data ingestion operations in-house. Hudi seems to be a de facto choice for CDC data lake features. Notion migrated the insert heavy workload from Snowflake to Hudi.

Data Engineering 98

Data Engineering Data Engineer Engineering Data Lake 98

Grouparoo

JANUARY 11, 2022

This article looks at the options available for storing and processing big data, which is too large for conventional databases to handle. There are two main options available, a data lake and a data warehouse. What is a Data Warehouse? What is a Data Lake?

Data Lake 52

Data Lake Data Warehouse Unstructured Data Raw Data 52

Cloudera

MARCH 31, 2021

Customers who have chosen Google Cloud as their cloud platform can now use CDP Public Cloud to create secure governed data lakes in their own cloud accounts and deliver security, compliance and metadata management across multiple compute clusters. Data Preparation (Apache Spark and Apache Hive) .

Google Cloud 100

Google Cloud Cloud Amazon Web Services Cloud Storage 100

Data Engineering Podcast

SEPTEMBER 11, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. How does the garbage in/garbage out problem manifest in data warehouse/data lake environments?

Systems 100

Systems Metadata Building MongoDB 100

Cloudera

NOVEMBER 5, 2020

When you register an Environment in CDP, a Data Lake is automatically deployed for that environment. Data Lake security and governance is managed by a shared set of services running within a Data Lake cluster. Apache Atlas — metadata management and governance: lineage, analytics, attributes.

Cloud 76

Cloud Data Lake Data Warehouse Metadata 76

Cloudera

JANUARY 21, 2021

With CDW, as an integrated service of CDP, your line of business gets immediate resources needed for faster application launches and expedited data access, all while protecting the company’s multi-year investment in centralized data management, security, and governance. Proprietary file formats mean no one else is invited in!

IT 96

IT Data Lake Data Warehouse Cloud Storage 96

ProjectPro

AUGUST 11, 2021

“Data Lake vs Data Warehouse = Load First, Think Later vs Think First, Load Later” The terms data lake and data warehouse are frequently stumbled upon when it comes to storing large volumes of data. Data Warehouse Architecture What is a Data lake?

Data Lake 52

Data Lake Data Warehouse Cloud Hadoop 52

ProjectPro

FEBRUARY 8, 2023

It offers a simple and efficient solution for data processing in organizations. It offers users a data integration tool that organizes data from many sources, formats it, and stores it in a single repository, such as data lakes, data warehouses, etc., being data exactly matches the classifier, and 0.0

AWS 98

AWS Scala Metadata Data Lake 98

Cloudera

FEBRUARY 8, 2024

Note that the materialized view definition contains the ‘stored by iceberg’ clause. The snapshotId of the source tables involved in the materialized view are also maintained in the metadata. Let’s create a materialized view that joins the three tables, has filter conditions, and does grouped aggregation.

Metadata 96

Metadata Data Warehouse BI AWS 96

Cloudera

JUNE 7, 2023

By now, almost everyone across the tech landscape has heard of the Zero Trust (ZT) security model, which assumes that every device, application, or user attempting to access a network is not to be trusted (see NIST definitions below). Understanding your data is critical to protecting the data.

Government 88

Government Metadata Data Lake Data 88

Data Engineering Podcast

NOVEMBER 20, 2021

In the same way that application performance monitoring ensures reliable software and keeps application downtime at bay, Monte Carlo solves the costly problem of broken data pipelines. Start trusting your data with Monte Carlo today! What (if any) option do you see for language agnostic/multi-language pipeline definitions in Dagster?

Data Warehouse 100

Data Warehouse Data Lake BI Business Intelligence 100

Cloudera

SEPTEMBER 15, 2022

Mark: The first element in the process is the link between the source data and the entry point into the data platform. At Ramsey International (RI), we refer to that layer in the architecture as the foundation, but others call it a staging area, raw zone, or even a source data lake. What is a data fabric?

Data Lake 88

Data Lake Analytics Application Cloud Storage Architecture 88

LinkedIn Engineering

JULY 19, 2023

Open source data lakehouse deployments are built on the foundations of compute engines (like Apache Spark, Trino, Apache Flink), distributed storage (HDFS, cloud blob stores), and metadata catalogs / table formats (like Apache Iceberg, Delta, Hudi, Apache Hive Metastore). Tables are governed as per agreed upon company standards.

Big Data 78

Big Data Data Management Management Metadata 78

Monte Carlo

MAY 14, 2024

At its core, a table format is a sophisticated metadata layer that defines, organizes, and interprets multiple underlying data files. Table formats incorporate aspects like columns, rows, data types, and relationships, but can also include information about the structure of the data itself.

Data Lake 52

Data Lake Metadata Hadoop Data Governance 52

Towards Data Science

FEBRUARY 15, 2023

In this context, data management in an organization is a key point for the success of its projects involving data. One of the main aspects of correct data management is the definition of a data architecture. What is Delta Lake? The data became useless. show() The history object is a Spark Data Frame.

Data Lake 74

Data Lake Data Warehouse Hadoop Architecture 74

Cloudera

MARCH 17, 2023

The table format provides the necessary structure for the unstructured data that is missing in a data lake, using a schema or metadata definition, to bring it closer to a data warehouse. Some of the popular table formats are Apache Iceberg, Delta Lake, Hudi, and Hive ACID.

Education 65

Education Unstructured Data Data Lake Data Warehouse 65

AltexSoft

AUGUST 22, 2022

A data fabric is an architecture design presented as an integration and orchestration layer built on top of multiple disjointed data sources like relational databases , data warehouses , data lakes, data marts , IoT , legacy systems, etc., to provide a unified view of all enterprise data.

Architecture 52

Architecture Metadata Data Lake Machine Learning 52

Monte Carlo

MARCH 4, 2024

To do this, data observability solutions like Monte Carlo will piece together information from query logs, data aggregates and system metadata. All of this is to say that how table formats are structured, and specifically the metadata, impacts how they can be monitored and observed.

Metadata 52

Metadata Data Lake Data Engineering Data Engineer 52

AltexSoft

NOVEMBER 23, 2021

If the transformation step comes after loading (for example, when data is consolidated in a data lake or a data lakehouse ), the process is known as ELT. You can learn more about how such data pipelines are built in our video about data engineering. Abstraction layer.

Process 69

Process Data Lake Metadata Data Warehouse 69

Snowflake

JUNE 4, 2024

This includes pipelines and transformations with Snowpark, Streams, Tasks and Dynamic Tables (public preview soon); extending AI and ML to Iceberg with Snowflake Cortex AI; performing storage maintenance with capabilities like automatic clustering and compaction; as well as securely collaborating on live data shares.

Government 87

Government Data Ingestion Data PostgreSQL 87

Advancing Analytics: Data Engineering

SEPTEMBER 7, 2021

1 SCD Type 1 in SQL and Python Introduction With the move to cloud based Data Lake platforms there has often been criticism from the more traditional Data Warehousing community. Sometimes just a change in personnel can begin the chain reaction of orphaned datasets and forgotten definitions. "By

Data Lake 52

Data Lake Datasets Data Warehouse SQL 52

Snowflake

NOVEMBER 11, 2024

Data stewards can also set up Request for Access (private preview) by setting a new visibility property on objects along with contact details so the right person can easily be reached to grant access. Support for auto-refresh and Iceberg metadata generation is coming soon to Delta Lake Direct.

Data Architecture 85

Data Architecture Architecture Data Lake Kafka 85

DataKitchen

OCTOBER 11, 2021

First, you must understand the existing challenges of the data team, including the data architecture and end-to-end toolchain. Second, you must establish a definition of “done.” In DataOps, the definition of done includes more than just some working code. Definition of Done. Monitoring Job Metadata.

Engineering 64

Engineering BI Business Analyst Metadata 64

AltexSoft

OCTOBER 30, 2021

One data scientist usually needs two or three data engineers. Depending on the complexity of a data pipeline or the experience of your data engineers, there could be more of them. Managing data and metadata. Data engineers control how data is stored and structured within those locations.

Data Engineering 98

Data Engineering Data Engineer Engineering Machine Learning 98

Grouparoo

OCTOBER 6, 2021

This provided a nice overview of the breadth of topics that are relevant to data engineering including data warehouses/lakes, pipelines, metadata, security, compliance, quality, and working with other teams. For example, grouping the ones about metadata, discoverability, and column naming might have made a lot of sense.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Ascend.io

APRIL 22, 2024

This initiative is more than just an upgrade; it’s a reimagining of what a Data Automation Platform can be: dynamic, extensible, and highly intelligent. A unified platform that combines a powerful metadata core, an extensible plugin architecture, DataAware automation, and multiple AI Assistants.

Project 52

Project Metadata Data Pipeline Data Engineering 52

Monte Carlo

DECEMBER 12, 2024

It means defining that data by documenting relationships between creator and context (like customers and their orders), establishing clear business definitions (what exactly counts as an “active user”?), and maintaining metadata about data freshness, quality, and lineage (more on that in a moment).

Government 52

Government Data Cloud Datasets 52

Data Engineering Weekly

JANUARY 15, 2023

SiliconANGLE theCUBE: Analyst Predictions 2023 - The Future of Data Management By far one of the best analyses of trends in Data Management. 2023 predictions from the panel are; Unified metadata becomes kingmaker. The author walked through various strategies, from sync to async job submission and batch job submission strategy.

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

phData: Data Engineering

NOVEMBER 1, 2024

Chief Technology Officer, Information Technology Industry Organizations have spent the past decade accumulating, maintaining, and securing data lakes/warehouses/fabrics that will now be expected to drive AI/LLM use cases. The technology for metadata management, data quality management, etc., No problem!

Government 52

Government Data Governance Finance Metadata 52

ProjectPro

JANUARY 24, 2023

You can leverage AWS Glue to discover, transform, and prepare your data for analytics. In addition to databases running on AWS, Glue can automatically find structured and semi-structured data kept in your data lake on Amazon S3, data warehouse on Amazon Redshift, and other storage locations.

AWS 52

AWS Data Lake ETL Tools Scala 52

Monte Carlo

AUGUST 10, 2023

What is Data observability? The five pillars My data observability definition has not changed since I first coined it in 2019: Data observability refers to an organization’s comprehensive understanding of the health and performance of the data within their systems. What is data pipeline observability?

Data Pipeline 98

Data Pipeline Software Engineer Software Engineering Machine Learning 98

Monte Carlo

MARCH 12, 2024

Google Cloud’s Dataplex is a data fabric tool that enables organizations to discover, manage, monitor, and govern their data across all of their data systems, including their data lakes, data warehouses, data lakehouses, and data marts. Dataplex works with your metadata.

Google Cloud 52

Google Cloud Metadata SQL Data Lake 52

DareData

AUGUST 12, 2021

This data will be at the root of your data ecosystem. In the data ecosystem, this role is taken by the data lake. There are, however, a few core ideas whose proper implementation, or lack thereof, has definitely felt very impactful in the different data ecosystems I have worked in.

Transportation 52

Transportation Data Lake Data Warehouse Data 52

AltexSoft

JULY 29, 2022

a runtime environment (sandbox) for classic business intelligence (BI), advanced analysis of large volumes of data, predictive maintenance , and data discovery and exploration; a store for raw data; a tool for large-scale data integration ; and. a suitable technology to implement data lake architecture.

Hadoop 59

Hadoop Big Data Google Cloud NoSQL 59

AltexSoft

JUNE 26, 2023

External or third-party data sources, on the other hand, deal with outside information, which comes from partners, competitors, social media, general market studies, databases with publicly available datasets , etc. Our experience shows that you definitely need both internal and external data to make accurate forecasts.

Data Collection 52

Data Collection Machine Learning Unstructured Data Non-relational Database 52

Ascend.io

DECEMBER 11, 2023

Data orchestration is the process of efficiently coordinating the movement and processing of data across multiple, disparate systems and services within a company. However, this approach quickly shows its limitations as data volume escalates. So, why is data orchestration a big deal?

Data Workflow 52

Data Workflow Data Pipeline Data Lake Data 52

Ascend.io

JULY 19, 2023

But something about data mesh feels different, doesn’t it? For one, data mesh tackles the real headaches caused by an overburdened data lake and the annoying game of tag that’s too often played between the people who make data, the ones who use it, and everyone else caught in the middle.

Data Governance 40

Data Governance Government Metadata Data 40