Snowflake

NOVEMBER 2, 2023

Over the years, the technology landscape for data management has given rise to various architecture patterns, each thoughtfully designed to cater to specific use cases and requirements. These patterns include both centralized storage patterns like data warehouse , data lake and data lakehouse , and distributed patterns such as data mesh.

Data Lake 114

Data Lake Data Warehouse Cloud Unstructured Data 114

Data Engineering Podcast

SEPTEMBER 1, 2021

Summary The Presto project has become the de facto option for building scalable open source analytics in SQL for the data lake. Another area that has been seeing a lot of activity is data lakes and projects to make them more manageable and feature complete (e.g. Hudi, Delta Lake, Iceberg, Nessie, LakeFS, etc.).

Data Lake 100

Data Lake Cloud AWS SQL 100

Towards Data Science

SEPTEMBER 27, 2023

Data Access API over Data Lake Tables Without the Complexity Build a robust GraphQL API service on top of your S3 data lake files with DuckDB and Go Photo by Joshua Sortino on Unsplash 1. This data might be primarily used for internal reporting, but might also be valuable for other services in our organization.

Data Lake 98

Data Lake Accessible Accessibility SQL 98

Edureka

APRIL 22, 2025

It incorporates elements from several Microsoft products working together, like Power BI, Azure Synapse Analytics, Data Factory, and OneLake, into a single SaaS experience. It provides real multi-cloud flexibility in its operations on AWS , Azure, and Google Cloud.

BI 52

BI Pipeline-centric Data Lake Google Cloud 52

ProjectPro

FEBRUARY 8, 2023

Do ETL and data integration activities seem complex to you? AWS Glue is here to put an end to all your worries! Read this blog to understand everything about AWS Glue that makes it one of the most popular data integration solutions in the industry. Did you know the global big data market will likely reach $268.4

AWS 98

AWS Scala Metadata Data Lake 98

Data Engineering Podcast

FEBRUARY 4, 2024

In this episode Yingjun Wu explains how it is architected to power analytical workflows on continuous data flows, and the challenges of making it responsive and scalable. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

SQL 173

SQL Data Lake High Quality Data Machine Learning 173

Data Engineering Podcast

JUNE 16, 2024

In this episode Kevin Liu shares some of the interesting features that they have built by combining those technologies, as well as the challenges that they face in supporting the myriad workloads that are thrown at this layer of their data platform.

Data Lake 147

Data Lake High Quality Data Metadata Machine Learning 147

Data Engineering Podcast

NOVEMBER 26, 2023

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Precisely

MARCH 9, 2023

Data warehouse vs. data lake, each has their own unique advantages and disadvantages; it’s helpful to understand their similarities and differences. In this article, we’ll focus on a data lake vs. data warehouse. It is often used as a foundation for enterprise data lakes.

Data Lake 58

Data Lake Data Warehouse Hadoop Raw Data 58

Data Engineering Podcast

MAY 14, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. Rudderstack]([link] RudderStack provides all your customer data pipelines in one platform.

Data Lake 147

Data Lake Machine Learning Data Warehouse AWS 147

Cloudyard

DECEMBER 13, 2022

Read Time: 4 Minute, 23 Second During this post we will discuss how AWS S3 service and Snowflake integration can be used as Data Lake in current organizations. How customer has migrated On Premises EDW to Snowflake to leverage snowflake Data Lake capabilities. Create S3 bucket to hold the tables data.

Data Lake 52

Data Lake AWS Metadata Data 52

Data Engineering Podcast

MARCH 10, 2024

Summary Data lakehouse architectures are gaining popularity due to the flexibility and cost effectiveness that they offer. The link that bridges the gap between data lake and warehouse capabilities is the catalog. Data lakes are notoriously complex. What is involved in integrating Nessie into a given data stack?

Data Lake 147

Data Lake High Quality Data Architecture Machine Learning 147

Snowflake

DECEMBER 11, 2023

The authorization furthers Snowflake’s commitment to helping our government customers secure and mobilize their mission-critical data It’s a milestone moment for Snowflake to have achieved FedRAMP High authorization on the AWS GovCloud (US-West and US-East Regions).

AWS 110

AWS Education Hospitality Government 110

Monte Carlo

SEPTEMBER 25, 2024

A data lake is essentially a vast digital dumping ground where companies toss all their raw data, structured or not. A modern data stack can be built on top of this data storage and processing layer, or a data lakehouse or data warehouse, to store data and process it before it is later transformed and sent off for analysis.

Data Lake 52

Data Lake Food Google Cloud AWS 52

Monte Carlo

APRIL 24, 2023

Data lakes are useful, flexible data storage repositories that enable many types of data to be stored in its rawest state. Traditionally, after being stored in a data lake, raw data was then often moved to various destinations like a data warehouse for further processing, analysis, and consumption.

Data Lake 52

Data Lake Google Cloud Data Warehouse AWS 52

Sync Computing

JANUARY 27, 2025

This brings us to todays topic: exploring strategies to manage your organizations data infrastructure in the most efficient and cost-efficient way possible. Databricks clusters and AWS EC2 In todays landscape, big data, which is data too large to fit into a single node machine, is transformed and managed by clusters.

AWS 52

AWS Data Lake Machine Learning Big Data 52

phData: Data Engineering

NOVEMBER 8, 2024

Note : Cloud Data warehouses like Snowflake and Big Query already have a default time travel feature. However, this feature becomes an absolute must-have if you are operating your analytics on top of your data lake or lakehouse. It can also be integrated into major data platforms like Snowflake. Contact phData Today!

Architecture 52

Architecture Systems Data Lake Google Cloud 52

DataKitchen

DECEMBER 9, 2024

Establish the Infrastructure The project began with establishing the data infrastructure. They opted for Snowflake, a cloud-native data platform ideal for SQL-based analysis. AWS Redshift, GCP Big Query, or Azure Synapse work well, too. It is necessary to have more than a data lake and a database.

Pharmaceutical 65

Pharmaceutical Data Lake Cloud Storage Project 65

Cloudera

SEPTEMBER 15, 2023

Next — working backwards — collaboration tooling such as dashboards and developer environments like those on Amazon Web Services (AWS), are also key components to consider. Underneath all of this are the data assets themselves, with appropriate governance (e.g.

Telecommunication 80

Telecommunication AWS Amazon Web Services Retail 80

Cloudera

OCTOBER 23, 2024

One of the most important innovations in data management is open table formats, specifically Apache Iceberg , which fundamentally transforms the way data teams manage operational metadata in the data lake. Try Cloudera’s open data lakehouse on AWS for 5 days for free here , or try Snowflake for free for 30 days here.

Metadata 84

Metadata BI Data Lake Business Intelligence 84

Data Engineering Podcast

MAY 22, 2022

Acryl Data provides DataHub as an easy to consume SaaS product which has been adopted by several companies. Signup for the SaaS product at dataengineeringpodcast.com/acryl RudderStack helps you build a customer data platform on your warehouse or data lake. Stop struggling to speed up your data lake.

Machine Learning 130

Machine Learning Data Engineering Data Engineer Cloud 130

Snowflake

JANUARY 24, 2023

CMS Chief Information Security Officer Rob Wood and Chief information Architect Amine Raounak led an in-depth presentation on their security data lake initiative with Snowflake. The security data lake project enables data-driven decision-making for system and product security at scale within CMS.

Data Lake 59

Data Lake IT Insurance AWS 59

phData: Data Engineering

SEPTEMBER 19, 2023

With the amount of data companies are using growing to unprecedented levels, organizations are grappling with the challenge of efficiently managing and deriving insights from these vast volumes of structured and unstructured data. What is a Data Lake? Consistency of data throughout the data lake.

Data Lake 52

Data Lake Process Metadata Data Warehouse 52

Data Engineering Podcast

NOVEMBER 20, 2022

Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. RudderStack helps you build a customer data platform on your warehouse or data lake. runs natively on data lakes and warehouses and in AWS, Google Cloud and Microsoft Azure.

Data Lake 130

Data Lake Data Ingestion MongoDB MySQL 130

Towards Data Science

APRIL 26, 2023

An open-source implementation of a Data Lake with DuckDB and AWS Lambdas A duck in the cloud. Moreover, the data will need to leave the cloud env to go on our machine, which is not exactly secure and auditable. The idea is to start from a Data Lake where our data are stored.

Engineering 96

Engineering Data Lake AWS BI 96

AltexSoft

AUGUST 29, 2023

In 2010, a transformative concept took root in the realm of data storage and analytics — a data lake. The term was coined by James Dixon , Back-End Java, Data, and Business Intelligence Engineer, and it started a new era in how organizations could store, manage, and analyze their data. What is a data lake?

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

ProjectPro

AUGUST 11, 2021

“Data Lake vs Data Warehouse = Load First, Think Later vs Think First, Load Later” The terms data lake and data warehouse are frequently stumbled upon when it comes to storing large volumes of data. Data Warehouse Architecture What is a Data lake? What is a Data lake?

Data Lake 52

Data Lake Data Warehouse Cloud Hadoop 52

U-Next

SEPTEMBER 7, 2022

The terms “ Data Warehouse ” and “ Data Lake ” may have confused you, and you have some questions. Structuring data refers to converting unstructured data into tables and defining data types and relationships based on a schema. What is Data Lake? . Athena on AWS. .

Data Lake 52

Data Lake Data Warehouse Unstructured Data Amazon Web Services 52

Snowflake

DECEMBER 4, 2023

With this public preview, those external catalog options are either “GLUE”, where Snowflake can retrieve table metadata snapshots from AWS Glue Data Catalog, or “OBJECT_STORE”, where Snowflake retrieves metadata snapshots directly from the specified cloud storage location. With these three options, which one should you use?

Building 120

Building Metadata Cloud Storage AWS 120

Data Engineering Weekly

FEBRUARY 23, 2025

[link] Alireza Sadeghi: Open Source Data Engineering Landscape 2025 This article comprehensively overviews the 2025 open-source data engineering landscape, highlighting key trends, active projects, and emerging technologies.

Data Engineer 62

Data Engineer Data Engineering Engineering Kafka 62

ProjectPro

JUNE 16, 2022

An AWS data pipeline helps businesses move and unify their data to support several data-driven initiatives. It enables flow from a data lake to an analytics database or an application to a data warehouse. AWS CLI is an excellent tool for managing Amazon Web Services.

Data Pipeline 59

Data Pipeline AWS Amazon Web Services Data Consolidation 59

Data Engineering Weekly

FEBRUARY 2, 2025

[link] Affirm: Expressive Time Travel and Data Validation for Financial Workloads Affirm migrated from daily MySQL snapshots to Change Data Capture (CDC) replay using Apache Iceberg for its data lake, improving data integrity and governance.

Data Engineering 57

Data Engineering Data Engineer Engineering Data Lake 57

phData: Data Engineering

APRIL 4, 2023

Fivetran today announced support for Amazon Simple Storage Service (Amazon S3) with Apache Iceberg data lake format. Amazon S3 is an object storage service from Amazon Web Services (AWS) that offers industry-leading scalability, data availability, security, and performance.

Data Lake 52

Data Lake Amazon Web Services Data Cleanse Data Warehouse 52

Edureka

JULY 16, 2024

However, going from data to the shape of a model in production can be challenging as it comprises data preprocessing, training, and deployment at a large scale. Amazon SageMaker, an AWS-managed AI service, is created to support enterprises on this journey and make it efficient and easy. Table of Content What is Amazon SageMaker?

AWS 52

AWS Algorithm Machine Learning Amazon Web Services 52

Data Engineering Podcast

OCTOBER 16, 2022

Summary The "data lakehouse" architecture balances the scalability and flexibility of data lakes with the ease of use and transaction support of data warehouses. Mention the podcast to get a free "In Data We Trust World Tour" t-shirt.

Data Lake 100

Data Lake Food MongoDB MySQL 100

Knowledge Hut

SEPTEMBER 25, 2023

Notably, there are several prominent cloud service providers in the market, such as Amazon AWS, Google GCP, and Microsoft Azure. In this blog, we will focus on the certifications offered by Amazon AWS. We will specifically talk about the AWS certification exam, AWS certification cost, and Amazon certification path.

AWS 52

AWS Certification Amazon Web Services Cloud Computing 52

Cloudera

JUNE 18, 2022

Cloudera customers run some of the biggest data lakes on earth. These lakes power mission critical large scale data analytics, business intelligence (BI), and machine learning use cases, including enterprise data warehouses. On data warehouses and data lakes.

Data Lake 94

Data Lake Data Warehouse BI SQL 94