Snowflake

DECEMBER 4, 2023

With this public preview, those external catalog options are either “GLUE”, where Snowflake can retrieve table metadata snapshots from AWS Glue Data Catalog, or “OBJECT_STORE”, where Snowflake retrieves metadata snapshots directly from the specified cloud storage location. With these three options, which one should you use?

Building 122

Building Metadata Cloud Storage AWS 122

ProjectPro

JUNE 6, 2025

This blog is your roadmap to building a data integration bridge out of chaos, leading to a world of streamlined insights. The data integration aspect of the project is highlighted in the utilization of relational databases, specifically PostgreSQL and MySQL , hosted on AWS RDS (Relational Database Service). Fear not, data engineers!

Data Integration 40

Data Integration Project Data Lake Hospitality 40

DataKitchen

DECEMBER 9, 2024

This blog dives into the remarkable journey of a data team that achieved unparalleled efficiency using DataOps principles and software that transformed their analytics and data teams into a hyper-efficient powerhouse. They opted for Snowflake, a cloud-native data platform ideal for SQL-based analysis.

Pharmaceutical 65

Pharmaceutical Data Lake Cloud Storage Cloud 65

ProjectPro

JUNE 6, 2025

To help you prepare for your data warehouse engineer interview, we have included a list of some popular Snowflake interview questions and answers in this blog. The data is organized in a columnar format in the Snowflake cloud storage. Briefly explain about Snowflake AWS. Yes, AWS glue and Snowflake can connect.

Amazon Web Services 40

Amazon Web Services Data Warehouse ETL Tools AWS 40

ProjectPro

JUNE 6, 2025

This shift presents abundant career opportunities, especially in big data and cloud computing , as businesses increasingly rely on cloud technologies. Therefore, gaining hands-on experience through practical projects in cloud computing is now essential for anyone looking to excel in this field.

Cloud Computing 40

Cloud Computing Cloud Project Google Cloud 40

ProjectPro

JUNE 6, 2025

This blog explores the top MLOps certifications, training courses, and the best resources to help you prepare for this journey. This blog is the perfect guide to exploring the secrets of this booming field, from understanding the hottest MLOps certifications to getting hands-on experience with real-world MLOps project examples.

Certification 53

Certification Google Cloud AWS Machine Learning 53

ProjectPro

JUNE 6, 2025

Learning Snowflake data Warehouse is like gaining a superpower for handling and analyzing data in the cloud. This blog is a definitive guide to mastering how to learn Snowflake data warehouse for all aspiring data engineers. This detailed blog guides you from the basics to a practical tutorial on how to learn Snowflake.

Data Warehouse 40

Data Warehouse SQL AWS Big Data 40

ProjectPro

JUNE 6, 2025

Snowflake vs BigQuery, both cloud data warehouses undoubtedly have unique capabilities, but deciding which is the best will depend on the user's requirements and interests. This blog will present a detailed comparison of Snowflake vs. BigQuery to help you select the best data warehouse solution for your next data engineering project.

Data Warehouse 40

Data Warehouse Cloud Google Cloud Cloud Storage 40

ProjectPro

JUNE 6, 2025

Migrating to a public, private, hybrid, or multi-cloud environment requires businesses to find a reliable, economical, and effective data migration project approach. From migrating data to the cloud to consolidating databases, this blog will cover a variety of data migration project ideas with best practices for successful data migration.

Project 40

Project Google Cloud AWS MongoDB 40

ProjectPro

JUNE 6, 2025

Cloud computing solves numerous critical business problems, which is why working as a cloud data engineer is one of the highest-paying jobs, making it a career of interest for many. Several businesses, such as Google and AWS , focus on providing their customers with the ultimate cloud experience.

Data Engineer 40

Data Engineer Data Engineering Google Cloud Engineering 40

Knowledge Hut

NOVEMBER 16, 2023

While cloud computing is pushing the boundaries of science and innovation into a new realm, it is also laying the foundation for a new wave of business start ups. 5 Reasons Your Startup Should Switch To Cloud Storage Immediately 1) Cost-effective Probably the strongest argument in cloud’s favor I is the cost-effectiveness that it offers.

Cloud Computing 98

Cloud Computing Cloud Cloud Storage AWS 98

ProjectPro

JUNE 6, 2025

This blog post compares these two platforms, Snowflake vs. Databricks, to help you choose the right platform for your next big data and data engineering projects. However, unlike Snowflake, Databricks lacks a storage layer because it functions on top of object-level storage such as AWS S3, Azure Blob Storage, Google Cloud Storage, and others.

Google Cloud 40

Google Cloud Cloud Storage Data Lake Data Storage 40

phData: Data Engineering

NOVEMBER 8, 2024

Open Table Format (OTF) architecture now provides a solution for efficient data storage, management, and processing while ensuring compatibility across different platforms. In this blog, we will discuss: What is the Open Table format (OTF)? Amazon S3, Azure Data Lake, or Google Cloud Storage). Why should we use it?

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Edureka

APRIL 22, 2025

In contrast to conventional warehouses, it keeps computation and storage apart, allowing for cost-effectiveness and dynamic scaling. It provides real multi-cloud flexibility in its operations on AWS , Azure, and Google Cloud. Snowflake: Offers multi-cloud support, which is present on AWS, Azure, and Google Cloud.

BI 52

BI Pipeline-centric Data Lake Google Cloud 52

DataKitchen

NOVEMBER 5, 2024

By storing data in its native state in cloud storage solutions such as AWS S3, Google Cloud Storage, or Azure ADLS, the Bronze layer preserves the full fidelity of the data. This foundational layer is a repository for various data types, from transaction logs and sensor data to social media feeds and system logs.

Architecture 79

Architecture Raw Data Pipeline-centric Data Ingestion 79

ProjectPro

JUNE 6, 2025

And if you are now searching for a list of that highlights those skills, head over to the next section of this blog. Project Idea: PySpark ETL Project-Build a Data Pipeline using S3 and MySQL Experience Hands-on Learning with the Best AWS Data Engineering Course and Get Certified! as they are required for processing large datasets.

Data Engineer 49

Data Engineer Data Engineering Engineering Hadoop 49

Scott Logic

APRIL 10, 2024

This blog post serves as a dev diary of the process, covering our challenges, contributions made and attempts to validate them. Further research We struggled to find more official information about how object storage is implemented and measured, so we decided to look at an object storage system that could be deployed locally called MinIO.

Cloud Storage 52

Cloud Storage Cloud AWS Metadata 52

ProjectPro

JUNE 6, 2025

Are you looking to choose the best cloud data warehouse for your next big data project? This blog presents a detailed comparison of two of the very famous cloud warehouses - Redshift vs. BigQuery - to help you pick the right solution for your data warehousing needs. The global data warehousing market will likely reach $51.18

Data Warehouse 40

Data Warehouse Data Mining Google Cloud PostgreSQL 40

ProjectPro

JUNE 6, 2025

The journey begins with understanding the fundamentals of cloud computing, which can take approximately six to twelve months for beginners to transition to an intermediate level. How to Learn Cloud Computing Step-by-Step? Now, how can you start learning cloud computing?

Cloud Computing 40

Cloud Computing Cloud Google Cloud AWS 40

ProjectPro

JUNE 6, 2025

This blog is your comprehensive guide to Google BigQuery, its architecture, and a beginner-friendly tutorial on how to use Google BigQuery for your data warehousing activities. BigQuery can process upto 20 TB of data per day and has a storage limit of 1PB per table. Search no more! Did you know ? What is Google BigQuery Used for?

Bytes 40

Bytes Google Cloud Data Warehouse Cloud Storage 40

ProjectPro

JUNE 6, 2025

Read this blog till the end to learn everything you need to know about Airflow DAG. This blog will dive into the details of Apache Airflow DAGs, exploring how they work and multiple examples of using Airflow DAGs for data processing and automation workflows. Apache Airflow DAGs are your one-stop solution!

Data Pipeline 63

Data Pipeline PostgreSQL Python Database 63

ProjectPro

JUNE 6, 2025

Read this blog for a detailed comparison between ELT vs. ETL to find out. Source Code- Yelp Data Analysis using Azure Databricks YouTube Data Analytics using AWS Glue, Lambda, and Athena In this Python ETL project, you'll develop an ETL Data Pipeline for YouTube data using Athena , Glue, and Lambda. But which method is better?

Data Engineer 40

Data Engineer Data Engineering Engineering Data Lake 40

Data Engineering Podcast

SEPTEMBER 22, 2019

What are the cases where it makes sense to use MinIO in place of a cloud-native object store such as S3 or Google Cloud Storage? What are the cases where it makes sense to use MinIO in place of a cloud-native object store such as S3 or Google Cloud Storage? What do you have planned for the future of MinIO?

Google Cloud 100

Google Cloud AWS Cloud Storage Data Lake 100

Cloudera

MARCH 31, 2021

In this first Google Cloud release, CDP Public Cloud provides built-in Data Hub definitions (see screenshot for more details) for: Data Ingestion (Apache NiFi, Apache Kafka). Google Cloud Storage buckets – in the same subregion as your subnets . Data Preparation (Apache Spark and Apache Hive) .

Google Cloud 94

Google Cloud Cloud Amazon Web Services Cloud Storage 94

ProjectPro

JUNE 6, 2025

This blog will give you an in-depth knowledge of what is a data pipeline and also explore other aspects such as data pipeline architecture, data pipeline tools, use cases, and so much more. AWS Glue You can easily extract and load your data for analytics using the fully managed extract, transform, and load (ETL) service AWS Glue.

Data Pipeline 40

Data Pipeline Architecture Kafka Data Lake 40

Netflix Tech

SEPTEMBER 24, 2021

Our previous tech blog Packaging award-winning shows with award-winning technology detailed our packaging technology deployed on the streaming side. From chunk encoding to assembly and packaging, the result of each previous processing step must be uploaded to cloud storage and then downloaded by the next processing step.

Cloud 97

Cloud Bytes Cloud Storage Media 97

ProjectPro

SEPTEMBER 6, 2021

Are you confused about choosing the best cloud platform for your next data engineering project ? AWS vs. GCP blog compares the two major cloud platforms to help you choose the best one. So, are you ready to explore the differences between two cloud giants, AWS vs. google cloud?

AWS 52

AWS Amazon Web Services Google Cloud Cloud Storage 52

ProjectPro

JUNE 6, 2025

In this blog, we’ll explore what an ETL developer does, the ETL developer skills required to excel in this role, the exciting ETL developer job opportunities available today, and the guide to becoming an ETL Developer. Works on data storage and retrieval, data processing, and data visualization.

ETL Tools 40

ETL Tools Data Cleanse Data Warehouse Big Data 40

Cloudera

DECEMBER 8, 2021

In this blog, we’ll share how CDP Operational Database can deliver high performance for your applications when running on AWS S3. CDP Operational Database allows developers to use Amazon Simple Storage Service (S3) as its main persistence layer for saving table data. AWS EC2 instance configurations. Test Environment.

Database 93

Database AWS Cloud Storage Datasets 93

Knowledge Hut

SEPTEMBER 28, 2023

The relevance of the AWS Cloud Practitioner Certification was something I couldn't ignore as I started on my path to gaining expertise in cloud computing. Anyone entering the cloud technology domain has to start with this fundamental credential. What is AWS Cloud Practitioner Certification?

AWS 52

AWS Certification Cloud Amazon Web Services 52

Cloudera

SEPTEMBER 28, 2021

The Ranger Authorization Service (RAZ) is a new service added to help provide fine-grained access control (FGAC) for cloud storage. We covered the value this new capability provides in a previous blog. Create an IDBroker mapping for each CDP user like Bob to a unique AWS IAM role.

Accessibility 62

Accessibility Accessible Cloud Cloud Storage 62

Cloudera

SEPTEMBER 1, 2020

YARN allows you to use various data processing engines for batch, interactive, and real-time stream processing of data stored in HDFS or cloud storage like S3 and ADLS. For the examples presented in this blog, we assume you have a CDP account already. aws s3 cp --recursive backups/ s3://dde-bucket/backups/.

Cloud Storage 71

Cloud Storage Unstructured Data AWS Analytics Application 71

Cloudera

DECEMBER 21, 2021

Early in the year we expanded our Public Cloud offering to Azure providing customers the flexibility to deploy on both AWS and Azure alleviating vendor lock-in. A new capability called Ranger Authorization Service (RAZ) provides fine grained authorization on cloud storage. Test Drive CDP Pubic Cloud.

Data Engineer 118

Data Engineer Data Engineering Engineering Data Pipeline 118

Cloudera

JUNE 25, 2021

Many Cloudera customers are making the transition from being completely on-prem to cloud by either backing up their data in the cloud, or running multi-functional analytics on CDP Public cloud in AWS or Azure. This blog post is not a substitute for that. For context, the setup used is as follows. Troubleshooting.

Cloud 73

Cloud Data Lake Cloud Storage Metadata 73

Cloudera

SEPTEMBER 15, 2020

introduces fine-grained authorization for access to Azure Data Lake Storage using Apache Ranger policies. Cloudera and Microsoft have been working together closely on this integration, which greatly simplifies the security administration of access to ADLS-Gen2 cloud storage. Cloudera Data Platform 7.2.1 What’s next?

Accessibility 79

Accessibility Accessible Cloud Cloud Storage 79

Cloudera

OCTOBER 26, 2020

*For clarity, the scope of the current certification covers CDP-Private Cloud Base. Certification of CDP-Private Cloud Experiences will be considered in the future. The certification process is designed to validate Cloudera products on a variety of Cloud, Storage & Compute Platforms.

Certification 87

Certification Cloud Kafka Cloud Storage 87

Knowledge Hut

FEBRUARY 29, 2024

File systems can store small datasets, while computer clusters or cloud storage keeps larger datasets. The designer must decide and understand the data storage, and inter-relation of data elements. It offers various blogs based on above mentioned technology in alphabetical order.

Data Science 98

Data Science Database Design Machine Learning Programming Language 98