Cloudera

MARCH 31, 2021

One of our customers, Commerzbank, has used the CDP Public Cloud trial to prove that they can combine both Google Cloud and CDP to accelerate their migration to Google Cloud without compromising data security or governance. . Data Preparation (Apache Spark and Apache Hive) .

Google Cloud 93

Google Cloud Cloud Amazon Web Services Cloud Storage 93

DareData

JANUARY 30, 2023

In this post, we'll discuss some key data engineering concepts that data scientists should be familiar with, in order to be more effective in their roles. These concepts include concepts like data pipelines, data storage and retrieval, data orchestrators or infrastructure-as-code.

Data Engineer 52

Data Engineer Data Engineering NoSQL Engineering 52

Ascend.io

FEBRUARY 23, 2024

If your core data systems are still running in a private data center or pushed to VMs in the cloud, you have some work to do. To take advantage of cloud-native services, some of your data must be replicated, copied, or otherwise made available to native cloud storage and databases.

Cloud Storage 52

Cloud Storage Data Ingestion Machine Learning Algorithm 52

Ascend.io

JUNE 6, 2023

In this blog post, we delve deep into the factors that contribute to how Ascend runs in the cloud, focusing on the primary areas that contribute to costs in this context: storage, compute, networking, and retries. But if reading is not your thing, dive right into the fascinating details of how we master cloud costs in the video below.

Cloud 52

Cloud Data Pipeline Data Ingestion Cloud Storage 52

phData: Data Engineering

AUGUST 4, 2023

Data storage is a vital aspect of any Snowflake Data Cloud database. Within Snowflake, data can either be stored locally or accessed from other cloud storage systems. What are the Different Storage Layers Available in Snowflake? They are flexible, secure, and provide exceptional performance.

Cloud Storage 52

Cloud Storage Google Cloud Amazon Web Services Data Storage 52

RandomTrees

SEPTEMBER 6, 2020

The architecture is three layered: Database Storage: Snowflake has a mechanism to reorganize the data into its internal optimized, compressed and columnar format and stores this optimized data in cloud storage. The data objects are accessible only through SQL query operations run using Snowflake.

Cloud Storage 52

Cloud Storage Data Ingestion Data Cleanse Data Warehouse 52

Towards Data Science

OCTOBER 9, 2024

Get started with Airbyte and Cloud Storage Coding the connectors yourself? Think very carefully Creating and maintaining a data platform is a hard challenge. Data connectors are an essential part of such a platform. Of course, how else are we going to get the data? So, do what is best for your application.

Coding 53

Coding Cloud Storage Data Data Ingestion 53

WeCloudData

OCTOBER 19, 2021

The client needed to build its own internal data pipeline with enough flexibility to meet the business requirements for a job market analysis platform & dashboard. The client intends to build on and improve this data pipeline by moving towards a more serverless architecture and adding DevOps tools & workflows.

Consulting 52

Consulting Raw Data Data Lake Data Pipeline 52

WeCloudData

OCTOBER 19, 2021

The client needed to build its own internal data pipeline with enough flexibility to meet the business requirements for a job market analysis platform & dashboard. The client intends to build on and improve this data pipeline by moving towards a more serverless architecture and adding DevOps tools & workflows.

Consulting 52

Consulting Raw Data Data Lake Data Pipeline 52

Knowledge Hut

JUNE 26, 2023

From exploratory data analysis (EDA) and data cleansing to data modeling and visualization, the greatest data engineering projects demonstrate the whole data process from start to finish. Data pipeline best practices should be shown in these initiatives.

Data Engineer 98

Data Engineer Data Engineering Coding Project 98

ProjectPro

AUGUST 24, 2021

Data Sourcing: Building pipelines to source data from different company data warehouses is fundamental to the responsibilities of a data engineer. So, work on projects that guide you on how to build end-to-end ETL/ELT data pipelines. Google BigQuery receives the structured data from workers.

Data Engineer 52

Data Engineer Data Engineering Coding Project 52

Knowledge Hut

APRIL 25, 2023

Data engineering is a field that requires a range of technical skills, including database management, data modeling, and programming. Data engineering tools can help automate many of these processes, allowing data engineers to focus on higher-level tasks like extracting insights and building data pipelines.

Data Engineer 52

Data Engineer Data Engineering Engineering Google Cloud 52

Knowledge Hut

SEPTEMBER 26, 2023

Born out of the minds behind Apache Spark, an open-source distributed computing framework, Databricks is designed to simplify and accelerate data processing, data engineering, machine learning, and collaborative analytics tasks. This flexibility allows organizations to ingest data from virtually anywhere.

Data Lake 52

Data Lake Database-centric Pipeline-centric Machine Learning 52

Rockset

MARCH 15, 2021

This includes making the data available for query as soon as it is ingested, creating proper indexes on the data so that the query latency is very low, and much more. Before it can be ingested, there’s usually a data pipeline for transforming incoming data. latency, indexing, etc.),

MongoDB 40

MongoDB Data Ingestion Analytics Application Kafka 40

Cloudera

OCTOBER 15, 2020

The scenario is the same as it was in the previous blog but the ingest pipeline differs. Spark as the ingest pipeline tool for Search (i.e. In this post, we demonstrated how Cloudera Data Platform components can collaborate with each other, while still being resource isolated and managed separately.

AWS 116

AWS Data Cloud Cloud Storage 116

Monte Carlo

FEBRUARY 15, 2023

Second, to reduce your time-to-detection you need to be end-to-end across your entire data system which may include warehouses or lakes from other vendors or other components of the modern data stack. Finally, where and how the data pipeline broke isn’t always obvious. They need to be transformed.

Data Warehouse 52

Data Warehouse Unstructured Data AWS Business Intelligence 52

Rockset

NOVEMBER 12, 2020

Elasticsearch is one tool to which reads can be offloaded, and, because both MongoDB and Elasticsearch are NoSQL in nature and offer similar document structure and data types, Elasticsearch can be a popular choice for this purpose. This blog post will examine the various tools that can be used to sync data between MongoDB and Elasticsearch.

MongoDB 40

MongoDB NoSQL Data Pipeline Data Storage 40

AltexSoft

JULY 29, 2022

To facilitate data ingestion, there are Apache Flume aggregating log data from multiple servers and Apache Sqoop designed to transport information between Hadoop and relational (SQL) databases. You are welcome to explore them yourself to find the right instruments for your requirements and for data engineering of any complexity.

Hadoop 59

Hadoop Big Data Google Cloud NoSQL 59

Confluent

OCTOBER 16, 2019

After scraping these manually, they’re produced directly into a Kafka topic: kafkacat -b localhost:9092 -t canx_reason_code -P -K: < Data wrangling with stream processing. These transformations make the data usable for both applications and analytics which subscribe to the Kafka topics populated. variation_status" : "LATE".

Kafka 19

Kafka Building Data Coding 19

ProjectPro

MAY 31, 2021

Below is a list of Big Data project ideas and an idea of the approach you could take to develop them; hoping that this could help you learn more about Big Data and even kick-start a career in Big Data. In that case, you must inform the warehouse team to check the stock availability and commit to fulfilling the order.

Big Data 52

Big Data Coding Project Hadoop 52

Towards Data Science

NOVEMBER 4, 2023

I’d like to discuss some popular Data engineering questions: Modern data engineering (DE). Does your DE work well enough to fuel advanced data pipelines and Business intelligence (BI)? Are your data pipelines efficient? and parallel data processing. What is it? ML model training using Airflow.

Data Engineer 80

Data Engineer Data Engineering Engineering BI 80

Monte Carlo

AUGUST 19, 2023

We’ll cover: What is a data platform? Recently, there’s been a lot of discussion around whether to go with open source or closed source solutions (the dialogue between Snowflake and Databricks’ marketing teams really brings this to light) when it comes to building your data platform.

Building 52

Building BI Data Lake Data Governance 52

Monte Carlo

OCTOBER 5, 2023

Aligning with stakeholders: SLAs, SLIs, and SLOs Many organizations adopt an approach to setting data quality standards that will be familiar to stakeholders: SLAs (service-level agreements), SLIs (service-level indicators), and SLOs (service-level objectives). This helps drastically improve time-to-detection.

Government 52

Government Data Governance Data Datasets 52

AltexSoft

MARCH 30, 2023

Databricks architecture Databricks provides an ecosystem of tools and services covering the entire analytics process — from data ingestion to training and deploying machine learning models. Besides that, it’s fully compatible with various data ingestion and ETL tools. Let’s see what exactly Databricks has to offer.

Scala 64

Scala Data Lake BI Google Cloud 64