Christophe Blefari

JANUARY 20, 2024

The main difference between both is the fact that your computation resides in your warehouse with SQL rather than outside with a programming language loading data in memory. In this category I recommend also to have a look at data ingestion (Airbyte, Fivetran, etc.), workflows (Airflow, Prefect, Dagster, etc.)

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Cloudera

AUGUST 13, 2021

Every enterprise is trying to collect and analyze data to get better insights into their business. Whether it is consuming log files, sensor metrics, and other unstructured data, most enterprises manage and deliver data to the data lake and leverage various applications like ETL tools, search engines, and databases for analysis.

Data Pipeline 111

Data Pipeline Data Lake ETL Tools Unstructured Data 111

Cloudera

APRIL 22, 2021

Apache Ozone is one of the major innovations introduced in CDP, which provides the next generation storage architecture for Big Data applications, where data blocks are organized in storage containers for larger scale and to handle small objects. Cloudera will publish separate blog posts with results of performance benchmarks.

Pipeline-centric 102

Pipeline-centric Data Lake Hadoop Metadata 102

Cloudera

DECEMBER 14, 2021

Then, the company used Cloudera’s Data Platform as a foundation to build its own Network Real-time Analytics Platform (NRAP) and created the proper infrastructure to collect and analyze large-scale big data in real-time. . For this, the RTA transformed its data ingestion and management processes. .

Transportation 112

Transportation Telecommunication Banking Data Lake 112

Data Engineering Podcast

JUNE 19, 2022

RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. In fact, while only 3.5%

Metadata 130

Metadata MongoDB MySQL Scala 130

Christophe Blefari

MARCH 4, 2023

I'll try to think about it in the following weeks to understand where I go for the third year of the newsletter and the blog. This is funny to see their offering because they offer a "managed data warehouse storage", which means without the compute. So thank you for that. Stay tuned and let's jump to the content.

Machine Learning 130

Machine Learning AWS Data Data Lake 130

Striim

SEPTEMBER 11, 2024

Data Collection/Ingestion The next component in the data pipeline is the ingestion layer, which is responsible for collecting and bringing data into the pipeline. By efficiently handling data ingestion, this component sets the stage for effective data processing and analysis.

Data Pipeline 98

Data Pipeline Designing Data Lake Data Warehouse 98

Snowflake

OCTOBER 16, 2024

For organizations who are considering moving from a legacy data warehouse to Snowflake, are looking to learn more about how the AI Data Cloud can support legacy Hadoop use cases, or are struggling with a cloud data warehouse that just isn’t scaling anymore, it often helps to see how others have done it.

Digital Media 82

Digital Media Media Data Lake Data Warehouse 82

Rockset

OCTOBER 11, 2022

In this blog, we’ll compare and contrast how Elasticsearch and Rockset handle data ingestion as well as provide practical techniques for using these systems for real-time analytics. That’s because Elasticsearch can only write data to one index.

Data Ingestion 52

Data Ingestion Kafka Relational Database PostgreSQL 52

Cloudera

MARCH 31, 2021

Customers who have chosen Google Cloud as their cloud platform can now use CDP Public Cloud to create secure governed data lakes in their own cloud accounts and deliver security, compliance and metadata management across multiple compute clusters. Data Preparation (Apache Spark and Apache Hive) .

Google Cloud 95

Google Cloud Cloud Amazon Web Services Cloud Storage 95

Cloudera

JUNE 30, 2022

Over the past decade, Cloudera has enabled multi-function analytics on data lakes through the introduction of the Hive table format and Hive ACID. Companies, on the other hand, have continued to demand highly scalable and flexible analytic engines and services on the data lake, without vendor lock-in.

Data Lake 89

Data Lake Business Intelligence Metadata Data Warehouse 89

Data Engineering Podcast

JULY 10, 2022

RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. In fact, while only 3.5%

Data Engineering 100

Data Engineering Data Engineer Engineering MongoDB 100

Cloudera

AUGUST 26, 2020

All of these happen continuously and repetitively on a daily basis, amounting to petabytes worth of information and data. This requires massive amounts of data ingestion, messaging, and processing within a data-in-motion context. From a data ingestion standpoint, NiFi is designed for this purpose.

Banking 85

Banking Data Ingestion Kafka Data Lake 85

Cloudera

FEBRUARY 16, 2021

In the previous blog post , we looked at some of the application development concepts for the Cloudera Operational Database (COD). In this blog post, we’ll see how you can use other CDP services with COD. Integrated across the Enterprise Data Lifecycle . Cloudera Shared Data Experience (SDX) . Conclusion.

Database 84

Database Machine Learning Kafka Data Lake 84

Data Engineering Weekly

APRIL 21, 2024

The blog narrates how Chronon fits into Stripe’s online and offline requirements. RevenueCat writes about solving such challenges with the ingestion table & consolidation table pattern. Grab narrates how it integrated Debeizium, Kafka, and Apache Hudi to enable near real-time data analytics on the data lake.

Data Engineering 52

Data Engineering Data Engineer Engineering Medical 52

Cloudera

SEPTEMBER 9, 2022

analyst Sumit Pal, in “Exploring Lakehouse Architecture and Use Cases,” published January 11, 2022: “Data lakehouses integrate and unify the capabilities of data warehouses and data lakes, aiming to support AI, BI, ML, and data engineering on a single platform.” According to Gartner, Inc.

Architecture 103

Architecture Metadata Machine Learning Unstructured Data 103

Cloudera

FEBRUARY 4, 2019

Cloudera DataFlow (CDF) is a scalable, real-time streaming data platform that collects, curates, and analyzes data so customers gain key insights for immediate actionable intelligence. CDF, as an end-to-end streaming data platform, emerges as a clear solution for managing data from the edge all the way to the enterprise.

Data Ingestion 73

Data Ingestion Retail Kafka Data Lake 73

Snowflake

JUNE 4, 2024

This includes pipelines and transformations with Snowpark, Streams, Tasks and Dynamic Tables (public preview soon); extending AI and ML to Iceberg with Snowflake Cortex AI; performing storage maintenance with capabilities like automatic clustering and compaction; as well as securely collaborating on live data shares.

Government 85

Government Data Ingestion Data PostgreSQL 85

Cloudera

AUGUST 9, 2022

Standby systems can be designed to meet storage requirements during typical periods with burstable compute for failover scenarios using new features such as Data Lake Scaling. Automating the healing, recovery, scaling, and rebalancing of core data services such as our Operational Database. Cloudera Data Platform.

Data Lake 79

Data Lake Data Warehouse Architecture Data Ingestion 79

Edureka

FEBRUARY 9, 2023

As a certified Azure Data Engineer, you have the skills and expertise to design, implement and manage complex data storage and processing solutions on the Azure cloud platform. As the demand for data engineers grows, having a well-written resume that stands out from the crowd is critical.

Data Engineering 52

Data Engineering Data Engineer Engineering Amazon Web Services 52

Cloudera

DECEMBER 15, 2022

Since we announced the general availability of Apache Iceberg in Cloudera Data Platform (CDP), Cloudera customers, such as Teranet , have built open lakehouses to future-proof their data platforms for all their analytical workloads. Cloudera partners are also benefiting from Apache Iceberg in CDP.

Cloud 80

Cloud Metadata Data Warehouse Google Cloud 80

ProjectPro

FEBRUARY 8, 2023

Do ETL and data integration activities seem complex to you? Read this blog to understand everything about AWS Glue that makes it one of the most popular data integration solutions in the industry. Did you know the global big data market will likely reach $268.4 Businesses are leveraging big data now more than ever.

AWS 98

AWS Scala Metadata Data Lake 98

Confluent

SEPTEMBER 26, 2019

In the early days, many companies simply used Apache Kafka ® for data ingestion into Hadoop or another data lake. ® , Go, and Python SDKs where an application can use SQL to query raw data coming from Kafka through an API (but that is a topic for another blog).

Kafka 21

Kafka SQL BI Hadoop 21

Databand.ai

AUGUST 30, 2023

DataOps is a collaborative approach to data management that combines the agility of DevOps with the power of data analytics. It aims to streamline data ingestion, processing, and analytics by automating and integrating various data workflows.

Architecture 52

Architecture Data Ingestion Data Governance Data Cleanse 52

Ascend.io

DECEMBER 21, 2022

21, 2022 – Ascend.io , The Data Automation Cloud, today announced they have partnered with Snowflake , the Data Cloud company, to launch Free Ingest , a new feature that will reduce an enterprise’s data ingest cost and deliver data products up to 7x faster by ingesting data from all sources into the Snowflake Data Cloud quickly and easily.

Data Ingestion 52

Data Ingestion Google Cloud Data Lake Cloud 52

phData: Data Engineering

NOVEMBER 8, 2024

Open Table Format (OTF) architecture now provides a solution for efficient data storage, management, and processing while ensuring compatibility across different platforms. In this blog, we will discuss: What is the Open Table format (OTF)? It can also be integrated into major data platforms like Snowflake.

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Cloudera

FEBRUARY 16, 2022

In the case of CDP Public Cloud, this includes virtual networking constructs and the data lake as provided by a combination of a Cloudera Shared Data Experience (SDX) and the underlying cloud storage. Each project consists of a declarative series of steps or operations that define the data science workflow.

Machine Learning 86

Machine Learning Algorithm Government Metadata 86

Data Engineering Weekly

OCTOBER 30, 2022

I found the blog helpful in understanding the generative model’s historical development and the path forward. link] Sponsored- [New eBook] The Ultimate Data Observability Platform Evaluation Guide Considering investing in a data quality solution? to solve some of the challenges with self-serving.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Ingestion 52

Cloudera

JULY 18, 2018

Now organizations can reap all the benefits of having an enterprise data lake, in addition to an advanced analytics solution enabling them to put machine learning and AI into action at massive scale to improve health outcomes for individuals and entire populations alike.

Machine Learning 51

Machine Learning Healthcare Medical Electronics 51

Cloudera

AUGUST 21, 2020

Over time, additional use cases and functions expanded from original EDW and Data Lake related functions to support increasing demands from the business. More sources, data, and functionality were added to these platforms, expanding their value but adding to the complexity, such as: Streaming data ingestion. .

Hadoop 77

Hadoop Big Data Cloud Kafka 77

Rockset

JULY 28, 2022

As capable as it is, there are still instances where MongoDB alone can't satisfy all of the requirements for an application, so getting a copy of the data into another platform via a change data capture (CDC) solution is required. If this is not already part of your architecture then it may add another layer of complexity and cost.

MongoDB 52

MongoDB Kafka NoSQL Data Lake 52

Snowflake

AUGUST 16, 2023

Read our Summit recap blog for highlights across industries or watch Summit sessions now on-demand. Applications Snowflake Native App Framework now available in AWS – public preview Snowflake Native Apps are an entirely new way to put data to work. Learn more about ML-Powered Functions in our blog or in Snowflake documentation.

Transportation 52

Transportation Scala Kafka Data Lake 52

Mutt Data

JULY 16, 2023

The project applied best practices in different components of their data pipelines such as data ingestion, transformation, visualization, governance, observability, and infrastructure as code. Implementing a modern data lake house architecture with layers to separate different quality and granularities of data.

Data Lake 40

Data Lake Banking Data Ingestion Data Pipeline 40

Knowledge Hut

MARCH 28, 2024

This demonstrates the increasing need for Microsoft Certified Data Engineers. In this blog, I will explore Azure data engineer jobs and the top 10 job roles in this field where you can begin your career. Role Level Intermediate Responsibilities Design and develop data pipelines to ingest, process, and transform data.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Mutt Data

JULY 16, 2023

The project applied best practices in different components of their data pipelines such as data ingestion, transformation, visualization, governance, observability, and infrastructure as code. Implementing a modern data lake house architecture with layers to separate different quality and granularities of data.

Data Lake 40

Data Lake AWS Banking Data Ingestion 40

Snowflake

MARCH 30, 2023

Data Vault as a practice does not stipulate how you transform your data, only that you follow the same standards to populate business vault link and satellite tables as you would to populate raw vault link and satellite tables. Feature engineering: Data is transformed to support ML model training. ML workflow, ubr.to/3EJHjvm

Engineering 86

Engineering Raw Data Data Science Machine Learning 86

Cloudera

JANUARY 22, 2019

Forrester describes Big Data Fabric as, “A unified, trusted, and comprehensive view of business data produced by orchestrating data sources automatically, intelligently, and securely, then preparing and processing them in big data platforms such as Hadoop and Apache Spark, data lakes, in-memory, and NoSQL.”.

Big Data 45

Big Data NoSQL Hadoop Data Lake 45