Pinterest Engineering

APRIL 7, 2025

In this blog post, well discuss our experiences in identifying the challenges associated with EC2 network throttling. In the remainder of this blog post, well share how we root cause and mitigate the aboveissues. In the database service, the application reads data (e.g. 4xl with up to 12.5

AWS 57

AWS Bytes Database Data Ingestion 57

Hepta Analytics

FEBRUARY 14, 2022

DE Zoomcamp 2.2.1 – Introduction to Workflow Orchestration Following last weeks blog , we move to data ingestion. We already had a script that downloaded a csv file, processed the data and pushed the data to postgres database. This week, we got to think about our data ingestion design.

Data Ingestion 52

Data Ingestion Data Engineering Data Engineer Engineering 52

Data Engineering Weekly

MARCH 23, 2025

[link] Georg Heiler: Upskilling data engineers What should I prefer for 2028, or how can I break into data engineering? I honestly don’t have a solid answer, but this blog is an excellent overview of upskilling. These are common LinkedIn requests. and then to Nuage 3.0, The article highlights Nuage 3.0's

Data Engineering 63

Data Engineering Data Engineer Engineering Data 63

Snowflake

JUNE 4, 2024

Iceberg tables (now generally available), when combined with the capabilities of the Snowflake platform, allow you to build various open architectures, including a data lakehouse and data mesh. Parquet Direct (private preview) allows you to use Iceberg without rewriting or duplicating Parquet files — even as new Parquet files arrive.

Government 112

Government Data Ingestion Data PostgreSQL 112

Rockset

MAY 3, 2023

lower latency than Elasticsearch for streaming data ingestion. In this blog, we’ll walk through the benchmark framework, configuration and results. We’ll also delve under the hood of the two databases to better understand why their performance differs when it comes to search and analytics on high-velocity data streams.

Data Ingestion 59

Data Ingestion Kafka Database Architecture 59

Cloudera

FEBRUARY 9, 2021

Cloudera Operational Database is now available in three different form-factors in Cloudera Data Platform (CDP). . If you are new to Cloudera Operational Database, see this blog post. In this blog post, we’ll look at both Apache HBase and Apache Phoenix concepts relevant to developing applications for Cloudera Operational Database.

Database 104

Database Java SQL Data Ingestion 104

Cloudera

AUGUST 26, 2021

In addition to big data workloads, Ozone is also fully integrated with authorization and data governance providers namely Apache Ranger & Apache Atlas in the CDP stack. While we walk through the steps one by one from data ingestion to analysis, we will also demonstrate how Ozone can serve as an ‘S3’ compatible object store.

Data Science 113

Data Science Cloud Hadoop Metadata 113

Striim

APRIL 18, 2025

Siloed storage : Critical business data is often locked away in disconnected databases, preventing a unified view. Delayed data ingestion : Batch processing delays insights, making real-time decision-making impossible. Heres why: AI Models Require Clean Data: Machine learning models are only as good as their training data.

High Quality Data 52

High Quality Data Business Intelligence Unstructured Data Data Pipeline 52

Snowflake

APRIL 18, 2024

This blog post explores how Snowflake can help with this challenge. Legacy SIEM cost factors to keep in mind Data ingestion: Traditional SIEMs often impose limits to data ingestion and data retention. Now there are a few ways to ingest data into Snowflake.

Data Lake 108

Data Lake Data Ingestion Bytes Cloud Computing 108

Striim

JANUARY 30, 2025

Todays organizations have access to more data than ever before, and consequently are faced with the challenge of determining how to transform this tremendous stream of real-time information into actionable insights. Safeguarding Personally Identifiable Information (PII) Oftentimes, crisis data includes sensitive details (e.g.,

Systems 52

Systems Management Hospitality Healthcare 52

Cloudera

FEBRUARY 8, 2021

This is part 2 in this blog series. You can read part 1, here: Digital Transformation is a Data Journey From Edge to Insight. The first blog introduced a mock connected vehicle manufacturing company, The Electric Car Company (ECC), to illustrate the manufacturing data path through the data lifecycle.

Data Pipeline 118

Data Pipeline Building Manufacturing Data Warehouse 118

Snowflake

APRIL 9, 2024

However, that data must be ingested into our Snowflake instance before it can be used to measure engagement or help SDR managers coach their reps — and the existing ingestion process had some pain points when it came to data transformation and API calls.

BI 103

BI Data Ingestion Data Aggregated Data 103

phData: Data Engineering

NOVEMBER 8, 2024

Open Table Format (OTF) architecture now provides a solution for efficient data storage, management, and processing while ensuring compatibility across different platforms. In this blog, we will discuss: What is the Open Table format (OTF)? They also support ACID transactions, ensuring data integrity and stored data reliability.

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Cloudera

NOVEMBER 19, 2020

Furthermore, the same tools that empower cybercrime can drive fraudulent use of public-sector data as well as fraudulent access to government systems. In financial services, another highly regulated, data-intensive industry, some 80 percent of industry experts say artificial intelligence is helping to reduce fraud.

Government 119

Government Machine Learning Algorithm Raw Data 119

Cloudera

JULY 9, 2021

If you’ve followed Cloudera for a while, you know we’ve long been singing the praises—or harping on the importance, depending on perspective—of a solid, standalone enterprise data strategy. The ways data strategies are implemented, the resulting outcomes and the lessons learned along the way provide important guardrails.

Government 114

Government Building Cloud Data Ingestion 114

Cloudera

AUGUST 17, 2022

?. What if you could access all your data and execute all your analytics in one workflow, quickly with only a small IT team? CDP One is a new service from Cloudera that is the first data lakehouse SaaS offering with cloud compute, cloud storage, machine learning (ML), streaming analytics, and enterprise grade security built-in.

Cloud Computing 116

Cloud Computing Cloud Storage Data Science Government 116

Rockset

OCTOBER 11, 2022

In this blog, we’ll compare and contrast how Elasticsearch and Rockset handle data ingestion as well as provide practical techniques for using these systems for real-time analytics. Or, they can periodically scan their relational database to get access to the most up to date records and reindex the data in Elasticsearch.

Data Ingestion 52

Data Ingestion Kafka Relational Database PostgreSQL 52

Cloudera

APRIL 9, 2021

This is part 4 in this blog series. This blog series follows the manufacturing and operations data lifecycle stages of an electric car manufacturer – typically experienced in large, data-driven manufacturing companies. The second blog dealt with creating and managing Data Enrichment pipelines.

Machine Learning 98

Machine Learning Manufacturing Data Collection Data Science 98

Cloudera

DECEMBER 14, 2021

During the economic upheaval brought about by the COVID pandemic, the government of Australia was working with the Commonwealth Bank of Australia (CBA) to access real-time data about everyday financial transactions to understand the economic and social impacts of the crisis. Commonwealth Bank of Australia.

Transportation 111

Transportation Telecommunication Banking Data Lake 111

Cloudera

OCTOBER 11, 2021

The platform converges data cataloging, data ingestion, data profiling, data tagging, data discovery, and data exploration into a unified platform, driven by metadata. Modak Nabu automates repetitive tasks in the data preparation process and thus accelerates the data preparation by 4x.

Data Engineering 95

Data Engineering Data Engineer Cloud Engineering 95

Cloudera

NOVEMBER 17, 2020

The data and the techniques presented in this prototype are still applicable as creating a PCA feature store is often part of the machine learning process. . The process followed in this prototype covers several steps that you should follow: Data Ingest – move the raw data to a more suitable storage location.

Deep Learning 115

Deep Learning Machine Learning Raw Data Data Ingestion 115

Snowflake

SEPTEMBER 13, 2023

Snowpark External Access – public preview External Access is in public preview on AWS regions. Users can now easily connect to external network locations, including external LLMs, from their Snowpark code while maintaining high security and governance over their data. Snowpark Python Updates Snowpark support for Python 3.9

Python 104

Python SQL Data Pipeline Data Ingestion 104

Cloudera

FEBRUARY 16, 2021

In the previous blog post , we looked at some of the application development concepts for the Cloudera Operational Database (COD). In this blog post, we’ll see how you can use other CDP services with COD. Integrated across the Enterprise Data Lifecycle . Cloudera Shared Data Experience (SDX) . Conclusion.

Database 83

Database Machine Learning Kafka Aggregated Data 83

Striim

SEPTEMBER 11, 2024

Understanding the essential components of data pipelines is crucial for designing efficient and effective data architectures. Data Collection/Ingestion The next component in the data pipeline is the ingestion layer, which is responsible for collecting and bringing data into the pipeline.

Data Pipeline 98

Data Pipeline Designing Data Lake Data Warehouse 98

Cloudera

AUGUST 13, 2021

By controlling the processing of the data flows, the cybersecurity team can now run Splunk searches 55% faster and get faster insight into potential fraud. Here is another example: the scoring system takes 70 minutes to identify a perpetrator getting access to the computer system and the time to detect the intrusion.

Data Pipeline 111

Data Pipeline Data Lake ETL Tools Unstructured Data 111

Cloudera

APRIL 20, 2021

It can access data from inside the business, like ERP and asset management, outside sources, like edge devices and external assets, and correlate them for real-time predictive maintenance. A modern streaming architecture consists of critical components that provide data ingestion, security and governance, and real-time analytics.

Kafka 95

Kafka Hospitality Retail Data Ingestion 95

Cloudera

SEPTEMBER 9, 2022

The promise of a modern data lakehouse architecture. Imagine having self-service access to all business data, anywhere it may be, and being able to explore it all at once. Imagine quickly answering burning business questions nearly instantly, without waiting for data to be found, shared, and ingested.

Architecture 102

Architecture Metadata Machine Learning Unstructured Data 102

Cloudera

JUNE 7, 2021

It calls out that Cloudera DataFlow “ includes streaming flow and streaming data processing unified with Cloudera Data Platform ”. Hundreds of customers across multiple industry verticals are leveraging Cloudera DataFlow today for various streaming use cases like Clickstreams, log ingestion/analysis, social stream analysis, etc.

Kafka 103

Kafka Data Ingestion Cloud Architecture 103

Cloudera

FEBRUARY 15, 2024

Some of the key benefits of a modern data architecture for regulatory compliance include: Enhanced data governance and compliance: Modern data architecture incorporates data governance practices and security controls to ensure data privacy, regulatory compliance, and protection against unauthorized access or breaches.

Insurance 88

Insurance Banking Data Architecture Data Ingestion 88

Ascend.io

OCTOBER 28, 2024

Data transformation helps make sense of the chaos, acting as the bridge between unprocessed data and actionable intelligence. You might even think of effective data transformation like a powerful magnet that draws the needle from the stack, leaving the hay behind.

Raw Data 52

Raw Data Datasets Aggregated Data Data Pipeline 52

Confluent

FEBRUARY 6, 2019

The blog posts How to Build and Deploy Scalable Machine Learning in Production with Apache Kafka and Using Apache Kafka to Drive Cutting-Edge Machine Learning describe the benefits of leveraging the Apache Kafka ® ecosystem as a central, scalable and mission-critical nervous system. For now, we’ll focus on Kafka.

Machine Learning 107

Machine Learning Python Kafka Java 107

Cloudera

MAY 30, 2024

By leveraging the flexibility of a data lake and the structured querying capabilities of a data warehouse, an open data lakehouse accommodates raw and processed data of various types, formats, and velocities. Learn more about the Cloudera Open Data Lakehouse here.

Data Lake 74

Data Lake Data Warehouse Programming Language Data Ingestion 74

Cloudera

MARCH 31, 2021

One of our customers, Commerzbank, has used the CDP Public Cloud trial to prove that they can combine both Google Cloud and CDP to accelerate their migration to Google Cloud without compromising data security or governance. . Data Preparation (Apache Spark and Apache Hive) .

Google Cloud 94

Google Cloud Cloud Amazon Web Services Cloud Storage 94

Cloudera

DECEMBER 20, 2021

The right set of tools helps businesses utilize data to drive insights and value. But balancing a strong layer of security and governance with easy access to data for all users is no easy task. You can become a data hero too. Retrofitting existing solutions to ever-changing policy and security demands is one option.

Government 91

Government Data Security Banking Metadata 91

phData: Data Engineering

MARCH 8, 2023

Fivetran, a cloud-based automated data integration platform, has emerged as a leading choice among businesses looking for an easy and cost-effective way to unify their data from various sources. With over 160 data connectors available, Fivetran makes it easy to move data out of, into, and across any cloud data platform in the market.

IT 96

IT Data Warehouse Data Ingestion Data Integration 96

Cloudera

FEBRUARY 10, 2022

In this use case NiFi deployments on CDF-PC are the bridge between streaming data and services relying on data being available in ADLS Gen2. Data Ingest for Microsoft Sentinel . Figure 4: Moving data from network infrastructure devices to Microsoft Sentinel.

Cloud 116

Cloud Kafka AWS Data Ingestion 116

Netflix Tech

MARCH 25, 2019

We adopted the following mission statement to guide our investments: “Provide a complete and accurate data lineage system enabling decision-makers to win moments of truth.” As a result, a single consolidated and centralized source of truth does not exist that can be leveraged to derive data lineage truth. push or pull.

Building 90

Building Metadata Transportation Data Ingestion 90