Christophe Blefari

JANUARY 20, 2024

Data engineering inherits from years of data practices in US big companies. Hadoop initially led the way with Big Data and distributed computing on-premise to finally land on Modern Data Stack — in the cloud — with a data warehouse at the center. This is often linked to real-time.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Cloudera

APRIL 13, 2021

What’s the fastest and easiest path towards powerful cloud-native analytics that are secure and cost-efficient? In our humble opinion, we believe that’s Cloudera Data Platform (CDP). And sure, we’re a little biased—but only because we’ve seen firsthand how CDP helps our customers realize the full benefits of public cloud. .

Cloud 79

Cloud Pharmaceutical Medical Data Warehouse 79

Monte Carlo

MAY 28, 2024

A data ingestion architecture is the technical blueprint that ensures that every pulse of your organization’s data ecosystem brings critical information to where it’s needed most. Data Transformation : Clean, format, and convert extracted data to ensure consistency and usability for both batch and real-time processing.

Data Ingestion 52

Data Ingestion Architecture Designing Hadoop 52

Cloudera

DECEMBER 15, 2022

For example, Modak Nabu is helping their enterprise customers accelerate data ingestion, curation, and consumption at petabyte scale. Today, we are thrilled to share some new advancements in Cloudera’s integration of Apache Iceberg in CDP to help accelerate your multi-cloud open data lakehouse implementation.

Cloud 85

Cloud Metadata Google Cloud Data Warehouse 85

Hepta Analytics

FEBRUARY 14, 2022

DE Zoomcamp 2.2.1 – Introduction to Workflow Orchestration Following last weeks blog , we move to data ingestion. We already had a script that downloaded a csv file, processed the data and pushed the data to postgres database. This week, we got to think about our data ingestion design.

Data Ingestion 52

Data Ingestion Data Engineering Data Engineer Engineering 52

Cloudera

SEPTEMBER 17, 2018

These trends and demands lead to stress for existing data warehouse solutions – scale, efficiency, security integrations, IT budgets, ease of access. Cloudera recently launched Cloudera Data Warehouse, a modern data warehousing solution. No need to be a 100% expert on your data.

Data Warehouse 57

Data Warehouse SQL Portfolio MySQL 57

Striim

SEPTEMBER 11, 2024

Dmitriy Rudakov , Director of Solutions Architecture at Striim, describes it as “a program that moves data from source to destination and provides transformations when data is inflight.” Benjamin Kennedy, Cloud Solutions Architect at Striim, emphasizes the outcome-driven nature of data pipelines. “A

Data Pipeline 98

Data Pipeline Designing Data Lake Data Warehouse 98

Cloudera

AUGUST 31, 2021

Cloudera and Accenture demonstrate strength in their relationship with an accelerator called the Smart Data Transition Toolkit for migration of legacy data warehouses into Cloudera Data Platform. Accenture’s Smart Data Transition Toolkit . Are you looking for your data warehouse to support the hybrid multi-cloud?

Data Warehouse 87

Data Warehouse Database-centric Metadata Cloud 87

Snowflake

OCTOBER 16, 2024

Many of our customers — from Marriott to AT&T — start their journey with the Snowflake AI Data Cloud by migrating their data warehousing workloads to the platform. Today we’re focusing on customers who migrated from a legacy data warehouse to Snowflake and some of the benefits they saw.

Digital Media 84

Digital Media Media Data Lake Data Warehouse 84

phData: Data Engineering

MARCH 8, 2023

In today’s increasingly data-driven world, businesses need reliable and efficient ways to collect, analyze, and draw insights from their data. With so many data integration tools available in the market, it can be difficult to determine which one is the best fit for your organization. What is Fivetran Used For?

IT 96

IT Data Warehouse Data Ingestion Data Integration 96

Snowflake

SEPTEMBER 14, 2023

The Snowflake Data Cloud gives you the flexibility to build a modern architecture of choice to unlock value from your data. Snowflake was built from the ground up in the cloud. Snowflake’s platform provides industry-leading features that ensure the highest standards of governance for your account, users, and data.

Data Architecture 103

Data Architecture Architecture Hadoop Data Warehouse 103

Christophe Blefari

MARCH 4, 2023

How to run dbt with BigQuery in GitHub Actions — When you're starting with dbt you don't need any orchestrator or dbt Cloud, a CI/CD do it for sure. When you activate the query acceleration service when Snowflake thinks that a query can be accelerated it will launch more compute than actually specified by your warehouse.

Machine Learning 130

Machine Learning AWS Data Data Lake 130

Rockset

AUGUST 4, 2021

Organizations that depend on data for their success and survival need robust, scalable data architecture, typically employing a data warehouse for analytics needs. Snowflake is often their cloud-native data warehouse of choice. Data ingestion must be performant to handle large amounts of data.

Data Ingestion 40

Data Ingestion Cloud Storage Data Warehouse Architecture 40

Data Engineering Podcast

NOVEMBER 6, 2022

Datafold shows how a change in SQL code affects your data, both on a statistical level and down to individual rows and values before it gets merged to production. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. In fact, while only 3.5%

MongoDB 130

MongoDB MySQL Scala Machine Learning 130

Cloudera

JULY 21, 2022

Cloudera users can securely connect Rill to a source of event stream data, such as Cloudera DataFlow , model data into Rill’s cloud-based Druid service, and share live operational dashboards within minutes via Rill’s interactive metrics dashboard or any connected BI solution. Cloudera Data Warehouse). Apache Hive.

BI 92

BI Digital Media Data Warehouse Kafka 92

Data Engineering Podcast

NOVEMBER 16, 2020

At Isima they decided to reimagine the entire ecosystem from the ground up and built a single unified platform to allow end-to-end self service workflows from data ingestion through to analysis. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows.

Data Management 100

Data Management Management BI Business Intelligence 100

Cloudera

JUNE 30, 2022

No more lock-in, unnecessary data transformations, or data movement across tools and clouds just to extract insights out of the data. With Shared Data Experience (SDX) which is built in to CDP right from the beginning, customers benefit from a common metadata, security, and governance model across all their data. .

Data Lake 94

Data Lake Business Intelligence Metadata Data Warehouse 94

Cloudyard

MARCH 4, 2023

Read Time: 5 Minute, 16 Second As we know Snowflake has introduced latest badge “Data Cloud Deployment Framework” which helps to understand knowledge in designing, deploying, and managing the Snowflake landscape. Respective Cloud would consume/Store the data in bucket or containers.

Architecture 52

Architecture Cloud Metadata Data Ingestion 52

Cloudera

SEPTEMBER 9, 2022

analyst Sumit Pal, in “Exploring Lakehouse Architecture and Use Cases,” published January 11, 2022: “Data lakehouses integrate and unify the capabilities of data warehouses and data lakes, aiming to support AI, BI, ML, and data engineering on a single platform.” Iceberg handles massive data born in the cloud.

Architecture 108

Architecture Metadata Machine Learning Unstructured Data 108

Data Engineering Podcast

NOVEMBER 13, 2022

Datafold shows how a change in SQL code affects your data, both on a statistical level and down to individual rows and values before it gets merged to production. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. In fact, while only 3.5%

MongoDB 100

MongoDB Google Cloud MySQL Scala 100

Data Engineering Podcast

OCTOBER 30, 2022

Datafold shows how a change in SQL code affects your data, both on a statistical level and down to individual rows and values before it gets merged to production. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. In fact, while only 3.5%

Engineering 100

Engineering MongoDB MySQL Scala 100

Data Engineering Podcast

SEPTEMBER 25, 2022

Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. Data teams are increasingly under pressure to deliver. Data teams are increasingly under pressure to deliver.

Food 100

Food MongoDB MySQL Scala 100

Data Engineering Podcast

AUGUST 21, 2022

Select Star’s data discovery platform solves that out of the box, with an automated catalog that includes lineage from where the data originated, all the way to which dashboards rely on it and who is viewing them every day. In fact, while only 3.5% That’s where our friends at Ascend.io In fact, while only 3.5%

Lambda Architecture 100

Lambda Architecture MongoDB MySQL Scala 100

Data Engineering Podcast

JULY 3, 2022

In order to quickly identify if and how two data systems are out of sync Gleb Mezhanskiy and Simon Eskildsen partnered to create the open source data-diff utility. report having current investments in automation, 85% of data teams plan on investing in automation in the next 12 months. In fact, while only 3.5%

Data Integration 100

Data Integration MongoDB MySQL Scala 100

Data Engineering Podcast

OCTOBER 16, 2022

Summary The "data lakehouse" architecture balances the scalability and flexibility of data lakes with the ease of use and transaction support of data warehouses. Sign up free… or just get the free t-shirt for being a listener of the Data Engineering Podcast at dataengineeringpodcast.com/rudder.

Data Lake 100

Data Lake Food MongoDB MySQL 100

Data Engineering Podcast

AUGUST 6, 2022

Summary The optimal format for storage and retrieval of data is dependent on how it is going to be used. For analytical systems there are decades of investment in data warehouses and various modeling techniques. Data teams are increasingly under pressure to deliver. In fact, while only 3.5%

Machine Learning 100

Machine Learning Database MySQL PostgreSQL 100

Cloudera

AUGUST 26, 2020

All of these happen continuously and repetitively on a daily basis, amounting to petabytes worth of information and data. This requires massive amounts of data ingestion, messaging, and processing within a data-in-motion context. From a data ingestion standpoint, NiFi is designed for this purpose.

Banking 89

Banking Data Ingestion Kafka Data Lake 89

Cloudera

AUGUST 9, 2022

As customers import their mainframe and legacy data warehouse workloads, there is an expectation on the platform that it can meet, if not exceed, the resilience of the prior system and its associated dependencies. The goal is to minimize the impact to a customer’s data-driven decision making in the time of an operational crisis.

Data Lake 84

Data Lake Data Warehouse Architecture Data Ingestion 84

Data Engineering Podcast

AUGUST 13, 2022

Select Star’s data discovery platform solves that out of the box, with an automated catalog that includes lineage from where the data originated, all the way to which dashboards rely on it and who is viewing them every day. In fact, while only 3.5% That’s where our friends at Ascend.io In fact, while only 3.5%

Metadata 100

Metadata MongoDB MySQL Scala 100

Data Engineering Podcast

OCTOBER 2, 2022

Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. Data teams are increasingly under pressure to deliver. Data teams are increasingly under pressure to deliver.

IT 100

IT Food PostgreSQL MongoDB 100

Data Engineering Podcast

JUNE 5, 2022

Datafold shows how a change in SQL code affects your data, both on a statistical level and down to individual rows and values before it gets merged to production. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. In fact, while only 3.5%

Data Security 100

Data Security Metadata MongoDB MySQL 100

Data Engineering Podcast

AUGUST 28, 2022

Select Star’s data discovery platform solves that out of the box, with an automated catalog that includes lineage from where the data originated, all the way to which dashboards rely on it and who is viewing them every day. In fact, while only 3.5% That’s where our friends at Ascend.io In fact, while only 3.5%

Building 100

Building MongoDB MySQL Scala 100

Cloudera

AUGUST 31, 2020

Given the prohibitive cost of scaling it, in addition to the new business focus on data science and the need to leverage public cloud services to support future growth and capability roadmap, SMG decided to migrate from the legacy data warehouse to Cloudera’s solution using Hive LLAP. The case for a new Data Warehouse?

Management 67

Management Hadoop Data Warehouse Data Science 67

Cloudera

FEBRUARY 16, 2021

In the following sections, we see how the Cloudera Operational Database is integrated with other services within CDP that provide unified governance and security, data ingest capabilities, and expand compatibility with Cloudera Runtime components to cater to your specific use cases. . Integrated across the Enterprise Data Lifecycle .

Database 88

Database Machine Learning Kafka Aggregated Data 88

Data Engineering Podcast

AUGUST 13, 2022

Select Star’s data discovery platform solves that out of the box, with an automated catalog that includes lineage from where the data originated, all the way to which dashboards rely on it and who is viewing them every day. In fact, while only 3.5% That’s where our friends at Ascend.io In fact, while only 3.5%

Machine Learning 100

Machine Learning Pipeline-centric Database-centric MongoDB 100

Towards Data Science

JULY 24, 2023

A data engineering manager at a Fortune 500 company expressed the pain of on-prem limitations to me by saying: “Our analysts were unable to run the queries they wanted to run when they wanted to run them. Why are these things related, and more importantly, why should data leaders care? Again, that answer will depend on who you are.

Metadata 94

Metadata Data Warehouse Data Lake Data Science 94

Snowflake

APRIL 9, 2024

For a more in-depth exploration, plus advice from Snowflake’s Travis Henry, Director of Sales Development Ops and Enablement, and Ryan Huang, Senior Marketing Data Analyst, register for our Snowflake on Snowflake webinar on boosting market efficiency by leveraging data from Outreach. Each of these sources may store data differently.

BI 80

BI Data Ingestion Data Aggregated Data 80