Data Engineering Podcast

JUNE 16, 2024

Your host is Tobias Macey and today I'm interviewing Kevin Liu about his use of Trino and Iceberg for Stripe's data lakehouse Interview Introduction How did you get involved in the area of data management? Can you describe what role Trino and Iceberg play in Stripe's data architecture?

Data Lake 147

Data Lake High Quality Data Metadata Machine Learning 147

Snowflake

NOVEMBER 11, 2024

Data and AI architecture matter “Before focusing on AI/ML use cases such as hyper personalization and fraud prevention, it is important that the data and data architecture are organized and structured in a way which meets the requirements and standards of the local regulators around the world.

Banking 98

Banking Finance Retail Food 98

Precisely

JUNE 6, 2023

At Precisely’s Trust ’23 conference, Chief Operating Officer Eric Yau hosted an expert panel discussion on modern data architectures. The group kicked off the session by exchanging ideas about what it means to have a modern data architecture.

Data Architecture 52

Data Architecture Architecture Metadata Data Lake 52

Cloudera

OCTOBER 23, 2024

In August, we wrote about how in a future where distributed data architectures are inevitable, unifying and managing operational and business metadata is critical to successfully maximizing the value of data, analytics, and AI. Both platforms are free to try today.

Metadata 84

Metadata BI Data Lake Business Intelligence 84

Striim

MARCH 21, 2025

In-flight data processing reduces the time needed for data preparation as it delivers the data in a consumable form.

Data Warehouse 52

Data Warehouse MongoDB MySQL Hadoop 52

Monte Carlo

OCTOBER 31, 2024

Key Differences Between AI Data Engineers and Traditional Data Engineers While traditional data engineers and AI data engineers have similar responsibilities, they ultimately differ in where they focus their efforts.

Data Engineering 52

Data Engineering Data Engineer Engineering Unstructured Data 52

Knowledge Hut

MARCH 22, 2024

Did you know that Amazon Web Services (AWS) has a 33% market share in cloud computing? With this leadership status in the domain, the job roles associated with AWS have also gained traction. AWS solutions architect career opportunities have grown multiplefold. Businesses in every sector realize cloud adoption.

AWS 52

AWS Amazon Web Services Cloud Computing Computer Science 52

Data Engineering Weekly

DECEMBER 2, 2023

AWS S3 Express One Zone sparks some delight in the data infrastructure. In case you missed it, please read the AWS announcement here. S3 Express One Zone can improve data access speeds by 10x and reduce request costs by 50% compared to S3 Standard and scales to process millions of requests per minute.

IT 85

IT BI AWS Kafka 85

Edureka

SEPTEMBER 3, 2024

Indeed, one of the solutions that has evolved into a best practice for organizations actively seeking a way to update the organization’s data architecture is the AWS Database Migration Service, or AWS DMS abbreviation. If you are looking to deepen your knowledge, consider enrolling in our comprehensive AWS Course.

AWS 40

AWS Database MySQL PostgreSQL 40

Cloudera

NOVEMBER 25, 2021

At Cloudera, supporting our customers through their complete data journey also means providing access to game-changing technologies with trusted partners like Amazon Web Services (AWS). . Cloudera and AWS: Harnessing the Power of Data and Cloud . Common Use Cases for Cloud and Data Solutions .

Amazon Web Services 90

Amazon Web Services Cloud AWS Machine Learning 90

Cloudera

JUNE 7, 2022

Sure we can help you secure, manage, and analyze PetaBytes of structured and unstructured data. We do that on-prem with almost 1 ZB of data under management – nearly 20% of that global total. We can also do it with your preferred cloud – AWS, Azure or GCP. Don’t just take our word for it, look at the stats.

IT 112

IT Unstructured Data Data Architecture Government 112

Data Engineering Weekly

FEBRUARY 23, 2025

It allows different data platforms to access and share the same underlying data without copying, treating OTFs as a storage-layer abstraction. link] Sponsored: Webinar - The State of Airflow 2025 We asked 5,000+ data engineers how Airflow is shaping the modern DataOps landscape.

Data Engineering 62

Data Engineering Data Engineer Engineering Kafka 62

Data Engineering Podcast

SEPTEMBER 22, 2019

We have partnered with organizations such as O’Reilly Media, Dataversity, Corinium Global Intelligence, and Data Council. Upcoming events include the O’Reilly AI conference, the Strata Data conference, the combined events of the Data Architecture Summit and Graphorum, and Data Council in Barcelona.

AWS 100

AWS Google Cloud Cloud Storage Data Lake 100

Data Engineering Podcast

DECEMBER 8, 2019

Links SnowflakeDB Free Trial Stack Overflow Data Warehouse Oracle DB MPP == Massively Parallel Processing Shared Nothing Architecture Multi-Cluster Shared Data Architecture Google BigQuery AWS Redshift AWS Redshift Spectrum Presto Podcast Episode SnowflakeDB Semi-Structured Data Types Hive ACID == Atomicity, Consistency, Isolation, Durability 3rd Normal (..)

Data Warehouse 100

Data Warehouse Cloud AWS Relational Database 100

Data Engineering Podcast

OCTOBER 2, 2021

What are the driving factors for building a real-time data platform? How is Aerospike being incorporated in application and data architectures? How have the design and architecture changed or evolved since it was first created? What are the driving factors for building a real-time data platform?

Building 100

Building BI Data Architecture Architecture 100

AltexSoft

FEBRUARY 11, 2023

This specialist works closely with people on both business and IT sides of a company to understand the current needs of the stakeholders and help them unlock the full potential of data. To get a better understanding of a data architect’s role, let’s clear up what data architecture is.

Data Architect 87

Data Architect Certification Generalist Big Data 87

Edureka

APRIL 9, 2025

Companies with expertise in Microsoft Fabric are in high demand, including Microsoft, Accenture, AWS, and Deloitte Are you prepared to influence the data-driven future? Data Analytics: Capability to effectively use tools and techniques for analyzing data and drawing insights.

Engineering 40

Engineering Data Ingestion Data Lake Programming Language 40

Cloudera

NOVEMBER 7, 2023

Are you struggling to manage the ever-increasing volume and variety of data in today’s constantly evolving landscape of modern data architectures? S3 Any cloud-native S3 workload built to access S3 storage using either the AWS CLI, Boto S3 client, or other S3 client library can access Ozone via the S3 protocol.

Systems 105

Systems Hadoop Unstructured Data Media 105

Data Engineering Podcast

SEPTEMBER 1, 2021

How does this activity change the calculus for organizations who are deciding on a lake or warehouse for their data architecture? Contact Info LinkedIn @dborkar on Twitter Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

Data Lake 100

Data Lake Cloud AWS SQL 100

phData: Data Engineering

NOVEMBER 8, 2024

Additionally, the optimized query execution and data pruning features reduce the compute cost associated with querying large datasets. Scaling data infrastructure while maintaining efficiency is one of the primary challenges of modern data architecture.

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Towards Data Science

DECEMBER 4, 2023

Anyways, I wasn’t paying enough attention during university classes, and today I’ll walk you through data layers using —  guess what  —  an example. Business Scenario & Data Architecture Imagine this: next year, a new team on the grid, Red Thunder Racing, will call us (yes, me and you) to set up their new data infrastructure.

Data Engineering 98

Data Engineering Data Engineer Engineering Data Lake 98

Data Engineering Podcast

JULY 3, 2022

Your host is Tobias Macey and today I’m interviewing Colleen Tartow about her views on the forces shaping the current generation of data architectures Interview Introduction How did you get involved in the area of data management? What are some of the supporting services that are helpful in these undertakings?

Architecture 100

Architecture Metadata MongoDB Data Warehouse 100

Data Engineering Podcast

OCTOBER 14, 2019

They provide an AWS-native, serverless, data infrastructure that installs in your VPC. Datacoral helps data engineers build and manage the flow of data pipelines without having to manage any infrastructure. They provide an AWS-native, serverless, data infrastructure that installs in your VPC.

Data Warehouse 100

Data Warehouse PostgreSQL AWS Programming Language 100

Pinterest Engineering

JULY 25, 2023

By executing jobs and accessing data as individual users, we could narrowly grant each user access to different data resources without creating large supersets of shared permissions or fragmenting clusters. Access Tokens:  —  AWS STS , which grants access to AWS services such as S3. Tokens have built-in expiration dates.

Big Data 98

Big Data Accessibility Accessible Hadoop 98

Snowflake

DECEMBER 12, 2023

To give customers flexibility for how they fit Snowflake into their data architecture, Iceberg Tables can be configured to use either Snowflake or an external service such as AWS Glue as the table’s catalog to track metadata, with an easy, one-line SQL command to convert the table’s catalog to Snowflake in a metadata-only operation.

Metadata 118

Metadata Python Government AWS 118

phData: Data Engineering

FEBRUARY 21, 2025

The Cloud represents an iteration beyond the on-prem data warehouse, where computing resources are delivered over the Internet and are managed by a third-party provider. Examples include: Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP).

Cloud 52

Cloud Data Warehouse Amazon Web Services Data Ingestion 52

DataKitchen

NOVEMBER 5, 2024

This architecture is valuable for organizations dealing with large volumes of diverse data sources, where maintaining accuracy and accessibility at every stage is a priority. It sounds great, but how do you prove the data is correct at each layer? How do you ensure data quality in every layer ?

Architecture 78

Architecture Raw Data Pipeline-centric Data Ingestion 78

Cloudera

DECEMBER 21, 2021

Early in the year we expanded our Public Cloud offering to Azure providing customers the flexibility to deploy on both AWS and Azure alleviating vendor lock-in. We are excited to offer in Tech Preview this born-in-the-cloud table format that will help future proof data architectures at many of our public cloud customers.

Data Engineering 118

Data Engineering Data Engineer Engineering Data Pipeline 118

Snowflake

DECEMBER 20, 2023

The team is moving fast to make Snowpark Container Services available across all AWS regions, with support for other clouds to follow. More diverse instance types: We introduced high-memory instances and dynamic GPU allocation for intensive workloads. Ready to get started?

Scala 142

Scala Government Java Cloud 142

Snowflake

AUGUST 1, 2024

Today we are excited to announce the general availability (GA) of Snowpark Container Services in all AWS commercial regions and Azure Public Preview in all Azure commercial regions. Customers can get fast access to GPU infrastructure without needing to self-procure instances or make reservations with their public cloud provider.

Deep Learning 126

Deep Learning Government AWS Architecture 126

Data Engineering Podcast

OCTOBER 7, 2019

They provide an AWS-native, serverless, data infrastructure that installs in your VPC. Datacoral helps data engineers build and manage the flow of data pipelines without having to manage any infrastructure. They provide an AWS-native, serverless, data infrastructure that installs in your VPC.

Structured Data 100

Structured Data Cloud SQL Programming Language 100

Data Engineering Podcast

OCTOBER 21, 2019

This week’s episode is also sponsored by Datacoral, an AWS-native, serverless, data infrastructure that installs in your VPC. We have partnered with organizations such as O’Reilly Media, Dataversity, Corinium Global Intelligence, Alluxio, and Data Council.

Cloud 100

Cloud Hadoop Data Lake Programming Language 100

Data Engineering Podcast

OCTOBER 28, 2019

This week’s episode is also sponsored by Datacoral, an AWS-native, serverless, data infrastructure that installs in your VPC. We have partnered with organizations such as O’Reilly Media, Dataversity, Corinium Global Intelligence, Alluxio, and Data Council.

Building 100

Building Data Pipeline Programming Language Kafka 100

Data Engineering Podcast

NOVEMBER 4, 2019

This week’s episode is also sponsored by Datacoral, an AWS-native, serverless, data infrastructure that installs in your VPC. We have partnered with organizations such as O’Reilly Media, Dataversity, Corinium Global Intelligence, Alluxio, and Data Council.

Programming Language 100

Programming Language Kafka Data Engineering Data Engineer 100

Data Engineering Podcast

NOVEMBER 11, 2019

This week’s episode is also sponsored by Datacoral, an AWS-native, serverless, data infrastructure that installs in your VPC. We have partnered with organizations such as O’Reilly Media, Dataversity, Corinium Global Intelligence, Alluxio, and Data Council.

Designing 100

Designing Data Pipeline Programming Language Data 100

Data Engineering Podcast

NOVEMBER 18, 2019

This week’s episode is also sponsored by Datacoral, an AWS-native, serverless, data infrastructure that installs in your VPC. We have partnered with organizations such as O’Reilly Media, Dataversity, Corinium Global Intelligence, Alluxio, and Data Council.

Data Lake 100

Data Lake Scala Data Warehouse Hadoop 100

Cloudera

OCTOBER 18, 2022

Like all of our customers, Cloudera depends on the Cloudera Data Platform (CDP) to manage our day-to-day analytics and operational insights. Many aspects of our business live within this modern data architecture, providing all Clouderans the ability to ask, and answer, important questions for the business.

Cloud 89

Cloud IT Data Warehouse Professional Services 89