Data Engineering Podcast

JUNE 16, 2024

In this episode Kevin Liu shares some of the interesting features that they have built by combining those technologies, as well as the challenges that they face in supporting the myriad workloads that are thrown at this layer of their data platform. Can you describe what role Trino and Iceberg play in Stripe's data architecture?

Data Lake 147

Data Lake High Quality Data Metadata Machine Learning 147

Cloudera

AUGUST 30, 2022

Data is the fuel that drives government, enables transparency, and powers citizen services. Citizens who have negative experiences with government services are less likely to use those services in the future. Modern data architectures. Deploying modern data architectures. Forrester ).

Data Architecture 79

Data Architecture Architecture Data Lake NoSQL 79

Precisely

JUNE 6, 2023

At Precisely’s Trust ’23 conference, Chief Operating Officer Eric Yau hosted an expert panel discussion on modern data architectures. The group kicked off the session by exchanging ideas about what it means to have a modern data architecture.

Data Architecture 52

Data Architecture Architecture Metadata Data Lake 52

phData: Data Engineering

APRIL 4, 2023

Today we want to introduce Fivetran’s support for Amazon S3 with Apache Iceberg, investigate some of the implications of this feature, and learn how it fits into the modern data architecture as a whole. Fivetran today announced support for Amazon Simple Storage Service (Amazon S3) with Apache Iceberg data lake format.

Data Lake 52

Data Lake Amazon Web Services Data Cleanse Data Warehouse 52

Data Engineering Podcast

JULY 22, 2019

Summary The current trend in data management is to centralize the responsibilities of storing and curating the organization’s information to a data engineering team. This organizational pattern is reinforced by the architectural pattern of data lakes as a solution for managing storage and access.

Data Lake 100

Data Lake Hadoop Data Architecture 100

Snowflake

JUNE 21, 2024

Snowflake is now making it even easier for customers to bring the platform’s usability, performance, governance and many workloads to more data with Iceberg tables (now generally available), unlocking full storage interoperability. Iceberg tables provide compute engine interoperability over a single copy of data.

Data Lake 114

Data Lake BI Business Intelligence Metadata 114

Monte Carlo

FEBRUARY 21, 2023

Today, as data sources become increasingly varied, data management becomes more complex, and agility and scalability become essential traits for data leaders, data fabric is quickly becoming the future of data architecture. If data fabric is the future, how can you get your organization up-to-speed?

Data Architecture 52

Data Architecture Architecture Metadata Unstructured Data 52

Monte Carlo

FEBRUARY 21, 2023

Today, as data sources become increasingly varied, data management becomes more complex, and agility and scalability become essential traits for data leaders, data fabric is quickly becoming the future of data architecture. If data fabric is the future, how can you get your organization up-to-speed?

Data Architecture 52

Data Architecture Architecture Metadata Unstructured Data 52

Striim

FEBRUARY 12, 2024

For instance, analytical data may uncover customer preferences or seasonal product trends, helping teams in strategic planning. This data is commonly gathered through an ETL process and stored in a central location, such as a data lake. Want to see how Striim’s Data Mesh and AI can benefit your organization?

Data Architecture 52

Data Architecture Architecture Data Data Lake 52

AltexSoft

AUGUST 29, 2023

In 2010, a transformative concept took root in the realm of data storage and analytics — a data lake. The term was coined by James Dixon , Back-End Java, Data, and Business Intelligence Engineer, and it started a new era in how organizations could store, manage, and analyze their data. What is a data lake?

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

Monte Carlo

JANUARY 24, 2024

Data is among your company’s most valuable commodities, but only if you know how to manage it. More data, more access to data, and more regulations mean data governance has become a higher-stakes game. Data Governance Trends The biggest data governance trend isn’t really a trend at all—rather, it’s a state of mind.

Data Governance 52

Data Governance Government Data Lake Data Architecture 52

Monte Carlo

APRIL 1, 2021

Over the past few years, data lakes have emerged as a must-have for the modern data stack. But while the technologies powering our access and analysis of data have matured, the mechanics behind understanding this data in a distributed environment have lagged behind. Data discovery tools and platforms can help.

Data Lake 52

Data Lake Unstructured Data Data Warehouse Metadata 52

Cloudera

OCTOBER 23, 2024

In August, we wrote about how in a future where distributed data architectures are inevitable, unifying and managing operational and business metadata is critical to successfully maximizing the value of data, analytics, and AI. They are free to choose the infrastructure best suited for each workload.

Metadata 83

Metadata BI Data Lake Business Intelligence 83

Striim

SEPTEMBER 11, 2024

Data pipelines are the backbone of your business’s data architecture. Implementing a robust and scalable pipeline ensures you can effectively manage, analyze, and organize your growing data. Understanding the essential components of data pipelines is crucial for designing efficient and effective data architectures.

Data Pipeline 98

Data Pipeline Designing Data Lake Data Warehouse 98

Cloudera

AUGUST 20, 2024

Whether it’s rapidly rising costs, an inefficient and outdated data infrastructure, or serious gaps in data governance, there are myriad reasons why organizations are struggling to move past adoption and achieve AI at scale in their enterprises. Ensuring data is trustworthy comes with its own complications.

Data Lake 78

Data Lake Government Data Governance Data Architecture 78

Towards Data Science

DECEMBER 4, 2023

Anyways, I wasn’t paying enough attention during university classes, and today I’ll walk you through data layers using —  guess what  —  an example. Business Scenario & Data Architecture Imagine this: next year, a new team on the grid, Red Thunder Racing, will call us (yes, me and you) to set up their new data infrastructure.

Data Engineer 82

Data Engineer Data Engineering Engineering Data Lake 82

Monte Carlo

MAY 14, 2024

As organizations seek greater value from their data, data architectures are evolving to meet the demand — and table formats are no exception. This development was crucial for enabling both batch and streaming data workflows in dynamic environments, ensuring consistency and durability in big data processing.

Data Lake 52

Data Lake Metadata Hadoop Data Governance 52

Cloudera

NOVEMBER 21, 2024

Powered by integrated platform services, Cloudera Data Engineering ensures data governance with robust data handling and automated lifecycle lineage tracking. Simplified Workflows and Iterative Collaborations : With Apache Airflow, Cloudera Data Engineering provides API integrations for external data tools like dbt.

Pipeline-centric 52

Pipeline-centric Data Engineer Data Engineering Engineering 52

Cloudera

MAY 9, 2023

Cloudera Data Platform (CDP) will enable SoftBank to increase resources flexibly as needed and adjust resources to meet business needs. In addition, it has functions to review and update user access controls regularly as part of data governance.

Data Security 90

Data Security Telecommunication Data Lake Data Governance 90

Precisely

JULY 18, 2024

Key Takeaways Data Fabric is a modern data architecture that facilitates seamless data access, sharing, and management across an organization. Data management recommendations and data products emerge dynamically from the fabric through automation, activation, and AI/ML analysis of metadata.

Data Management 52

Data Management Management Metadata Database-centric 52

Cloudera

NOVEMBER 17, 2022

With Cloudera Data Platform, we aim to unlock value faster and offer consistent data security and governance to meet this goal. Aqeel Ahmed Jatoi, Lead – Architecture, Governance and Control, Habib Bank Limited. Future-proof data capabilities. See other customers’ success here .

Banking 85

Banking Management Data Lake Professional Services 85

Cloudera

JUNE 30, 2022

With Cloudera’s vision of hybrid data , enterprises adopting an open data lakehouse can easily get application interoperability and portability to and from on premises environments and any public cloud without worrying about data scaling. Why integrate Apache Iceberg with Cloudera Data Platform?

Data Lake 88

Data Lake Business Intelligence Metadata Data Warehouse 88

Data Engineering Weekly

MARCH 3, 2024

The migration enhanced data quality, lineage visibility, performance improvements, cost reductions, and better reliability and scalability, setting a robust foundation for future expansions and onboarding. This approach helps maintain accuracy, relevance, and compliance in generative AI applications.

Data Engineering 67

Data Engineering Data Engineer Engineering Pipeline-centric 67

Cloudera

JUNE 6, 2024

To keep pace as banking becomes increasingly digitized in Southeast Asia, OCBC was looking to utilize AI/ML to make more data-driven decisions to improve customer experience and mitigate risks. While these are great proof points to demonstrate how business value can be driven by AI/ML, this was only made possible with trusted data.

Banking 82

Banking Data Lake Finance Data Governance 82

Knowledge Hut

MARCH 28, 2024

Role Level Intermediate Responsibilities Design and develop data pipelines to ingest, process, and transform data. Implemented and managed data storage solutions using Azure services like Azure SQL Database , Azure Data Lake Storage, and Azure Cosmos DB. GDPR, HIPAA), and industry standards.

Data Engineer 52

Data Engineer Data Engineering Engineering Data Warehouse 52

Cloudera

JANUARY 30, 2023

One estimate states that “ on average, people will produce 463 exabytes of data per day by 2025.” Now consider that the federal government has approximately 2.8 This dark data resides everywhere in the enterprise, siloed in multiple data repositories, from laptops and mobile devices to data lakes and applications.

Media 83

Media Government Data Lake Professional Services 83

Monte Carlo

AUGUST 30, 2024

It’s our goal at Monte Carlo to provide data observability and quality across the enterprise by monitoring every system vital in the delivery of data from source to consumption. We started with popular modern data warehouses and quickly expanded our support as data lakes became data lakehouses.

SQL 40

SQL Engineering Data Lake High Quality Data 40

Data Engineering Podcast

SEPTEMBER 23, 2018

Using the metaphor of a museum curator carefully managing the precious resources on display and in the vaults, he discusses the various layers of an enterprise data strategy. Can you walk through the stages of an ideal lifecycle for data within the context of an organizations uses for it?

Data Lake 100

Data Lake Data Warehouse Data Architecture Architecture 100

Monte Carlo

OCTOBER 31, 2024

Key Differences Between AI Data Engineers and Traditional Data Engineers While traditional data engineers and AI data engineers have similar responsibilities, they ultimately differ in where they focus their efforts. Data Storage Solutions As we all know, data can be stored in a variety of ways.

Data Engineer 52

Data Engineer Data Engineering Engineering Unstructured Data 52

Cloudera

MARCH 17, 2023

As the use of ChatGPT becomes more prevalent, I frequently encounter customers and data users citing ChatGPT’s responses in their discussions. I love the enthusiasm surrounding ChatGPT and the eagerness to learn about modern data architectures such as data lakehouses, data meshes, and data fabrics.

Education 64

Education Unstructured Data Data Lake Data Warehouse 64

Towards Data Science

FEBRUARY 15, 2023

Data by itself has no value, it needs to be organized, standardized, and clean. Governance is needed. In this context, data management in an organization is a key point for the success of its projects involving data. One of the main aspects of correct data management is the definition of a data architecture.

Data Lake 76

Data Lake Data Warehouse Hadoop Architecture 76

DataKitchen

AUGUST 3, 2021

The data mesh design pattern breaks giant, monolithic enterprise data architectures into subsystems or domains, each managed by a dedicated team. First-generation – expensive, proprietary enterprise data warehouse and business intelligence platforms maintained by a specialized team drowning in technical debt.

Pharmaceutical 98

Pharmaceutical Data Lake Data Architecture Architecture 98

Cloudera

SEPTEMBER 12, 2022

Combining and analyzing both structured and unstructured data is a whole new challenge to come to grips with, let alone doing so across different infrastructures. Both obstacles can be overcome using modern data architectures, specifically data fabric and data lakehouse. Unified data fabric. Better together.

Unstructured Data 75

Unstructured Data Data Lake Data Architecture Data 75

Cloudera

JUNE 13, 2024

After countless open-source innovations ushered in the Big Data era, including the first commercial distribution of HDFS (Apache Hadoop Distributed File System), commonly referred to as Hadoop, the two companies joined forces, giving birth to an entire ecosystem of technology and tech companies.

Hadoop 102

Hadoop Banking Big Data Insurance 102

Databand.ai

AUGUST 30, 2023

A DataOps architecture is the structural foundation that supports the implementation of DataOps principles within an organization. It encompasses the systems, tools, and processes that enable businesses to manage their data more efficiently and effectively. As a result, they can be slow, inefficient, and prone to errors.

Architecture 52

Architecture Data Ingestion Data Governance Data Cleanse 52

RandomTrees

SEPTEMBER 27, 2024

Data Factory, Data Activator, Power BI, Synapse Real-Time Analytics, Synapse Data Engineering, Synapse Data Science, and Synapse Data Warehouse are some of them. With One Lake serving as a primary multi-cloud repository, Fabric is designed with an open, lake-centric architecture.

Database-centric 52

Database-centric Pipeline-centric IT BI 52

Cloudera

SEPTEMBER 15, 2022

Mark: The first element in the process is the link between the source data and the entry point into the data platform. At Ramsey International (RI), we refer to that layer in the architecture as the foundation, but others call it a staging area, raw zone, or even a source data lake. What is a data fabric?

Data Lake 82

Data Lake Analytics Application Cloud Storage Architecture 82