Snowflake

NOVEMBER 2, 2023

Over the years, the technology landscape for data management has given rise to various architecture patterns, each thoughtfully designed to cater to specific use cases and requirements. Each of these architectures has its own unique strengths and tradeoffs. The schema of semi-structured data tends to evolve over time.

Data Lake 97

Data Lake Data Warehouse Cloud Unstructured Data 97

Cloudera

AUGUST 30, 2022

Modern data architectures. To eliminate or integrate these silos, the public sector needs to adopt robust data management solutions that support modern data architectures (MDAs). Towards Data Science ). Deploying modern data architectures. Forrester ).

Data Architecture 84

Data Architecture Architecture Data Lake NoSQL 84

Towards Data Science

DECEMBER 4, 2023

A Glossary with Use Cases for First-Timers in Data Engineering An happy Data Engineer at work Are you a data engineering rookie interested in knowing more about modern data infrastructures? In this guide Data Engineering meets Formula 1. I bet you are, this article is for you!

Data Engineering 81

Data Engineering Data Engineer Engineering Data Lake 81

Snowflake

NOVEMBER 11, 2024

Whether it’s unifying transactional and analytical data with Hybrid Tables, improving governance for an open lakehouse with Snowflake Open Catalog or enhancing threat detection and monitoring with Snowflake Horizon Catalog , Snowflake is reducing the number of moving parts to give customers a fully managed service that just works.

Data Architecture 85

Data Architecture Architecture Data Lake Kafka 85

Data Engineering Podcast

JUNE 17, 2023

In this episode Satish Jayanthi explores the benefits of incorporating column-aware tooling in the data modeling process. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake.

Data Lake 130

Data Lake Machine Learning Metadata Data Architecture 130

Snowflake

JUNE 21, 2024

Snowflake is now making it even easier for customers to bring the platform’s usability, performance, governance and many workloads to more data with Iceberg tables (now generally available), unlocking full storage interoperability. Iceberg tables provide compute engine interoperability over a single copy of data.

Data Lake 117

Data Lake BI Business Intelligence Metadata 117

Data Engineering Weekly

AUGUST 6, 2024

TL;DR Aswin and I are thrilled to announce the release of the first version of our comprehensive guide for evaluating Change Data Capture. Why CDC is More Relevant in Unified Data Architecture As we advance into the Gen AI era, Change Data Capture (CDC) systems are emerging as crucial components of the ever-evolving data architecture.

Data Lake 126

Data Lake Data Warehouse Database Data Architecture 126

AltexSoft

OCTOBER 30, 2021

If you’re an executive who has a hard time understanding the underlying processes of data science and get confused with terminology, keep reading. We will try to answer your questions and explain how two critical data jobs are different and where they overlap. Data science vs data engineering.

Data Engineering 98

Data Engineering Data Engineer Engineering Machine Learning 98

Data Engineering Weekly

MARCH 3, 2024

Editor’s Note: Chennai, India Meetup - March-08 Update We are thankful to Ideas2IT to host our first Data Hero’s meetup. There will be food, networking, and real-world talks around data engineering.

Data Engineering 69

Data Engineering Data Engineer Pipeline-centric Engineering 69

Data Engineering Podcast

SEPTEMBER 23, 2018

This includes modeling the lifecycle of your information as a pipeline from the raw, messy, loosely structured records in your data lake, through a series of transformations and ultimately to your data warehouse. Can you walk through the stages of an ideal lifecycle for data within the context of an organizations uses for it?

Data Lake 100

Data Lake Data Warehouse Data Architecture Architecture 100

Monte Carlo

APRIL 1, 2021

Over the past few years, data lakes have emerged as a must-have for the modern data stack. But while the technologies powering our access and analysis of data have matured, the mechanics behind understanding this data in a distributed environment have lagged behind.

Data Lake 52

Data Lake Unstructured Data Data Warehouse Metadata 52

Cloudera

JUNE 30, 2022

Today’s general availability announcement covers Iceberg running within key data services in the Cloudera Data Platform (CDP) — including Cloudera Data Warehousing ( CDW ), Cloudera Data Engineering ( CDE ), and Cloudera Machine Learning ( CML ). We selected change data capture as our first use case on Iceberg.

Data Lake 94

Data Lake Business Intelligence Metadata Data Warehouse 94

Monte Carlo

OCTOBER 31, 2024

One job that has become increasingly popular across enterprise data teams is the role of the AI data engineer. Demand for AI data engineers has grown rapidly in data-driven organizations. But what does an AI data engineer do? Table of Contents What Does an AI Data Engineer Do?

Data Engineering 52

Data Engineering Data Engineer Engineering Unstructured Data 52

Data Engineering Podcast

AUGUST 20, 2021

Summary Data lakes have been gaining popularity alongside an increase in their sophistication and usability. Despite improvements in performance and data architecture they still require significant knowledge and experience to deploy and manage. Can you describe what Cuelake is and the story behind it?

Building 100

Building Data Lake Data Warehouse SQL 100

Monte Carlo

FEBRUARY 21, 2023

Despite its prevalence, data can be messy, siloed, ungovernable, and inaccessible—especially to the non-technical employees who rely on it. Enter data fabric: a data management architecture designed to serve the needs of the business, not just those of data engineers.

Data Architecture 52

Data Architecture Architecture Metadata Unstructured Data 52

Monte Carlo

FEBRUARY 21, 2023

Despite its prevalence, data can be messy, siloed, ungovernable, and inaccessible—especially to the non-technical employees who rely on it. Enter data fabric: a data management architecture designed to serve the needs of the business, not just those of data engineers.

Data Architecture 52

Data Architecture Architecture Metadata Unstructured Data 52

Data Engineering Weekly

SEPTEMBER 18, 2024

The Rise of Data Observability Data observability has become increasingly critical as companies seek greater visibility into their data processes. This growing demand has found a natural synergy with the rise of the data lake. As a result, monitoring data in real time was often an afterthought.

Data Lake 98

Data Lake Data Pipeline Unstructured Data Datasets 98

Knowledge Hut

JULY 5, 2024

Large enterprises heavily rely on data for informed decision-making, and this reliance is where data engineers step in. Data engineers like myself play a pivotal role in assessing infrastructure and taking relevant actions. Looking ahead, the future of data engineering appears promising.

Data Engineering 52

Data Engineering Data Engineer Engineering Hadoop 52

Cloudera

JUNE 18, 2022

Cloudera customers run some of the biggest data lakes on earth. These lakes power mission critical large scale data analytics, business intelligence (BI), and machine learning use cases, including enterprise data warehouses. On data warehouses and data lakes.

Data Lake 98

Data Lake Data Warehouse BI SQL 98

DataKitchen

NOVEMBER 29, 2021

Data Gets Meshier. 2022 will bring further momentum behind modular enterprise architectures like data mesh. The data mesh addresses the problems characteristic of large, complex, monolithic data architectures by dividing the system into discrete domains managed by smaller, cross-functional teams.

Data Lake 98

Data Lake Manufacturing Architecture Data Architecture 98

DataKitchen

AUGUST 3, 2021

The data mesh design pattern breaks giant, monolithic enterprise data architectures into subsystems or domains, each managed by a dedicated team. First-generation – expensive, proprietary enterprise data warehouse and business intelligence platforms maintained by a specialized team drowning in technical debt.

Pharmaceutical 98

Pharmaceutical Data Lake Data Architecture Architecture 98

phData: Data Engineering

APRIL 4, 2023

Today we want to introduce Fivetran’s support for Amazon S3 with Apache Iceberg, investigate some of the implications of this feature, and learn how it fits into the modern data architecture as a whole. Fivetran today announced support for Amazon Simple Storage Service (Amazon S3) with Apache Iceberg data lake format.

Data Lake 52

Data Lake Amazon Web Services Data Cleanse Data Warehouse 52

Monte Carlo

NOVEMBER 22, 2024

Your data engineering pipeline started simple: a few CSV exports, some Python scripts, and manual updates every week. You’re left wondering if there’s a breaking point where your DIY data solution won’t cut it anymore—and honestly, you might be there already. It means you’re scaling!

Data Engineering 52

Data Engineering Data Engineer Building Engineering 52

Data Engineering Weekly

APRIL 28, 2024

Intuit: The Data Mesh Strategy Behind Intuit’s Global Financial Technology Platform The Data Product Builder platform is becoming increasingly important in enterprise data engineering. It offers more targeted and customized data asset building than the general-purpose data stack.

Data Engineering 52

Data Engineering Data Engineer Engineering Hospitality 52

Knowledge Hut

MARCH 28, 2024

Data engineering is one of them. According to AnalytixLabs , the data science market is expected to be worth USD 230.80 All these numbers point to one thing–increased job roles and careers, especially when we talk about data engineering jobs in Azure, which are on the rise every year. Let’s get started.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Data Engineering Podcast

NOVEMBER 18, 2019

Summary With the constant evolution of technology for data management it can seem impossible to make an informed decision about whether to build a data warehouse, or a data lake, or just leave your data wherever it currently rests. How does it influence the relevancy of data warehouses or data lakes?

Data Lake 100

Data Lake Scala Data Warehouse Hadoop 100

AltexSoft

AUGUST 29, 2023

In 2010, a transformative concept took root in the realm of data storage and analytics — a data lake. The term was coined by James Dixon , Back-End Java, Data, and Business Intelligence Engineer, and it started a new era in how organizations could store, manage, and analyze their data. What is a data lake?

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

Edureka

FEBRUARY 9, 2023

Azure Data Engineering is a rapidly growing field that involves designing, building, and maintaining data processing systems using Microsoft Azure technologies. Contents: What is the role of an Azure Data Engineer?

Data Engineering 52

Data Engineering Data Engineer Engineering Amazon Web Services 52

DataKitchen

NOVEMBER 4, 2021

Data organizations often have a mix of centralized and decentralized activity. DataOps concerns itself with the complex flow of data across teams, data centers and organizational boundaries. It expands beyond tools and data architecture and views the data organization from the perspective of its processes and workflows.

Process 98

Process Data Process Pharmaceutical Data Lake 98

Knowledge Hut

NOVEMBER 17, 2023

The demand for data-related professions, including data engineering, has indeed been on the rise due to the increasing importance of data-driven decision-making in various industries. Becoming an Azure Data Engineer in this data-centric landscape is a promising career choice.

Data Engineering 52

Data Engineering Data Engineer Engineering Scala 52

Towards Data Science

FEBRUARY 15, 2023

In this context, data management in an organization is a key point for the success of its projects involving data. One of the main aspects of correct data management is the definition of a data architecture. What is Delta Lake? The data became useless. The Lakehouse architecture was one of them.

Data Lake 74

Data Lake Data Warehouse Hadoop Architecture 74

Data Engineering Podcast

SEPTEMBER 18, 2019

This was an eye opening conversation about how stateful computation of data streams from edge devices can reduce cost and complexity as compared to batch oriented workflows. We have partnered with organizations such as O’Reilly Media, Dataversity, Corinium Global Intelligence, and Data Council.

Hadoop 100

Hadoop Data Lake BI Kafka 100

Data Engineering Podcast

OCTOBER 21, 2019

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Cloud 100

Cloud Hadoop Data Lake Programming Language 100

Data Engineering Podcast

SEPTEMBER 22, 2019

Summary Object storage is quickly becoming the unifying layer for data intensive applications and analytics. Modern, cloud oriented data warehouses and data lakes both rely on the durability and ease of use that it provides.

AWS 100

AWS Google Cloud Cloud Storage Data Lake 100

Knowledge Hut

NOVEMBER 17, 2023

Azure Data Engineers play an important role in building efficient, secure, and intelligent data solutions on Microsoft Azure's powerful platform. The position of Azure Data Engineers is becoming increasingly important as businesses attempt to use the power of data for strategic decision-making and innovation.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Silectis

JANUARY 3, 2021

If you’re new to data engineering or are a practitioner of a related field, such as data science, or business intelligence, we thought it might be helpful to have a handy list of commonly used terms available for you to get up to speed. Data Engineering Data engineering is a process by which data engineers make data useful.

Data Engineering 40

Data Engineering Data Engineer Engineering Hadoop 40

Data Engineering Weekly

JANUARY 15, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Sign up free to test out the tool today.

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52