Data Engineering Weekly

AUGUST 6, 2024

TL;DR Aswin and I are thrilled to announce the release of the first version of our comprehensive guide for evaluating Change Data Capture. CDC Evaluation Guide Google Sheet Link: [link] CDC Evaluation Guide Github Link: [link] Change Data Capture (CDC) is a powerful technology in data engineering that allows for continuously capturing changes (inserts, updates, and deletes) made to source systems.

Data Lake 125

Data Lake Data Warehouse Database Data Architecture 125

ArcGIS

AUGUST 6, 2024

Explore how moving from ArcMap to ArcGIS Pro and user types can make GIS workflows better, improve collaboration, and make big changes within your organization.

127

127

databricks

AUGUST 6, 2024

Fueled by the exponential growth in external data and AI for innovation, organizations across all industries are looking for effective ways to collaborate.

Data 120

Data 120

KDnuggets

AUGUST 6, 2024

Learn statistics through interactive books, code examples, cheat sheets, guides, and tools documentation.

Coding 119

Coding Data Science Data 119

Advertisement

Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?

Business Intelligence

Confluent

AUGUST 6, 2024

Using SQL-based BigQuery Continuous Queries w/Confluent lets you stream your warehouse data in real-time, sending it downstream for analytics use cases & more.

SQL 69

SQL Data IT Cloud 69

KDnuggets

AUGUST 6, 2024

Harness the simplicity and effectiveness of Hugging Face's Datasets library to efficiently load datasets, regardless of their source

Datasets 87

Datasets Data 87

KDnuggets

AUGUST 6, 2024

Masked arrays in NumPy are specialized array structures that allow you to handle missing or invalid data efficiently. They are particularly useful in scenarios where you must perform computations on datasets containing unreliable entries.

Datasets 72

Datasets Data Python 72

Monte Carlo

AUGUST 6, 2024

Data warehouses are the centralized repositories that store and manage data from various sources. They are integral to an organization’s data strategy, ensuring data accessibility, accuracy, and utility. However, beneath their surface lies a host of invisible risks embedded within the data warehouse layers. These “hidden threats” can silently undermine your data quality and reliability and often remain undetected until they trigger significant problems such as incorrect busines

Data Warehouse 52

Data Warehouse Raw Data BI Machine Learning 52

RandomTrees

AUGUST 6, 2024

Manufacturing has always been at the cutting edge of technology since it drives economic growth and societal changes. As a result, in recent times, the development of Generative Artificial Intelligence (GenAI) has opened up new possibilities for innovation in this critical area. GenAI is an artificial intelligence subset dedicated to generating new content and designs.

Manufacturing 52

Manufacturing Transportation Datasets High Quality Data 52

Hevo

AUGUST 6, 2024

In today’s dynamic business environment, companies often need to migrate their databases for many different reasons, ranging from scaling their operations to modernizing their technology stack or moving to the cloud to enjoy numerous benefits.

AWS 52

AWS Database Cloud Technology 52

Speaker: Jay Allardyce, Deepak Vittal, and Terrence Sheflin

As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.

Data

DataKitchen

AUGUST 6, 2024

Summary In this episode of the Data Engineering Podcast, host Tobias Macey welcomes back Chris Berg, CEO of DataKitchen, to discuss his ongoing mission to…

Data Engineering 52

Data Engineering Data Engineer Engineering Data 52

Hevo

AUGUST 6, 2024

Introduction In the data-driven modern world, organizations are quite dependent on ETL tools that help them integrate their data efficiently. These are the tools that base their guarantee of a smooth flow of data from sources to destination for supporting businesses in making decisions.

ETL Tools 52

ETL Tools Data 52

DataKitchen

AUGUST 6, 2024

DataKitchen’s Data Quality TestGen found 18 potential data quality issues in a few minutes (including install time) on data.boston.gov building permit data! Imagine a free tool that you can point at any dataset and find actionable data quality issues immediately! It sure beats having your data consumers tell you about problems they find when you are trying to enjoy your weekend.

Datasets 50

Datasets SQL Data Building 50

Hevo

AUGUST 6, 2024

We’ve all been there: your business is growing, and your data is expanding across various systems. You’re trying to keep everything in sync, but manual updates and batch processing don’t cut it anymore. You need a reliable way to keep your data up-to-date across all platforms.

AWS 52

AWS SQL Database Systems 52

Speaker: Nikhil Joshi, Founder & President of Snic Solutions

Is your manufacturing operation reaching its efficiency potential? A Manufacturing Execution System (MES) could be the game-changer, helping you reduce waste, cut costs, and lower your carbon footprint. Join Nikhil Joshi, Founder & President of Snic Solutions, in this value-packed webinar as he breaks down how MES can drive operational excellence and sustainability.

Manufacturing

DataKitchen

AUGUST 6, 2024

Chris Bergh joins me to chat about all things DataOps. We also discuss lean, removing waste from data processes and teams, and much more.

Data Engineering 52

Data Engineering Data Engineer Engineering Data 52

Cloudera

AUGUST 6, 2024

Over the past several years, data leaders asked many questions about where they should keep their data and what architecture they should implement to serve an incredible breadth of analytic use cases. Vendors with proprietary formats and query engines made their pitches, and over the years the market listened, and data leaders made their decisions. The most interesting thing about their choices is that, despite the millions of marketing dollars vendors spent trying to convince customers that the

Metadata 81

Metadata Government Datasets Architecture 81