Seattle Data Guy

FEBRUARY 10, 2024

Photo by Tiger Lily Data warehouses and data lakes play a crucial role for many businesses. It gives businesses access to the data from all of their various systems. As well as often integrating data so that end-users can answer business critical questions.

Data Lake 162

Data Lake Data Warehouse Data Accessibility 162

Cloudera

NOVEMBER 7, 2023

Are you struggling to manage the ever-increasing volume and variety of data in today’s constantly evolving landscape of modern data architectures? Apache Ozone is compatible with Amazon S3 and Hadoop FileSystem protocols and provides bucket layouts that are optimized for both Object Store and File system semantics.

Systems 107

Systems Hadoop Unstructured Data Media 107

The Pragmatic Engineer

NOVEMBER 21, 2023

If you had a continuous deployment system up and running around 2010, you were ahead of the pack: but today it’s considered strange if your team would not have this for things like web applications.  We dabbled in network engineering, database management, and system administration. and hand-rolled C -code.

Engineering 301

Engineering Bytes Cloud Computing AWS 301

Data Engineering Podcast

MAY 18, 2024

Summary The purpose of business intelligence systems is to allow anyone in the business to access and decode data to help them make informed decisions. The team at Zenlytic have leaned on the promise of large language models to build an AI agent that lets you converse with your data. Data lakes are notoriously complex.

Building 278

Building Data Lake High Quality Data Business Intelligence 278

Analytics Vidhya

FEBRUARY 6, 2023

Introduction HDFS (Hadoop Distributed File System) is not a traditional database but a distributed file system designed to store and process big data. It provides high-throughput access to data and is optimized for […] The post A Dive into the Basics of Big Data Storage with HDFS appeared first on Analytics Vidhya.

Data Storage 185

Data Storage Big Data Hadoop Datasets 185

Data Engineering Podcast

FEBRUARY 4, 2024

Summary Stream processing systems have long been built with a code-first design, adding SQL as a layer on top of the existing framework. In this episode Yingjun Wu explains how it is architected to power analytical workflows on continuous data flows, and the challenges of making it responsive and scalable.

SQL 173

SQL Data Lake High Quality Data Data Pipeline 173

Data Engineering Podcast

FEBRUARY 11, 2024

Summary Sharing data is a simple concept, but complicated to implement well. There are also numerous technical considerations to be made, particularly if the producer and consumer of the data aren't using the same platforms. Dagster offers a new approach to building and running data platforms and data pipelines.

Data Lake 147

Data Lake High Quality Data Government Data Pipeline 147

Data Engineering Podcast

APRIL 7, 2024

Summary Maintaining a single source of truth for your data is the biggest challenge in data engineering. Different roles and tasks in the business need their own ways to access and analyze the data in the organization. Dagster offers a new approach to building and running data platforms and data pipelines.

Data Lake 162

Data Lake High Quality Data BI Data Workflow 162

Data Engineering Podcast

JANUARY 21, 2024

A substantial amount of the data that is being managed in these systems is related to customers and their interactions with an organization. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Want to see Starburst in action?

Data Lake 147

Data Lake High Quality Data NoSQL Data Warehouse 147

Precisely

DECEMBER 16, 2024

Key Takeaways: Centralized visibility of data is key. Modern IT environments require comprehensive data for successful AIOps, that includes incorporating data from legacy systems like IBM i and IBM Z into ITOps platforms. Tool overload can lead to inefficiencies and data silos. Legacy systems operate in isolation.

Systems 59

Systems IT Machine Learning Insurance 59

Data Engineering Podcast

JUNE 30, 2024

Petr shares his journey from being an engineer to founding Synq, emphasizing the importance of treating data systems with the same rigor as engineering systems. He discusses the challenges and solutions in data reliability, including the need for transparency and ownership in data systems.

Pipeline-centric 147

Pipeline-centric Engineering Data Lake High Quality Data 147

KDnuggets

JUNE 23, 2022

Essential Linux commands to improve the data science workflow. It will give you the power to automate tasks, build pipelines, access file systems, and enhance development operations.

Data Science 149

Data Science Data Accessibility Accessible 149

Data Engineering Podcast

JANUARY 7, 2024

Summary Data processing technologies have dramatically improved in their sophistication and raw throughput. Unfortunately, the volumes of data that are being generated continue to double, requiring further advancements in the platform capabilities to keep up. What are the open questions today in technical scalability of data engines?

Data Process 147

Data Process Process Data Lake High Quality Data 147

Data Engineering Podcast

JUNE 16, 2024

Summary Stripe is a company that relies on data to power their products and business. In this episode Kevin Liu shares some of the interesting features that they have built by combining those technologies, as well as the challenges that they face in supporting the myriad workloads that are thrown at this layer of their data platform.

Data Lake 147

Data Lake High Quality Data Metadata Machine Learning 147

Data Engineering Podcast

JUNE 2, 2024

Summary Modern businesses aspire to be data driven, and technologists enjoy working through the challenge of building data systems to support that goal. Data governance is the binding force between these two parts of the organization. What are some of the misconceptions that you encounter about data governance?

Data Governance 147

Data Governance Government Data Lake High Quality Data 147

Data Engineering Podcast

MARCH 24, 2024

Summary A core differentiator of Dagster in the ecosystem of data orchestration is their focus on software defined assets as a means of building declarative workflows. Data lakes are notoriously complex. Your first 30 days are free! Want to see Starburst in action? Can you describe what the focus of Dagster+ is and the story behind it?

Data Lake 162

Data Lake High Quality Data Hadoop Data Pipeline 162

Data Engineering Podcast

MAY 12, 2024

Summary Building a data platform is a substrantial engineering endeavor. The services and systems need to be kept up to date, but so does the code that controls their behavior. As someone who listens to the Data Engineering Podcast, you know that the road from tool selection to production readiness is anything but smooth or straight.

Management 147

Management Data Lake High Quality Data Machine Learning 147

The Pragmatic Engineer

MARCH 12, 2024

They come up with test cases and data. Brooks discusses software in the context of producing operating systems, pre-internet. Building operating systems and embedded software remains challenging, but software updates over the internet are common and bug fixes can be delivered later. The tester.   The program clerk.

Engineering 273

Engineering Programming Language Hospitality Programming 273

Data Engineering Podcast

DECEMBER 24, 2023

Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. Data lakes are notoriously complex.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Data Engineering Weekly

AUGUST 6, 2024

TL;DR Aswin and I are thrilled to announce the release of the first version of our comprehensive guide for evaluating Change Data Capture. Why CDC is More Relevant in Unified Data Architecture As we advance into the Gen AI era, Change Data Capture (CDC) systems are emerging as crucial components of the ever-evolving data architecture.

Data Lake 125

Data Lake Data Warehouse Database Data Architecture 125

Data Engineering Podcast

FEBRUARY 11, 2023

Summary Data is a team sport, but it's often difficult for everyone on the team to participate. For a long time the mantra of data tools has been "by developers, for developers", which automatically excludes a large portion of the business members who play a crucial role in the success of any data project.

PostgreSQL 130

PostgreSQL SQL Machine Learning Data Warehouse 130

The Pragmatic Engineer

OCTOBER 17, 2024

The startup was able to start operations thanks to getting access to an EU grant called NGI Search grant. Storing data: data collected is stored to allow for historical comparisons. Visualizing the data: the frontend that allows querying of live and historic data. Each benchmarking task is evaluated sequentially.

Cloud 277

Cloud AWS Metadata Cloud Computing 277

Data Engineering Podcast

JANUARY 28, 2024

Summary Monitoring and auditing IT systems for security events requires the ability to quickly analyze massive volumes of unstructured log data. Cliff Crosland co-founded Scanner to provide fast querying of high scale log data for security auditing. SIEM) A query engine is useless without data to analyze.

Data Lake 147

Data Lake Building High Quality Data AWS 147

Data Engineering Weekly

JULY 21, 2024

Editor’s Note: A New Series on Data Engineering Tools Evaluation There are plenty of data tools and vendors in the industry. Data Engineering Weekly is launching a new series on software evaluation focused on data engineering to better guide data engineering leaders in evaluating data tools.

Data Engineering 117

Data Engineering Data Engineer Engineering Generalist 117

Snowflake

FEBRUARY 6, 2024

Maverick replies, “The data on the MIG is inaccurate.” While flying may be more automated now, the importance of accurate and diverse data for aviation safety remains — and is likely even more critical. In two recent airplane accidents, automated systems aboard a Boeing 737 MAX made decisions based on inaccurate data.

Unstructured Data 107

Unstructured Data Retail Data Manufacturing 107

The Pragmatic Engineer

APRIL 11, 2024

From Sella’s status page : “Following the installation of an update to the operating system and related firmware which led to an unstable situation. Still, I’m puzzled by how long the system has been down. If it was an update to Oracle, or to the operating system, then why not roll back the update?

Banking 220

Banking Utilities Database Engineering 220

Christophe Blefari

FEBRUARY 3, 2023

Delivering the data news ( credits ) Hey you, it's already February. The article shows our Netflix art creators are using past data to create new artworks. ebay, Variable Hub a data access layer for risk decisioning — Looks like a feature store but for risk topics. Every week same analysis for me.

BI 130

BI Google Cloud Machine Learning SQL 130

Christophe Blefari

SEPTEMBER 28, 2023

Make your data stack take-off ( credits ) Hello, another edition of Data News. This week, we're going to take a step back and look at the current state of data platforms. What are the current trends and why are people fighting around the concept of the modern data stack. Is the modern data stack dying?

Big Data 147

Big Data Cloud Storage Hadoop SQL 147

Snowflake

JANUARY 29, 2024

Imagine seamlessly integrating your Google Analytics data with Snowflake, allowing you to combine it effortlessly with other key sources like CRM, ERP, social media metrics, email campaign data, and whatever data sources compose the full scope of your data estate.

Raw Data 116

Raw Data Aggregated Data Cloud Data 116

Data Engineering Podcast

MARCH 17, 2024

Summary A significant portion of data workflows involve storing and processing information in database engines. In this episode Gleb Mezhanskiy, founder and CEO of Datafold, discusses the different error conditions and solutions that you need to know about to ensure the accuracy of your data. Data lakes are notoriously complex.

Database 147

Database Data Lake High Quality Data Data Workflow 147

Christophe Blefari

MARCH 31, 2023

Google Data Cloud & AI Summit Two days ago Google announced new things at their Data Cloud & AI Summit. Big changes—or should I say BigChanges—the flat-rate pricing will not be accessible starting on July 5 and will be replaced by a capacity pricing similar to what Snowflake is doing.

Bytes 130

Bytes Data Google Cloud Education 130

Christophe Blefari

APRIL 14, 2023

Anyway, here the weekly Data News, written faster than usual. This is just a normal evolution of deep learning systems. Hot takes on the Modern Data Stack — Matt gives 5 hot takes about the MDS. This time he writes about the new marketing approach of the modern data stack ecosystem.

Datasets 130

Datasets Data Deep Learning SQL 130

Data Engineering Podcast

MARCH 10, 2024

Summary Data lakehouse architectures are gaining popularity due to the flexibility and cost effectiveness that they offer. The link that bridges the gap between data lake and warehouse capabilities is the catalog. Data lakes are notoriously complex. Join us at the top event for the global data community, Data Council Austin.

Data Lake 147

Data Lake High Quality Data Architecture Data Pipeline 147

Snowflake

JANUARY 15, 2024

Leading companies around the world rely on Informatica data management solutions to manage and integrate data across various platforms from virtually any data source and on any cloud. Now, Informatica customers in the Snowflake ecosystem have an even easier way to integrate data to and from the Snowflake Data Cloud.

Data Integration 104

Data Integration BI Cloud Data 104

Data Engineering Podcast

APRIL 21, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. From a product perspective, what are the data challenges that are posed by email?

Data Lake 182

Data Lake High Quality Data Data Pipeline Machine Learning 182

Data Engineering Weekly

JUNE 16, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. Regulative requirements and privacy concerns are often a big hurdle to training context-rich data.

Data Engineering 104

Data Engineering Data Engineer Engineering Kafka 104

Snowflake

APRIL 1, 2024

Snowflake and Salesforce are happy to share that bidirectional data sharing between Snowflake, the Data Cloud company and Salesforce Data Cloud is now generally available. Enterprises generate valuable customer data inside Salesforce applications. It has opened up new frontiers of data collaboration.

Cloud 119

Cloud Retail Entertainment Media 119