Seattle Data Guy

AUGUST 9, 2023

Learning about how to data models from basic star schemas on the internet is like learning data science using the IRIS data set. Data modeling in real life requires you fully understand the data sources and your business use cases.… It works great as a toy example. But it doesn’t match real life at all.

Data Science 246

Data Science Data Consulting IT 246

Seattle Data Guy

NOVEMBER 12, 2024

A key responsibility for any data team is to understand the core metrics driving their business. Many companies, therefore, break down these top-line metrics into more… Read more The post How Data Teams Drive Business Success by Understanding Core Metrics appeared first on Seattle Data Guy.

Data 130

Data Data Science 130

Seattle Data Guy

NOVEMBER 13, 2024

Much of the data we have used for analysis in traditional enterprises has been structured data. However, much of the data that is being created and will be created comes in some form of unstructured format. However, the digital era… Read more The post What is Unstructured Data?

Unstructured Data 130

Unstructured Data Process Structured Data Data 130

Seattle Data Guy

FEBRUARY 10, 2024

Photo by Tiger Lily Data warehouses and data lakes play a crucial role for many businesses. It gives businesses access to the data from all of their various systems. As well as often integrating data so that end-users can answer business critical questions.

Data Lake 162

Data Lake Data Warehouse Data Accessible 162

Analytics Vidhya

AUGUST 7, 2024

Introduction Apache Airflow is a crucial component in data orchestration and is known for its capability to handle intricate workflows and automate data pipelines. Many organizations have chosen it due to its flexibility and strong scheduling capabilities.

Data Pipeline 187

Data Pipeline Data Process Data Workflow 187

Confessions of a Data Guy

MAY 30, 2024

Of all the duties that Data Engineers take on during the regular humdrum of business and work, it’s usually filled with the same old, same old. Build new pipeline, update pipeline, new data model, fix bug, etc, etc. It’s never-ending.

Building 184

Building Data Warehouse Data Data Engineering 184

Analytics Vidhya

APRIL 3, 2023

Introduction Companies can access a large pool of data in the modern business environment, and using this data in real-time may produce insightful results that can spur corporate success. Real-time dashboards such as GCP provide strong data visualization and actionable information for decision-makers.

Data Engineering 251

Data Engineering Data Engineer Engineering Data 251

Start Data Engineering

APRIL 22, 2024

Most data projects use Docker to set up the data infra locally (and often in production). Communicate between containers and local OS 2.2.2. Start containers with docker CLI or compose 3. Conclusion 1. Introduction Docker can be overwhelming to start with.

Data Engineering 225

Data Engineering Data Engineer Engineering Data 225

Analytics Vidhya

NOVEMBER 8, 2023

In the data-driven world […] The post Monitoring Data Quality for Your Big Data Pipelines Made Easy appeared first on Analytics Vidhya. Determine success by the precision of your charts, the equipment’s dependability, and your crew’s expertise. A single mistake, glitch, or slip-up could endanger the trip.

Big Data 246

Big Data Data Pipeline Data IT 246

Start Data Engineering

SEPTEMBER 18, 2024

Parts of data engineering 3.1. Identify what tool to use to process data 3.3. Data flow architecture 3. Introduction 2. Requirements 3.1.1. Understand input datasets available 3.1.2. Define what the output dataset will look like 3.1.3. Define SLAs so stakeholders know what to expect 3.1.4.

Project 240

Project Building Datasets Architecture 240

Seattle Data Guy

JULY 2, 2024

Running a successful data team is hard. Data teams are expected to juggle a combination of ad-hoc requests, big bet projects, migrations, etc. All while keeping up with the latest changes in technology.

Data Management 130

Data Management Management Data Project 130

Christophe Blefari

OCTOBER 6, 2024

Still I hope this edition finds you well, it will be a mix of personal news, OpenAI saga and usual data engineering stuff that I enjoy reading. Forward Data Conference ✨ As a reminder, on November 25th I'm organising the Forward Data Conference. First things first, yes, I'm co-founding a company.

Data 130

Data SQL Hadoop Deep Learning 130

Seattle Data Guy

AUGUST 1, 2024

What would you do if you became the head or director of data for a 1,000-person company? … Read more The post How To Run A Data Team As A New Head Of Data appeared first on Seattle Data Guy. … Read more The post How To Run A Data Team As A New Head Of Data appeared first on Seattle Data Guy.

Data 130

Data Consulting Big Data Data Analytics 130

Confessions of a Data Guy

JUNE 20, 2024

After 10 years of Data Engineering work, I think it’s time to hang up the proverbial hat and ride off into the sunset, never to be seen again. Sometimes I wonder if I’ve learned anything […] The post What I’ve Learned After A Decade Of Data Engineering appeared first on Confessions of a Data Guy.

Data Engineering 223

Data Engineering Data Engineer Engineering Data 223

Precisely

OCTOBER 31, 2024

Key Takeaways: Data mesh is a decentralized approach to data management, designed to shift creation and ownership of data products to domain-specific teams. Data fabric is a unified approach to data management, creating a consistent way to manage, access, and share data across distributed environments.

Data Architecture 69

Data Architecture Architecture Metadata Government 69

KDnuggets

NOVEMBER 20, 2024

Can SQL be used for advanced data manipulation in data science? It sure can with these seven techniques.

Data Science 82

Data Science SQL Data IT 82

Cloudera

NOVEMBER 13, 2024

We are excited to announce the acquisition of Octopai , a leading data lineage and catalog platform that provides data discovery and governance for enterprises to enhance their data-driven decision making. This dampens confidence in the data and hampers access, in turn impacting the speed to launch new AI and analytic projects.

Metadata 82

Metadata Management Data Governance Government 82

databricks

NOVEMBER 18, 2024

While GenAI is the focus today, most enterprises have been working for a decade or longer to make data intelligence a reality within.

Data 52

Data 52

Data Engineering Weekly

JULY 21, 2024

Editor’s Note: A New Series on Data Engineering Tools Evaluation There are plenty of data tools and vendors in the industry. Data Engineering Weekly is launching a new series on software evaluation focused on data engineering to better guide data engineering leaders in evaluating data tools.

Data Engineering 117

Data Engineering Data Engineer Engineering Generalist 117

Precisely

NOVEMBER 13, 2024

Key Takeaways : MDM delivers a unified holistic view of your data across domains, so you can make faster, more accurate decisions. Today, you have more data than ever. But to be truly data-driven , you need to break down the data silos that hold you back. What is Master Data Management? What is Data Integrity?

Data Integration 64

Data Integration Data Management Management IT 64

Precisely

JULY 12, 2024

Data can be your organization’s most valuable asset, but only if it’s data you can trust. When companies work with data that is untrustworthy for any reason, it can result in incorrect insights, skewed analysis, and reckless recommendations to become data integrity vs data quality.

Data Integration 105

Data Integration Datasets Data Data Governance 105

Cloudyard

NOVEMBER 17, 2024

Read Time: 4 Minute, 21 Second Snowflake’s Data Metric Functions (DMF) are powerful tools designed to ensure data quality and governance. By enabling automated checks and validations, DMFs allow organizations to monitor their data continuously and enforce business rules. Null values and duplicate entries for customer IDs.

Datasets 52

Datasets Data Government Data Validation 52

Data Engineering Weekly

JULY 28, 2024

Switching from Apache Spark to Ray improves compact 12X larger datasets than Apache Spark, improves cost efficiency by 91%, and processes 13X more data per hour. Adopting LLM in SQL-centric workflow is particularly interesting since companies increasingly try text-2-SQL to boost data usage.

Data Engineering 109

Data Engineering Data Engineer Engineering Database-centric 109

Data Engineering Weekly

NOVEMBER 11, 2024

To identify express queries, we developed a method using historical data to predict whether an upcoming query is an express query. The blog is an excellent overview of building dynamic data pipelines with Airflow datasets and pub/sub. Uber writes about express queries , such as any Presto query, that can be finished within 2 minutes.

Data Engineering 57

Data Engineering Data Engineer Engineering Datasets 57

Snowflake

OCTOBER 22, 2024

Founder and CEO Chuck Frisbie about how synthetic data is the answer to balancing the need for data privacy with the need for data access, and some of the unexpected benefits of their Snowflake Native App. Well, we’re taking those products and making them portable, more intelligent and consumable in an enterprise data setting.

Data 80

Data Data Schemas Datasets Machine Learning 80

Data Engineering Weekly

JUNE 24, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. A few highlights from the report Unstructured data goes mainstream. AI-driven code development is going mainstream now.

Data Engineering 109

Data Engineering Data Engineer Engineering Google Cloud 109

Data Engineering Weekly

NOVEMBER 3, 2024

The challenges around memory, data size, and runtime are exciting to read. Sampling is an obvious strategy for data size, but the layered approach and dynamic inclusion of dependencies are some key techniques I learned with the case study. This count helps to ensure data consistency when deleting and compacting segments.

Data Engineering 62

Data Engineering Data Engineer Engineering Pipeline-centric 62

KDnuggets

NOVEMBER 8, 2024

From learning Python to creating analytical reports, learn about ten easy steps to become a data scientist.

Python 131

Python Data 131

Data Engineering Weekly

JUNE 30, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. Looming regulatory requirements, data quality, governance issues & model accuracy keep failing enterprises.

Data Engineering 102

Data Engineering Data Engineer Engineering Unstructured Data 102

Snowflake

MARCH 19, 2024

A company’s data strategy is always in motion. Some emerging approaches may be seen in our newly released Snowflake Data Trends 2024 , looking at how users in the Data Cloud are working with their data. For the report, we focused on two broad aspects of data strategy. The most marked finding was around governance.

Data Governance 120

Data Governance Government Unstructured Data Programming Language 120