Data Engineering Podcast

NOVEMBER 19, 2023

Summary The dbt project has become overwhelmingly popular across analytics and data engineering teams. Dustin Dorsey and Cameron Cyr co-authored a practical guide to building your dbt project. In this episode they share their hard-won wisdom about how to build and scale your dbt projects. What was your path to adoption of dbt?

Project 147

Project Data Lake SQL High Quality Data 147

KDnuggets

JANUARY 22, 2025

Are you looking for an easier way to manage files across different storage systems? fsspec is a Python library that simplifies file handling by providing a unified interface for file management.

Python 109

Python Management Project Systems 109

Seattle Data Guy

JULY 2, 2024

Data teams are expected to juggle a combination of ad-hoc requests, big bet projects, migrations, etc. All while keeping up with the latest changes in technology.

Data Management 130

Data Management Management Data Project 130

KDnuggets

AUGUST 5, 2024

Get noticed by recruiters and hiring managers by creating and documenting the following AI projects.

Portfolio 130

Portfolio Project Recruitment Management 130

The Pragmatic Engineer

SEPTEMBER 19, 2024

Without the backing of management, a large-scale rewrite is likely to fail. This project helped onboard me to the software, its structure, its build, and our issue tracking and version control workflows. My first project was supporting i18n (internationalization) in the app. Big rewrites need heavyweight support.

Recruitment 297

Recruitment Java Coding Project 297

KDnuggets

DECEMBER 4, 2024

Learn to build, run, and manage data engineering pipelines both locally and in the cloud using popular tools.

Data Engineer 150

Data Engineer Data Engineering Engineering Project 150

The Pragmatic Engineer

OCTOBER 10, 2024

Wordpress is the most popular content management system (CMS),  estimated  to power around 43% of all websites; a staggering number! Automattic generates most of its revenue by offering managed Wordpress hosting. WP Engine also sells managed Wordpress services, making it a direct competitor to Automattic.

Consulting 347

Consulting AWS Engineering Software Engineering 347

KDnuggets

SEPTEMBER 17, 2024

In this post, you'll learn what BigQuery is, understand its capabilities, and set up a project in Google Cloud which we will later use to practice using BigQuery for loading, querying, and analyzing data.

Project 122

Project Google Cloud Cloud Management 122

Start Data Engineering

SEPTEMBER 23, 2024

Steps to decide on a data project to build 2.1. Potential referral/hiring manager research 2.2.3. Introduction 2. Objective 2.2. Research 2.2.1. Job description 2.2.2. Company research 2.3. Data 2.3.1. Dataset Search 2.3.2. Generate fake data 2.4. Outcome 2.4.1. Visualization 2.5. Presentation 3. Conclusion 4. Read these 1.

Portfolio 130

Portfolio Project Datasets Data 130

Precisely

FEBRUARY 26, 2025

Lets take a closer look at these exciting innovations and explore how theyll help you tackle six top data management challenges. Automated metadata management – AI-generated catalog asset descriptions significantly reduce manual efforts and improve metadata quality – enabling teams to focus on more strategic tasks.

Data Integration 63

Data Integration Data Management Management Data Governance 63

Data Engineering Podcast

MAY 12, 2024

Once it is running, the next challenge is figuring out how to address release management for all of the different component parts. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management This episode is supported by Code Comments, an original podcast from Red Hat.

Management 147

Management Data Lake High Quality Data Machine Learning 147

The Pragmatic Engineer

OCTOBER 18, 2024

Corporate conflict recap Automattic is the creator of open source WordPress content management system (CMS), and WordPress powers an incredible 43% of webpages and 65% of CMSes. But leveraging a supposedly neutral platform (the WordPress plugin manager) should not be the way to win in business – at least not in open source.

Engineering 357

Engineering Government Project AWS 357

Data Engineering Podcast

MARCH 31, 2024

To bring observability to dbt projects the team at Elementary embedded themselves into the workflow. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Can you start by outlining what elements of observability are most relevant for dbt projects?

Project 130

Project Data Lake High Quality Data Data Workflow 130

KDnuggets

OCTOBER 17, 2024

This article will discuss how to set up and streamline data science projects using Monday.com for project management among teams.

Data Science 119

Data Science Project Data Management 119

The Pragmatic Engineer

SEPTEMBER 22, 2023

The npm or pnpm package managers do this additional check and so are slower, though arguably more reliable. It begins with a clean state, and can ship something that works for, say, 90% of existing Node projects, and break the remaining 10%. The project would simply need volunteer contributors, or sponsors who pay contributions.

Programming Language 316

Programming Language Project Coding Engineering 316

dbt Developer Hub

APRIL 20, 2025

In particular, we expect both Business Intelligence and Data Engineering will be driven by AI operating on top of the context defined in your dbt Projects. Weve known for a while that the combination of structured data from your dbt project + LLMs is a potent combo (particularly when using the dbt Semantic Layer).

Structured Data 98

Structured Data SQL BI Project 98

Knowledge Hut

SEPTEMBER 24, 2024

What is management ? What is the difference between governing body and management? What is the relevance of meetings in management? Does the management layer need to conduct so many meetings? I am sure most of us have attended or conducted meetings as a part of management governance.

Government 98

Government Management Project Certification 98

KDnuggets

NOVEMBER 30, 2023

The blog discusses five platforms designed for data scientists with specialized capabilities in managing large datasets, models, workflows, and collaboration beyond what GitHub offers.

Data Science 148

Data Science Project Datasets Data 148

RandomTrees

DECEMBER 4, 2024

As one of the most important sectors of the global economy, the food and beverage (F&B) industry works in highly volatile conditions and ensures its success by reducing waste and managing inventories. Managing production and consumption, meeting deadlines, cutting waste, and being environmentally friendly are always a challenge.

Food 52

Food Management Manufacturing Machine Learning 52

Netflix Tech

DECEMBER 19, 2024

Many of these projects are under constant development by dedicated teams with their own business goals and development best practices, such as the system that supports our content decision makers , or the system that ranks which language subtitles are most valuable for a specific piece ofcontent.

Machine Learning 82

Machine Learning Project Data Warehouse Coding 82

The Pragmatic Engineer

MARCH 12, 2024

In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. This is how Brooks put it: “Programming managers have long recognized wide productivity variations between good programmers and poor ones. (.) Progress does not. Two secretaries.

Engineering 329

Engineering Programming Language Hospitality Programming 329

The Pragmatic Engineer

OCTOBER 17, 2024

In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. Internal comms: Chat: Slack Coordination / project management: Linear 3. Building such a service would help sustain our open-source projects after the EU grant runs out. Source: Spare Cores 2.

Cloud 332

Cloud AWS Metadata Cloud Computing 332

Snowflake

APRIL 4, 2024

As companies collect more data, from disparate sources and in disparate formats, building and managing transformations has become exponentially more complex and time-consuming. This expands the number of users who can contribute to data projects and enhances collaboration. The Coalesce platform is uniquely built for Snowflake.

Cloud 131

Cloud Management Data Pipeline SQL 131

Analytics Vidhya

MARCH 7, 2023

Introduction Apache Cassandra is a NoSQL database management system that is open-source and distributed. Facebook created Cassandra, which ultimately became an Apache Software Foundation project. It is well-known for its rapid write […] The post Top 6 Cassandra Interview Questions appeared first on Analytics Vidhya.

NoSQL 259

NoSQL Database Project Systems 259

Analytics Vidhya

FEBRUARY 6, 2023

Introduction While working with multiple projects, there are chances of issues with versions of packages in python; for example, a project needs a new version of a package, and another requires a different version. Sometimes the python version itself changes from project to project.

Python 225

Python Project Management Data Engineer 225

The Pragmatic Engineer

JUNE 29, 2023

In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. They’ll need to familiarize themselves with Squarespace’s domain management capabilities. It’s been a domain registrar since 2020, and already manages 1 million domain names.

AWS 65

AWS Software Engineering Software Engineer Engineering 65

The Pragmatic Engineer

AUGUST 22, 2023

In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. Tommy has built his own video games, consulted on a wide variety of game projects, and for a decade has taught game development at various universities. To get the full issues, twice a week, subscribe here.

Software Engineer 235

Software Engineer Software Engineering Consulting Entertainment 235

The Pragmatic Engineer

NOVEMBER 21, 2023

In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. After Zynga, he rejoined Amazon, and was the General Manager (GM) for Compute services at AWS, and later chief of staff, and advisor to AWS executives like Charlie Bell and Andy Jassy (Amazon’s current CEO.)

Engineering 345

Engineering Bytes Cloud Computing AWS 345

Data Engineering Podcast

APRIL 23, 2023

In this episode DeVaris Brown discusses the types of applications that are possible when teams don't have to manage the complex infrastructure necessary to support continuous data flows. If you've learned something or tried out a project from the show then tell us about it! Who are the target customers for Meroxa?

Data Lake 277

Data Lake Kafka Machine Learning Data Warehouse 277

The Pragmatic Engineer

AUGUST 1, 2023

In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. He said: “The Senior Engineer role at Wise is the equivalent to an Engineering Manager 1 on the IC track. The SDE3 level expects leadership on projects in which this engineer is involved.

Software Engineer 246

Software Engineer Software Engineering Engineering Designing 246

Data Engineering Podcast

FEBRUARY 18, 2024

Multiple open source projects and vendors have been working together to make this vision a reality. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines.

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262