Data Engineering Podcast

JUNE 11, 2023

Summary Data engineering is all about building workflows, pipelines, systems, and interfaces to provide stable and reliable data. Datafold has invested a lot of time into integrating with the workflow of dbt projects to add early verification that the changes you are making are correct. What are the parallels to that in data projects?

Project 130

Project Building Data Lake Machine Learning 130

Knowledge Hut

APRIL 12, 2024

Project management plays a significant role in the success of every organization. It ensures that the project is on track, aids in efficient management of resources, and also keeps the stakeholders know what is project and what's happening in it. What is Project Management Structures, and Why is it Important?

Project 98

Project Management Certification Designing 98

Data Engineering Podcast

SEPTEMBER 17, 2023

Summary A significant amount of time in data engineering is dedicated to building connections and semantic meaning around pieces of information. In this episode Brian Platz explains how JSON-LD can be used as a shared representation of linked data for building semantic data products. Hex brings everything together.

Building 189

Building SQL BI Python 189

Knowledge Hut

APRIL 30, 2024

Management is paramount to making any project successful, irrespective of the industry and the work scope. Moreover, there are varied aspects to manage to ensure successful project delivery. All the tasks in the project management process involve multiple steps and the use of various tools that help bring efficiency.

Project 98

Project Management Certification Programming 98

Data Engineering Podcast

OCTOBER 1, 2023

Now that AI has reached the level of sophistication seen in the various generative models it is being used to build new ETL workflows. In this episode Jay Mishra shares his experiences and insights building ETL pipelines with the help of generative AI. How can you get the best results for your use case?

Building 162

Building BI SQL Machine Learning 162

Knowledge Hut

APRIL 16, 2024

Over several decades, projects have been initiated or undertaken due to market demands, business needs, at the behest of customer request, technological advancements and to comply with regulatory requirements. We will look at some of the key project management methods followed in today’s world. style="height: 514px;"> 2.

Project 98

Project Management Programming Designing 98

Tweag

FEBRUARY 15, 2023

opam is a source-based package manager for OCaml. It is the de-facto standard for package management in the OCaml ecosystem. Like many other language-specific package managers (e.g. Provide those dependencies such that the build system can find them. Run the build system. cargo, cabal, etc.), The solution to this?

Project 145

Project Programming Language Building Systems 145

Knowledge Hut

APRIL 16, 2024

Over the years I have found that my most popular blog posts are those that speak to entry-level project managers. Project management is a vast practice area and for somebody who has only recently started managing projects , it can seem overwhelming.

Project 97

Project Management Certification Process 97

Data Engineering Podcast

OCTOBER 15, 2023

Summary Building streaming applications has gotten substantially easier over the past several years. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. How can you get the best results for your use case? Closing Announcements Thank you for listening!

Process 182

Process Building SQL BI 182

Knowledge Hut

MAY 6, 2024

Release Planning / Goal Plan & Prioritize your Backlog (Define User Stories like Taking a Course, Complete Application, Apply & Book for the exam) Set the release goal: “As a Project Management Professional, I want to Obtain X-Certification in order to gain more knowledge in Y-Industry.” Validate your Experience.

Certification 97

Certification Project Management Technology 97

Data Engineering Podcast

NOVEMBER 19, 2023

Summary The dbt project has become overwhelmingly popular across analytics and data engineering teams. Dustin Dorsey and Cameron Cyr co-authored a practical guide to building your dbt project. In this episode they share their hard-won wisdom about how to build and scale your dbt projects.

Project 147

Project Data Lake High Quality Data SQL 147

Data Engineering Podcast

FEBRUARY 26, 2023

Jean-Georges Perrin was tasked with designing a new data platform implementation at PayPal and wound up building a data mesh. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Are you tired of dealing with the headache that is the 'Modern Data Stack'? We feel your pain.

Building 147

Building Machine Learning Metadata Data Integration 147

Tweag

JULY 5, 2023

Buck2 is a from-scratch rewrite of Buck , a polyglot, monorepo build system that was developed and used at Meta (Facebook), and shares a few similarities with Bazel. As you may know, the Scalable Builds Group at Tweag has a strong interest in such scalable build systems. invoke build buck2 build //starlark-rust/starlark 6.

Systems 141

Systems Building Java Programming Language 141

Data Engineering Podcast

MAY 12, 2024

Summary Building a data platform is a substrantial engineering endeavor. Once it is running, the next challenge is figuring out how to address release management for all of the different component parts. If you've learned something or tried out a project from the show then tell us about it!

Management 147

Management Data Lake High Quality Data Machine Learning 147

Data Engineering Podcast

APRIL 28, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. If you've learned something or tried out a project from the show then tell us about it! Your first 30 days are free!

Building 147

Building Data Lake High Quality Data Machine Learning 147

KDnuggets

DECEMBER 4, 2024

Learn to build, run, and manage data engineering pipelines both locally and in the cloud using popular tools.

Data Engineering 150

Data Engineering Data Engineer Engineering Project 150

The Pragmatic Engineer

MARCH 12, 2024

In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. This is how Brooks put it: “Programming managers have long recognized wide productivity variations between good programmers and poor ones. (.) Progress does not. A most interesting addition! The tester.

Engineering 299

Engineering Programming Language Hospitality Programming 299

Data Engineering Podcast

MARCH 31, 2024

Identifying and accounting for those errors is a critical piece of building trust in the organization that your data is accurate and up to date. To bring observability to dbt projects the team at Elementary embedded themselves into the workflow. Dagster offers a new approach to building and running data platforms and data pipelines.

Project 130

Project Data Lake High Quality Data Data Workflow 130

Start Data Engineering

SEPTEMBER 23, 2024

Steps to decide on a data project to build 2.1. Potential referral/hiring manager research 2.2.3. Introduction 2. Objective 2.2. Research 2.2.1. Job description 2.2.2. Company research 2.3. Data 2.3.1. Dataset Search 2.3.2. Generate fake data 2.4. Outcome 2.4.1. Visualization 2.5. Presentation 3. Conclusion 4. Read these 1.

Portfolio 130

Portfolio Project Datasets Data 130

Data Engineering Podcast

MAY 5, 2024

In this episode he explains his approach to building AI in a more human-like fashion and the emphasis on learning rather than statistical prediction. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines.

Building 147

Building Data Lake High Quality Data Machine Learning 147

Data Engineering Podcast

MARCH 5, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Hey there podcast listener, are you tired of dealing with the headache that is the 'Modern Data Stack'? It's supposed to make building smarter, faster, and more flexible data infrastructures a breeze. We feel your pain.

Building 147

Building Database Design Machine Learning Metadata 147

The Pragmatic Engineer

SEPTEMBER 22, 2023

Bun was mostly built by Jared Sumner , a former Stripe engineer, and recipient of the Thiel Fellowship (a grant of $100,000 for young people to drop out of school and build things, founded by venture capitalist, Peter Thiel). The npm or pnpm package managers do this additional check and so are slower, though arguably more reliable.

Programming Language 281

Programming Language Project Coding Engineering 281

Data Engineering Podcast

APRIL 16, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. How have the recent breakthroughs in large language models (LLMs) improved your ability to build features in Zenlytic?

Business Intelligence 147

Business Intelligence Building Data Lake BI 147

Data Engineering Podcast

JANUARY 28, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Contact Info LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? What do you have planned for the future of Scanner?

Data Lake 147

Data Lake Building High Quality Data AWS 147

The Pragmatic Engineer

SEPTEMBER 19, 2024

Without the backing of management, a large-scale rewrite is likely to fail. This project helped onboard me to the software, its structure, its build, and our issue tracking and version control workflows. My first project was supporting i18n (internationalization) in the app. Big rewrites need heavyweight support.

Recruitment 260

Recruitment Java Coding Project 260

The Pragmatic Engineer

OCTOBER 10, 2024

Wordpress is the most popular content management system (CMS),  estimated  to power around 43% of all websites; a staggering number! Automattic generates most of its revenue by offering managed Wordpress hosting. WP Engine also sells managed Wordpress services, making it a direct competitor to Automattic.

Consulting 316

Consulting AWS Engineering Software Engineering 316

Data Engineering Podcast

JULY 23, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Building 130

Building Machine Learning SQL Python 130

Data Engineering Podcast

FEBRUARY 18, 2024

Multiple open source projects and vendors have been working together to make this vision a reality. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Your first 30 days are free!

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Engineering at Meta

DECEMBER 6, 2023

This is the most significant milestone yet for this project, which began in earnest after Mark Zuckerberg outlined his vision for it in 2019. Throughout the project, we have consulted with a diverse range of external parties to ensure that we’re making the right set of tradeoffs.

Building 145

Building Designing Consulting Accessible 145

The Pragmatic Engineer

MAY 11, 2023

Prometheus is part of the Cloud Native Foundation, membership of which indicates that it’s safe to build on top of Prometheus, as it’s actively maintained and will continue to be. Source: Grafana.org Clickhouse : log management. In the case of Coinbase, building in-house following a $65M bill was a clear no-brainer.

AWS 342

AWS Software Engineer Software Engineering Google Cloud 342

Data Engineering Podcast

AUGUST 27, 2023

In the era of the cloud most developers rely on hosted services to manage their databases, but what if you are a cloud service? Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. Closing Announcements Thank you for listening!

Database 130

Database Building PostgreSQL BI 130

Knowledge Hut

MAY 6, 2024

The scope keeps changing as the project makes progress on the agreed timeline, so, the estimation becomes a challenging task for the product owner. Agile software estimation is used for realistic planning to implementing the project requirements as per commitments. Master the art of project management and achieve new heights.

Project 98

Project Programming Certification Database 98

The Pragmatic Engineer

NOVEMBER 21, 2023

In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers.   He then worked at the casual games company Zynga, building their in-game advertising platform. We dabbled in network engineering, database management, and system administration.

Engineering 325

Engineering Bytes Cloud Computing AWS 325

Tweag

NOVEMBER 15, 2023

tl;dr: We’re pleased to announce the beta release of Organist , a tool designed to ease the definition of reliable and low-friction development environments and workflows, building on the combined strengths of Nix and Nickel. Modern development projects are synthesizer-like these days. Be tailored to your project.

Management 120

Management Python Programming Language Project 120

Engineering at Meta

MAY 22, 2024

In recent years, Meta’s data management systems have evolved into a composable architecture that creates interoperability, promotes reusability, and improves engineering efficiency. A few years ago we embarked on a journey to address these shortcomings by rethinking how our data management systems were designed.

Data Management 104

Data Management Management Data SQL 104