Data Engineering Podcast

DECEMBER 3, 2023

With Datafold, you can seamlessly plan, translate, and validate data across systems, massively accelerating your migration project. Your most recent project involves operationalizing a generalized data transfer service. What was the design of your initial solution to the problem?

Systems 130

Systems Designing Data Lake SQL 130

Data Engineering Podcast

NOVEMBER 19, 2023

Summary The dbt project has become overwhelmingly popular across analytics and data engineering teams. Dustin Dorsey and Cameron Cyr co-authored a practical guide to building your dbt project. In this episode they share their hard-won wisdom about how to build and scale your dbt projects. What was your path to adoption of dbt?

Project 147

Project Data Lake High Quality Data SQL 147

Knowledge Hut

MAY 21, 2024

The average salary for a UI/UX Designer is $73,000 (Source:Indeed.com) compared to the average salary of a Front End Developer which is $62,000. So you want to be a designer or developer? But which one should you choose: Front end Developer vs UX designer or UI designer? Almost every business today has a website.

Designing 98

Designing Programming Language Portfolio Media 98

Data Engineering Podcast

MARCH 31, 2024

To bring observability to dbt projects the team at Elementary embedded themselves into the workflow. Can you start by outlining what elements of observability are most relevant for dbt projects? Can you describe what Elementary is and how it is designed to enhance the development and maintenance work in dbt projects?

Project 130

Project Data Lake High Quality Data Data Workflow 130

Knowledge Hut

APRIL 29, 2024

In project management, project scheduling encompasses listing activities, defining milestones and scheduling deliverables for delivery. This indicates that every project schedule must include a planned start date and planned finish date, estimated resources assigned to each activity and estimated duration of each activity.

Project 98

Project Management Certification Algorithm 98

Knowledge Hut

JANUARY 25, 2024

With over a decade of my experience in Project management, I might have crashed about 80% of my Project. Project Crashing is not a negative or a bad thing like it sounds, instead it serves as a strategy in project management, aimed at expediting project timelines without compromising the project's scope.

Project 98

Project Management Process Certification 98

Knowledge Hut

APRIL 16, 2024

Over several decades, projects have been initiated or undertaken due to market demands, business needs, at the behest of customer request, technological advancements and to comply with regulatory requirements. We will look at some of the key project management methods followed in today’s world. Know more about the project description.

Project 98

Project Management Programming Designing 98

Knowledge Hut

APRIL 12, 2024

Project management plays a significant role in the success of every organization. It ensures that the project is on track, aids in efficient management of resources, and also keeps the stakeholders know what is project and what's happening in it. What is Project Management Structures, and Why is it Important?

Project 98

Project Management Certification Designing 98

Knowledge Hut

OCTOBER 29, 2023

Choosing the best computer science project topic is critical to the success of any computer science student or employee. After all, the more engaging and interesting topic, the more likely it is that students or employees will be able to stay motivated and focused throughout the duration of the project.

Computer Science 111

Computer Science Coding Project Hospitality 111

KDnuggets

NOVEMBER 30, 2023

The blog discusses five platforms designed for data scientists with specialized capabilities in managing large datasets, models, workflows, and collaboration beyond what GitHub offers.

Data Science 151

Data Science Project Datasets Data 151

The Pragmatic Engineer

AUGUST 1, 2023

Senior Engineers are not only expected to lead significant projects in their teams, but they have a say in whether that feature is worth building or not. We put a lot of emphasis on communication and prioritization and the ability to unblock yourself or your team – this comes on top of the programming and design skills.

Software Engineer 245

Software Engineer Software Engineering Engineering Designing 245

Knowledge Hut

MAY 6, 2024

Before proceeding, it is important to know that management of the task and a lengthy project is imperative for every development team for the precision of task and quality output. Earlier, the software development team used to take the assistance of the waterfall project management for the completion of the project management task.

Project 98

Project Management Pharmaceutical Programming 98

Knowledge Hut

MARCH 7, 2024

Creating small to complex projects requires some tactics. Working on complex projects at a single time can be overwhelming. You must manage the projects using proper management teams and skills. It helps in setting a timeline for your project. What is a Project Timeline? You must sort tasks out via discussions.

Project 98

Project Certification Management Designing 98

Start Data Engineering

MAY 28, 2024

Project demo 3. Distributed systems are scalable, resilient to failures, & designed for high availability 4.5. Introduction 2. Building efficient data pipelines with DuckDB 4.1. Use DuckDB to process data, not for multiple users to access data 4.2. Cost calculation: DuckDB + Ephemeral VMs = dirt cheap data processing 4.3.

Data Pipeline 219

Data Pipeline Python Building Data 219

Knowledge Hut

MAY 6, 2024

Release Planning / Goal Plan & Prioritize your Backlog (Define User Stories like Taking a Course, Complete Application, Apply & Book for the exam) Set the release goal: “As a Project Management Professional, I want to Obtain X-Certification in order to gain more knowledge in Y-Industry.”

Certification 97

Certification Project Management Technology 97

Knowledge Hut

APRIL 29, 2024

P roject Communication Plan is an important step in Project Management Plan or what is simply called Project Plan. For more details on Project Management please refer to PMP certification course. What is Project Communication Plan?

Project 98

Project Consulting BI Algorithm 98

Knowledge Hut

MAY 9, 2024

The Lean Six Sigma Green Belt certification is an important step in becoming a master of the lean six sigma technique and leading improvement projects for a company. Before we jump to the six sigma green belt project examples and ideas, let us learn some of the basic aspects of Six Sigma Green Belt. It should not need help.

Project 98

Project Manufacturing Certification Consulting 98

Data Engineering Podcast

MAY 18, 2024

For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst is an end-to-end data lakehouse platform built on Trino, the query engine Apache Iceberg was designed for, with complete support for all table formats including Apache Iceberg, Hive, and Delta Lake. Want to see Starburst in action?

Building 278

Building Data Lake High Quality Data Business Intelligence 278

Data Engineering Podcast

JUNE 25, 2023

SQLMesh was designed as a unifying tool that is simple to work with but powerful enough for large-scale transformations and complex projects. How have the design and goals evolved since you first started working on it? If you've learned something or tried out a project from the show then tell us about it!

Data Engineering 225

Data Engineering Data Engineer Python Engineering 225

The Pragmatic Engineer

NOVEMBER 21, 2023

Joshua also writes an excellent Substack newsletter about how to design products which customers love, how to operate live services at scale, grow and optimize your technology orgs, and the history of the tech industry. This pioneering project was conceived during the Cold War between the US and the USSR. Subscribe here.

Engineering 302

Engineering Bytes Cloud Computing AWS 302

The Pragmatic Engineer

AUGUST 22, 2023

Tommy has built his own video games, consulted on a wide variety of game projects, and for a decade has taught game development at various universities. Each project typically takes several years to create, with shifting hardware specifications and emerging competitors and trends to anticipate and react to, during the process.

Software Engineer 207

Software Engineer Software Engineering Consulting Entertainment 207

Christophe Blefari

MARCH 1, 2023

dbt Labs also develop dbt Cloud which is a cloud product that hosts and runs dbt Core projects. a dbt project — a dbt project is a folder that contains all the dbt objects needed to work. You can initialise a project with the CLI command: dbt init. In a dbt project you can define YAML file everywhere.

Data Warehouse 278

Data Warehouse SQL Metadata Raw Data 278

The Pragmatic Engineer

MARCH 12, 2024

The title of the book takes aim at the “myth” that software development can be measured in “man months,” which Brooks disproves in the pages that follow: “Cost [of the software project] does indeed vary as the product of the number of men and the number of months. Progress does not. Two secretaries.

Engineering 274

Engineering Programming Language Hospitality Programming 274

Data Engineering Podcast

FEBRUARY 18, 2024

Multiple open source projects and vendors have been working together to make this vision a reality. What are the differences in terms of pipeline design/access and usage patterns when using a Trino/Iceberg lakehouse as compared to other popular warehouse/lakehouse structures? Email hosts@dataengineeringpodcast.com ) with your story.

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

The Pragmatic Engineer

SEPTEMBER 22, 2023

It begins with a clean state, and can ship something that works for, say, 90% of existing Node projects, and break the remaining 10%. I tip my hat to all volunteer open source contributors and maintainers — both for Node, and for other projects. Bun has no such constraint. If you are one of these people: thank you!

Programming Language 255

Programming Language Project Coding Engineering 255

The Pragmatic Engineer

NOVEMBER 26, 2024

A next-generation code editor designed for collaboration with AI. Those mentioned were IntelliJ, PyCharm, PHPStorm, and RubyMind. Windsurf Editor. An “agentic IDE” released by Codeium just two weeks ago, which is attracting users fast. Neovim with Copilot.

Software Engineer 290

Software Engineer Software Engineering Engineering Coding 290

The Pragmatic Engineer

SEPTEMBER 19, 2024

This project helped onboard me to the software, its structure, its build, and our issue tracking and version control workflows. My first project was supporting i18n (internationalization) in the app. They hired a manager who had done this kind of project before, and set a target date of nearly two years.

Recruitment 176

Recruitment Java Coding Project 176

Data Engineering Podcast

FEBRUARY 4, 2024

Summary Stream processing systems have long been built with a code-first design, adding SQL as a layer on top of the existing framework. How have the design and goals/scope changed since you first started working on it? What are the core design philosophies that you rely on to prioritize the ongoing development of the project?

SQL 173

SQL Data Lake High Quality Data Data Pipeline 173

The Pragmatic Engineer

APRIL 9, 2024

a YouTuber doing design videos has designers watching, or a software engineer like me writing on software engineering has developers reading) These "creators" would vet people looking for jobs And they would also vet companies looking to hire.

Software Engineer 271

Software Engineer Software Engineering Engineering Recruitment 271

Data Engineering Podcast

JUNE 16, 2024

For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst is an end-to-end data lakehouse platform built on Trino, the query engine Apache Iceberg was designed for, with complete support for all table formats including Apache Iceberg, Hive, and Delta Lake. Want to see Starburst in action?

Data Lake 147

Data Lake High Quality Data Metadata Machine Learning 147

The Pragmatic Engineer

OCTOBER 17, 2024

  This grant is designed to “support entrepreneurs, tech-geeks, developers, and socially engaged people, who are capable of challenging the way we search and discover information and resources on the internet” The team is tiny; only three people. Internal comms: Chat: Slack Coordination / project management: Linear 3.

Cloud 278

Cloud AWS Metadata Cloud Computing 278

Data Engineering Podcast

DECEMBER 24, 2023

When a cluster's usage expands beyond the original designed capacity, what are the options/procedures for expanding that capacity? If you've learned something or tried out a project from the show then tell us about it! If you've learned something or tried out a project from the show then tell us about it!

Kafka 245

Kafka Data Lake High Quality Data SQL 245