Data Engineering Podcast

JUNE 11, 2023

Datafold has invested a lot of time into integrating with the workflow of dbt projects to add early verification that the changes you are making are correct. What are some of the error conditions/failure modes that data-diff can help identify in a dbt project? What are the parallels to that in data projects?

Project 130

Project Building Data Lake Machine Learning 130

The Pragmatic Engineer

NOVEMBER 21, 2023

Joshua is currently VP of Product & Strategy at VMware, a cloud computing and virtualization technology company. Joshua also writes an excellent Substack newsletter about how to design products which customers love, how to operate live services at scale, grow and optimize your technology orgs, and the history of the tech industry.

Engineering 345

Engineering Bytes Cloud Computing AWS 345

WeCloudData

MARCH 20, 2025

Natural Language Processing (NLP) has transformed technology by allowing machines to understand, decode, and generate human language. NLP plays a crucial role in multiple domains and NLP projects ranging from its automating customer service, improving search engines, or analyzing social media sentiments.

Project 52

Project Media Technology Engineering 52

The Pragmatic Engineer

SEPTEMBER 22, 2023

It begins with a clean state, and can ship something that works for, say, 90% of existing Node projects, and break the remaining 10%. I tip my hat to all volunteer open source contributors and maintainers — both for Node, and for other projects. Bun has no such constraint. If you are one of these people: thank you!

Programming Language 316

Programming Language Project Coding Engineering 316

The Pragmatic Engineer

SEPTEMBER 19, 2024

This project helped onboard me to the software, its structure, its build, and our issue tracking and version control workflows. My first project was supporting i18n (internationalization) in the app. They hired a manager who had done this kind of project before, and set a target date of nearly two years.

Recruitment 297

Recruitment Java Coding Project 297

Christophe Blefari

MARCH 1, 2023

dbt Labs also develop dbt Cloud which is a cloud product that hosts and runs dbt Core projects. a dbt project — a dbt project is a folder that contains all the dbt objects needed to work. You can initialise a project with the CLI command: dbt init. In a dbt project you can define YAML file everywhere.

Data Warehouse 278

Data Warehouse SQL Metadata Raw Data 278

Snowflake

DECEMBER 4, 2024

In the enterprise technology space, both the greatest certainties and the most significant potential surprises come from one area: the rapidly advancing field of artificial intelligence. But businesses will continue to hesitate to put in front of customers a technology that may display bias or provide inaccurate responses.

Unstructured Data 97

Unstructured Data Data Lake Deep Learning Structured Data 97

Knowledge Hut

MAY 6, 2024

If you are a software professional or someone who is adept in the knowledge of the technology and the latest software development, then you have probably come across the term ‘Agile’ several times. Agile management ensures the quick and precise development of the target and the project status.

Project 98

Project Management Pharmaceutical Programming 98

Knowledge Hut

MAY 23, 2024

You can find the online PMP exam application on the Project Management Institute (PMI)® website. Check Project Management professional preparation course to get started with your PMP preparation. Work Experience Your project management expertise is questioned in the next area of the online form.

Project 98

Project Certification Education Telecommunication 98

Engineering at Meta

APRIL 2, 2025

In close collaboration with the open source community, we shared knowledge, introduced new projects, and enhanced existing ones. In this post, we look at our portfolio of open source projects through numbers to give a better view of the scale of the community we interact with daily.

Portfolio 86

Portfolio Media Project Building 86

Analytics Vidhya

MARCH 26, 2023

There may be a number of reasons why you’d want to bring this rising technology […] The post How to Choose a Machine Learning Consulting Firm in 2023? Introduction Artificial intelligence (AI) and machine learning (ML) are in the best swing to help businesses sharpen their edge over their competitors in the market.

Consulting 239

Consulting Machine Learning Technology Data Analysis 239

Data Engineering Podcast

FEBRUARY 18, 2024

Multiple open source projects and vendors have been working together to make this vision a reality. Contact Info LinkedIn dain on GitHub Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? What do you have planned for the future of Trino/Starburst?

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

The Pragmatic Engineer

AUGUST 1, 2023

Senior Engineers are not only expected to lead significant projects in their teams, but they have a say in whether that feature is worth building or not. Senior engineers are expected to tackle problems where the business problem (or customer case) is well defined, but the technology strategy is not. A lead role. Coach and mentor.

Software Engineering 246

Software Engineering Software Engineer Engineering Designing 246

Data Engineering Podcast

APRIL 23, 2023

Contact Info LinkedIn @devarispbrown on Twitter Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? If you've learned something or tried out a project from the show then tell us about it! What do you have planned for the future of Meroxa?

Data Lake 277

Data Lake Kafka Machine Learning Data Warehouse 277

Snowflake

FEBRUARY 13, 2025

Lastly, companies have historically collaborated using inefficient and legacy technologies requiring file retrieval from FTP servers, API scraping and complex data pipelines. To remain nimble and have the flexibility to use the best tools for different workstreams, customers should not be locked into a specific vendor or technology.

Management 93

Management Government Cloud Unstructured Data 93

Data Engineering Podcast

JUNE 25, 2023

SQLMesh was designed as a unifying tool that is simple to work with but powerful enough for large-scale transformations and complex projects. Contact Info tobymao on GitHub @captaintobs on Twitter Website Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

Data Engineering 225

Data Engineering Data Engineer Python Engineering 225

The Pragmatic Engineer

AUGUST 22, 2023

Tommy has built his own video games, consulted on a wide variety of game projects, and for a decade has taught game development at various universities. Each project typically takes several years to create, with shifting hardware specifications and emerging competitors and trends to anticipate and react to, during the process.

Software Engineering 235

Software Engineering Software Engineer Consulting Entertainment 235

Precisely

JANUARY 30, 2025

Align people, processes, and technology Successful data governance requires a holistic approach. Woods echoed this sentiment, emphasizing the importance of a roadmap for both data governance and quality initiatives: A roadmap should include technology, people, and processes.

Data Governance 103

Data Governance Government Metadata Data 103

Data Engineering Podcast

OCTOBER 29, 2023

Learn more about Datafold by visiting dataengineeringpodcast.com/datafold Data projects are notoriously complex. I especially like the ability to combine your technical diagrams with data documentation and dependency mapping, allowing your data engineers and data consumers to communicate seamlessly about your projects.

Database 189

Database SQL BI Machine Learning 189

Ascend.io

DECEMBER 4, 2024

The rapid evolution of technology and the increasing demand for data-driven insights have placed immense pressure on these teams. As we look towards 2025, it’s clear that data teams must evolve to meet the demands of evolving technology and opportunities. So, why do so many automation projects fail to deliver?

Data Pipeline 52

Data Pipeline Metadata Data Workflow Data 52

Snowflake

APRIL 23, 2025

Free comprehensive teaching resources and a no-hassle setup: Teaching AI/ML, data science, apps and data cloud technologies shouldnt be bogged down by logistical challenges. Access to cutting-edge data technology: Snowflake is widely used by more than 10,000 enterprises across industries such as healthcare, financial services and retail.

Education 59

Education Certification Portfolio Retail 59

Seattle Data Guy

JULY 2, 2024

Data teams are expected to juggle a combination of ad-hoc requests, big bet projects, migrations, etc. All while keeping up with the latest changes in technology. Running a successful data team is hard.

Data Management 130

Data Management Management Data Project 130

Precisely

NOVEMBER 18, 2024

A long-term approach to your data strategy is key to success as business environments and technologies continue to evolve. The rapid pace of technological change has made data-driven initiatives more crucial than ever within modern business strategies. Overall, AI success truly depends on a business outcome-driven approach. “We

Data Analytics 72

Data Analytics Data Governance Data Integration Government 72

Data Engineering Podcast

JUNE 16, 2024

In this episode Kevin Liu shares some of the interesting features that they have built by combining those technologies, as well as the challenges that they face in supporting the myriad workloads that are thrown at this layer of their data platform. If you've learned something or tried out a project from the show then tell us about it!

Data Lake 147

Data Lake High Quality Data Metadata Machine Learning 147

Data Engineering Podcast

DECEMBER 24, 2023

Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. Contact Info LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Operating it at scale, however, is notoriously challenging.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Data Engineering Podcast

JUNE 2, 2024

Nicola Askham found her way into data governance by accident, and stayed because of the benefit that she was able to provide by serving as a bridge between the technology and business. What impact has the evolution of data technologies had on the implementation of governance practices? Are there any trends that concern you?

Data Governance 147

Data Governance Government Data Lake High Quality Data 147

The Pragmatic Engineer

OCTOBER 10, 2023

They called it Office 365, and in 2010, this was a really exciting project to work on. After all, we already  had open source technology to use – M3 – and we also had a clear path on what we wanted to build, similar to the roadmap the observability team at Uber used to have.  We needed $11M to get started.

Software Engineer 327

Software Engineer Software Engineering Architecture Media 327

Data Engineering Podcast

OCTOBER 15, 2023

As more people start using AI for projects, two things are clear: It’s a rapidly advancing field, but it’s tough to navigate. Contact Info esammer on GitHub LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? How can you get the best results for your use case?

Process 182

Process Building SQL BI 182

Snowflake

MARCH 12, 2025

Consumers attention spans are at an all-time low, while competition for sports fan attention remains high as teams seek to secure a piece of the projected $8.12 Sports entity data teams are often mighty but small making complex technology solutions unrealistic to leverage. Personalization matters more than ever before.

Media 80

Media Cloud Data Collection Programming 80

The Pragmatic Engineer

JUNE 29, 2023

I suspect most respondents use these registrars for side projects, and some use them at work. This service is ideal for saving money; for example, when buying domains for a side project. I asked on Twitter “what domain registrars would you recommend / do you use” , and more than 250 techies replied.

AWS 65

AWS Software Engineering Software Engineer Engineering 65

Scott Logic

MARCH 18, 2025

This massive event sprawled across half of the ExCeL centre, bringing together industry vendors, academics, and innovators across multiple technology domains. Not far off what weve been exploring through previous blog posts and my personal podcast, Architect Tomorrow , where sustainable technology solutions are a common topic.

Cloud Computing 52

Cloud Computing Technology Data Government 52

Data Engineering Podcast

MARCH 24, 2024

With their launch of Dagster+ as the redesigned commercial companion to the open source project they are investing in that capability with a suite of new features. What are the notable enhancements beyond the Dagster Core project that this updated platform provides? What problems are you trying to solve with Dagster+?

Data Lake 162

Data Lake High Quality Data Hadoop Machine Learning 162

Data Engineering Podcast

SEPTEMBER 10, 2023

Summary Data systems are inherently complex and often require integration of multiple technologies. In this episode Nick Schrock, creator of Dagster, shares his perspective on the state of data orchestration technology and its application to help inform its implementation in your environment.

BI 208

BI SQL Machine Learning Data 208

Snowflake

MARCH 6, 2025

And to create significant technology and team efficiencies, organizations need to consider opportunities to integrate LLM pipelines with existing structured data workflows. Being able to flexibly switch LLMs helps businesses optimize costs by right-sizing models for each use case and easily upgrading as models improve.

Unstructured Data 97

Unstructured Data Medical Media Data Workflow 97

RandomTrees

NOVEMBER 25, 2024

The scope of telecom services is growing in size and complexity, owing to technologies such as 5G, the Internet of Things (IoT), and cloud technology. And one technology that has potential to transform the telecom sector is Generative AI , or GAI, which lies in the focus of creating new things, be it content, ideas or solutions.

Telecommunication 52

Telecommunication IT Unstructured Data Data Mining 52