Data Engineering Podcast

SEPTEMBER 17, 2023

Summary A significant amount of time in data engineering is dedicated to building connections and semantic meaning around pieces of information. In this episode Brian Platz explains how JSON-LD can be used as a shared representation of linked data for building semantic data products. Hex brings everything together. What is JSON-LD?

Building 189

Building SQL BI Python 189

Engineering at Meta

MARCH 12, 2024

By the end of 2024, we’re aiming to continue to grow our infrastructure build-out that will include 350,000 NVIDIA H100 GPUs as part of a portfolio that will feature compute power equivalent to nearly 600,000 H100s. RSC has accelerated our open and responsible AI research by helping us build our first generation of advanced AI models.

Building 145

Building Portfolio Utilities Data Storage 145

Data Engineering Podcast

OCTOBER 1, 2023

Now that AI has reached the level of sophistication seen in the various generative models it is being used to build new ETL workflows. In this episode Jay Mishra shares his experiences and insights building ETL pipelines with the help of generative AI. How can you get the best results for your use case?

Building 162

Building BI SQL Machine Learning 162

Data Engineering Podcast

APRIL 28, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Go to dataengineeringpodcast.com/dagster today to get started. Your first 30 days are free! Data lakes are notoriously complex.

Building 147

Building Data Lake High Quality Data Machine Learning 147

Data Engineering Podcast

FEBRUARY 6, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

SQL 100

SQL Building IT Google Cloud 100

The Pragmatic Engineer

JULY 20, 2023

In this issue, we cover: How Akita was founded On cofounders Raising funding Pivoting and growing the company On hiring The tech stack The biggest challenges of building a startup For this article, I interviewed Jean directly. Startups that are running out of money, and are unable to raise more capital, are having to forcibly shut down.

Building 204

Building Programming Language Programming Python 204

Engineering at Meta

APRIL 6, 2023

Buck2, our new open source, large-scale build system , is now available on GitHub. Buck2 is an extensible and performant build system written in Rust and designed to make your build experience faster and more efficient. In our internal tests at Meta, we observed that Buck2 completed builds 2x as fast as Buck1.

Building 144

Building Systems Java Coding 144

Confessions of a Data Guy

MARCH 29, 2024

Have you ever wondered at a high level what it’s like to build production-level data pipelines on Databricks? The post Building Databricks Data Pipelines 101 appeared first on Confessions of a Data Guy. What does it look like, what tools do you use?

Data Pipeline 100

Data Pipeline Building Data IT 100

Analytics Vidhya

FEBRUARY 3, 2023

Imagine scheduling your ML tasks to run automatically without the need for manual […] The post How to Build and Monitor Systems Using Airflow? Are you looking for a way to automate and simplify the process? Airflow can help you manage your workflow and make your life easier with its monitoring and notifications features.

Systems 214

Systems Building Machine Learning Management 214

Data Engineering Podcast

MAY 5, 2024

In this episode he explains his approach to building AI in a more human-like fashion and the emphasis on learning rather than statistical prediction. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines.

Building 147

Building Data Lake High Quality Data Machine Learning 147

Confessions of a Data Guy

JUNE 11, 2024

In a show of solidarity to the open-source community, I thought I would publish a PYPI package that will probably be used by 5 people […] The post Building Open-Source Python Packages – SparklePop appeared first on Confessions of a Data Guy.

Python 100

Python Building Software Engineer Software Engineering 100

ThoughtSpot

OCTOBER 18, 2024

In the fast-paced world of software development, the efficiency of build processes plays a crucial role in maintaining productivity and code quality. At ThoughtSpot , while Gradle has been effective, the growing complexity of our projects demanded a more sophisticated approach to understanding and optimizing our builds.

Building 52

Building Process Pipeline-centric Database-centric 52

Data Engineering Podcast

FEBRUARY 26, 2023

Jean-Georges Perrin was tasked with designing a new data platform implementation at PayPal and wound up building a data mesh. It's supposed to make building smarter, faster, and more flexible data infrastructures a breeze. We feel your pain. It ends up being anything but that. But don't worry, there is a better way.

Building 147

Building Machine Learning Metadata Data Integration 147

Data Engineering Podcast

FEBRUARY 20, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. What are the core goals of the Fugue project?

Python 100

Python Data Process IT Building 100

Data Engineering Podcast

OCTOBER 15, 2023

Summary Building streaming applications has gotten substantially easier over the past several years. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. How can you get the best results for your use case?

Process 182

Process Building SQL BI 182

Data Engineering Podcast

JANUARY 28, 2024

For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics. for the prroduct?

Data Lake 147

Data Lake Building High Quality Data AWS 147

Seattle Data Guy

JANUARY 4, 2024

I once had an engineer tell me that they essentially didn’t want to consider cost as they were building a solution. Don’t get me wrong, yes, when you’re building, you iterate and aim to improve your solutions cost. I was baffled.

IT 130

IT Data Engineering Building 130

Data Engineering Podcast

JULY 23, 2023

RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. What was your decision process for building Dozer as open source? Who are you building Dozer for and what were the motivations for that vision?

Building 130

Building Machine Learning SQL Python 130

Data Engineering Podcast

MARCH 5, 2023

It's supposed to make building smarter, faster, and more flexible data infrastructures a breeze. By bringing all the layers of the data stack together, TimeXtender helps you build data solutions up to 10 times faster and saves you 70-80% on costs. We feel your pain. It ends up being anything but that.

Building 147

Building Database Design Machine Learning Metadata 147

Tweag

JULY 5, 2023

Buck2 is a from-scratch rewrite of Buck , a polyglot, monorepo build system that was developed and used at Meta (Facebook), and shares a few similarities with Bazel. As you may know, the Scalable Builds Group at Tweag has a strong interest in such scalable build systems. invoke build buck2 build //starlark-rust/starlark 6.

Systems 141

Systems Building Java Programming Language 141

Knowledge Hut

JANUARY 31, 2024

The status of working is continually changing, responding quickly to changing environments and, as a result, influencing our insight into the role of technology in our daily lives. This eventually leads to a decrease in both IT efficiency and customer satisfaction. What Is a Service Catalog? What Is in a Service catalog?

IT 98

IT Certification Portfolio Accessible 98

Knowledge Hut

MAY 6, 2024

The adoption was necessitated by this particular project where design and build activities were to be iterated based on changing requirements. The approach allowed necessary flexibility to implement varied design approaches and pass them quickly to build phase to avoid large-scale changes once the complete design was done.

IT 98

IT Banking Retail Project 98

Ascend.io

JULY 11, 2024

Metadata is the information that provides context and meaning to data, ensuring it’s easily discoverable, organized, and actionable. It enhances data quality, governance, and automation, transforming raw data into valuable insights. Imagine a library with millions of books but no catalog system to organize them. Chaos, right? What is Metadata?

Metadata 52

Metadata IT Government High Quality Data 52

Engineering at Meta

DECEMBER 6, 2023

This is the most significant milestone yet for this project, which began in earnest after Mark Zuckerberg outlined his vision for it in 2019. Bringing E2EE to Messenger has been a complex process, with every feature and product goal revealing further challenges that required careful consideration.

Building 144

Building Designing Consulting Coding 144

Knowledge Hut

FEBRUARY 14, 2024

The Entry Certificate in Business Analysis (ECBA) certification is a valuable career-building tool for any business owner or manager. Yes, ECBA certification is the first step in building your career as an accounting professional. However, is ECBA certification worth it for advancing as a business owner?

Certification 52

Certification IT Business Analyst Finance 52

Zalando Engineering

OCTOBER 16, 2024

Introduction Context and Purpose Our team is part of the Transport teams within the Logistics department, where we build and manage software for internal users, including finance teams, warehouses, and, in the future, our third-party partners. Modularity was key to enabling independent feature development and deployment across teams.

Building 68

Building Transportation Coding Designing 68

Knowledge Hut

MAY 2, 2024

Solutions: As part of team-building talk to employees and share their cultures and activities so that everyone knows a little about the team If you meet someone with the same cultural background, bonding can also improve which will together increase productivity 4. Working remotely is the new trend that is transcending industries.

IT 98

IT Management Project Building 98

Snowflake

NOVEMBER 15, 2023

BUILD 2023 is where AI gets real. Join our two-day virtual global conference and learn how to build with the app dev innovations you heard about at Snowflake Summit and Snowday. As always, BUILD is created by builders, for builders, so you’ll find plenty to spark your interest and boost your skill set.

Building 112

Building Accessibility Accessible IT 112

Netflix Tech

NOVEMBER 6, 2023

We have built an internal system that allows someone to perform in-video search across the entire Netflix video catalog, and we’d like to share our experience in building this system. Building in-video search To build such a visual search engine, we needed a machine learning system that can understand visual elements.

Building 92

Building Media Machine Learning Utilities 92

Knowledge Hut

NOVEMBER 16, 2023

One of the biggest challenges faced by corporations today when it comes to cloud adoption is the lack of cloud expertise. There is a clear shortage of professionals certified with Amazon Web Services (AWS). As far as AWS certifications are concerned, there is always a certain debate surrounding them. What is AWS?

AWS 98

AWS Certification IT Amazon Web Services 98