Tweag

FEBRUARY 15, 2023

Provide those dependencies such that the build system can find them. Run the build system. However, there are some problems with step (4): The build is not properly isolated: it can (in theory) fetch arbitrary things from the network, access arbitrary files on the filesystem, and even modify other packages.

Project 144

Project Programming Language Building Systems 144

Data Engineering Podcast

SEPTEMBER 17, 2023

Summary A significant amount of time in data engineering is dedicated to building connections and semantic meaning around pieces of information. In this episode Brian Platz explains how JSON-LD can be used as a shared representation of linked data for building semantic data products. Hex brings everything together.

Building 189

Building SQL BI Python 189

Data Engineering Podcast

DECEMBER 3, 2023

Data transfer systems are a critical component of data enablement, and building them to support large volumes of information is a complex endeavor. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Systems 130

Systems Designing Data Lake SQL 130

Engineering at Meta

MARCH 12, 2024

By the end of 2024, we’re aiming to continue to grow our infrastructure build-out that will include 350,000 NVIDIA H100 GPUs as part of a portfolio that will feature compute power equivalent to nearly 600,000 H100s. RSC has accelerated our open and responsible AI research by helping us build our first generation of advanced AI models.

Building 145

Building Portfolio Utilities Data Storage 145

Data Engineering Podcast

OCTOBER 15, 2023

Summary Building streaming applications has gotten substantially easier over the past several years. Decodable was built with a mission of eliminating all of the painful aspects of developing and deploying stream processing systems for engineering teams. How can you get the best results for your use case?

Process 182

Process Building SQL BI 182

The Pragmatic Engineer

MARCH 12, 2024

The title of the book takes aim at the “myth” that software development can be measured in “man months,” which Brooks disproves in the pages that follow: “Cost [of the software project] does indeed vary as the product of the number of men and the number of months. Progress does not. The toolsmith.

Engineering 274

Engineering Programming Language Hospitality Programming 274

Data Engineering Podcast

FEBRUARY 26, 2023

Jean-Georges Perrin was tasked with designing a new data platform implementation at PayPal and wound up building a data mesh. It's supposed to make building smarter, faster, and more flexible data infrastructures a breeze. What are the core problems that you were addressing with this project? We feel your pain.

Building 147

Building Machine Learning Metadata Data Integration 147

Data Engineering Podcast

MAY 5, 2024

In this episode he explains his approach to building AI in a more human-like fashion and the emphasis on learning rather than statistical prediction. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines.

Building 147

Building Data Lake High Quality Data Machine Learning 147

Data Engineering Podcast

APRIL 28, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. If you've learned something or tried out a project from the show then tell us about it! Your first 30 days are free!

Building 147

Building Data Lake High Quality Data Machine Learning 147

Data Engineering Podcast

MARCH 31, 2024

Identifying and accounting for those errors is a critical piece of building trust in the organization that your data is accurate and up to date. To bring observability to dbt projects the team at Elementary embedded themselves into the workflow. Dagster offers a new approach to building and running data platforms and data pipelines.

Project 130

Project Data Lake High Quality Data Data Workflow 130

Data Engineering Podcast

DECEMBER 25, 2022

Recognizing this shortcoming and the capabilities that could be unlocked by a robust solution Rishabh Poddar helped to create Opaque Systems as an outgrowth of his PhD studies. RudderStack helps you build a customer data platform on your warehouse or data lake. Sign up free at dataengineeringpodcast.com/rudder Build Data Pipelines.

Machine Learning 130

Machine Learning Systems Data Lake Data Warehouse 130

Data Engineering Podcast

JANUARY 28, 2024

Summary Monitoring and auditing IT systems for security events requires the ability to quickly analyze massive volumes of unstructured log data. If you've learned something or tried out a project from the show then tell us about it! The Machine Learning Podcast helps you go from idea to production with machine learning.

Data Lake 147

Data Lake Building High Quality Data AWS 147

Data Engineering Podcast

FEBRUARY 18, 2024

Multiple open source projects and vendors have been working together to make this vision a reality. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Your first 30 days are free!

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Engineering at Meta

DECEMBER 6, 2023

This is the most significant milestone yet for this project, which began in earnest after Mark Zuckerberg outlined his vision for it in 2019. Throughout the project, we have consulted with a diverse range of external parties to ensure that we’re making the right set of tradeoffs.

Building 145

Building Designing Consulting Accessible 145

The Pragmatic Engineer

NOVEMBER 21, 2023

If you had a continuous deployment system up and running around 2010, you were ahead of the pack: but today it’s considered strange if your team would not have this for things like web applications.    He then worked at the casual games company Zynga, building their in-game advertising platform.

Engineering 302

Engineering Bytes Cloud Computing AWS 302

Knowledge Hut

FEBRUARY 1, 2024

Not only could this recommendation system save time browsing through lists of movies, it can also give more personalized results so users don’t feel overwhelmed by too many options. What are Movie Recommendation Systems? Recommender systems have two main categories: content-based & collaborative filtering.

Systems 98

Systems Entertainment Algorithm Datasets 98

Data Engineering Podcast

APRIL 16, 2023

As the capabilities of these systems has improved and become more accessible, the target of what self-serve means changes. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. Who are the target users?

Business Intelligence 147

Business Intelligence Building Data Lake BI 147

The Pragmatic Engineer

OCTOBER 17, 2024

A €150K ($165K) grant, three people, and 10 months to build it. We recently covered how CockroachDB joins the trend of moving from open source to proprietary and why Oxide decided to keep using it with self-support , regardless Web hosting:  Netlify : chosen thanks to their super smooth preview system with SSR support.

Cloud 278

Cloud AWS Metadata Cloud Computing 278

The Pragmatic Engineer

AUGUST 1, 2023

Senior Engineers are not only expected to lead significant projects in their teams, but they have a say in whether that feature is worth building or not. They strive to build things fast, but also know when we have to slow down to go faster.” This project should require multiple team members’ contributions.

Software Engineer 245

Software Engineer Software Engineering Engineering Designing 245

Lyft Engineering

JUNE 28, 2023

In early 2022, Lyft already had a comprehensive Machine Learning Platform called LyftLearn composed of model serving , training , CI/CD, feature serving , and model monitoring systems. However, streaming data was not supported as a first-class citizen across many of the platform’s systems — such as training, complex monitoring, and others.

Machine Learning 117

Machine Learning Building Metadata Kafka 117

Knowledge Hut

OCTOBER 29, 2023

Choosing the best computer science project topic is critical to the success of any computer science student or employee. After all, the more engaging and interesting topic, the more likely it is that students or employees will be able to stay motivated and focused throughout the duration of the project.

Computer Science 111

Computer Science Coding Project Hospitality 111

The Pragmatic Engineer

SEPTEMBER 22, 2023

Bun was mostly built by Jared Sumner , a former Stripe engineer, and recipient of the Thiel Fellowship (a grant of $100,000 for young people to drop out of school and build things, founded by venture capitalist, Peter Thiel). I tip my hat to all volunteer open source contributors and maintainers — both for Node, and for other projects.

Programming Language 255

Programming Language Project Coding Engineering 255

Engineering at Meta

SEPTEMBER 7, 2023

We’re introducing Arcadia, Meta’s unified system that simulates the compute, memory, and network performance of AI training clusters. We need a systemized source of truth that can simulate various performance factors across compute, storage, and network collectively. For instance, the AI Research SuperCluster for AI research.

Systems 112

Systems Architecture Designing Utilities 112

Data Engineering Podcast

AUGUST 27, 2023

Summary Data persistence is one of the most challenging aspects of computer systems. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. Why Postgres?

Database 130

Database Building PostgreSQL BI 130

Knowledge Hut

APRIL 16, 2024

Over several decades, projects have been initiated or undertaken due to market demands, business needs, at the behest of customer request, technological advancements and to comply with regulatory requirements. We will look at some of the key project management methods followed in today’s world. Know more about the project description.

Project 98

Project Management Programming Designing 98

Engineering at Meta

DECEMBER 10, 2024

Were explaining the end-to-end systems the Facebook app leverages to deliver relevant content to people. At Facebooks scale, the systems built to support and overcome these challenges require extensive trade-off analyses, focused optimizations, and architecture built to allow our engineers to push for the same user and business outcomes.

Systems 74

Systems Architecture Engineering Data Pipeline 74

Knowledge Hut

MAY 6, 2024

The scope keeps changing as the project makes progress on the agreed timeline, so, the estimation becomes a challenging task for the product owner. Agile software estimation is used for realistic planning to implementing the project requirements as per commitments. Master the art of project management and achieve new heights.

Project 98

Project Programming Certification Database 98

Data Engineering Podcast

JANUARY 15, 2023

In this episode Srujan Akula explains how the system is implemented and how you can start using it today with your existing data systems. Materialize breaks down those barriers with a true cloud-native streaming database - not simply a database that connects to streaming systems.

Coding 130

Coding Building PostgreSQL Data Lake 130

Data Engineering Podcast

JANUARY 29, 2023

The team at Omni have taken a new approach by automatically building models based on the queries that are executed. Materialize breaks down those barriers with a true cloud-native streaming database - not simply a database that connects to streaming systems. What are the core goals that you are trying to achieve with building Omni?

Business Intelligence 130

Business Intelligence PostgreSQL Building BI 130

Knowledge Hut

MAY 6, 2024

Applying systems thinking views a system as a set of interconnected and interdependent components defined by its limits and more than the sum of their parts (subsystems). When one component of a system is altered, the effects frequently spread across the entire system. are the main objectives of systems thinking.

Systems 98

Systems Transportation Certification Healthcare 98

Data Engineering Podcast

DECEMBER 24, 2023

Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Knowledge Hut

APRIL 12, 2024

Project management plays a significant role in the success of every organization. It ensures that the project is on track, aids in efficient management of resources, and also keeps the stakeholders know what is project and what's happening in it. What is Project Management Structures, and Why is it Important?

Project 98

Project Management Certification Designing 98

Data Engineering Podcast

OCTOBER 29, 2023

RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. Learn more about Datafold by visiting dataengineeringpodcast.com/datafold Data projects are notoriously complex.

Database 189

Database BI SQL Machine Learning 189

Knowledge Hut

JANUARY 11, 2024

For most of us, the role of a Project Manager is quite well defined. But how many of us know the role a project manager plays in an Agile project? Some other questions that often boggle budding Agilists are, exactly how different a product owner is different from a project manager? And are these roles interchangeable?

Project 98

Project Management Certification Process 98

The Pragmatic Engineer

OCTOBER 10, 2023

He’s solved interesting engineering challenges along the way, too – like building observability for Amazon’s EC2 offering, and being one of the first engineers on Uber’s observability platform. They called it Office 365, and in 2010, this was a really exciting project to work on.

Software Engineer 268

Software Engineer Software Engineering Architecture Media 268

The Pragmatic Engineer

AUGUST 22, 2023

Q: “I play video games, and I was wondering how building games compares to more ‘standard’ software development. At Skype, I worked on the team building Skype for the Xbox One console, and several engineers on that team had previously worked in games. To get the full issues, twice a week, subscribe here.

Software Engineer 207

Software Engineer Software Engineering Consulting Entertainment 207

The Pragmatic Engineer

MAY 11, 2023

A very popular open-source solution for systems and services monitoring. Prometheus is part of the Cloud Native Foundation, membership of which indicates that it’s safe to build on top of Prometheus, as it’s actively maintained and will continue to be. It evaluates rules and can trigger alerts. But why is this?

AWS 318

AWS Software Engineer Software Engineering Google Cloud 318