This site uses cookies to improve your experience. To help us insure we adhere to various privacy regulations, please select your country/region of residence. If you do not select a country, we will assume you are from the United States. Select your Cookie Settings or view our Privacy Policy and Terms of Use.
Cookies and similar technologies are used on this website for proper function of the website, for tracking performance analytics and for marketing purposes. We and some of our third-party providers may use cookie data for various purposes. Please review the cookie settings below and choose your preference.
Used for the proper function of the website
Used for monitoring website traffic and interactions
Cookies and similar technologies are used on this website for proper function of the website, for tracking performance analytics and for marketing purposes. We and some of our third-party providers may use cookie data for various purposes. Please review the cookie settings below and choose your preference.
The Pragmatic Engineer
JUNE 1, 2023
I’d like to share a story about an educational side project which could prove fruitful for a software engineer who’s seeking a new job. Juraj included system monitoring parts which monitor the server’s capacity he runs the app on: The monitoring page on the Rides app And it doesn’t end here. Persistence.
363 
Data Engineering Podcast
MAY 26, 2024
Summary Any software system that survives long enough will require some form of migration or evolution. When that system is responsible for the data layer the process becomes more challenging. Sriram Panyam has been involved in several projects that required migration of large volumes of data in high traffic environments.
130 
Tweag
JULY 5, 2023
Buck2 is a from-scratch rewrite of Buck , a polyglot, monorepo build system that was developed and used at Meta (Facebook), and shares a few similarities with Bazel. As you may know, the Scalable Builds Group at Tweag has a strong interest in such scalable build systems. Meta recently announced they have made Buck2 open-source.
141 
Data Engineering Podcast
DECEMBER 3, 2023
Data transfer systems are a critical component of data enablement, and building them to support large volumes of information is a complex endeavor. With Datafold, you can seamlessly plan, translate, and validate data across systems, massively accelerating your migration project.
130 
Tweag
FEBRUARY 15, 2023
Provide those dependencies such that the build system can find them. Run the build system. This allows for irreproducible builds, and makes it easy to forget to explicitly list a system dependency if it happens to be installed on the author’s system. Also, the templates are suited to building opam projects.
144 
Engineering at Meta
APRIL 6, 2023
Buck2, our new open source, large-scale build system , is now available on GitHub. Buck2 is an extensible and performant build system written in Rust and designed to make your build experience faster and more efficient. In particular, we support Sapling-based file systems. Why rebuild Buck?
145 
ArcGIS
NOVEMBER 30, 2023
Learn how to filter coordinate systems based on a spatial extent, GCS, or projection property.
130 
Data Engineering Podcast
MARCH 31, 2024
To bring observability to dbt projects the team at Elementary embedded themselves into the workflow. Can you start by outlining what elements of observability are most relevant for dbt projects? Over the past ~3 years there were numerous data observability systems/products created. How is Elementary designed/implemented?
130 
Engineering at Meta
DECEMBER 10, 2024
Were explaining the end-to-end systems the Facebook app leverages to deliver relevant content to people. At Facebooks scale, the systems built to support and overcome these challenges require extensive trade-off analyses, focused optimizations, and architecture built to allow our engineers to push for the same user and business outcomes.
76 
Data Engineering Podcast
JUNE 11, 2023
Summary Data engineering is all about building workflows, pipelines, systems, and interfaces to provide stable and reliable data. Datafold has invested a lot of time into integrating with the workflow of dbt projects to add early verification that the changes you are making are correct. What are the parallels to that in data projects?
130 
Knowledge Hut
APRIL 16, 2024
Over several decades, projects have been initiated or undertaken due to market demands, business needs, at the behest of customer request, technological advancements and to comply with regulatory requirements. We will look at some of the key project management methods followed in today’s world. Know more about the project description.
98 
Knowledge Hut
OCTOBER 29, 2023
Choosing the best computer science project topic is critical to the success of any computer science student or employee. After all, the more engaging and interesting topic, the more likely it is that students or employees will be able to stay motivated and focused throughout the duration of the project.
111 
The Pragmatic Engineer
MARCH 12, 2024
The title of the book takes aim at the “myth” that software development can be measured in “man months,” which Brooks disproves in the pages that follow: “Cost [of the software project] does indeed vary as the product of the number of men and the number of months. Progress does not. Two secretaries.
274 
Knowledge Hut
MAY 6, 2024
Applying systems thinking views a system as a set of interconnected and interdependent components defined by its limits and more than the sum of their parts (subsystems). When one component of a system is altered, the effects frequently spread across the entire system. are the main objectives of systems thinking.
98 
Knowledge Hut
MAY 6, 2024
The scope keeps changing as the project makes progress on the agreed timeline, so, the estimation becomes a challenging task for the product owner. Agile software estimation is used for realistic planning to implementing the project requirements as per commitments. Master the art of project management and achieve new heights.
98 
Knowledge Hut
APRIL 12, 2024
Project management plays a significant role in the success of every organization. It ensures that the project is on track, aids in efficient management of resources, and also keeps the stakeholders know what is project and what's happening in it. What is Project Management Structures, and Why is it Important?
98 
The Pragmatic Engineer
NOVEMBER 21, 2023
If you had a continuous deployment system up and running around 2010, you were ahead of the pack: but today it’s considered strange if your team would not have this for things like web applications. We dabbled in network engineering, database management, and system administration. and hand-rolled C -code.
302 
Analytics Vidhya
FEBRUARY 3, 2023
Say Harish and Lisa are two people working on the same project but on two different systems(say windows and […] The post Getting Started with The Basics of Docker appeared first on Analytics Vidhya. ” If you read the above quote, you must think, what does this all mean? Well, my friend, this is what Docker is.
257 
Analytics Vidhya
MARCH 7, 2023
Introduction Apache Cassandra is a NoSQL database management system that is open-source and distributed. Facebook created Cassandra, which ultimately became an Apache Software Foundation project. It is well-known for its rapid write […] The post Top 6 Cassandra Interview Questions appeared first on Analytics Vidhya.
252 
The Pragmatic Engineer
OCTOBER 18, 2024
Corporate conflict recap Automattic is the creator of open source WordPress content management system (CMS), and WordPress powers an incredible 43% of webpages and 65% of CMSes. OpenAI’s impossible business projections. According This event is shameful and unprecedented in the history of open source on the web.
326 
Start Data Engineering
MAY 28, 2024
Project demo 3. Distributed systems are scalable, resilient to failures, & designed for high availability 4.5. Introduction 2. Building efficient data pipelines with DuckDB 4.1. Use DuckDB to process data, not for multiple users to access data 4.2. Cost calculation: DuckDB + Ephemeral VMs = dirt cheap data processing 4.3.
219 
The Pragmatic Engineer
AUGUST 1, 2023
Senior Engineers are not only expected to lead significant projects in their teams, but they have a say in whether that feature is worth building or not. The SDE3 level expects leadership on projects in which this engineer is involved. It’s not a checklist, but some expectations that could be considered: Lead a complex project.
Yelp Engineering
MARCH 25, 2024
We present a high level overview of our in-house phone masking system and dive into the details of the engineering challenge of optimizing the usage of proxy phone number resources at Yelp’s scale. Background Every year, millions of requests for quotes, consultations or other messages are sent to businesses on Yelp.
103 
The Pragmatic Engineer
SEPTEMBER 19, 2024
In the early 90’s, DOS programs like the ones my company made had its own Text UI screen rendering system. This rendering system was easy for me to understand, even on day one. Our rendering system was very memory inefficient, but that could be fixed. By doing so, I got to see every screen of the system.
176 
Data Engineering Podcast
FEBRUARY 18, 2024
Multiple open source projects and vendors have been working together to make this vision a reality. What are the pain points that are still prevalent in lakehouse architectures as compared to warehouse or vertically integrated systems? If you've learned something or tried out a project from the show then tell us about it!
262 
The Pragmatic Engineer
SEPTEMBER 22, 2023
It begins with a clean state, and can ship something that works for, say, 90% of existing Node projects, and break the remaining 10%. I tip my hat to all volunteer open source contributors and maintainers — both for Node, and for other projects. Bun has no such constraint. If you are one of these people: thank you!
The Pragmatic Engineer
OCTOBER 17, 2024
We recently covered how CockroachDB joins the trend of moving from open source to proprietary and why Oxide decided to keep using it with self-support , regardless Web hosting: Netlify : chosen thanks to their super smooth preview system with SSR support. Internal comms: Chat: Slack Coordination / project management: Linear 3.
278 
Data Engineering Podcast
OCTOBER 29, 2023
Learn more about Datafold by visiting dataengineeringpodcast.com/datafold Data projects are notoriously complex. I especially like the ability to combine your technical diagrams with data documentation and dependency mapping, allowing your data engineers and data consumers to communicate seamlessly about your projects.
189 
The Pragmatic Engineer
OCTOBER 10, 2024
Wordpress is the most popular content management system (CMS), estimated to power around 43% of all websites; a staggering number! As a response, cloud providers started the Valkey project, which could become the “new and still permissive Redis.”
306 
The Pragmatic Engineer
APRIL 11, 2024
From Sella’s status page : “Following the installation of an update to the operating system and related firmware which led to an unstable situation. Still, I’m puzzled by how long the system has been down. If it was an update to Oracle, or to the operating system, then why not roll back the update?
221 
Cloudera
NOVEMBER 15, 2024
Unlike Other OCR systems, which can often miss context or require multiple steps to clean the data, Claude 3 enables customers to perform complex document understanding tasks directly. You can launch it from the AMP catalog within your Cloudera AI (Formerly Cloudera Machine Learning) workspace, or start a new project with the repository URL.
The Pragmatic Engineer
AUGUST 22, 2023
Tommy has built his own video games, consulted on a wide variety of game projects, and for a decade has taught game development at various universities. Each project typically takes several years to create, with shifting hardware specifications and emerging competitors and trends to anticipate and react to, during the process.
Data Engineering Podcast
MAY 18, 2024
Summary The purpose of business intelligence systems is to allow anyone in the business to access and decode data to help them make informed decisions. If you've learned something or tried out a project from the show then tell us about it! If you've learned something or tried out a project from the show then tell us about it!
278 
Precisely
SEPTEMBER 26, 2024
However, achieving success in AI projects isn’t just about deploying advanced algorithms or machine learning models. The real challenge lies in ensuring that the data powering your projects is AI-ready. AI systems require high-quality, well-governed data to avoid missteps.
75 
Data Engineering Podcast
FEBRUARY 4, 2024
Summary Stream processing systems have long been built with a code-first design, adding SQL as a layer on top of the existing framework. There are numerous stream processing engines, near-real-time database engines, streaming SQL systems, etc. If you've learned something or tried out a project from the show then tell us about it!
173 
Data Engineering Podcast
DECEMBER 24, 2023
Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. If you've learned something or tried out a project from the show then tell us about it! If you've learned something or tried out a project from the show then tell us about it!
245 
Netflix Tech
JUNE 1, 2023
To do this, we devised a novel way to simulate the projected traffic weeks ahead of launch by building upon the traffic migration framework described here. Replay traffic enabled us to test our new systems and algorithms at scale before launch, while also making the traffic as realistic as possible.
140 
Data Engineering Podcast
JANUARY 7, 2024
What are the experimental methods that you are using to gain understanding in the opportunities and practical limits of those systems? As you strive to push the limits of technical capacity in data systems, how does that impact the usability of the resulting systems? Email hosts@dataengineeringpodcast.com ) with your story.
147 
Data Engineering Podcast
JANUARY 21, 2024
A substantial amount of the data that is being managed in these systems is related to customers and their interactions with an organization. Data projects are notoriously complex. Find simplicity in your most complex projects with Miro. If you've learned something or tried out a project from the show then tell us about it!
147 
The Pragmatic Engineer
SEPTEMBER 28, 2023
Willem Spruijt is a software engineer whom I worked on the same team with at Uber in Amsterdam, building payments systems. So we had a quarterly planning process to ensure all project dependencies were incorporated into each team’s roadmap. We cover one out of four topics in today’s subscriber-only The Pulse issue.
The Pragmatic Engineer
OCTOBER 10, 2023
They called it Office 365, and in 2010, this was a really exciting project to work on. I wrote code for drivers on Windows, and started to put a basic observability system in place. EC2 had no observability system back then: people would spin up EC2 instances but have no idea whether or not they worked.
The Pragmatic Engineer
APRIL 19, 2023
I still remember being in a meeting where a Very Respected Engineer was explaining how they are building a project, and they said something along the lines of "and, of course, idempotency is non-negotiable." I was sceptical that any system would automatically reject resumes, because I never saw this as a hiring manager.
Data Engineering Podcast
MARCH 3, 2024
What are the skills and systems that need to be in place to effectively execute on an AI program? "AI" Organizations can easily fall into the trap of green-lighting an AI project before they have done the work of ensuring they have the necessary data and the ability to process it. When is AI the wrong choice?
162 
Data Engineering Podcast
OCTOBER 15, 2023
Decodable was built with a mission of eliminating all of the painful aspects of developing and deploying stream processing systems for engineering teams. As more people start using AI for projects, two things are clear: It’s a rapidly advancing field, but it’s tough to navigate. How can you get the best results for your use case?
182
Expert insights. Personalized for you.
We have resent the email to
Are you sure you want to cancel your subscriptions?
Let's personalize your content