The Pragmatic Engineer

NOVEMBER 21, 2023

If you had a continuous deployment system up and running around 2010, you were ahead of the pack: but today it’s considered strange if your team would not have this for things like web applications.  Joshua is currently VP of Product & Strategy at VMware, a cloud computing and virtualization technology company.

Engineering 325

Engineering Bytes Cloud Computing AWS 325

databricks

NOVEMBER 12, 2024

Monolithic to Modular The proof of concept (POC) of any new technology often starts with large, monolithic units that are difficult to characterize.

Systems 120

Systems Engineering Technology Data 120

Knowledge Hut

MARCH 22, 2024

I have comprehensively analyzed the area of physical security, particularly the ongoing discussion surrounding fail safe vs fail-safe secure electric strike locking systems. On the other hand, fail-secure systems focus on maintaining continuous security, keeping doors locked even in difficult conditions to protect assets.

Systems 105

Systems Electronics Hospitality Architecture 105

The Pragmatic Engineer

JUNE 1, 2023

Juraj included system monitoring parts which monitor the server’s capacity he runs the app on: The monitoring page on the Rides app And it doesn’t end here. Juraj created a systems design explainer on how he built this project, and the technologies used: The systems design diagram for the Rides application The app uses: Node.js

Education 363

Education Project PostgreSQL Software Engineering 363

Knowledge Hut

MAY 6, 2024

Applying systems thinking views a system as a set of interconnected and interdependent components defined by its limits and more than the sum of their parts (subsystems). When one component of a system is altered, the effects frequently spread across the entire system. are the main objectives of systems thinking.

Systems 98

Systems Transportation Certification Healthcare 98

Analytics Vidhya

MARCH 5, 2023

Introduction Microsoft Azure HDInsight(or Microsoft HDFS) is a cloud-based Hadoop Distributed File System version. A distributed file system runs on commodity hardware and manages massive data collections. It is a fully managed cloud-based environment for analyzing and processing enormous volumes of data.

Hadoop 253

Hadoop Data Collection Cloud Technology 253

Data Engineering Podcast

MAY 18, 2024

Summary The purpose of business intelligence systems is to allow anyone in the business to access and decode data to help them make informed decisions. Contact Info Ryan LinkedIn Paul LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

Building 278

Building Data Lake High Quality Data Business Intelligence 278

Precisely

MARCH 5, 2025

The Precisely Women in Technology (PWIT) network was created to connect women in the organization to offer support, guidance, mentorship, and more opportunities. Continue reading to learn more about Sravani Malempati, Senior Support Engineer II, and her 16+ year career in technology. Why did you choose to pursue a career in technology?

Technology 59

Technology Computer Science Electronics Certification 59

The Pragmatic Engineer

SEPTEMBER 19, 2024

In the early 90’s, DOS programs like the ones my company made had its own Text UI screen rendering system. This rendering system was easy for me to understand, even on day one. Our rendering system was very memory inefficient, but that could be fixed. By doing so, I got to see every screen of the system.

Recruitment 260

Recruitment Java Coding Project 260

Precisely

DECEMBER 16, 2024

Modern IT environments require comprehensive data for successful AIOps, that includes incorporating data from legacy systems like IBM i and IBM Z into ITOps platforms. As technology continues its rapid ongoing evolution, IT environments have become increasingly complex which leaves businesses needing to adapt at unprecedented speeds.

Systems 59

Systems IT Machine Learning Insurance 59

Data Engineering Podcast

FEBRUARY 18, 2024

What are the pain points that are still prevalent in lakehouse architectures as compared to warehouse or vertically integrated systems? Contact Info LinkedIn dain on GitHub Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Jesse Anderson

OCTOBER 5, 2023

Data quality problems are solved by more than just technology. Data products need to be exposed with the right technologies for the various use cases. In another post , I introduced a concept from Justin Coffey about how much systems can be changed from their original design. Flink is an excellent piece of technology.

Kafka 195

Kafka Finance Cloud Designing 195

The Pragmatic Engineer

SEPTEMBER 22, 2023

There are people whose company pays them to maintain Node for their own, company systems — such as IBM paying to maintain Node so that it stays compatible with IBM AIX — a proprietary Unix operating system designed to run on IBM® Power® servers.

Programming Language 281

Programming Language Project Coding Engineering 281

Data Engineering Podcast

DECEMBER 24, 2023

Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. Contact Info LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Closing Announcements Thank you for listening!

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Data Engineering Podcast

APRIL 21, 2024

Summary Generative AI has rapidly transformed everything in the technology sector. How do you manage the personalization of the AI functionality in your system for each user/team? Contact Info LinkedIn Blog Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

Data Lake 182

Data Lake High Quality Data Machine Learning Data Pipeline 182

Data Engineering Podcast

SEPTEMBER 10, 2023

Summary Data systems are inherently complex and often require integration of multiple technologies. In this episode Nick Schrock, creator of Dagster, shares his perspective on the state of data orchestration technology and its application to help inform its implementation in your environment.

BI 208

BI SQL Machine Learning Data 208

The Pragmatic Engineer

OCTOBER 10, 2023

I wrote code for drivers on Windows, and started to put a basic observability system in place. EC2 had no observability system back then: people would spin up EC2 instances but have no idea whether or not they worked. With my team, we built the basics of what is now called AWS Systems Manager. We needed $11M to get started.

Software Engineering 293

Software Engineering Software Engineer Architecture Media 293

The Pragmatic Engineer

APRIL 19, 2023

I was sceptical that any system would automatically reject resumes, because I never saw this as a hiring manager. I've observed several hype cycles in tech when people get involved in new areas in technologies, when they lack understanding of how things down. " that hundreds of sites took over on the internet.

Software Engineering 319

Software Engineering Software Engineer Engineering Media 319

Data Engineering Podcast

MAY 21, 2023

The batch world has been the default for years because of the complexities of running a reliable streaming system at scale. In this episode David Yaffe and Johnny Graettinger share the story behind the business and technology and how you can start using it today to build a real-time data lake without all of the headache.

Data Lake 162

Data Lake Machine Learning Kafka Data Warehouse 162

Data Engineering Podcast

JUNE 9, 2024

To address this shortcoming Datorios created an observability platform for Flink that brings visibility to the internals of this popular stream processing system. How have the requirements of generative AI shifted the demand for streaming data systems? What role does Flink play in the architecture of generative AI systems?

Process 147

Process Data Lake High Quality Data Machine Learning 147

Data Engineering Podcast

APRIL 7, 2024

In order to enable this use case, while maintaining a single point of access, the semantic layer has evolved as a technological solution to the problem. What are the benefits of having a discrete service that offers the business metrics/semantic mappings as opposed to implementing those concepts as part of a more general system?

Data Lake 162

Data Lake High Quality Data BI Data Workflow 162

Data Engineering Podcast

OCTOBER 29, 2023

When a team is selecting a database technology for a given task, what are the types of questions that they should be asking? What is driving the broad adoption of columnar stores as a separate environment from transactional systems? When building analytical systems there are numerous moving parts with intricate dependencies.

Database 189

Database SQL BI Machine Learning 189

Data Engineering Podcast

JUNE 2, 2024

Summary Modern businesses aspire to be data driven, and technologists enjoy working through the challenge of building data systems to support that goal. Nicola Askham found her way into data governance by accident, and stayed because of the benefit that she was able to provide by serving as a bridge between the technology and business.

Data Governance 147

Data Governance Government Data Lake High Quality Data 147

Data Engineering Podcast

FEBRUARY 4, 2024

Summary Stream processing systems have long been built with a code-first design, adding SQL as a layer on top of the existing framework. There are numerous stream processing engines, near-real-time database engines, streaming SQL systems, etc. Can you describe what RisingWave is and the story behind it?

SQL 173

SQL Data Lake High Quality Data Machine Learning 173

Data Engineering Podcast

JUNE 30, 2024

Petr shares his journey from being an engineer to founding Synq, emphasizing the importance of treating data systems with the same rigor as engineering systems. He discusses the challenges and solutions in data reliability, including the need for transparency and ownership in data systems. When is Synq the wrong choice?

Pipeline-centric 147

Pipeline-centric Engineering Data Lake High Quality Data 147

The Pragmatic Engineer

MAY 11, 2023

A quick summary of these technologies: Prometheus : a time series database. A very popular open-source solution for systems and services monitoring. A fast and open-source column-oriented database management system, which is a popular choice for log management. It evaluates rules and can trigger alerts.

AWS 342

AWS Software Engineering Software Engineer Google Cloud 342

Engineering at Meta

JANUARY 22, 2025

Data lineage is an instrumental part of Metas Privacy Aware Infrastructure (PAI) initiative, a suite of technologies that efficiently protect user privacy. It is a critical and powerful tool for scalable discovery of relevant data and data flows, which supports privacy controls across Metas systems.

Data Warehouse 109

Data Warehouse SQL Programming Language Data 109

Data Engineering Podcast

MARCH 3, 2024

Summary Artificial intelligence technologies promise to revolutionize business and produce new sources of value. What are the skills and systems that need to be in place to effectively execute on an AI program? "AI" In order to make those promises a reality there is a substantial amount of strategy and investment required.

Programming 162

Programming Data Lake High Quality Data Machine Learning 162

Precisely

JANUARY 9, 2025

Disclaimer: Throughout this post, I discuss a variety of complex technologies but avoid trying to explain how these technologies work. The goal of this post is to understand how data integrity best practices have been embraced time and time again, no matter the technology underpinning. But simply moving the data wasnt enough.

Data Integration 119

Data Integration Hadoop Data Warehouse Data Lake 119

Data Engineering Podcast

OCTOBER 15, 2023

Decodable was built with a mission of eliminating all of the painful aspects of developing and deploying stream processing systems for engineering teams. Contact Info esammer on GitHub LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

Process 182

Process Building SQL BI 182

Snowflake

NOVEMBER 16, 2023

The rise of new technologies such as 5G, cloud computing, and the Internet of Things (IoT) is putting pressure on telecom operators to find new ways to improve the performance of their networks, reduce costs and provide better customer service. These systems can make it difficult and expensive to integrate gen AI solutions.

Cloud Computing 115

Cloud Computing Cloud Systems Technology 115

Data Engineering Podcast

JUNE 16, 2024

In this episode Kevin Liu shares some of the interesting features that they have built by combining those technologies, as well as the challenges that they face in supporting the myriad workloads that are thrown at this layer of their data platform. What are the other systems that feed into and rely on the Trino/Iceberg service?

Data Lake 147

Data Lake High Quality Data Metadata Machine Learning 147

Data Engineering Podcast

MARCH 24, 2024

Contact Info Twitter LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Contact Info Twitter LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

Data Lake 162

Data Lake High Quality Data Hadoop Machine Learning 162

Netflix Tech

JUNE 1, 2023

In this blog post, we’ll discuss the methods we used to ensure a successful launch, including: How we tested the system Netflix technologies involved Best practices we developed Realistic Test Traffic Netflix traffic ebbs and flows throughout the day in a sinusoidal pattern. Basic with ads was launched worldwide on November 3rd.

Algorithm 138

Algorithm Kafka Metadata Systems 138