The Pragmatic Engineer

JANUARY 21, 2025

In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. Middle manager burnout incoming?    A Forbes article suggests a broader middle manager burnout to come across most professional sectors.

Software Engineering 339

Software Engineering Software Engineer Engineering Coding 339

Precisely

FEBRUARY 6, 2025

The traditional ways of operations management are over modernization and holistic approaches are now essential. Success in tackling modernization of IT operations management starts with assessing where your team is. The post New Year, New Approaches to Tackling IT Operations Management appeared first on Precisely. Whats next?

IT 64

IT Management Datasets Systems 64

Precisely

FEBRUARY 26, 2025

Lets take a closer look at these exciting innovations and explore how theyll help you tackle six top data management challenges. Automated metadata management – AI-generated catalog asset descriptions significantly reduce manual efforts and improve metadata quality – enabling teams to focus on more strategic tasks.

Data Integration 64

Data Integration Data Management Management Data Governance 64

KDnuggets

JANUARY 28, 2025

Python environment management can sometimes give you that awful feeling in the pit of your stomach. So don't do it: just use Docker containers.

Python 137

Python Management AWS IT 137

Start Data Engineering

MARCH 1, 2025

Introduction 2.Strategies Strategies for data teams to handle changing schemas 2.1. Meetings are the most straightforward approach 2.2. Upstream dumps data, data team deals with it 2.3. The data team as upstream reviewer leads to issue prevention 2.4. Validating input before processing saves on debug time 3. Conclusion 4. Recommended reading 1.

Data Schemas 130

Data Schemas Management Data Process 130

The Pragmatic Engineer

OCTOBER 10, 2024

Wordpress is the most popular content management system (CMS),  estimated  to power around 43% of all websites; a staggering number! Automattic generates most of its revenue by offering managed Wordpress hosting. WP Engine also sells managed Wordpress services, making it a direct competitor to Automattic.

Consulting 347

Consulting AWS Engineering Software Engineering 347

Simon Späti

JULY 28, 2024

A Vim-Inspired Approach to Efficient Note Management with Obsidian and Markdown

Management 361

Management 361

Analytics Vidhya

DECEMBER 30, 2024

GenAI is already transforming how data is managed, analyzed, and utilized, paving the way for […] The post Top 11 GenAI Powered Data Engineering Tools to Follow in 2025 appeared first on Analytics Vidhya. How will generative AI shape the tools and processes Data Engineers rely on today?

Data Engineering 145

Data Engineering Data Engineer Engineering Utilities 145

KDnuggets

FEBRUARY 20, 2025

Whatever role is best for youdata scientist, data engineer, or technology managerNorthwestern University's MS in Data Science program will help you to prepare for the jobs of today and the jobs of the future.

Technology 127

Technology Data Engineering Data Engineer Engineering 127

Precisely

JANUARY 9, 2025

In the beginning, there was a data warehouse The data warehouse (DW) was an approach to data architecture and structured data management that really hit its stride in the early 1990s. There was no easy way to consolidate and analyze this data to more effectively manage our business. Data management best practices havent changed.

Data Integration 119

Data Integration Hadoop Data Warehouse Data Lake 119

The Pragmatic Engineer

SEPTEMBER 19, 2024

Without the backing of management, a large-scale rewrite is likely to fail. In 2004, I was hired by ISO-NE, a non-profit that manages the electric grid in New England. They hired a manager who had done this kind of project before, and set a target date of nearly two years. Big rewrites need heavyweight support.

Recruitment 297

Recruitment Java Coding Project 297

The Pragmatic Engineer

OCTOBER 18, 2024

Corporate conflict recap Automattic is the creator of open source WordPress content management system (CMS), and WordPress powers an incredible 43% of webpages and 65% of CMSes. But leveraging a supposedly neutral platform (the WordPress plugin manager) should not be the way to win in business – at least not in open source.

Engineering 357

Engineering Government Project AWS 357

Snowflake

MARCH 31, 2025

Despite the best efforts of many ML teams, most models still never make it to production due to disparate tooling, which often leads to fragmented data and ML pipelines and complex infrastructure management. Snowflake has continuously focused on making it easier and faster for customers to bring advanced models into production.

Healthcare 96

Healthcare Medical Government Food 96

The Pragmatic Engineer

MAY 25, 2023

In every issue, I cover topics related to Big Tech and high-growth startups through the lens of engineering managers and senior engineers. ” I managed to talk to someone in this company’s HR department, who confirmed that the leadership set a goal to improve the business’s Glassdoor rating.

Software Engineering 354

Software Engineering Software Engineer AWS Engineering 354

Seattle Data Guy

APRIL 21, 2025

On the surface BigQuery is Google Clouds fully-managed, serverless data warehouse. If you work in data, then youve likely used BigQuery and youve likely used it without really thinking about how it operates under the hood. Its the Redshift of GCP except we like it a little more. The question becomes, how does it work?…

IT 130

IT Google Cloud Data Warehouse Data 130

The Pragmatic Engineer

MAY 12, 2023

OpsGenie is Atlassian's incident management tool, which is widespread thanks to distribution. Incident management. At incident.io, we believe that we’re the most sensible incident response and management tool for companies looking to do more than just alert. However, there is little awareness of these.

Systems 284

Systems Management Engineering IT 284

Snowflake

JANUARY 8, 2025

AI is both a contributor to the problem more data to secure, more attack surface and a potential boon, providing tools to manage amounts of data that humans cant grasp on their own. Compared to the traditional security incident and event management tools, security data lakes are generally more flexible, scalable and cost effective.

Data Lake 105

Data Lake Data Security Machine Learning Technology 105

The Pragmatic Engineer

MAY 11, 2023

Prometheus can be self-hosted, but several cloud providers also offer managed Prometheus services: both Google Cloud and AWS have this service in production, while Azure has it in preview. Source: Grafana.org Clickhouse : log management. Grafana : the frontend for visualizing metrics. Shopify letting go most staff in Germany.

AWS 347

AWS Software Engineering Software Engineer Google Cloud 347

Data Engineering Weekly

DECEMBER 28, 2024

Investment in an Agent Management System (AMS) is crucial, as it offers a framework for scaling, monitoring, and refining AI agents. AI engineers, in particular, will find their skills in high demand as they navigate managing and optimizing agents to ensure reliability within enterprise systems.

Unstructured Data 113

Unstructured Data Metadata Data Government 113

Waitingforcode

FEBRUARY 18, 2025

Using cloud managed services is often a love and hate story. On one hand, they abstract a lot of tedious administrative work to let you focus on the essentials. From another, they often have quotas and limits that you, as a data engineer, have to take into account in your daily work.

Data Engineer 130

Data Engineer Data Engineering Cloud Data 130

The Pragmatic Engineer

DECEMBER 18, 2023

I get asked every now and then if I offer 1:1 mentoring for either software engineers or engineering managers or leaders. I do believe great mentors are useful for professional growth , and have benefitted both informal and formal mentorship from experienced engineers and managers.

Software Engineering 265

Software Engineering Software Engineer Engineering Management 265

Snowflake

APRIL 2, 2025

Without adopting Iceberg tables, data teams are forced to spend significant time and resources managing migrations and governance before being able to capture the opportunities new technologies and solutions offer. Apache Hive) encounter due to their strict reliance on a physical directory structure. Why should you care about Iceberg?

Data Lake 97

Data Lake Cloud Storage Metadata Data Warehouse 97

The Pragmatic Engineer

JUNE 29, 2023

In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. They’ll need to familiarize themselves with Squarespace’s domain management capabilities. It’s been a domain registrar since 2020, and already manages 1 million domain names.

AWS 65

AWS Software Engineering Software Engineer Engineering 65

The Pragmatic Engineer

SEPTEMBER 22, 2023

The npm or pnpm package managers do this additional check and so are slower, though arguably more reliable. For most commercial open source companies, the approach tends to be to offer a managed, dedicated service of the open source software, or to provide additional features on top of it.

Programming Language 316

Programming Language Project Coding Engineering 316

The Pragmatic Engineer

OCTOBER 17, 2024

In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. Internal comms: Chat: Slack Coordination / project management: Linear 3. Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. To get full issues twice a week, subscribe here.

Cloud 332

Cloud AWS Metadata Cloud Computing 332

Data Engineering Podcast

APRIL 23, 2023

In this episode DeVaris Brown discusses the types of applications that are possible when teams don't have to manage the complex infrastructure necessary to support continuous data flows. Can you describe what Meroxa is and the story behind it? How have the focus and goals of the platform and company evolved over the past 2 years?

Data Lake 277

Data Lake Kafka Machine Learning Data Warehouse 277

Precisely

OCTOBER 31, 2024

Key Takeaways: Data mesh is a decentralized approach to data management, designed to shift creation and ownership of data products to domain-specific teams. Data fabric is a unified approach to data management, creating a consistent way to manage, access, and share data across distributed environments.

Data Architecture 69

Data Architecture Architecture Metadata Government 69

Analytics Vidhya

MARCH 5, 2023

A distributed file system runs on commodity hardware and manages massive data collections. It is a fully managed cloud-based environment for analyzing and processing enormous volumes of data. Introduction Microsoft Azure HDInsight(or Microsoft HDFS) is a cloud-based Hadoop Distributed File System version.

Hadoop 254

Hadoop Data Collection Cloud Technology 254

Snowflake

NOVEMBER 19, 2024

Managed Apache Spark environments — such as Databricks, Amazon EMR, and certain Cloudera deployments — can present teams with a plethora of pain points, which may include complexity, unpredictable costs, security concerns, or performance issues. But managing complex infrastructure diverted data teams from model building, causing delays.

Data Governance 98

Data Governance Government Healthcare Building 98

The Pragmatic Engineer

NOVEMBER 21, 2023

In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. After Zynga, he rejoined Amazon, and was the General Manager (GM) for Compute services at AWS, and later chief of staff, and advisor to AWS executives like Charlie Bell and Andy Jassy (Amazon’s current CEO.)

Engineering 345

Engineering Bytes Cloud Computing AWS 345