The Pragmatic Engineer

AUGUST 14, 2024

I (Gergely) sometimes get reachouts to do talks at events in Amsterdam (where I am based,) the Netherlands, or somewhere in Europe. Talks about domain-driven design, event storming, domain storytelling, software architecture, continuous deployments and tech leadership. See his past talks. Maciej Jedrzejewski (Switzerland.)

Software Engineer 130

Software Engineer Software Engineering Education Architecture 130

Snowflake

OCTOBER 9, 2023

Easily collect and store digital events directly to create a complete composable customer data platform (CDP) Marketers are increasingly leveraging the Snowflake Data Cloud as the foundation for all of their customer data analytics and activation.

Cloud 125

Cloud Building SQL Data 125

The Pragmatic Engineer

OCTOBER 18, 2024

And yet, substitute Apple with Automattic, App Store with WordPress.org and Spotify with one of the most popular WordPress plugins: and Automattic’s CEO is accused of orchestrating events similar to above. This event is shameful and unprecedented in the history of open source on the web. Open source theft? Source: X What next?

Engineering 357

Engineering Government Project AWS 357

Netflix Tech

NOVEMBER 12, 2024

By: Rajiv Shringi , Oleksii Tkachuk , Kartik Sathyanarayanan Introduction In our previous blog post, we introduced Netflix’s TimeSeries Abstraction , a distributed service designed to store and query large volumes of temporal event data with low millisecond latencies. Today, we’re excited to present the Distributed Counter Abstraction.

Datasets 100

Datasets Computer Science Systems Kafka 100

Waitingforcode

APRIL 24, 2024

As a data engineer you're certainly familiar with data skew. Yes, this bad phenomena where one task takes considerably more input than the others and often causes unexpected latency or failures. Turns out, stream processing also has its skew but more related to time.

Process 130

Process Data Engineering Data Engineer Engineering 130

Confluent

FEBRUARY 19, 2025

Event-driven architecture can overcome the challenges of coordinating agentic AI agents to create scalable and efficient reasoning systems. See examples of multi-agent patterns.

Systems 90

Systems Architecture Kafka 90

Engineering at Meta

JANUARY 21, 2025

For example, a profiler takes a sample every N events (or milliseconds in the case of time profilers) to understand where that event occurs or what is happening at the moment of that event. With a CPU-cycles event, for example, the profile will be CPU time spent in functions or function call stacks executing on the CPU.

Technology 114

Technology Metadata Utilities Engineering 114

KDnuggets

FEBRUARY 5, 2024

If you are new to generative AI or an expert who wants to learn more, O’Reilly offers a range of resources to kickstart your generative AI journey.

137

137

Snowflake

JANUARY 8, 2025

Compared to the traditional security incident and event management tools, security data lakes are generally more flexible, scalable and cost effective. Within the security data lake, teams can bring machine learning and advanced analytics to bear. And by making it more affordable to hold on to more data longer, teams can run better forensics.

Data Lake 105

Data Lake Data Security Machine Learning Technology 105

The Pragmatic Engineer

DECEMBER 2, 2023

A year ago, I spent months doing an investigative report on how UK events tech company Pollen had its staff work for free, as it had run out of money but still kept operating. So I looked at other conferences by the same organizer, Dev Events. Three featured speakers listed at DevTernity 2021, 2022 and 2023, and JDKon 2024.

Software Engineering 343

Software Engineering Software Engineer Insurance Coding 343

Data Engineering Podcast

APRIL 23, 2023

Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools. You can collect, transform, and route data across your entire stack with its event streaming, ETL, and reverse ETL pipelines.

Data Lake 277

Data Lake Kafka Machine Learning Data Warehouse 277

Analytics Vidhya

JANUARY 30, 2023

Introduction Azure Functions is a serverless computing service provided by Azure that provides users a platform to write code without having to provision or manage infrastructure in response to a variety of events. Azure functions allow developers […] The post How to Develop Serverless Code Using Azure Functions?

Coding 245

Coding Database Management Process 245

Netflix Tech

APRIL 8, 2025

As noted in our previous blog post, our initial attribution approach relied on Sonar , an internal IP address tracking service that emits an event whenever an IP address in Netflixs AWS VPCs is assigned or unassigned to a workload. Additionally, event timestamps may be inaccurate depending on how they are captured.

AWS 75

AWS Kafka Cloud Programming 75

Snowflake

MARCH 19, 2025

These industry-specific virtual events are ideal for IT professionals and business leaders who want to bridge the gap between perception and reality, build robust data foundations and accelerate their AI initiatives. The events will also feature demos of key use cases and best practices. Why attend Accelerate Retail and Consumer Goods?

Retail 96

Retail Media Entertainment Unstructured Data 96

Data Engineering Podcast

FEBRUARY 18, 2024

Join in with the event for the global data community, Data Council Austin. Don't miss out on their only event this year! Data Council Logo]([link] Join us at the event for the global data community, Data Council Austin. Don't miss out on our only event this year! Don't miss out on their only event this year!

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Start Data Engineering

JUNE 14, 2024

Event-driven pipelines 3.5. Introduction 2. Run Data Pipelines 2.1. Run on codespaces 2.2. Run locally 3. Projects 3.1. Projects from least to most complex 3.2. Batch pipelines 3.3. Stream pipelines 3.4. LLM RAG pipelines 4. Conclusion 1.

Data Engineering 239

Data Engineering Data Engineer Project Engineering 239

The Pragmatic Engineer

MARCH 3, 2023

I asked Googlers the reason why these events have been canceled and one thing became clear: most of the program managers who worked on the coding competitions were recently let go in Google’s historic job cuts. So knowing these lead up signs, the cancellation of the event is not that surprising, considering the info we had beforehand.

Coding 218

Coding IT Software Engineer Software Engineering 218

Data Engineering Podcast

JUNE 25, 2023

Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools. You can collect, transform, and route data across your entire stack with its event streaming, ETL, and reverse ETL pipelines.

Data Engineering 225

Data Engineering Data Engineer Python Engineering 225

Engineering at Meta

FEBRUARY 13, 2025

We also celebrated the security research done by our bug bounty community as part of our annual bug bounty summit and many other industry events. As part of our defense-in-depth strategy , we continued to collaborate with the security research community in the areas of GenAI, AR/VR, ads tools, and more.

Programming 109

Programming Designing Accessible Accessibility 109

The Pragmatic Engineer

NOVEMBER 9, 2023

From to this lawsuit, we get an inside look at how events unfolded inside Frank. Pollen: an engineer told to double charge customers by the CEO Last year, I published my first – and to date only– investigative article on how events tech startup Pollen raised $200M and then collapsed , owing months of wages to staff.

Software Engineering 290

Software Engineering Software Engineer Engineering Coding 290

Netflix Tech

MARCH 28, 2025

To harness this data effectively, we employ a process of interaction tokenization, ensuring meaningful events are identified and redundancies are minimized. Even with such strategies, interaction histories from active users can span thousands of events, exceeding the capacity of transformer models with standard self attention layers.

Metadata 80

Metadata Bytes Data Mining Entertainment 80

Cloudera

JANUARY 7, 2025

Airports are an interconnected system where one unforeseen event can tip the scale into chaos. Not all the time, but thats why we support this broader thinking with data so people can plan for erroneous events and better understand the shifts. That’s part of the group that brings in events into Halifax.

Finance 81

Finance Building Cloud Engineering 81

Seattle Data Guy

JANUARY 17, 2023

With all the recent data events I have put together I inevitably run into new data engineers who are either finishing up college or looking to transition into a data engineer or data scientist position. In fact I have talked to several newly graduated engineers who are struggling to find work.

Data Engineer 223

Data Engineer Data Engineering Engineering Data 223

Analytics Vidhya

MARCH 7, 2023

Introduction Apache Flume is a tool/service/data ingestion mechanism for gathering, aggregating, and delivering huge amounts of streaming data from diverse sources, such as log files, events, and so on, to centralized data storage. Flume is a tool that is very dependable, distributed, and customizable. einsteinerupload of.

Data Ingestion 223

Data Ingestion Data Storage Hadoop Data 223

Netflix Tech

FEBRUARY 14, 2025

Collecting Raw Impression Events As Netflix members explore our platform, their interactions with the user interface spark a vast array of raw events. These events are promptly relayed from the client side to our servers, entering a centralized event processing queue.

Kafka 66

Kafka Datasets Metadata Utilities 66

Snowflake

APRIL 2, 2025

Learn more Join Snowflake at Iceberg Summit , a two-day event taking place in San Francisco on April 8 and virtually April 9. We are excited to support the community as a headlining sponsor for the inaugural event.

Data Lake 97

Data Lake Metadata Cloud Storage Data Warehouse 97

The Pragmatic Engineer

APRIL 11, 2024

Confirming this suspicion is Fabrick’s status page that says Fabrick’s tech staff are involved: “We would like to inform you that the event is currently still ongoing, continuous monitoring is in place, and any slowdowns are affecting only the services provided jointly with the Sella group.

Banking 271

Banking Utilities Database Engineering 271

Seattle Data Guy

JANUARY 17, 2023

Our team is putting together an all day event focused on helping answer some… Read more The post What Is The State Of Data Engineering And Infrastructure In 2023 appeared first on Seattle Data Guy. Are Snowflake and Databricks still fighting over total cost of ownership? Is everyone switching to DuckDB?

Data Engineer 211

Data Engineer Data Engineering Engineering Data 211

Edureka

APRIL 22, 2025

Enter Amazon EventBridge, a fully managed serverless event bus service that makes it easier to build event-driven applications using data from your AWS services, custom applications, or SaaS providers. It is a fully managed, serverless event bus service that allows applications to communicate with each other using events.

AWS 40

AWS Architecture Media Cloud 40

The Pragmatic Engineer

OCTOBER 10, 2024

  The  end of the latest zero interest rate period  (ZIRP) has been one of biggest events in the tech industry in the past 20 years, by slowing down the growth of tech companies, and leading valuations to drop. In 2021, it was valued at $7.5B. Zero percent interest rates have ended: profits matter now. 

Consulting 347

Consulting AWS Engineering Software Engineer 347

Confluent

JANUARY 14, 2025

PodPrep AI, an AI-powered research assistant, leverages EDA and real-time streaming data using Confluent and Flink, in order to help its author with podcast preparation.

Kafka 80

Kafka Building IT Data 80

Snowflake

NOVEMBER 4, 2024

Get hands-on with tools like pandas, Document AI and Snowflake Notebooks Up-close, hands-on sessions and demos — created for builders, by builders — is what sets this event apart from other dev conferences. Go in-depth on some of Snowflake’s most popular features, like Document AI.

Unstructured Data 105

Unstructured Data Python Machine Learning Data Pipeline 105

Snowflake

NOVEMBER 26, 2024

Processing some 90,000 tables per day, the team oversees the ingestion of more than 100 terabytes of data from upward of 8,500 events daily. Serving a company that has games available in more than 190 countries and employs more than 8,000 people, its data engineering team is always running. million in cost savings annually.

Data Warehouse 105

Data Warehouse Cloud PostgreSQL Hadoop 105

Data Engineering Podcast

DECEMBER 24, 2023

Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. In the event of these different cluster errors, what are the strategies for mitigating and recovering from those failures? Operating it at scale, however, is notoriously challenging.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Snowflake

FEBRUARY 11, 2025

Join David Fisher, Industry Principal Media & Entertainment UK & EMEA, Snowflake; David Wells, Industry Principal AdTech & MarTech, Snowflake; and Erin Foxworthy, Industry Principal Agencies & Advertisers, Snowflake, for the 2025 AI + Data Predictions for Advertising, Media and Entertainment event.

Entertainment 96

Entertainment Media Healthcare Technology 96

Snowflake

JUNE 10, 2024

Leveraging Snowflake’s Query History, Event Tables , Alerts and Notifications as the telemetry foundation, Snowflake Trail provides enhanced visibility into data quality, pipelines and applications. Events are all within Snowflake with no need for additional data transfer. Explore logs in your event table easily within Snowsight.

Python 122

Python Java Hadoop Coding 122