The Pragmatic Engineer

JULY 20, 2023

Venture funding is on a downward trend , and we seem to be at the start – or the middle – of a “startup purge” event. This news is hot off the press, publicly announced by Postman and by Akita yesterday, and you are among the early ones to hear about this event. We pivoted to API observability in 2020.

Building 204

Building Programming Language Programming Python 204

Data Engineering Podcast

OCTOBER 1, 2023

Now that AI has reached the level of sophistication seen in the various generative models it is being used to build new ETL workflows. In this episode Jay Mishra shares his experiences and insights building ETL pipelines with the help of generative AI. How can you get the best results for your use case?

Building 162

Building BI SQL Machine Learning 162

Data Engineering Podcast

NOVEMBER 26, 2019

Summary The team at Sentry has built a platform for anyone in the world to send software errors and events. Go to dataengineeringpodcast.com/conferences to learn more about these and other events, and take advantage of our partner discounts to save money when you register today. What did the previous system look like?

Data Warehouse 100

Data Warehouse Building PostgreSQL Kafka 100

Netflix Tech

FEBRUARY 18, 2022

To this end, we developed a Rapid Event Notification System (RENO) to support use cases that require server initiated communication with devices in a scalable and extensible manner. In this blog post, we will give an overview of the Rapid Event Notification System at Netflix and share some of the learnings we gained along the way.

Systems 133

Systems Architecture Portfolio Designing 133

Snowflake

OCTOBER 30, 2023

We are excited to announce the general availability of Snowflake Event Tables for logging and tracing, an essential feature to boost application observability and supportability for Snowflake developers. Developers can then query Event Tables to troubleshoot their applications or gain insights into performance and code behavior.

Java 98

Java Scala Hadoop Data Ingestion 98

Data Engineering Podcast

OCTOBER 15, 2023

Summary Building streaming applications has gotten substantially easier over the past several years. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. How can you get the best results for your use case?

Process 182

Process Building SQL BI 182

Precisely

APRIL 3, 2024

The Precisely Women in Technology (PWIT) network was first established to bring the women of Precisely together to create more opportunities for learning and engagement. Throughout the years, the program has grown, and it now provides mentorship opportunities, a book club, networking events, and more. How have you overcome them?

Technology 59

Technology Programming Project Engineering 59

LinkedIn Engineering

OCTOBER 19, 2023

Authors: Bingfeng Xia and Xinyu Liu Background At LinkedIn, Apache Beam plays a pivotal role in stream processing infrastructures that process over 4 trillion events daily through more than 3,000 pipelines across multiple production data centers.

Process 119

Process Lambda Architecture Kafka Machine Learning 119

Knowledge Hut

DECEMBER 26, 2023

Technology has come a long way. With UPI, even a vegetable vendor or a small business owner can access high-end new technology (Newest tech). Top 10 Latest Technology Trends in 2024 It is no surprise that the demand for tech-savvy professionals is increasing as the world gets digital.

Technology 98

Technology Cloud Computing Healthcare Data Science 98

Data Engineering Podcast

APRIL 16, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. How have the recent breakthroughs in large language models (LLMs) improved your ability to build features in Zenlytic? Who are the target users?

Business Intelligence 147

Business Intelligence Building Data Lake BI 147

Data Engineering Podcast

JANUARY 28, 2024

Summary Monitoring and auditing IT systems for security events requires the ability to quickly analyze massive volumes of unstructured log data. Contact Info LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? When is Scanner the wrong choice?

Data Lake 147

Data Lake Building High Quality Data AWS 147

Data Engineering Podcast

MARCH 5, 2023

It's supposed to make building smarter, faster, and more flexible data infrastructures a breeze. By bringing all the layers of the data stack together, TimeXtender helps you build data solutions up to 10 times faster and saves you 70-80% on costs. What do you have planned for the future of this topic at Data Council events?

Building 147

Building Database Design Machine Learning Metadata 147

Data Engineering Podcast

JUNE 11, 2023

Summary Data engineering is all about building workflows, pipelines, systems, and interfaces to provide stable and reliable data. In this episode Gleb Mezhanskiy shares some valuable advice and insights into how you can build reliable and well-tested data assets with dbt and data-diff. RudderStack also supports real-time use cases.

Project 130

Project Building Data Lake Machine Learning 130

François Nguyen

MARCH 22, 2021

Just an illustration – not the truth and you certainly can do it with other technologies. How do we build data products ? There are certainly many other ways to do the same with another technologies. More than the technology chosen, it is more interesting to talk about the spirit behind these choices.

Technology 130

Technology Architecture Google Cloud Metadata 130

Workfall

SEPTEMBER 26, 2023

Reading Time: 9 minutes In this blog, we will cover: What are Server-Sent Events? Why Stream Data Using Server-Sent Events (SSE)? Hands-On Conclusion What are Server-Sent Events? These events can carry information like notifications, updates, or alerts. What is FastAPI?

Python 85

Python Bytes Coding Project 85

Data Engineering Podcast

AUGUST 10, 2020

Summary Event based data is a rich source of information for analytics, unless none of the event structures are consistent. The team at Iteratively are building a platform to manage the end to end flow of collaboration around what events are needed, how to structure the attributes, and how they are captured.

Data Collection 100

Data Collection Data Engineering Data Engineer Education 100

Data Engineering Podcast

NOVEMBER 6, 2022

In this episode Nick King discusses how you can be intentional about data creation in your applications and services to reduce the friction and errors involved in building data products and ML applications. What are the organizational patterns that are required to support effective workflows for building data generation capabilities?

Building 130

Building IT Metadata MongoDB 130

Cloudera

SEPTEMBER 26, 2023

These solutions include on-premises and public cloud offerings with the data management, security and governance needed to build an enterprise data and analytic big data solution. The post IBM Technology Chooses Cloudera as its Preferred Partner for Addressing Real Time Data Movement Using Kafka appeared first on Cloudera Blog.

Kafka 93

Kafka Technology IT Government 93

Workfall

JUNE 27, 2023

In this blog, we’ll explore how to harness the power of Kafka to streamline event streaming within a microservices architecture and unlock its full potential for building scalable and responsive systems. In the event of server failure, Kafka can autonomously restart itself, ensuring uninterrupted data flow. Let’s get started!

Kafka 75

Kafka Architecture AWS Transportation 75

Striim

NOVEMBER 8, 2023

Data Mesh is revolutionizing event streaming architecture by enabling organizations to quickly and easily integrate real-time data, streaming analytics, and more. In this article, we will explore the advantages and limitations of data mesh, while also providing best practices for building and optimizing a data mesh with Striim.

Architecture 52

Architecture Generalist Government Datasets 52

Lyft Engineering

JUNE 28, 2023

On the flip side, there was a substantial appetite to build real-time ML systems from developers at Lyft. The Event Driven Decisions capability in particular turned out to be general enough as to be applicable to a wide range of use cases. One key component is the Analytics Event Abstraction layer.

Machine Learning 116

Machine Learning Building Metadata Kafka 116

Snowflake

OCTOBER 2, 2023

Building resiliency into every element with Snowgrid At DTCC, the notion of resiliency is built into all our initiatives, including how we go about modernizing our applications. We have conducted at least 15 disaster recovery exercises using Snowgrid technology for business continuity. trillion per day in U.S. Curious about Snowgrid?

AWS 91

AWS Technology Data Cloud 91

Confluent

FEBRUARY 27, 2019

What is more, as the world adopts the event-driven streaming architecture, how does it fit with serverless? Serverless functions provide a synergistic relationship with event streaming applications; they behave differently with respect to streaming workloads but are both event driven. Event-first FaaS? What is FaaS?

Kafka 109

Kafka AWS Architecture Cloud 109

Precisely

DECEMBER 28, 2023

Enterprise technology is having a watershed moment; no longer do we access information once a week, or even once a day. This article explores what streaming data pipelines are, how they work, and how to build this data pipeline architecture. But insights derived from day-old data don’t cut it. Now, information is dynamic.

Data Pipeline 91

Data Pipeline Building Kafka Big Data 91

Towards Data Science

FEBRUARY 9, 2024

How to build a modern, scalable data platform to power your analytics and data science projects (updated) Table of Contents: What’s changed? Orchestration I mentioned modularity as a core concept of building a modern data platform in my 2021 article, but I failed to emphasize the importance of data orchestration.

Building 78

Building Transportation Data Lake Metadata 78

Data Engineering Podcast

FEBRUARY 18, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Join in with the event for the global data community, Data Council Austin. Don't miss out on their only event this year!

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Confluent

MAY 9, 2019

Event-first thinking enables us to build a new atomic unit: the event. This model is completely free form, we can build anything provided that we apply mechanical sympathy with the underlying system behavior. Four pillars of event streaming. Building the KPay payment system. Putting it all together.

Kafka 94

Kafka Pipeline-centric Architecture Database-centric 94

Snowflake

OCTOBER 2, 2023

To help customers overcome these challenges, RudderStack and Snowflake recently launched Profiles , a new product that allows every data team to build a customer 360 directly in their Snowflake Data Cloud environment. Now teams can leverage their existing data engineering tools and workflows to build their customer 360.

Cloud 106

Cloud Building Insurance Data Engineering 106

Pinterest Engineering

JANUARY 4, 2024

In order to build a distributed and replicated service using RocksDB, we built a real time replicator library: Rocksplicator. Maintaining these disparate systems and building common functionality among them was adding a huge overhead to the teams. RocksDB is a single node key value store.

Database 93

Database Building Datasets Relational Database 93

Snowflake

DECEMBER 18, 2023

Data clean rooms have emerged as the technology to meet this need, enabling interoperability where multiple parties can collaborate on and analyze sensitive data in a governed way without exposing direct access to the underlying data and business logic. Snowflake’s acquisition of Samooha is subject to customary closing conditions.

Cloud 118

Cloud Building Entertainment Government 118

Knowledge Hut

MAY 20, 2024

Scala is now being used by many companies and individuals to build their own new ideas. Scala is now the next wave of computation engines and more importance has been given to the speed processing rather than the size of the batch, and the ability to process event streaming in real-time.

Scala 52

Scala Technology Kafka Hadoop 52

Workfall

SEPTEMBER 12, 2023

This makes it a powerful tool for building interactive and dynamic web applications that require instant data exchange between clients and servers, such as chat applications, online gaming, collaborative tools, and more. Its user-friendly API simplifies development, embracing an event-driven structure for custom interactions.

Building 89

Building MongoDB Programming Language Coding 89

Cloudera

SEPTEMBER 13, 2023

Coupled with amazing technology was an amazing team that only grew and improved with the merger with Cloudera. It’s the Apache Software Foundation’s biggest event of the year. If a career in innovative technology is what you want and you’re excited about the open-source community, Cloudera might be the place for you.

Technology 83

Technology Hadoop Kafka Project 83

LinkedIn Engineering

AUGUST 3, 2023

As I worked closely with iOS, I wanted to learn everything I could about the UI and dedicated my time to coding files and researching how to build apps. While browsing one day, I found an opportunity at LinkedIn for a software engineer on the events team. For example, if I wanted to learn more about what's happening in the U.S.

Building 68

Building Software Engineer Software Engineering Computer Science 68

Snowflake

APRIL 20, 2023

Snowflake users are already taking advantage of LLMs to build really cool apps with integrations to web-hosted LLM APIs using external functions , and using Streamlit as an interactive front end for LLM-powered apps such as AI plagiarism detection , AI assistant , and MathGPT. Join us in Vegas at our Summit to learn more.

Building 118

Building Unstructured Data Government Coding 118

Knowledge Hut

NOVEMBER 19, 2023

With advances in technology, wearable devices do provide some trace of your health. Furthermore, Data Scientists analyze this data, build reports and train ML models on top of it. This data can be used to build reports (batch or real-time) and train Machine Learning models. How do they collect, process, and analyze data for you?

Data Science 52

Data Science Technology Machine Learning Big Data 52

Confluent

FEBRUARY 13, 2019

Part 1 of this series discussed why you need to embrace event-first thinking, while this article builds a rationale for different styles of event-driven architectures and compares and contrasts scaling, persistence and runtime models. Event-driven architecture. Event-driven, reactive architecture. Data evolution.

Architecture 92

Architecture Programming Kafka Database-centric 92