The Pragmatic Engineer

NOVEMBER 21, 2023

If you had a continuous deployment system up and running around 2010, you were ahead of the pack: but today it’s considered strange if your team would not have this for things like web applications.  Joshua is currently VP of Product & Strategy at VMware, a cloud computing and virtualization technology company.

Engineering 302

Engineering Bytes Cloud Computing AWS 302

Data Engineering Podcast

DECEMBER 24, 2023

Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. In the event of these different cluster errors, what are the strategies for mitigating and recovering from those failures? Operating it at scale, however, is notoriously challenging.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Knowledge Hut

MAY 6, 2024

Applying systems thinking views a system as a set of interconnected and interdependent components defined by its limits and more than the sum of their parts (subsystems). When one component of a system is altered, the effects frequently spread across the entire system. are the main objectives of systems thinking.

Systems 98

Systems Transportation Certification Healthcare 98

Data Engineering Podcast

DECEMBER 25, 2022

Recognizing this shortcoming and the capabilities that could be unlocked by a robust solution Rishabh Poddar helped to create Opaque Systems as an outgrowth of his PhD studies. Can you describe what you are building at Opaque Systems and the story behind it? Sign up free at dataengineeringpodcast.com/rudder Build Data Pipelines.

Machine Learning 130

Machine Learning Systems Data Lake Data Warehouse 130

Knowledge Hut

MARCH 22, 2024

I have comprehensively analyzed the area of physical security, particularly the ongoing discussion surrounding fail safe vs fail-safe secure electric strike locking systems. On the other hand, fail-secure systems focus on maintaining continuous security, keeping doors locked even in difficult conditions to protect assets.

Systems 105

Systems Electronics Hospitality Architecture 105

Data Engineering Podcast

APRIL 24, 2022

WhyLogs is a powerful library for flexibly instrumenting all of your data systems to understand the entire lifecycle of your data from source to productionized model. Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool.

Machine Learning 100

Machine Learning Systems Data Lake Java 100

Data Engineering Podcast

MARCH 3, 2024

Summary Artificial intelligence technologies promise to revolutionize business and produce new sources of value. Join us at the top event for the global data community, Data Council Austin. Don't miss out on our only event this year! Visit dataengineeringpodcast.com/data-council and use code dataengpod20 to register today!

Programming 162

Programming Data Lake High Quality Data Data Pipeline 162

Snowflake

OCTOBER 2, 2023

The case for business continuity is particularly compelling for a company such as The Depository Trust & Clearing Corporation (DTCC) , which is designated as a systemically important financial market utility (SIFMU), a U.S. We have conducted at least 15 disaster recovery exercises using Snowgrid technology for business continuity.

AWS 93

AWS Technology Data Cloud 93

Data Engineering Podcast

MAY 21, 2023

The batch world has been the default for years because of the complexities of running a reliable streaming system at scale. In this episode David Yaffe and Johnny Graettinger share the story behind the business and technology and how you can start using it today to build a real-time data lake without all of the headache.

Data Lake 162

Data Lake Machine Learning Kafka Data Warehouse 162

Precisely

DECEMBER 16, 2024

Modern IT environments require comprehensive data for successful AIOps, that includes incorporating data from legacy systems like IBM i and IBM Z into ITOps platforms. As technology continues its rapid ongoing evolution, IT environments have become increasingly complex which leaves businesses needing to adapt at unprecedented speeds.

Systems 52

Systems IT Machine Learning Insurance 52

Netflix Tech

JUNE 1, 2023

In this blog post, we’ll discuss the methods we used to ensure a successful launch, including: How we tested the system Netflix technologies involved Best practices we developed Realistic Test Traffic Netflix traffic ebbs and flows throughout the day in a sinusoidal pattern. We used Elasticsearch dashboards to analyze results.

Algorithm 140

Algorithm Metadata Kafka Systems 140

Data Engineering Podcast

MAY 28, 2023

Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools. Have you found it more practical to hire outside talent to build out the first data systems, or grow that talent internally?

Data Lake 162

Data Lake Machine Learning Data Warehouse Education 162

Data Engineering Podcast

MARCH 17, 2024

Join us at the top event for the global data community, Data Council Austin. Don't miss out on our only event this year! Contact Info LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Don't miss out on our only event this year!

Database 147

Database Data Lake High Quality Data Data Workflow 147

Data Engineering Podcast

MARCH 10, 2024

Join us at the top event for the global data community, Data Council Austin. Don't miss out on our only event this year! Contact Info LinkedIn Twitter Alex's Article on Dremio's Blog Alex's Substack Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

Data Lake 147

Data Lake High Quality Data Data Pipeline Architecture 147

Data Engineering Podcast

MARCH 24, 2024

Contact Info Twitter LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Contact Info Twitter LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

Data Lake 162

Data Lake High Quality Data Hadoop Data Pipeline 162

The Pragmatic Engineer

OCTOBER 10, 2023

I wrote code for drivers on Windows, and started to put a basic observability system in place. EC2 had no observability system back then: people would spin up EC2 instances but have no idea whether or not they worked. With my team, we built the basics of what is now called AWS Systems Manager. It was a lot of fun!

Software Engineer 268

Software Engineer Software Engineering Architecture Media 268

Data Engineering Podcast

MARCH 18, 2023

Summary As with all aspects of technology, security is a critical element of data applications, and the different controls can be at cross purposes with productivity. He also explains why data security is distinct from application security and some methods for reducing the challenge of working across different data systems.

Data Security 130

Data Security Machine Learning Data Science Metadata 130

Data Engineering Podcast

OCTOBER 15, 2023

Decodable was built with a mission of eliminating all of the painful aspects of developing and deploying stream processing systems for engineering teams. Contact Info esammer on GitHub LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

Process 182

Process Building SQL BI 182

Data Engineering Podcast

APRIL 16, 2023

As the capabilities of these systems has improved and become more accessible, the target of what self-serve means changes. Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools.

Business Intelligence 147

Business Intelligence Building Data Lake BI 147

Data Engineering Podcast

MAY 7, 2023

The challenge is that most companies have a multitude of systems that contain fragments of the customer's interactions and stitching that together is complex and time consuming. Segment created the Unify product to reduce the burden of building a comprehensive view of customers and synchronizing it to all of the systems that need it.

Pipeline-centric 130

Pipeline-centric Data Lake Machine Learning Data Warehouse 130

Cloudera

MARCH 6, 2024

As we enter into a new month, the Cloudera team is getting ready to head off to the Gartner Data & Analytics Summit in Orlando, Florida for one of the most important events of the year for Chief Data Analytics Officers (CDAOs) and the field of data and analytics. Add a Human To The Loop: An Introduction to RLHF & DPO.

Data Analytics 114

Data Analytics Data Technology Data Engineering 114

Precisely

MAY 30, 2024

For IBM AIX and IBM i systems, solutions like Assure MIMIX provide seamless cloud transition, real-time data replication, and flexible recovery options to ensure continuous system protection. Are you considering leveraging the cloud alongside your existing IBM AIX and IBM i systems infrastructure?

Systems 69

Systems Cloud Cloud Computing Education 69

Data Engineering Podcast

JUNE 11, 2023

Summary Data engineering is all about building workflows, pipelines, systems, and interfaces to provide stable and reliable data. Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools. Sponsored By: Rudderstack : ![Rudderstack]([link]

Project 130

Project Building Data Lake Machine Learning 130

Data Engineering Podcast

SEPTEMBER 17, 2023

Linked data technologies provide a means of tightly coupling metadata with raw information. what are the characteristics that distinguish a knowledge graph from What are the layers/stages of applications and data that can/should incorporate JSON-LD as the representation for records and events?

Building 189

Building SQL BI Python 189

Jesse Anderson

DECEMBER 12, 2022

Google looked over the expanse of the growing internet and realized they’d need scalable systems. They were the first companies to commercialize open source big data technologies and pushed the marketing and commercialization of Hadoop. We lacked a scalable pub/sub system. Some people blamed the technologies.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Edureka

JUNE 7, 2024

In this article, we will take a closer look at two of the most talked about and widely used AI technologies of 2024 – generative AI and predictive AI. Table of Contents Generative AI vs Predictive AI – Comparison Table Generative AI 101: A Revolutionary Cocktail of Technology and Art How Does Generative AI Work?

Deep Learning 114

Deep Learning Media Manufacturing Algorithm 114

Cloudera

FEBRUARY 10, 2022

Apache NiFi’s rich processor library provides Azure focused processors like ADLS Gen2, Event Hub, Blob Storage or Cosmos DB out of the box. Figure 2: Moving application log data from Azure Event Hub to ADLS Gen2 and SIEM systems. Figure 3: Moving data from Azure Event Hub to ADLS Gen2. SIEM Optimization.

Cloud 121

Cloud Kafka AWS Data Ingestion 121

Data Engineering Podcast

JUNE 17, 2023

In data systems one of the core architectural exercises is data modeling, which can have significant impacts on what is and is not possible for downstream use cases. Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools.

Data Lake 130

Data Lake Machine Learning Metadata Data Architecture 130

Engineering at Meta

OCTOBER 22, 2024

Traditionally, WhatsApp has lacked the ability to store your contact list in a way that can be easily and automatically restored in the event you lose it. IPLS incorporates new privacy technology that protects your contact lists in a privacy-preserving fashion.

Consulting 109

Consulting Accessible Accessibility Technology 109

Data Engineering Podcast

NOVEMBER 6, 2022

He also describes the considerations involved in bringing behavioral data into your systems, and the ways that he and the rest of the Snowplow team are working to make that an easy addition to your platforms. What technical systems are required to generate and collect those interactions? Closing Announcements Thank you for listening!

Building 130

Building IT Metadata MongoDB 130

Data Engineering Podcast

JUNE 4, 2023

Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools. How have the design and goals of the system changed since you first started working on it? What does CI/CD look like for a data warehouse?

Data Engineering 130

Data Engineering Data Engineer Engineering Data Lake 130

Cloudera

DECEMBER 18, 2024

To amplify and support women’s voices, Cloudera has introduced the Women Leaders in Technology Initiative. Ive been fortunate to work with some of the most transformative brands in the technology industry. Having launched similar initiatives at HPE and ASG Technologies, Ive seen firsthand the transformative power they can have.

Technology 53

Technology Insurance Banking Programming 53

Christophe Blefari

AUGUST 24, 2024

Events ✨ As you may know I'm co-organising the Forward Data Conference on November 25th in Paris. Get tickets for Forward Data Conference Transition to another event. Some personal news, in September things will be changing professionally on my side, I'll be slowly leaving the freelancing world.

BI 130

BI Food Data SQL 130

Knowledge Hut

MAY 3, 2024

Dean Wampler (Renowned author of many big data technology-related books) Dean Wampler makes an important point in one of his webinars. And hence, there is a need to understand the concept of “stream processing “and the technology behind it. Kafka offers better fault tolerance because of its event-driven processing.

Kafka 98

Kafka Scala Java Amazon Web Services 98

Data Engineering Podcast

JANUARY 28, 2024

Summary Monitoring and auditing IT systems for security events requires the ability to quickly analyze massive volumes of unstructured log data. Contact Info LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? When is Scanner the wrong choice?

Data Lake 147

Data Lake Building High Quality Data AWS 147

Snowflake

NOVEMBER 16, 2023

The rise of new technologies such as 5G, cloud computing, and the Internet of Things (IoT) is putting pressure on telecom operators to find new ways to improve the performance of their networks, reduce costs and provide better customer service. These systems can make it difficult and expensive to integrate gen AI solutions.

Cloud Computing 105

Cloud Computing Cloud Systems Technology 105

Knowledge Hut

DECEMBER 22, 2023

Implementing voice recognition technology and APIs to accurately transcribe and process voice inputs. Event websites featuring event details, schedules, and registration forms on a single page. JavaScript Technology This technology is at the heart of almost all major tech giants.

Entertainment 105

Entertainment Machine Learning Utilities Google Cloud 105