KDnuggets
DECEMBER 1, 2022
In this blog, I shared my story on getting 4 data science job offers including Airbnb, Lyft and Twitter after being laid off. Any data scientist who was laid off due to the pandemic or who is actively looking for a data science position can find something here to which they can relate.
Confluent
NOVEMBER 29, 2022
Apache Kafka’s Streams API embeds Machine Learning into any app or microservice (Java, Docker, Kubernetes, etc.) to add business value.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Confessions of a Data Guy
DECEMBER 1, 2022
Nothing captures the imagination and heart like a tale of betrayal and heartbreak, and that is a tale I want to bring to you today. It’s a tale of Databricks Workflows and Jobs, version changes, new features, API’s, and insidious little hidden gems that will make you pull your hair out when you find them. […] The post A Tale of Betrayal and Heartbreak – Databricks Workflows and Jobs. appeared first on Confessions of a Data Guy.
Data Engineering Podcast
NOVEMBER 27, 2022
Summary The most expensive part of working with massive data sets is the work of retrieving and processing the files that contain the raw information. FeatureBase (formerly Pilosa) avoids that overhead by converting the data into bitmaps. In this episode Matt Jaffee explains how to model your data as bitmaps and the benefits that this representation provides for fast aggregate computation.
Advertisement
In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate
KDnuggets
DECEMBER 1, 2022
The latest KDnuggets exclusive cheatsheet covers the essentials of machine learning with Scikit-learn.
Confluent
DECEMBER 2, 2022
ksqlDB use case: see how apps can use ksqlDB to ingest, filter, enrich, aggregate, and query data directly with Kafka—no complex architectures or data stores needed.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
|
Data Engineering Podcast
NOVEMBER 27, 2022
Summary The data ecosystem has been growing rapidly, with new communities joining and bringing their preferred programming languages to the mix. This has led to inefficiencies in how data is stored, accessed, and shared across process and system boundaries. The Arrow project is designed to eliminate wasted effort in translating between languages, and Voltron Data was created to help grow and support its technology and community.
KDnuggets
NOVEMBER 29, 2022
First steps to learning data science & machine learning are the foundations.
Confluent
DECEMBER 2, 2022
Broadcom's Mainframe Operational Intelligence Product (MOI) collects and analyzes data at mass scale, using ksqlDB to improve anomaly detection and custom alarm filtering.
DoorDash Engineering
NOVEMBER 29, 2022
As DoorDash’s business grows, engineers strive for a better network infrastructure to ensure more third-party services could be integrated into our system while keeping data securely transmitted. Due to security and compliance concerns, some vendors handling such sensitive data cannot expose services to the public Internet and therefore host their own on-premise data centers.
Speaker: Tamara Fingerlin, Developer Advocate
Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.
Cloudera
DECEMBER 1, 2022
Full-stack observability is a critical requirement for effective modern data platforms to deliver the agile, flexible, and cost-effective environment organizations are looking for. For analytic applications to properly leverage a hybrid, multi-cloud ecosystem to support modern data architectures, data observability has become even more important. I spoke to Mark Ramsey of Ramsey International (RI) to dive deeper into that last subject.
KDnuggets
DECEMBER 2, 2022
The data science field is full of job opportunities, yet there is still a lot of confusion about what data scientists actually do. This confusion is largely due to the many myths that exist about the role of a data scientist. In this article, we will bust the top 10 myths about data science. By the end of this article, you will have a better understanding of the role of a data scientist and what it takes to be one.
Confluent
DECEMBER 2, 2022
Major improvements to the Kafka consumer, Streams, and ksqlDB for incremental cooperative rebalancing while maintaining at-least-once and exactly-once guarantees.
Engineering at Meta
NOVEMBER 30, 2022
UPM is our internal standalone library to perform static analysis of SQL code and enhance SQL authoring. UPM takes SQL code as input and represents it as a data structure called a semantic tree. Infrastructure teams at Meta leverage UPM to build SQL linters, catch user mistakes in SQL code, and perform data lineage analysis at scale. Executing SQL queries against our data warehouse is important to the workflows of many engineers and data scientists at Meta for analytics and monitoring use cases
Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage
There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.
Cloudera
NOVEMBER 30, 2022
What is CDP Operational Database (COD). CDP Operational Database enables developers to quickly build future-proof applications that are architected to handle data evolution. It helps developers automate and simplify database management with capabilities like auto-scale, and is fully integrated with Cloudera Data Platform (CDP). For more information and to get started with COD, refer to Getting Started with Cloudera Data Platform Operational Database (COD).
KDnuggets
NOVEMBER 28, 2022
Everything you need to know to start your career in Data Engineering.
Confluent
DECEMBER 2, 2022
Here's a deep dive on how we implemented Bincover, a simple, open source tool for measuring code coverage of Golang binaries.
Booking.com Engineering
DECEMBER 2, 2022
Booking.com’s mission is to make it easier for everyone to experience the world. To help people discover destinations, we are a leading travel advertiser on Google Pay Per Click (PPC). Booking Holdings, as a whole, spent $4.7 billion in marketing across all brands in the first nine months of 2022[1]. How do we run PPC at our scale, and efficiently? In this article, we want to illustrate our extensive use of the public cloud, specifically Google Cloud Platform (GCP).
Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives
Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri
Cloudera
NOVEMBER 30, 2022
What is Cloudera Operational Database (COD). Cloudera Operational Database enables developers to quickly build future-proof applications that are architected to handle data evolution. It helps developers automate and simplify database management with capabilities like auto-scale, and is fully integrated with Cloudera Data Platform (CDP). For more information and to get started with COD, refer to our article Getting Started with Cloudera Data Platform Operational Database (COD).
KDnuggets
NOVEMBER 29, 2022
Secure major savings on DataCamp’s Black Friday deal and Cyber Monday deal!
Confluent
NOVEMBER 30, 2022
What is stream processing, or complex event processing (CEP), and how does it work? Learn about real-time data and event stream analytics in this tutorial.
Pinterest Engineering
DECEMBER 2, 2022
Grey Skold | (former Android Video Engineer) ; Lin Wang | Android Performance Engineer; Sheng Liu | Android Performance Engineer Pinterest Android App offers a rare experience with a mix of images and videos on a two-column grid. In order to maintain a performant video experience on Android devices, we focused on: Warming up Configurations Pooling players Warming Up In order to reduce the startup latency, we establish a video network connection by sending a dummy HTTP HEAD request during the ear
Advertisement
With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you
U-Next
DECEMBER 2, 2022
Introduction . The rise of cryptocurrency can be attributed to the increasing popularity of blockchain technology. Blockchain is the underlying technology that powers cryptocurrency, and it is seen as a highly secure and transparent way of conducting transactions. With more and more businesses and organizations beginning to explore the use of blockchain, the demand for cryptocurrency is likely to continue to grow. .
KDnuggets
DECEMBER 2, 2022
This article discusses the different python libraries used for data visualization with examples.
Confluent
DECEMBER 2, 2022
Build fast, break nothing. Learn about the unique challenges Confluent's engineering team has faced building ksqlDB and continuously shipping the latest, greatest features.
Ascend.io
DECEMBER 1, 2022
Data migration is one of the most common undertakings for data teams. Yet, many businesses underestimate the process—resulting in extra time and money spent. A data migration process usually takes longer than it should and requires several teams. In addition, it is highly visible to both users and executives. How can you keep from making the same mistake?
Speaker: Tamara Fingerlin, Developer Advocate
In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!
U-Next
DECEMBER 2, 2022
Introduction To Summarizing Data In Excel . Data for Excel is a way of summarizing large amounts of data into a few numbers. For example, if you have 3,000 sales at $50 each, you could summarize this by saying that total sales were $150,000. The summarization of data in Excel is doable in many ways. . Approximately 54% of businesses use Excel , which doesn’t include any other spreadsheet programs.
KDnuggets
NOVEMBER 29, 2022
Improve the model performance by balancing the dataset using the synthetic minority oversampling technique.
Confluent
DECEMBER 2, 2022
With over 4,700 stores, learn how Walmart used Kafka to build an event-driven architecture for real-time inventory management, providing a seamless omnichannel experience.
DataKitchen
DECEMBER 1, 2022
Data Team members, have you ever felt overwhelmed? The never-ending flow of new information can be stressful, and it’s hard to know where to start. Well, don’t worry because DataOps is here to help! In this post, we’ll discuss how DataOps Observability and Automation can relieve team stress and show you how to get started. So don’t wait any longer.
Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage
When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m
Expert insights. Personalized for you.
160 
Let's personalize your content