KDnuggets

DECEMBER 26, 2019

Getting ready to leap into the world of Data Science? Consider these top machine learning courses curated by experts to help you learn and thrive in this exciting field.

Machine Learning 160

Machine Learning Data Science Deep Learning Education 160

Uber Engineering

DECEMBER 19, 2019

Every day around the world, millions of trips take place across the Uber network, giving users more reliable transportation through ridesharing, bikes, and scooters, drivers and truckers additional opportunities to earn, employees and employers more convenient business travel, and hungry … The post Uber Infrastructure in 2019: Improving Reliability, Driving Customer Satisfaction appeared first on Uber Engineering Blog.

Transportation 145

Transportation Engineering Architecture 145

Confluent

DECEMBER 23, 2019

When KSQL was released, my first blog post about it showed how to use KSQL with Twitter data. Two years later, its successor ksqlDB was born, which we announced this […].

Data 27

Data IT PostgreSQL Process 27

Data Engineering Podcast

DECEMBER 30, 2019

Summary DataDog is one of the most successful companies in the space of metrics and monitoring for servers and cloud infrastructure. In order to support their customers, they need to capture, process, and analyze massive amounts of timeseries data with a high degree of uptime and reliability. Vadim Semenov works on their data engineering team and joins the podcast in this episode to discuss the challenges that he works through, the systems that DataDog has built to power their business, and how

Process 100

Process Building Hadoop Java 100

Advertisement

In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate

Data Pipeline

Netflix Tech

DECEMBER 3, 2019

by David Berg , Ravi Kiran Chirravuri , Romain Cledat , Savin Goyal , Ferras Hamad , Ville Tuulos tl;dr Metaflow is now open-source! Get started at metaflow.org. Netflix applies data science to hundreds of use cases across the company, including optimizing content delivery and video encoding. Data scientists at Netflix relish our culture that empowers them to work autonomously and use their judgment to solve problems independently.

Data Science 95

Data Science Amazon Web Services AWS Machine Learning 95

Teradata

DECEMBER 11, 2019

Teradata's team of experts are chiming in on their top technology and business predictions for 2020 - from AI to Customer Experience to the Cloud. Read more!

Cloud 72

Cloud Technology 72

Uber Engineering

DECEMBER 17, 2019

Uber’s busy 2019 included our billionth delivery of an Uber Eats order, 24 million miles covered by bike and scooter riders on our platform, and trips to top destinations such as the Empire State Building, the Eiffel Tower, and the … The post Uber’s Data Platform in 2019: Transforming Information to Intelligence appeared first on Uber Engineering Blog.

Data 144

Data Engineering Building Big Data 144

Confluent

DECEMBER 18, 2019

The amount of time it takes for a message to move through a system plays a big role in the performance of distributed systems like Apache Kafka®. In Kafka, the […].

Kafka 26

Kafka Systems IT 26

Data Engineering Podcast

DECEMBER 22, 2019

Summary Transactional databases used in applications are optimized for fast reads and writes with relatively simple queries on a small number of records. Data warehouses are optimized for batched writes and complex analytical queries. Between those use cases there are varying levels of support for fast reads on quickly changing data. To address that need more completely the team at Materialize has created an engine that allows for building queryable views of your data as it is continually update

SQL 100

SQL Engineering Building Java 100

Netflix Tech

DECEMBER 17, 2019

Andreas Andreakis , Ioannis Papapanagiotou Overview Change-Data-Capture (CDC) allows capturing committed changes from a database in real-time and propagating those changes to downstream consumers [1][2]. CDC is becoming increasingly popular for use cases that require keeping multiple heterogeneous datastores in sync (like MySQL and ElasticSearch) and addresses challenges that exist with traditional techniques like dual-writes and distributed transactions [3][4].

MySQL 88

MySQL PostgreSQL Database Transportation 88

Speaker: Tamara Fingerlin, Developer Advocate

Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.

Data

Teradata

DECEMBER 22, 2019

Machine Learning algorithms help computers understand human languages. Learn how this application in Vantage enables businesses to understand their customers.

Process 12

Process Algorithm Machine Learning 12

KDnuggets

DECEMBER 3, 2019

What follows is a set of broad recommendations, and it will inevitably require a lot of adjustments in each implementation. Given that caveat, here are our curriculum recommendations.

Data Science 159

Data Science Data IT Education 159

Uber Engineering

DECEMBER 10, 2019

Michelangelo , Uber’s machine learning (ML) platform, powers machine learning model training across various use cases at Uber, such as forecasting rider demand , fraud detection , food discovery and recommendation for Uber Eats , and improving the accuracy of … The post Productionizing Distributed XGBoost to Train Deep Tree Models with Large Data Sets at Uber appeared first on Uber Engineering Blog.

Food 123

Food Machine Learning Data Engineering 123

Confluent

DECEMBER 19, 2019

With Confluent Platform 5.3, we are actively embracing the rising DevOps movement by introducing CP-Ansible, our very own open source Ansible playbooks for deployment of Apache Kafka® and the Confluent […].

Kafka 22

Kafka 22

Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage

There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.

Systems

Data Engineering Podcast

DECEMBER 16, 2019

Summary Building clean datasets with reliable and reproducible ingestion pipelines is completely useless if it’s not possible to find them and understand their provenance. The solution to discoverability and tracking of data lineage is to incorporate a metadata repository into your data platform. The metadata repository serves as a data catalog and a means of reporting on the health and status of your datasets when it is properly integrated into the rest of your tools.

Metadata 100

Metadata PostgreSQL Datasets Data Warehouse 100

Netflix Tech

DECEMBER 10, 2019

Netflix Hack Day?—?Fall 2019 By Tom Richards , Carenina Garcia Motion , and Leslie Posada Hack Day at Netflix is an opportunity to build and show off a feature, tool, or quirky app. The goal is simple: experiment with new ideas/technologies, engage with colleagues across different disciplines, and have fun! We know even the silliest idea can spur something more.

Technology 84

Technology Project Systems Building 84

Teradata

DECEMBER 9, 2019

Identifying and focusing on priority analytic use cases within your organization will ensure you are asking the right business questions. Find out more.

Data Analytics 61

Data Analytics Data 61

KDnuggets

DECEMBER 24, 2019

This article is about Market Basket Analysis & the Apriori algorithm that works behind it.

Algorithm 159

Algorithm Data Mining IT Python 159

Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives

Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri

Manufacturing

Dataquest

DECEMBER 18, 2019

Today, Stacey Ustian is a data engineer. But the path that led her here wasn’t always easy, and there were a few bumps and twists along the way. Her journey to data science started in a rather unusual place: the law library. After earning her Master’s degree in Library and Information Science, Stacey had taken a job working in the library of a law firm.

SQL 52

SQL Python Data Engineer Data Engineering 52

Confluent

DECEMBER 16, 2019

On behalf of the Apache Kafka® community, it is my pleasure to announce the release of Apache Kafka 2.4.0. This release includes a number of key new features and improvements […].

Kafka 21

Kafka Scala IT 21

Data Engineering Podcast

DECEMBER 8, 2019

Summary Data warehouses have gone through many transformations, from standard relational databases on powerful hardware, to column oriented storage engines, to the current generation of cloud-native analytical engines. SnowflakeDB has been leading the charge to take advantage of cloud services that simplify the separation of compute and storage. In this episode Kent Graziano, chief technical evangelist for SnowflakeDB, explains how it is differentiated from other managed platforms and traditiona

Data Warehouse 100

Data Warehouse Cloud AWS Relational Database 100

Netflix Tech

DECEMBER 17, 2019

Andreas Andreakis , Ioannis Papapanagiotou Overview Change-Data-Capture (CDC) allows capturing committed changes from a database in real-time and propagating those changes to downstream consumers [1][2]. CDC is becoming increasingly popular for use cases that require keeping multiple heterogeneous datastores in sync (like MySQL and ElasticSearch) and addresses challenges that exist with traditional techniques like dual-writes and distributed transactions [3][4].

MySQL 83

MySQL PostgreSQL Database Transportation 83

Advertisement

Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.

Cloud

Teradata

DECEMBER 29, 2019

How do you organize your business for analytics? Here are six steps your enterprise should take when creating an analytics team. Read more!

59

59

KDnuggets

DECEMBER 6, 2019

Are you interested in studying machine learning over the holidays? This collection of 10 free top notch courses will allow you to do just that, with something for every approach to improving your machine learning skills.

Machine Learning 159

Machine Learning Deep Learning Python 159

Preset

DECEMBER 15, 2019

Announcing Elasticsearch in Superset, powered by a new open-source Python library from Preset

Python 40

Python 40

Confluent

DECEMBER 30, 2019

During the holiday season, it’s a particularly relevant time to pause, reflect, and celebrate, both the days past and those ahead. Here at Confluent, it’s a noticeably nostalgic moment, given […].

19

19

Advertisement

Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?

Business Intelligence

Data Engineering Podcast

DECEMBER 2, 2019

Summary The financial industry has long been driven by data, requiring a mature and robust capacity for discovering and integrating valuable sources of information. Citadel is no exception, and in this episode Michael Watson and Robert Krzyzanowski share their experiences managing and leading the data engineering teams that power the business. They shared helpful insights into some of the challenges associated with working in a regulated industry, organizing teams to deliver value rapidly and re

Data Engineering 100

Data Engineering Data Engineer Engineering Machine Learning 100

Netflix Tech

DECEMBER 2, 2019

Julie (Novak) Beckley, Andy Rhines, Jeffrey Wong, Matthew Wardrop, Toby Mao, Martin Tingley Ever wonder why Netflix works so well when you’re streaming at home, on the train, or in a foreign hotel? Behind the scenes, Netflix engineers are constantly striving to improve the quality of your streaming service. The goal is to bring you joy by delivering the content you love quickly and reliably every time you watch.

Datasets 58

Datasets Data Algorithm Engineering 58

Teradata

DECEMBER 29, 2019

How do you organize your business for analytics? Here are six steps your enterprise should take when creating an analytics team. Read more!

58

58

KDnuggets

DECEMBER 31, 2019

With so many buzzwords surrounding AI and machine learning, understanding which can bring business value and which are best left in the lab to mature is difficult. While machine learning offers significant power in driving digital transformations, a business must start with the right questions and leave the math to the development teams.

Data Science 157

Data Science Machine Learning Data 157

Advertisement

With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you

Data Engineering