KDnuggets

OCTOBER 28, 2019

Learn how Bayes Theorem is in Machine Learning for classification and regression!

Machine Learning 123

Machine Learning 123

Uber Engineering

OCTOBER 16, 2019

Michelangelo , Uber’s machine learning (ML) platform, supports the training and serving of thousands of models in production across the company. Designed to cover the end-to-end ML workflow, the system currently supports classical machine learning, time series forecasting, and deep … The post Evolving Michelangelo Model Representation for Flexibility at Scale appeared first on Uber Engineering Blog.

Machine Learning 111

Machine Learning Engineering Designing Systems 111

Confluent

OCTOBER 10, 2019

The Internet of Things (IoT) is getting more and more traction as valuable use cases come to light. A key challenge, however, is integrating devices and machines to process the data in real time and at scale. Apache Kafka ® and its surrounding ecosystem, which includes Kafka Connect, Kafka Streams, and KSQL, have become the technology of choice for integrating and processing these kinds of datasets.

Kafka 20

Kafka Google Cloud Architecture Machine Learning 20

Data Engineering Podcast

OCTOBER 28, 2019

Summary Despite the fact that businesses have relied on useful and accurate data to succeed for decades now, the state of the art for obtaining and maintaining that information still leaves much to be desired. In an effort to create a better abstraction for building data applications Nick Schrock created Dagster. In this episode he explains his motivation for creating a product for data management, how the programming model simplifies the work of building testable and maintainable pipelines, and

Building 100

Building Data Pipeline Programming Language Kafka 100

Advertisement

In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate

Data Pipeline

Teradata

OCTOBER 9, 2019

Teradata Workload Management automates changes in settings across processing windows as patterns of end user usage changes and as business requirements shift.

Process 16

Process Management 16

Netflix Tech

OCTOBER 16, 2019

By Ammar Khaku Introduction In a microservice architecture such as Netflix’s, propagating datasets from a single source to multiple downstream destinations can be challenging. These datasets can represent anything from service configuration to the results of a batch job, are often needed in-memory to optimize access and must be updated as they change over time.

Datasets 76

Datasets Metadata Bytes Machine Learning 76

Uber Engineering

OCTOBER 16, 2019

Michelangelo , Uber’s machine learning (ML) platform, supports the training and serving of thousands of models in production across the company. Designed to cover the end-to-end ML workflow, the system currently supports classical machine learning, time series forecasting, and deep … The post Evolving Michelangelo Model Representation for Flexibility at Scale appeared first on Uber Engineering Blog.

Machine Learning 110

Machine Learning Engineering Designing Systems 110

Confluent

OCTOBER 16, 2019

Trains are an excellent source of streaming data—their movements around the network are an unbounded series of events. Using this data, Apache Kafka ® and Confluent Platform can provide the foundations for both event-driven applications as well as an analytical platform. With tools like KSQL and Kafka Connect, the concept of streaming ETL is made accessible to a much wider audience of developers and data engineers.

Kafka 19

Kafka Building Data Coding 19

Data Engineering Podcast

OCTOBER 21, 2019

Summary The scale and complexity of the systems that we build to satisfy business requirements is increasing as the available tools become more sophisticated. In order to bridge the gap between legacy infrastructure and evolving use cases it is necessary to create a unifying set of components. In this episode Dipti Borkar explains how the emerging category of data orchestration tools fills this need, some of the existing projects that fit in this space, and some of the ways that they can work to

Cloud 100

Cloud Hadoop Data Lake Programming Language 100

Teradata

OCTOBER 3, 2019

Predict the future faster with predictive modeling. Learn more about use cases and how to get more value out of your data.

Data 69

Data 69

Speaker: Tamara Fingerlin, Developer Advocate

Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.

Data

Netflix Tech

OCTOBER 18, 2019

Faisal Siddiqi Infrastructure for Contextual Bandits and Reinforcement Learning?—? theme of the ML Platform meetup hosted at Netflix, Los Gatos on Sep 12, 2019. Contextual and Multi-armed Bandits enable faster and adaptive alternatives to traditional A/B Testing. They enable rapid learning and better decision-making for product rollouts. Broadly speaking, these approaches can be seen as a stepping stone to full-on Reinforcement Learning (RL) with closed-loop, on-policy evaluation and model objec

Algorithm 52

Algorithm Architecture Machine Learning Deep Learning 52

KDnuggets

OCTOBER 24, 2019

In this post, you will see 3 different techniques of how to do Feature Selection to your datasets and how to build an effective predictive model.

Datasets 123

Datasets Building Machine Learning 123

Dataquest

OCTOBER 16, 2019

Exciting news: we just launched a totally revamped Data Engineering path that offers from-scratch training for anyone who wants to become a data engineer or learn some data engineering skills. Looks cool, right? But it begs the question: why learn data engineering in the first place? Typically, data science teams are comprised of data analysts, data scientists, and data engineers.

Data Engineer 52

Data Engineer Data Engineering Engineering Data Science 52

Confluent

OCTOBER 7, 2019

This tutorial describes how to set up a sample Spring Boot application in Pivotal Application Service (PAS), which consumes and produces events to an Apache Kafka ® cluster running in Pivotal Container Service (PKS). With this tutorial, you can set up your PAS and PKS configurations so that they work with Kafka. For a tutorial on how to set up a Kafka cluster in PKS, please see How to Deploy Confluent Platform on Pivotal Container Service (PKS) with Confluent Operator.

Kafka 18

Kafka Java Coding Accessible 18

Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage

There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.

Systems

Data Engineering Podcast

OCTOBER 14, 2019

Summary Managing a data warehouse can be challenging, especially when trying to maintain a common set of patterns. Dataform is a platform that helps you apply engineering principles to your data transformations and table definitions, including unit testing SQL scripts, defining repeatable pipelines, and adding metadata to your warehouse to improve your team’s communication.

Data Warehouse 100

Data Warehouse PostgreSQL AWS Programming Language 100

Teradata

OCTOBER 21, 2019

With four new offerings, Teradata is helping companies move from analytics to answers wherever they are on their cloud journey. Read more.

Cloud 66

Cloud 66

Rockset

OCTOBER 10, 2019

In this blog, we walk through how to build a real-time dashboard for operational monitoring and analytics on streaming event data from Kafka, which often requires complex SQL, including filtering, aggregations, and joins with other data sets. Apache Kafka is a widely used distributed data log built to handle streams of unstructured and semi-structured event data at massive scales.

Kafka 52

Kafka SQL Building Data 52

KDnuggets

OCTOBER 21, 2019

If you are interested in learning more about the latest Youtube recommendation algorithm paper, read this post for details on its approach and improvements.

Algorithm 123

Algorithm IT Systems Engineering 123

Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives

Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri

Manufacturing

Dataquest

OCTOBER 16, 2019

We’ve got some really exciting news: we’ve just launched a total revamp of our Data Engineering learning path ! This revamped path is designed to be more like our other course paths. You can start it even if you have no prior experience with coding , and it’ll take you from total beginner to experienced practitioner with all of the core skills needed to become a data engineer.

Data Engineer 52

Data Engineer Data Engineering Engineering SQL 52

Confluent

OCTOBER 24, 2019

I’ve written an event sourcing bank simulation in Clojure (a lisp build for Java virtual machines or JVMs) called open-bank-mark , which you are welcome to read about in my previous blog post explaining the story behind this open source example. As a next step, specifically for this article I’ve added SSL and combined some topics together, using the subject name strategy option of Confluent Schema Registry , making it more production like, adding security, and making it possible to put multiple

Kafka 18

Kafka Java Banking Bytes 18

Data Engineering Podcast

OCTOBER 7, 2019

Summary The process of exposing your data through a SQL interface has many possible pathways, each with their own complications and tradeoffs. One of the recent options is Rockset, a serverless platform for fast SQL analytics on semi-structured and structured data. In this episode CEO Venkat Venkataramani and SVP of Product Shruti Bhat explain the origins of Rockset, how it is architected to allow for fast and flexible SQL analytics on your data, and how their serverless platform can save you th

Structured Data 100

Structured Data Cloud SQL Programming Language 100

Teradata

OCTOBER 15, 2019

Find out how our UX team is going to radically simplify the Teradata user experience. To be unveiled at Teradata Universe!

65

65

Advertisement

Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.

Cloud

Rockset

OCTOBER 21, 2019

At Rockset, our singular focus is to be the shortest (and most efficient) path from data to applications for our users. We recognize and truly believe that our success lies in the success of our users. We constantly think about improving our workflows, coming up with new ones and iterating on them in ways that takes the user experience to a whole new level.

Unstructured Data 40

Unstructured Data Designing Data Accessible 40

KDnuggets

OCTOBER 22, 2019

For full-stack data science mastery, you must understand data management along with all the bells and whistles of machine learning. This high-level overview is a road map for the history and current state of the expansive options for data storage and infrastructure solutions.

Data Management 123

Data Management Management Data Storage Machine Learning 123

Netflix Tech

OCTOBER 18, 2019

Faisal Siddiqi Infrastructure for Contextual Bandits and Reinforcement Learning?—? theme of the ML Platform meetup hosted at Netflix, Los Gatos on Sep 12, 2019. Contextual and Multi-armed Bandits enable faster and adaptive alternatives to traditional A/B Testing. They enable rapid learning and better decision-making for product rollouts. Broadly speaking, these approaches can be seen as a stepping stone to full-on Reinforcement Learning (RL) with closed-loop, on-policy evaluation and model objec

Algorithm 40

Algorithm Architecture Machine Learning Deep Learning 40

Confluent

OCTOBER 9, 2019

Last week, the Kafka Summit hosted nearly 2,000 people from 40 different countries and 595 companies—the largest Summit yet. By the numbers, we got to enjoy four keynote speakers, 56 sessions, 75 speakers, 38 sponsors, and one big party, including the classic Apache Kafka ® ice sculpture, per the traditions handed down to us. (I guess we handed those traditions down to ourselves in this case, but still.).

Kafka 17

Kafka Software Engineer Software Engineering Data Pipeline 17

Advertisement

Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?

Business Intelligence

Preset

OCTOBER 8, 2019

Preset Announcement

Systems 40

Systems 40

Teradata

OCTOBER 7, 2019

Companies must think about the future of personalization and how to modernize their current systems to combat over-choice in the market. Find out how.

Systems 12

Systems 12

Rockset

OCTOBER 1, 2019

Kubernetes at Rockset At Rockset, we use Kubernetes (k8s) for cluster orchestration. It runs all our production microservices — from our ingest workers to our query-serving tier. In addition to hosting all the production infrastructure, each engineer has their own Kubernetes namespace and dedicated resources that we use to locally deploy and test new versions of code and configuration.

SQL 40

SQL Systems Metadata Accessibility 40

KDnuggets

OCTOBER 16, 2019

This post is about various evaluation metrics and how and when to use them.

Data 123

Data Machine Learning Python 123

Advertisement

With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you

Data Engineering