Data Engineering Podcast

MAY 26, 2019

Summary Some problems in data are well defined and benefit from a ready-made set of tools. For everything else, there’s Pachyderm, the platform for data science that is built to scale. In this episode Joe Doliner, CEO and co-founder, explains how Pachyderm started as an attempt to make data provenance easier to track, how the platform is architected and used today, and examples of how the underlying principles manifest in the workflows of data engineers and data scientists as they collabor

Data Science 100

Data Science Data Pipeline Data Kafka 100

Teradata

MAY 30, 2019

Rob Armstrong explains three critical ways to get better answers from your data.

Data 99

Data 99

Confluent

MAY 29, 2019

In part 1 , we discussed an event streaming architecture that we implemented for a customer using Apache Kafka ® , KSQL from Confluent, and Kafka Streams. Now in part 2, we’ll discuss the challenges we faced developing, building, and deploying the KSQL portion of our application and how we used Gradle to address them. In part 3, we’ll explore using Gradle to build and deploy KSQL user-defined functions (UDFs) and Kafka Streams microservices.

Kafka 96

Kafka Management Bytes SQL 96

Netflix Tech

MAY 30, 2019

By Juliano Moraes , David Henry , Corey Grunewald & Jim Isaacs Recently Netflix has started building mobile apps to bring technology and innovation to our Studio Physical Productions , the portion of the business responsible for producing our TV shows and movies. Our very first mobile app is called Prodicle and was built for Android & iOS using the same reactive architecture in both platforms, which allowed us to build 2 apps from scratch in 3 months with 4 software engineers.

Architecture 62

Architecture Coding Software Engineer Software Engineering 62

Advertisement

Whether you’re creating complex dashboards or fine-tuning large language models, your data must be extracted, transformed, and loaded. ETL and ELT pipelines form the foundation of any data product, and Airflow is the open-source data orchestrator specifically designed for moving and transforming data in ETL and ELT pipelines. This eBook covers: An overview of ETL vs.

Architecture

Cloudera

MAY 28, 2019

Gartner states that “By 2022, 75% of new end-user solutions leveraging machine learning (ML) and AI techniques will be built with commercial instead of open source platforms” ¹. Spoiler alert: it’s not because data scientists will stop relying on open source for the latest innovation in ML algorithms and development environments. But rather as businesses look to operationalize machine learning capabilities at scale, they’ll turn increasingly to commercial platforms, with connectors to open so

Data Science 51

Data Science Transportation Machine Learning Algorithm 51

Teradata

MAY 28, 2019

Tom Casey discusses marketing personalization and why it's important to the modern customer experience.

IT 91

IT 91

Rockset

MAY 31, 2019

Live dashboards can help organizations make sense of their event data and understand what's happening in their businesses in real time. Marketing managers constantly want to know how many signups there were in the last hour, day, or week. Product managers are always looking to understand which product features are working well and most heavily utilized.

BI 40

BI Java Data SQL 40

Zalando Engineering

MAY 26, 2019

This blog post describes how we manage the process of proposing, reviewing and approving projects to become open source, while at the same time ensuring project code follows our compliance rules, and the maintainers of the projects are aware of their responsibilities. See our formal release guidelines Overview The process involves five steps that take the project from internal source code, through a review phase to our incubator, which eventually results in the project being graduated into our t

Project 40

Project PostgreSQL Coding Machine Learning 40

Confluent

MAY 28, 2019

Part 1 of this blog series by Gwen Shapira explained the benefits of schemas, contracts between services, and compatibility checking for schema evolution. In particular, using Confluent Schema Registry makes this really easy for developers to use schemas, and it is designed to be highly available. But it’s important to configure it properly from the start and manage it well, or else the schemas may not be available to the applications that need them.

Management 85

Management Kafka Java Certification 85