Data Engineering Podcast

MAY 6, 2018

Summary The Open Data Science Conference brings together a variety of data professionals each year in Boston. This week’s episode consists of a pair of brief interviews conducted on-site at the conference. First up you’ll hear from Alan Anders, the CTO of Applecart about their challenges with getting Spark to scale for constructing an entity graph from multiple data sources.

Data Science 100

Data Science Machine Learning Data Engineering Data Engineer 100

InData Labs

MAY 7, 2018

Digital Transformation Definition Digital transformation has been a big topic for a few years now, and it has many definitions. From a business perspective, digital transformation is about leveraging digital technologies to improve processes, competencies, and business models. It is also about changing the culture of the company because it requires letting go of old.

Process 52

Process Technology IT Data Science 52

Cloudera

MAY 10, 2018

Cloudera 2017 Data Impact Award Winners. We are excited to kick off the 2018 Data Impact Awards ! Nominations are now open and will remain open until July 20th. Since 2012, the Data Impact Awards have showcased how organizations are using Cloudera and the power of data to transform themselves and achieve dramatic results. Click to tweet : Nominations are now open for the sixth annual Cloudera Data Impact Awards!

Healthcare 41

Healthcare Machine Learning Media Data Science 41

Zalando Engineering

MAY 8, 2018

Moving from a more traditional internal IT setup to a product-driven culture I love building enterprise systems, because you get to work with your customers/users every day and literally see their lives change as you release new features. In my case, at Zalando, these are systems for fashion buying, supply chain management, inventory management and procure-to-pay processes (e.g. paying our suppliers for merchandise we bought from them).

Systems 40

Systems Management Recruitment Retail 40

Advertisement

In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate

Data Pipeline

Zalando Engineering

MAY 7, 2018

Real-time joins in event-driven microservices As discussed in my previous blog post , Kafka is one of the key components of our event-driven microservice architecture in Zalando’s Smart Product Platform. We use it for sequencing events and building an aggregated view of data hierarchies. This post expands on what I previously wrote about the one-to-many data model and introduces more complex many-to-many relationships.

Kafka 40

Kafka Scala Java Media 40