January, 2017

article thumbnail

The Rise of the Data Engineer

Maxime Beauchemin

I joined Facebook in 2011 as a business intelligence engineer. By the time I left in 2013, I was a data engineer. I wasn’t promoted or assigned to this new role. Instead, Facebook came to realize that the work we were doing transcended classic business intelligence. The role we’d created for ourselves was a new discipline entirely. My team was at forefront of this transformation.

article thumbnail

Dask with Matthew Rocklin - Episode 2

Data Engineering Podcast

Summary There is a vast constellation of tools and platforms for processing and analyzing your data. In this episode Matthew Rocklin talks about how Dask fills the gap between a task oriented workflow tool and an in memory processing framework, and how it brings the power of Python to bear on the problem of big data. Preamble Hello and welcome to the Data Engineering Podcast, the show about modern data infrastructure Go to dataengineeringpodcast.com to subscribe to the show, sign up for the news

Hadoop 100
article thumbnail

What is a Data Engineer?

Dataquest

From helping cars drive themselves to helping Facebook tag you in photos , data science has attracted a lot of buzz recently. Data scientists have become extremely sought after , and for good reason — a skilled data scientist can add incredible value to a business. But what about data engineers? Who are they, and what do they do? A data scientist is only as good as the data they have access to.

article thumbnail

About Akka Streams

Zalando Engineering

In many computer programs, the whole logic (or a vast part of it) is essentially the step-by-step processing of data. Of course, this includes the situation when we iterate over the data and just execute the processing logic on every piece of it. However, there are a couple of complications here: The processing logic may be quite complex, with various aggregations, merging, routing, error recoveries, etc.

Scala 52
article thumbnail

Apache Airflow® Best Practices for ETL and ELT Pipelines

Whether you’re creating complex dashboards or fine-tuning large language models, your data must be extracted, transformed, and loaded. ETL and ELT pipelines form the foundation of any data product, and Airflow is the open-source data orchestrator specifically designed for moving and transforming data in ETL and ELT pipelines. This eBook covers: An overview of ETL vs.

article thumbnail

Recap of Hadoop News for December

ProjectPro

News on Hadoop-December 2016 Telefonica Gains Real-Time Advantage With Big Data Analytics.Forbes.com,December 5, 2016. A leading telecommunications company Telefonica in Spain with 3 million customers and more than 17 million mobile customers is making huge profits from Big Data and Business Intelligence. With the help of Big Data and Business Intelligence the company is now able to analyze the preferences of its audience and improve customer satisfaction.

Hadoop 52
article thumbnail

Pachyderm with Daniel Whitenack - Episode 1

Data Engineering Podcast

Summary Do you wish that you could track the changes in your data the same way that you track the changes in your code? Pachyderm is a platform for building a data lake with a versioned file system. It also lets you use whatever languages you want to run your analysis with its container based task graph. This week Daniel Whitenack shares the story of how the project got started, how it works under the covers, and how you can get started using it today!

Data Lake 100

More Trending

article thumbnail

What is Hardcore Data Science – In Practice?

Zalando Engineering

This article originally appeared on oreilly.com. Data science has become widely accepted across a broad range of industries in the past few years. Originally more of a research topic, data science has early roots in scientists efforts to understand human intelligence and create artificial intelligence; it has since proven that it can add real business value.

article thumbnail

Your Lifelike Hologram using Structure and HoloLens for Hack Week

Zalando Engineering

Zalando recently held its annual Hack Week , during which teams can brainstorm and work on innovative ideas that bring about real business value. This year I went for something totally geeky compared to last year, which had me working on a refugee-related project. The name of it is VRify. VRify’s purpose is to create an immersive conference call experience that allows you to communicate with your remote colleagues more efficiently.

article thumbnail

The Role of UX in Hack Week

Zalando Engineering

Zalando’s annual Hack Week is the perfect opportunity for engineers, product specialists, UX designers, and user researchers to experiment and play around with ideas. Projects range in scope and focus on improving the customer experience of Zalando, but also rethink the way we work together across our tech department. Many Hack Week projects, especially when UX-savvy people are involved, not only solve design-related problems but also apply user centered design methods in the development process

article thumbnail

Rule Over Your Angular2 State Machine

Zalando Engineering

It is really important to control your application state. Extremely important. The application state binds all of your functionality together, allowing us to do the awesome things we love about programming. Today I want to write about the Angular2-state-machine , which helps you handle your Angular 2 application state and, hopefully, makes your life as a developer much easier.

article thumbnail

Apache Airflow®: The Ultimate Guide to DAG Writing

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

article thumbnail

In Search of the Perfect Fit – Insights from the UX Job Title Survey

Zalando Engineering

The results are in! -- One in three UX professionals aren’t happy with their job title. “I just wish my field was called something different” , they told us. Many of them avoid mentioning their title outside of a work context: “Service Designer? - Is that like designing call centers?”. But the confusion doesn’t stop there. Across the job market, it is hard to get an overview of which skills are expected from whom.

article thumbnail

App Migration to Swift 3

Zalando Engineering

We are a team of 10 iOS developers located in Zalando’s Helsinki hub , working on a Zalando iOS application called fleek: a fashion e-commerce app that connects mobile-savvy consumers with brands, retailers, and influencers. To ensure our app was ready for the latest release of Xcode 8/iOS 10, we started to plan our code migration to the new version of Swift.

Retail 40
article thumbnail

Sapphire Deep Learning Upskilling

Zalando Engineering

At Zalando’s Fashion Insights Centre in Dublin , we work in autonomous data science delivery teams. This means that each team has the responsibility to deliver technology from research work and MVPs, through to production code and operational systems. This gives us a great opportunity to make decisions about how we organise our work so that we can balance the investment of effort in developing new data science solutions and necessary experimentation work, as well as production-ready code and mai