Data Engineering Podcast

FEBRUARY 25, 2018

Summary One of the sources of data that often gets overlooked is the systems that we use to run our businesses. This data is not used to directly provide value to customers or understand the functioning of the business, but it is still a critical component of a successful system. Sam Stokes is an engineer at Honeycomb where he helps to build a platform that is able to capture all of the events and context that occur in our production environments and use them to answer all of your questions abou

Kafka 100

Kafka AWS Data Data Pipeline 100

Uber Engineering

FEBRUARY 16, 2018

In distributed systems, retries are inevitable. From network errors to replication issues and even outages in downstream dependencies, services operating at a massive scale must be prepared to encounter, identify, and handle failure as gracefully as possible. Given the scope … The post Building Reliable Reprocessing and Dead Letter Queues with Apache Kafka appeared first on Uber Engineering Blog.

Kafka 109

Kafka Building Engineering Systems 109

Zalando Engineering

FEBRUARY 21, 2018

Why FOSDEM is not your average conference I could get cheeky with semantics and point out that the “M” in FOSDEM stands for “Meeting”. But I’ll play nice and focus instead on the specifics of the event itself. FOSDEM has been running since 2001. In that time, it has grown to become the open source community event for Europe. Over a two-day event, thousands of attendees descend upon the ULB in Brussels to attend what is, in reality, a collection of conferences.

Software Engineer 52

Software Engineer Software Engineering Database Engineering 52

nodeSWAT

FEBRUARY 5, 2018

Our story begins like so many others with a code loving protagonist — someone we all can relate to. His days are largely filled with designing code, writing code and reading about code — keeping clients happy while learning and having fun. This has been going on for years now with both MySQL and Node.js among others and as such our protagonist considers himself quite proficient with both those technologies.

MySQL 52

MySQL Database Programming Coding 52

Advertisement

Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?

Business Intelligence

ProjectPro

FEBRUARY 1, 2018

News on Hadoop - Janaury 2018 Apache Hadoop 3.0 goes GA, adds hooks for cloud and GPUs.TechTarget.com, January 3, 2018. The latest update to the 11 year old big data framework Hadoop 3.0 allows cluster pooling on GPU resources , reduces storage requirements, and adds a novel federation scheme that lets YARN resource manager and the job scheduler expand the number of nodes which can run within a Hadoop cluster.

Hadoop 52

Hadoop Food Healthcare Cloud Computing 52

Cloudera

FEBRUARY 19, 2018

Running a large company is impossible without having an ERP system in place, and SAP business software remains at the forefront in this category. But when companies are looking towards new technologies such as data lakes, machine learning or predictive analytics, SAP alone is just not enough. To keep up with tech trends, businesses have to face the challenges of integrating SAP with non-SAP technologies and embark on a crusade against data silos.

Data Lake 46

Data Lake Finance Government Hadoop 46

Uber Engineering

FEBRUARY 22, 2018

In 2014, Uber Engineering built Schemaless , our fault-tolerant and scalable datastore, to facilitate the rapid growth of our company. For context, we deployed more than 40 Schemaless instances and many thousands of storage nodes in 2016 alone. As our … The post Code Migration in Production: Rewriting the Sharding Layer of Uber’s Schemaless Datastore appeared first on Uber Engineering Blog.

Coding 92

Coding Engineering Designing Architecture 92

Zalando Engineering

FEBRUARY 28, 2018

Apache’s lightning fast engine for data analysis and machine learning In recent years, there has been a massive shift in the industry towards data-oriented decision making backed by enormously large data sets. This means that we can serve our customers with more relevant, personalized content. We in the Digital Experience team are tasked with analysing Big Data in order to gather insights and support the product team with the decision making process.

Data Analysis 40

Data Analysis Hadoop Datasets SQL 40

Data Engineering Podcast

FEBRUARY 11, 2018

Summary As communications between machines become more commonplace the need to store the generated data in a time-oriented manner increases. The market for timeseries data stores has many contenders, but they are not all built to solve the same problems or to scale in the same manner. In this episode the founders of TimescaleDB, Ajay Kulkarni and Mike Freedman, discuss how Timescale was started, the problems that it solves, and how it works under the covers.

PostgreSQL 100

PostgreSQL NoSQL Google Cloud MongoDB 100

Data Engineering Podcast

FEBRUARY 3, 2018

Summary One of the critical components for modern data infrastructure is a scalable and reliable messaging system. Publish-subscribe systems have been popular for many years, and recently stream oriented systems such as Kafka have been rising in prominence. This week Rajan Dhabalia and Matteo Merli discuss the work they have done on Pulsar, which supports both options, in addition to being globally scalable and fast.

Kafka 100

Kafka Data Pipeline Data Science Data Engineering 100

Speaker: Jay Allardyce, Deepak Vittal, and Terrence Sheflin

As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.

Data

Data Engineering Podcast

FEBRUARY 3, 2018

Summary One of the critical components for modern data infrastructure is a scalable and reliable messaging system. Publish-subscribe systems have been popular for many years, and recently stream oriented systems such as Kafka have been rising in prominence. This week Rajan Dhabalia and Matteo Merli discuss the work they have done on Pulsar, which supports both options, in addition to being globally scalable and fast.

Kafka 100

Kafka Data Pipeline Data Science Data Engineering 100

Cloudera

FEBRUARY 16, 2018

In many ways, 2017 was a singular year for Cloudera, not least because we staged a successful IPO and joined the ranks of the world’s fastest-growing, publicly traded companies. We deeply appreciate the vote of confidence and trust our customers have placed in us and are proud of the hard work of our 1,600-plus employees. These are some of the year’s highlights.

Manufacturing 45

Manufacturing Cloud Computing Healthcare Machine Learning 45

Cloudera

FEBRUARY 14, 2018

I’ve spent the last four years here at Cloudera talking with our customers about how to run their businesses better using their data and Cloudera’s products and services. Now I get to put my money where my mouth is – and turn my focus internally on how we at Cloudera can become more data-driven. We aspire to and are on the journey to be the best-run company on data, and to be our own best reference.

Professional Services 43

Professional Services Finance Data Cloud 43

Cloudera

FEBRUARY 6, 2018

As cyber attacks continue to increase across the world, it has become more critical for countries to implement cyber operations from a defensive and offensive perspective to protect national secrets and their citizens. An Arizona State University research paper showed just how global this problem is when they discovered that if hackers discussed a zero-day exploit on the dark web in Chinese the likelihood of a hacker exploiting the vulnerability was 9%.

Government 40

Government Technology IT Management 40

Speaker: Nikhil Joshi, Founder & President of Snic Solutions

Is your manufacturing operation reaching its efficiency potential? A Manufacturing Execution System (MES) could be the game-changer, helping you reduce waste, cut costs, and lower your carbon footprint. Join Nikhil Joshi, Founder & President of Snic Solutions, in this value-packed webinar as he breaks down how MES can drive operational excellence and sustainability.

Manufacturing

Cloudera

FEBRUARY 12, 2018

What exactly can we expect for IoT in 2018, and how can you improve your organization with connected devices? That was the question Dave Shuman set out to answer when he sat down last month with John Gilroy at the Federal News Radio headquarters in Washington, D.C. Federal Tech Talk looks at the world of high technology in the federal government and, as its host, John speaks the language of federal CISOs, CIOs, and CTOs.

Big Data 40

Big Data Government Manufacturing Data 40

Zalando Engineering

FEBRUARY 19, 2018

Multi-functional teams make for a greater customer journey When I started in Zalando Tech, I hadn’t worked with a product manager before, and I had probably never seen a UX designer, a UI designer, a researcher or a business developer before either. My world was data science, more specifically, personalization and recommender systems. In this isolated bubble, data scientists often thought we could solve all problems without help, but in the last two years, I came to understand why we need to sto

Designing 40

Designing Data Science Management Building 40

Zalando Engineering

FEBRUARY 7, 2018

How We Built the Next Generation Product Search from Scratch using a Deep Neural Network Product search is one of the key components in an online retail store. A good product search can understand a user’s query in any language, retrieve as many relevant products as possible, and finally present the results as a list in which the preferred products should be at the top, and the less relevant products should be at the bottom.

Deep Learning 40

Deep Learning Architecture Retail Coding 40

Zalando Engineering

FEBRUARY 5, 2018

Solving the many small files problem for AVRO The Fashion Content Platform teams in Zalando Dublin handle large amounts of data on a daily basis. To make sense of it all, we utilise Hadoop (EMR) on AWS. Within this post, we discuss a system where a real-time system feeds the data. Due to the variance in data volumes and the period that these systems write to storage, there can be a large number of small files.

Hadoop 40

Hadoop Amazon Web Services AWS Utilities 40

Speaker: Anindo Banerjea, CTO at Civio & Tony Karrer, CTO at Aggregage

When developing a Gen AI application, one of the most significant challenges is improving accuracy. This can be especially difficult when working with a large data corpus, and as the complexity of the task increases. The number of use cases/corner cases that the system is expected to handle essentially explodes. 💥 Anindo Banerjea is here to showcase his significant experience building AI/ML SaaS applications as he walks us through the current problems his company, Civio, is solving.

Systems

Zalando Engineering

FEBRUARY 14, 2018

The benefits of Connexion: Zalando’s open source API-First framework In this article, I will show how quick and simple it can be to create a RESTful API for a machine learning model using Zalando’s open source Swagger/OpenAPI First framework called Connexion. Official documentation describes Connexion as the following: “Connexion is a framework on top of Flask that automagically handles HTTP requests based on OpenAPI 2.0 Specification (formerly known as Swagger Spec) of your API described in YAM

Machine Learning 40

Machine Learning Python Coding IT 40