Data Engineering Podcast

OCTOBER 28, 2018

Summary Jupyter notebooks have gained popularity among data scientists as an easy way to do exploratory analysis and build interactive reports. However, this can cause difficulties when trying to move the work of the data scientist into a more standard production environment, due to the translation efforts that are necessary. At Netflix they had the crazy idea that perhaps that last step isn’t necessary, and the production workflows can just run the notebooks directly.

Scala 100

Scala Python Data Engineering Data Engineer 100

Uber Engineering

OCTOBER 17, 2018

Uber is committed to delivering safer and more reliable transportation across our global markets. To accomplish this, Uber relies heavily on making data-driven decisions at every level, from forecasting rider demand during high traffic events to identifying and addressing bottlenecks … The post Uber’s Big Data Platform: 100+ Petabytes with Minute Latency appeared first on Uber Engineering Blog.

Big Data 109

Big Data Transportation Data Engineering 109

Cloudera

OCTOBER 3, 2018

We’ve just announced that Cloudera and Hortonworks have agreed to merge to form a single company. I want to explain the thinking behind the deal and the combination. Rob Bearden from Hortonworks has written up a post sharing his thoughts, as well. First, remember the history of Apache Hadoop. Google built an innovative scale-out platform for data storage and analysis in the late 1990s and early 2000s, and published research papers about their work.

Hadoop 75

Hadoop Cloud Data Storage Big Data 75

Zalando Engineering

OCTOBER 31, 2018

Why I advocate a practice of no estimates as a software engineer Before I get to the topic, I would like to clarify one thing: I don’t want to ban estimations generally from software development, as there are good and solid reasons for it. In a nutshell, business needs to be predictable. I want to show a software developer's view on how to reduce or even get rid of endless estimations meetings with doubtful outcomes.

Software Engineer 52

Software Engineer Software Engineering Coding Building 52

Advertisement

Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?

Business Intelligence

Netflix Tech

OCTOBER 31, 2018

Netflix Media Database?—?the Media Timeline Data Model In the previous post in this series, we described some important Netflix business needs as well as traits of the media data system?—?called “ N etflix M edia D ata B ase” (NMDB) that is used to address them. The curious reader might have noticed that a majority of these characteristics relate to properties of the data managed by NMDB.

Media 54

Media Metadata Data MongoDB 54

Rockset

OCTOBER 30, 2018

Prior to Rockset, I spent eight years at Facebook building out their big data infrastructure and online data infrastructure. All the software we wrote was deployed in Facebook's private data centers, so it was not till I started building on the public cloud that I fully appreciated its true potential. Facebook may be the very definition of a web-scale company, but getting hardware still required huge lead times and extensive capacity planning.

Cloud 40

Cloud IT MongoDB Hadoop 40

Uber Engineering

OCTOBER 30, 2018

Cluster management, a common software infrastructure among technology companies, aggregates compute resources from a collection of physical hosts into a shared resource pool, amplifying compute power and allowing for the flexible use of data center hardware. At Uber, cluster management … The post Peloton: Uber’s Unified Resource Scheduler for Diverse Cluster Workloads appeared first on Uber Engineering Blog.

Engineering 103

Engineering Management Technology Hadoop 103

Cloudera

OCTOBER 30, 2018

451 Research Analyst Sheryl Kingstone, and Cloudera’s Steve Totman recently discussed how a growing number of organizations are replacing legacy Customer 360 systems with Customer Insights Platforms ( watch the replay here ). In this blog post, Sheryl outlines how next-gen CIP applications are delivering a better customer experience, and why businesses are relying on CIPs as their preferred path to customer insights.

Unstructured Data 42

Unstructured Data Data Lake Algorithm Machine Learning 42

Zalando Engineering

OCTOBER 24, 2018

A Scala 3 Experiment I'll start this post by admitting that I’ve never gone deeply into any kind of Scala coding on the typelevel. It's not what I, as a common application (or microservice) developer, usually need. Having stated that, of course, I might be missing out on a whole world of opportunities for better code without knowing. And because of that, I put some effort into trying to understanding the features of Scala that might sound strange, overly-theoretical, and maybe even useless, at f

Scala 40

Scala Coding Database IT 40

ProjectPro

OCTOBER 5, 2018

Hadoop-as-a-Service: The Need Of The Hour For Superior Business Solutions.InsideBigData.com, September 7, 2018 Hadoop is the cornerstone of the big data industry, however, the challenges involved in maintaining the hadoop network has led to the development and growth of Hadoop-as-a-Service (HaaS) market.Industry research reveals that the global Hadoop-as-a-Service market is anticipated to reach $16.2 billion by 2020 growing a a compound annual growth rate of 70.8% from 2014 to 2020.With market l

Hadoop 40

Hadoop BI Big Data MongoDB 40

Speaker: Jay Allardyce, Deepak Vittal, and Terrence Sheflin

As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.

Data

Rockset

OCTOBER 19, 2018

I love open-source but open-source software for data infrastructure is on the way out. There, I said it. And you might think I've got a screw loose, given the broad adoption of open source today, but hear me out. Yes, open source is ubiquitous in data management today, but the era of open-source innovation is all but over. In the age of public cloud, there is no longer a reason to build or use open source for data infrastructure, and a new category of software I'm labeling open services will ren

MongoDB 40

MongoDB Hadoop Kafka Data Warehouse 40

Data Engineering Podcast

OCTOBER 14, 2018

Summary With the growth of the Hadoop ecosystem came a proliferation of implementations for the Hive table format. Unfortunately, with no formal specification, each project works slightly different which increases the difficulty of integration across systems. The Hive format is also built with the assumptions of a local filesystem which results in painful edge cases when leveraging cloud object storage for a data lake.

Data Lake 100

Data Lake Big Data Cloud Hadoop 100

Uber Engineering

OCTOBER 30, 2018

Cluster management, a common software infrastructure among technology companies, aggregates compute resources from a collection of physical hosts into a shared resource pool, amplifying compute power and allowing for the flexible use of data center hardware. At Uber, cluster management … The post Peloton: Uber’s Unified Resource Scheduler for Diverse Cluster Workloads appeared first on Uber Engineering Blog.

Engineering 62

Engineering Management Technology Hadoop 62

Cloudera

OCTOBER 1, 2018

What an evening! Last week Cloudera hosted over 150 attendees at our annual EMEA Partner Summit in Amsterdam with attendees from over 21 countries across the region. Representatives from across the Cloudera ecosystem came together to hear from company executives and EMEA leadership as well as interactive sessions on Machine Learning, AI and Data Analytics, Cloud and Platform as well training and certification opportunities.

Machine Learning 40

Machine Learning Consulting Cloud Certification 40

Speaker: Nikhil Joshi, Founder & President of Snic Solutions

Is your manufacturing operation reaching its efficiency potential? A Manufacturing Execution System (MES) could be the game-changer, helping you reduce waste, cut costs, and lower your carbon footprint. Join Nikhil Joshi, Founder & President of Snic Solutions, in this value-packed webinar as he breaks down how MES can drive operational excellence and sustainability.

Manufacturing

Zalando Engineering

OCTOBER 17, 2018

The six month journey of the customer inbox multi-disciplinary team The customer inbox multi-disciplinary area operates in the Fashion Store pillar of the Zalando platform organization. The purpose of the Customer Inbox Unit is to serve customers personal and practical fashion messages, through multiple channels, i.e. “Target the customers at the right time, at the right place.

Consulting 40

Consulting Management Engineering IT 40

Data Engineering Podcast

OCTOBER 9, 2018

Summary One of the most complex aspects of managing data for analytical workloads is moving it from a transactional database into the data warehouse. What if you didn’t have to do that at all? MemSQL is a distributed database built to support concurrent use by transactional, application oriented, and analytical, high volume, workloads on the same hardware.

PostgreSQL 100

PostgreSQL BI Machine Learning Data Warehouse 100

Cloudera

OCTOBER 30, 2018

Federated Learning is a technology that allows you to build machine learning systems when your datacenter can’t get direct access to model training data. The data remains in its original location, which helps to ensure privacy and reduces communication costs. Privacy and reduced communication make federated learning a great fit for smartphones and edge hardware, healthcare and other privacy-sensitive use cases, and industrial applications such as predictive maintenance.

Machine Learning 40

Machine Learning Healthcare Manufacturing Accessibility 40

Zalando Engineering

OCTOBER 9, 2018

How we revolutionized the way we worked agile One and a half years ago we started something new at Zalando. We asked all producers of our department to join one team with the purpose of helping us create great teams to get things done in the best way possible. Where did we start from? The producer role had been introduced at Zalando to provide a team with whatever it lacked at a certain moment in time, be it a roadmap, team building, process improvement, documentation or even testing.

Generalist 40

Generalist Retail Recruitment BI 40

Speaker: Anindo Banerjea, CTO at Civio & Tony Karrer, CTO at Aggregage

When developing a Gen AI application, one of the most significant challenges is improving accuracy. This can be especially difficult when working with a large data corpus, and as the complexity of the task increases. The number of use cases/corner cases that the system is expected to handle essentially explodes. 💥 Anindo Banerjea is here to showcase his significant experience building AI/ML SaaS applications as he walks us through the current problems his company, Civio, is solving.

Systems

Zalando Engineering

OCTOBER 3, 2018

Essential building blocks for strong leadership that enables people to grow and achieve results The story of how I ended up working for Zalando in Berlin starts with a LinkedIn message from Joseph Wilkinson, one of our tech recruiters. In tech, we get a lot of messages on LinkedIn, but this one was different and made me very interested to know more about Zalando.

Recruitment 40

Recruitment Building Engineering Designing 40