Data Engineering Podcast

DECEMBER 28, 2020

Summary One of the core responsibilities of data engineers is to manage the security of the information that they process. The team at Satori has a background in cybersecurity and they are using the lessons that they learned in that field to address the challenge of access control and auditing for data governance. In this episode co-founder and CTO Yoav Cohen explains how the Satori platform provides a proxy layer for your data, the challenges of managing security across disparate storage system

Data Governance 130

Data Governance Government Transportation Data 130

Team Data Science

DECEMBER 17, 2020

Organizations in several industries such as banking, healthcare, and automobiles are now acknowledging the value of data science in their mode of operation. Thus, an ideal and efficacious data science team are therefore expected to manage numerous volume of tasks. Even then, developing a team to successfully manage AI tasks is essential to tackle any challenges faced by organizations as regard data.

Data Engineering 130

Data Engineering Data Engineer Engineering Cloud Computing 130

Confluent

DECEMBER 28, 2020

Before you say it: Yes, we are right now three days past Christmas, but technically the 12 days of Christmas refer to the days between Christmas and Epiphany, which is—I […].

Kafka 145

Kafka IT 145

Netflix Tech

DECEMBER 14, 2020

Life of a Netflix Partner Engineer?—?The case of the extra 40 ms By: John Blair , Netflix Partner Engineering The Netflix application runs on hundreds of smart TVs, streaming sticks and pay TV set top boxes. The role of a Partner Engineer at Netflix is to help device manufacturers launch the Netflix application on their devices. In this article we talk about one particularly difficult issue that blocked the launch of a device in Europe.

Bytes 142

Bytes Engineering Coding Manufacturing 142

Advertisement

Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?

Business Intelligence

Cloudera

DECEMBER 15, 2020

It is crucial for organizations to focus on supporting the new way of work, enhancing productivity, and improving cost efficiency to ensure business survival in the post-pandemic world. However, those that are overly focused on these short-term goals risk losing sight of what’s truly important. As shared in my previous post , diverse teams can help organizations unlock innovations that allow them to adapt to market changes quickly and drive business growth.

Programming 111

Programming Building Designing Project 111

DataKitchen

DECEMBER 30, 2020

While 2020 has been a collectively difficult year, we want to take a moment to thank all of our employees for the hard work they put into continually developing our DataKitchen DataOps Platform for our customers. We also want to thank all of the data industry groups that have recognized our DataKitchen DataOps Platform and Transformation Advisory Services throughout the year.

Manufacturing 96

Manufacturing Big Data Data Pipeline Data Science 96

Teradata

DECEMBER 29, 2020

When considering your organization's move to the cloud, it's imperative to understand what the cloud can and cannot do, and how to best leverage its benefits.

Cloud 90

Cloud IT 90

Confluent

DECEMBER 15, 2020

Consuming messages in parallel is what Apache Kafka® is all about, so you may well wonder, why would we want anything else? It turns out that, in practice, there are […].

Process 144

Process Kafka IT 144

Netflix Tech

DECEMBER 23, 2020

By Fabio Kung , Sargun Dhillon , Andrew Spyker , Kyle , Rob Gulewich, Nabil Schear , Andrew Leung , Daniel Muino, and Manas Alekar As previously discussed on the Netflix Tech Blog, Titus is the Netflix container orchestration system. It runs a wide variety of workloads from various parts of the company?—?everything from the frontend API for netflix.com, to machine learning training workloads, to video encoders.

Media 116

Media Accessibility Accessible Systems 116

Cloudera

DECEMBER 11, 2020

In a previous blog post on CDW performance, we compared Azure HDInsight to CDW. In this blog post, we compare Cloudera Data Warehouse (CDW) on Cloudera Data Platform (CDP) using Apache Hive-LLAP to EMR 6.0 (also powered by Apache Hive-LLAP) on Amazon using the TPC-DS 2.9 benchmark. Amazon recently announced their latest EMR version 6.1.0 with support for ACID transactions.

Data Warehouse 110

Data Warehouse Metadata Datasets Data 110

Speaker: Jay Allardyce, Deepak Vittal, and Terrence Sheflin

As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.

Data

DataKitchen

DECEMBER 21, 2020

We understand that many folks would like to say goodbye and good riddance to 2020. But before we shut the door on such a turbulent, transformative year, we at DataKitchen would like to share the creme de la creme of our DataOps content in hopes that it can help you as you learn about and implement DataOps. We hope you and your family have happy holidays and we look forward to continuing your DataOps journey with you in the new year.

IT 84

IT 84

Data Engineering Podcast

DECEMBER 14, 2020

Summary As a data engineer you’re familiar with the process of collecting data from databases, customer data platforms, APIs, etc. At YipitData they rely on a variety of alternative data sources to inform investment decisions by hedge funds and businesses. In this episode Andrew Gross, Bobby Muldoon, and Anup Segu describe the self service data platform that they have built to allow data analysts to own the end-to-end delivery of data projects and how that has allowed them to scale their o

Data Analytics 100

Data Analytics Building MySQL Data 100

Start Data Engineering

DECEMBER 5, 2020

1.

SQL 130

SQL Data Data Process Process 130

Confluent

DECEMBER 11, 2020

This blog post shows how transactional machine learning (TML) integrates data streams with automated machine learning (AutoML), using Apache Kafka® as the data backbone, to create a frictionless machine learning […].

Machine Learning 141

Machine Learning Kafka Data Programming 141

Speaker: Nikhil Joshi, Founder & President of Snic Solutions

Is your manufacturing operation reaching its efficiency potential? A Manufacturing Execution System (MES) could be the game-changer, helping you reduce waste, cut costs, and lower your carbon footprint. Join Nikhil Joshi, Founder & President of Snic Solutions, in this value-packed webinar as he breaks down how MES can drive operational excellence and sustainability.

Manufacturing

Netflix Tech

DECEMBER 11, 2020

In our previous post and QConPlus talk , we discussed GraphQL Federation as a solution for distributing our GraphQL schema and implementation. In this post, we shift our attention to what is needed to run a federated GraphQL platform successfully?—?from our journey implementing it to lessons learned. Our Journey so Far Over the past year, we’ve implemented the core infrastructure pieces necessary for a federated GraphQL architecture as described in our previous post: Studio Edge Architecture The

IT 101

IT Architecture Java Designing 101

Cloudera

DECEMBER 15, 2020

Across nearly every sector working with complex data, Spark has quickly become the de-facto distributed computing framework for teams across the data and analytics lifecycle. One of most awaited features of Spark 3.0 is the new Adaptive Query Execution framework (AQE), which fixes the issues that have plagued a lot of Spark SQL workloads. Those were documented in early 2018 in this blog from a mixed Intel and Baidu team.

SQL 98

SQL Datasets Designing Process 98

Teradata

DECEMBER 16, 2020

Vantage Social Network Analysis (SNA) Framework helps us with useful insights to identify various risk factors associated with the spread of Covid-19. Read more.

66

66

Data Engineering Podcast

DECEMBER 7, 2020

Summary Building data products are complicated by the fact that there are so many different stakeholders with competing goals and priorities. It is also challenging because of the number of roles and capabilities that are necessary to go from idea to delivery. Different organizations have tried a multitude of organizational strategies to improve the success rate of these data teams with varying levels of success.

Building 100

Building Big Data Consulting Data Engineering 100

Speaker: Anindo Banerjea, CTO at Civio & Tony Karrer, CTO at Aggregage

When developing a Gen AI application, one of the most significant challenges is improving accuracy. This can be especially difficult when working with a large data corpus, and as the complexity of the task increases. The number of use cases/corner cases that the system is expected to handle essentially explodes. 💥 Anindo Banerjea is here to showcase his significant experience building AI/ML SaaS applications as he walks us through the current problems his company, Civio, is solving.

Systems

DataKitchen

DECEMBER 20, 2020

It is not the strongest of the species that survives, nor the most intelligent that survives. It is the one that is most adaptable to change. – Leon C. Megginson on Charles Darwin “Origin of Species”. Adapt or face decline. The agile alliance defines “ business agility ” as the ability of an organization to sense changes internally or externally and respond accordingly in order to deliver value to its customers.

Engineering 59

Engineering Manufacturing Data Analytics Data Workflow 59

Confluent

DECEMBER 16, 2020

We’re pleased to announce ksqlDB 0.14, one of the most feature-packed releases of the year. This version includes expanded query support over materialized views, incremental schema alteration, variable substitution, additional […].

Process 126

Process 126

Netflix Tech

DECEMBER 10, 2020

by Melody Dye *, Chaitanya Ekanadham *, Avneesh Saluja *, Ashish Rastogi * contributed equally Netflix is pioneering content creation at an unprecedented scale. Our catalog of thousands of films and series caters to 195M+ members in over 190 countries who span a broad and diverse range of tastes. Content, marketing, and studio production executives make the key decisions that aspire to maximize each series’ or film’s potential to bring joy to our subscribers as it progresses from pitch to play o

Machine Learning 96

Machine Learning Entertainment Metadata Consulting 96

Cloudera

DECEMBER 11, 2020

The Data Security and Governance category, at the annual Data Impact Awards, has never been so important. Consider for a moment, just how much 2020 brought about for businesses to deal with. The sudden rise in remote working, a huge influx in data as the world turned digital, not to mention the never-ending list of regulations businesses need to remain compliant with (how many acronyms can you name in full?

Data Lake 98

Data Lake Government Data Security Unstructured Data 98

Speaker: Donna Laquidara-Carr, PhD, LEED AP, Industry Insights Research Director at Dodge Construction Network

In today’s construction market, owners, construction managers, and contractors must navigate increasing challenges, from cost management to project delays. Fortunately, digital tools now offer valuable insights to help mitigate these risks. However, the sheer volume of tools and the complexity of leveraging their data effectively can be daunting. That’s where data-driven construction comes in.

Project

Teradata

DECEMBER 14, 2020

From COVID-19 to AI in industry, our Teradata experts offer their best predictions for the state of technology and business in 2021 and beyond. Read more.

Technology 64

Technology 64

Grouparoo

DECEMBER 22, 2020

A necessary practice and skill to building a successful product is having accurate, accessible, and actionable data. I’ve had the privilege to work at some great companies that are extremely data-focused, and I’ve learned from some of the best along the way. Quantitative user data At Zynga , we had millions of daily active users, where we tracked every click, action, and session across all of our games and apps.

Project 52

Project Entertainment Recruitment SQL 52

Data Council

DECEMBER 21, 2020

Here's our December 2020 roundup of links from across the web that could be relevant to you: 1. The Modern Data Stack (Fishtown Analytics) This long-form post on the dbt blog is a must-read. Titled “The Modern Data Stack: Past, Present, and Future,” it answers the question that Tristan Handy has been asking himself for the past two years: “What happened to the massive innovation we saw from 2012-2016?

Metadata 52

Metadata Architecture Data Technology 52

Confluent

DECEMBER 17, 2020

Microservice architectures continue to grow within engineering organizations as teams strive to increase development velocity. Microservices promote the idea of modularity as a first-class citizen in a distributed architecture, enabling […].

Architecture 122

Architecture Engineering Kafka 122

Speaker: Aindra Misra, Senior Manager, Product Management (Data, ML, and Cloud Infrastructure) at BILL

Join us for an insightful webinar that explores the critical intersection of data privacy and AI governance. In today’s rapidly evolving tech landscape, building robust governance frameworks is essential to fostering innovation while staying compliant with regulations. Our expert speaker, Aindra Misra, will guide you through best practices for ensuring data protection while leveraging AI capabilities.

Government

Netflix Tech

DECEMBER 21, 2020

By Anupom Syam Background At Netflix, our current data warehouse contains hundreds of Petabytes of data stored in AWS S3 , and each day we ingest and create additional Petabytes. At this scale, we can gain a significant amount of performance and cost benefits by optimizing the storage layout (records, objects, partitions) as the data lands into our warehouse.

Data Warehouse 83

Data Warehouse Metadata Algorithm Data 83

Cloudera

DECEMBER 10, 2020

Why worry about costs with cloud-native data warehousing? Have you been burned by the unexpected costs of a cloud data warehouse? If so, you know about the failed economics of some cloud-native solutions on the market today. If not, before adopting a cloud data warehouse, consider the true costs of a cloud-native data warehouse. Data warehouses have been broadly adopted to provide timely reports and valuable insights.

Data Warehouse 98

Data Warehouse Cloud Storage Metadata Data 98

Teradata

DECEMBER 27, 2020

A medical bill has hundreds of thousands of possible medical diagnoses & procedure combinations. This complexity presents a competitive advantage if overlaid with the right analytics.

Medical 52

Medical 52

Grouparoo

DECEMBER 21, 2020

Yesterday, I was writing some blog posts. In grand engineer tradition, I got distracted while blogging and spent a few hours writing tools to increase blogging efficiency. Specifically, I was having trouble knowing the correct width and height props to put on the screenshots I was making for the blog post. I would take the screenshot and then use image tools and even a spreadsheet to figure out the right ratio/dimensions for how I wanted it to show up in the UI.

Coding 52

Coding Engineering IT 52

Advertisement

Entity Resolution Sometimes referred to as data matching or fuzzy matching, entity resolution, is critical for data quality, analytics, graph visualization and AI. Learn what entity resolution is, why it matters, how it works and its benefits. Advanced entity resolution using AI is crucial because it efficiently and easily solves many of today’s data quality and analytics problems.

IT