Start Data Engineering

OCTOBER 12, 2020

Table of Contents Table of Contents Introduction Design Setup Prerequisites Clone repository Get data Code Move data and script to the cloud create an EMR cluster add steps and wait to complete terminate EMR cluster Run the DAG Conclusion Further reading Introduction I have been asked and seen the questions how others are automating apache spark jobs on EMR how to submit spark jobs to an EMR cluster from Airflow ?

Cloud 130

Cloud Coding Designing Data 130

Confluent

OCTOBER 16, 2020

Apache Kafka® is an event streaming platform used by more than 30% of the Fortune 500 today. There are numerous features of Kafka that make it the de-facto standard for […].

Kafka 145

Kafka IT 145

Cloudera

OCTOBER 1, 2020

Since 2013 the UK Government’s flagship ‘Cloud First’ policy has been at the forefront of enabling departments to shed their legacy IT architecture in order to meaningfully embrace digital transformation. The policy outlines that the cloud (and specifically, public cloud) be the default position for any new services; unless it can be demonstrated that other alternatives offer better value for money. .

Government 128

Government Cloud Data Storage Architecture 128

Data Engineering Podcast

OCTOBER 26, 2020

Summary One of the most challenging aspects of building a data platform has nothing to do with pipelines and transformations. If you are putting your workflows into production, then you need to consider how you are going to implement data security, including access controls and auditing. Different databases and storage systems all have their own method of restricting access, and they are not all compatible with each other.

Data Security 100

Data Security Coding Cloud MongoDB 100

Speaker: Aindra Misra, Senior Manager, Product Management (Data, ML, and Cloud Infrastructure) at BILL

Join us for an insightful webinar that explores the critical intersection of data privacy and AI governance. In today’s rapidly evolving tech landscape, building robust governance frameworks is essential to fostering innovation while staying compliant with regulations. Our expert speaker, Aindra Misra, will guide you through best practices for ensuring data protection while leveraging AI capabilities.

Government

Teradata

OCTOBER 13, 2020

The most frequently asked question of Finance departments today is, ‘whose data do we trust’? Here’s how to ensure Finance always has the correct answer.

Finance 117

Finance Data 117

Netflix Tech

OCTOBER 19, 2020

by Maulik Pandey Our Team?—? Kevin Lew , Narayanan Arunachalam , Elizabeth Carretto , Dustin Haffner , Andrei Ushakov, Seth Katz , Greg Burrell , Ram Vaithilingam , Mike Smith and Maulik Pandey “ @Netflixhelps Why doesn’t Tiger King play on my phone?”?—?a Netflix member via Twitter This is an example of a question our on-call engineers need to answer to help resolve a member issue?

Building 94

Building Transportation Metadata Java 94

Confluent

OCTOBER 23, 2020

Apache Kafka® is a distributed real-time processing platform that allows for the ingestion of huge volumes of data. ksqlDB is part of the Kafka ecosystem and offers a SQL-like language […].

Kafka 143

Kafka SQL Process Data 143

Cloudera

OCTOBER 26, 2020

Growing up, were you ever told you can’t have it all? That you can’t eat all the snacks in one sitting? That you can’t watch the complete Back to the Future trilogy as well as study for your science exam in one evening? Over time, we learn to set priorities, make a decision for one thing over the other, and compromise. Just like when it comes to data access in business.

Data Security 120

Data Security IT Unstructured Data Government 120

Data Engineering Podcast

OCTOBER 19, 2020

Summary In order for analytics and machine learning projects to be useful, they require a high degree of data quality. To ensure that your pipelines are healthy you need a way to make them observable. In this episode Barr Moses and Lior Gavish, co-founders of Monte Carlo, share the leading causes of what they refer to as data downtime and how it manifests.

Machine Learning 100

Machine Learning Data Engineering Data Engineer Data 100

Teradata

OCTOBER 10, 2020

Successful companies need to squeeze maximum value from all of their data & do it at the lowest possible cost. But they often get hit with unexpected budget overruns. Teradata can help.

Data Warehouse 98

Data Warehouse Cloud Data IT 98

Speaker: Christophe Louvion, Chief Product & Technology Officer of NRC Health and Tony Karrer, CTO at Aggregage

Christophe Louvion, Chief Product & Technology Officer of NRC Health, is here to take us through how he guided his company's recent experience of getting from concept to launch and sales of products within 90 days. In this exclusive webinar, Christophe will cover key aspects of his journey, including: LLM Development & Quick Wins 🤖 Understand how LLMs differ from traditional software, identifying opportunities for rapid development and deployment.

Designing

Domino Data Lab: Data Engineering

OCTOBER 6, 2020

Danger of Big Data Big data is the rage. This could be lots of rows (samples) and few columns (variables) like credit card transaction data, or lots of columns (variables) and few rows (samples) like genomic sequencing in life sciences research. The Curse of Dimensionality , or Large P, Small N, ((P >> N)) problem applies to the latter case of lots of variables measured on a relatively few number of samples.

Big Data 52

Big Data Deep Learning Data Datasets 52

Grouparoo

OCTOBER 15, 2020

Two of the major components of the @grouparoo/core application are a Node.js API server and a React frontend. We use Actionhero as the API server, and Next.JS for our React site generator. As we develop the Grouparoo application, we are constantly adding new API endpoints and changing existing ones. One of the great features of Typescript is that it can help not only to share type definitions within a codebase, but also across multiple codebases or services.

Utilities 52

Utilities Coding IT Data 52

Confluent

OCTOBER 1, 2020

Each month, we’ve announced a set of Confluent features organized around what we think are the key foundational traits of cloud-native data systems as part of Project Metamorphosis. Data systems […].

Project 142

Project Cloud Systems Data 142

Cloudera

OCTOBER 13, 2020

Recently, my colleague published a blog build on your investment by Migrating or Upgrading to CDP Data Center , which articulates great CDP Private Cloud Base features. Existing CDH and HDP customers can immediately benefit from this new functionality. This blog focuses on the process to accelerate your CDP journey to CDP Private Cloud Base for both professional services engagements and self-service upgrades.

Professional Services 115

Professional Services Cloud Relational Database Utilities 115

Speaker: Jamie Bernard

Success in product management goes beyond delivering great features - it’s about achieving measurable financial outcomes that resonate across the organization. By connecting your product’s journey with the company’s financial success, you’ll ensure that every feature, release, and innovation contributes to the bottom line, driving both customer satisfaction and business growth.

Management

Data Engineering Podcast

OCTOBER 12, 2020

Summary Business intelligence efforts are only as useful as the outcomes that they inform. Power BI aims to reduce the time and effort required to go from information to action by providing an interface that encourages rapid iteration. In this episode Rob Collie shares his enthusiasm for the Power BI platform and how it stands out from other options.

Business Intelligence 100

Business Intelligence BI Consulting Data Ingestion 100

Teradata

OCTOBER 12, 2020

In honor of World Mental Health Day this past weekend, Shehzeen Rehman writes on the importance of de-stigmatizing mental health and learning how to seek help.

98

98

Rockset

OCTOBER 27, 2020

I’m excited to soon be the newest member of Rockset. I will be joining a truly spectacular engineering team, working on a product that leverages deep technical insights to make real-time analytics easy. My passion is building infrastructure that makes things simpler for users, supporting people at higher levels of the stack by giving them clean APIs and predictable behavior.

Algorithm 40

Algorithm Utilities Cloud Coding 40

Preset

OCTOBER 1, 2020

Build a beautiful Slack dashboard using open source tools Meltano and Superset. Part 2 of 3.

Building 40

Building 40

Advertisement

Entity Resolution Sometimes referred to as data matching or fuzzy matching, entity resolution, is critical for data quality, analytics, graph visualization and AI. Learn what entity resolution is, why it matters, how it works and its benefits. Advanced entity resolution using AI is crucial because it efficiently and easily solves many of today’s data quality and analytics problems.

IT

Confluent

OCTOBER 27, 2020

As described in the blog post Apache Kafka® Needs No Keeper: Removing the Apache ZooKeeper Dependency, when KIP-500 lands next year, Apache Kafka will replace its usage of Apache ZooKeeper […].

Kafka 140

Kafka IT 140

Cloudera

OCTOBER 30, 2020

Being a woman in tech can be incredibly rewarding, lonely, frustrating and inspiring all at once. Each individual has their own experience and path that they’ve followed to get where they are. That is, after all, what makes us unique. Earlier this week Cindy Maike, VP Industry Solutions, hosted a panel discussion with women across the Cloudera EMEA business, working in a variety of different roles; each of us with diverse backgrounds and perspectives, which made for a wide-ranging discussion. .

Architecture 110

Architecture Technology Project Engineering 110

Data Engineering Podcast

OCTOBER 5, 2020

Summary Analytical workloads require a well engineered and well maintained data integration process to ensure that your information is reliable and up to date. Building a real-time pipeline for your data lakes and data warehouses is a non-trivial effort, requiring a substantial investment of time and energy. Meroxa is a new platform that aims to automate the heavy lifting of change data capture, monitoring, and data loading.

Data Integration 100

Data Integration Kafka Data Lake Machine Learning 100

Teradata

OCTOBER 5, 2020

Teradata Vantage on Google Cloud is now generally available! Vantage on Google Cloud is an as-a-service offer in which customers can get the most analytic value from their data. Read more.

Google Cloud 82

Google Cloud Cloud Data 82

Advertisement

The complexity of financial data, the need for real-time insight, and the demand for user-friendly visualizations can seem daunting when it comes to analytics - but there is an easier way. With Logi Symphony, we aim to turn these challenges into opportunities. Our platform empowers you to seamlessly integrate advanced data analytics, generative AI, data visualization, and pixel-perfect reporting into your applications, transforming raw data into actionable insights.

Raw Data

Rockset

OCTOBER 27, 2020

Today, Rockset announced $40M in Series B funding from Sequoia and Greylock , our two investors who have partnered with us right from the beginning. Additionally, we announced support for fully managed, secure private deployments of Rockset within a customer’s Amazon VPC. These are important milestones for both our company and product, but this announcement is less a celebration of Rockset than a recognition of our hundreds of beloved customers who have launched amazing real-time applications.

Building 40

Building Data Warehouse Transportation Retail 40

Preset

OCTOBER 7, 2020

In part 2, we'll start to visualize trends using Superset charts.

Building 40

Building 40

Confluent

OCTOBER 13, 2020

All around the world, companies are asking the same question: What is happening right now? We are inundated with pieces of data that have a fragment of the answer. But […].

Data 120

Data Process 120

Cloudera

OCTOBER 8, 2020

As Cloudera offices around the world continue to cope with the impact of COVID-19, we have worked hard to ease stress and adapt to remote working. People are the heart of our company and we’re investing in creative, new ways to make every Clouderan feel valued and appreciated. Clouderans are superstars at work and at home, and burn-out is unhealthy for employees, their families, and the company.

Designing 104

Designing Programming Coding Management 104

Speaker: Timothy Chan, PhD., Head of Data Science

Are you ready to move beyond the basics and take a deep dive into the cutting-edge techniques that are reshaping the landscape of experimentation? 🌐 From Sequential Testing to Multi-Armed Bandits, Switchback Experiments to Stratified Sampling, Timothy Chan, Data Science Lead, is here to unravel the mysteries of these powerful methodologies that are revolutionizing how we approach testing.

Data Science

Netflix Tech

OCTOBER 29, 2020

Netflix Android and iOS Studio Apps?—?now powered by Kotlin Multiplatform By David Henry & Mel Yahya Over the last few years Netflix has been developing a mobile app called Prodicle to innovate in the physical production of TV shows and movies. The world of physical production is fast-paced, and needs vary significantly between the country, region, and even from one production to the next.

Coding 111

Coding Architecture Technology Database 111

Teradata

OCTOBER 26, 2020

Our new global survey reveals how business leaders are changing the way they think about about data -- from their trust in to to the role it plays in a post-pandemic recovery.

Data 64

Data IT 64

Rockset

OCTOBER 8, 2020

Regardless of what database you pick to run your application—MongoDB, Postgres, Oracle, or Cassandra—you will eventually encounter the same issue: slow queries. Slow queries can be the result of inefficient query design, inefficient table design, or general infrastructure problems. Although it may be tempting to add more machines or further complicate your data infrastructure to speed up your queries, improving the queries themselves is usually the best place to start when you want to improve da

MongoDB 40

MongoDB Database Insurance Healthcare 40

Preset

OCTOBER 4, 2020

A high level tour of Apache Superset's REST API

40

40

Speaker: Conrado Morlan

In this engaging and witty talk, industry expert Conrado Morlan will explore how artificial intelligence can transform the daily tasks of product managers into streamlined, efficient processes. Using the lens of a superhero narrative, he’ll uncover how AI can be the ultimate sidekick, aiding in data management and reporting, enhancing productivity, and boosting innovation.

Management