Start Data Engineering

JUNE 11, 2020

I have seen, heard and been asked questions and comments like What is Kafka and When should I use it? I don’t understand why we have to use Kafka The objective of this post is to get you up to speed with what Apache Kafka is, when to use them and the foundational concepts of Apache Kafka with a simple example. What is Apache Kafka First let’s understand what Apache Kafka is.

Kafka 130

Kafka IT 130

Team Data Science

JUNE 6, 2020

Welcome back to this Toronto Specific data engineering project. We left off last time concluding finance has the largest demand for data engineers who have skills with AWS, and sketched out what our data ingestion pipeline will look like. I began building out the data ingestion pipeline by launching an EC2 instance. I should note that if you have created an AWS account, but have not yet created an Identity Access Management (IAM) admin role, and are therefore still using root credentials, I am s

Project 130

Project Management Data Ingestion AWS 130

Confluent

JUNE 9, 2020

tl;dr When a client wants to send or receive a message from Apache Kafka®, there are two types of connection that must succeed: The initial connection to a broker (the […].

Kafka 123

Kafka Java Python AWS 123

Teradata

JUNE 9, 2020

Teradata's own Sir Freek Cox on dedicating one's life to charity and good works. Read more.

105

105

Advertisement

Whether you’re creating complex dashboards or fine-tuning large language models, your data must be extracted, transformed, and loaded. ETL and ELT pipelines form the foundation of any data product, and Airflow is the open-source data orchestrator specifically designed for moving and transforming data in ETL and ELT pipelines. This eBook covers: An overview of ETL vs.

Architecture

Data Engineering Podcast

JUNE 8, 2020

Summary The landscape of data management and processing is rapidly changing and evolving. There are certain foundational elements that have remained steady, but as the industry matures new trends emerge and gain prominence. In this episode Astasia Myers of Redpoint Ventures shares her perspective as an investor on which categories she is paying particular attention to for the near to medium term.

Data Management 100

Data Management Management Machine Learning Portfolio 100

Monte Carlo

JUNE 11, 2020

One of our customers recently posed this question related to data quality metrics: I would like to set up an OKR for ourselves [the data team] around data availability. I’d like to establish a single data quality KPI that would summarize availability, freshness, quality. What’s the best way to do this? I can’t tell you how much joy this request brought me.

Data 59

Data Data Pipeline BI Data Engineer 59

Teradata

JUNE 9, 2020

Teradata has an extraordinary legacy; yet only in the IT world is legacy looked at as a negative. Learn how Teradata's legacy gives it the ultimate competitive advantage.

IT 75

IT 75

Rock the JVM

JUNE 10, 2020

FP fans discuss the challenge of type classes in pure functional programming with Scala: why are they difficult, and why do we really need them?

Scala 52

Scala Programming 52

Rockset

JUNE 9, 2020

In the course of implementing the Rockset connector to MongoDB , we did a fair amount of research on the MongoDB user experience, both online and through user interviews. We learned a lot about how organizations operated MongoDB in production and found that many of our discussions invariably touched upon what it took to achieve performance at scale.

MongoDB 52

MongoDB NoSQL Database Kafka 52

Confluent

JUNE 8, 2020

We are excited to announce the preview release of the fully managed MongoDB Atlas source and sink connectors in Confluent Cloud, our fully managed event streaming service based on Apache […].

MongoDB 77

MongoDB Cloud Management Kafka 77

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

Data

Teradata

JUNE 11, 2020

Why is Teradata uniquely positioned to help the modern enterprise unlock the business value of data, quickly and coherently? Read more.

Data 69

Data 69

Rock the JVM

JUNE 10, 2020

FP fans discuss the challenge of type classes in pure functional programming with Scala: why are they difficult, and why do we really need them?

Scala 52

Scala Programming 52

Preset

JUNE 7, 2020

Create a first chart with Superset

Building 52

Building 52

Confluent

JUNE 10, 2020

At Confluent, every now and then we like to take a day away from our normal sprint tasks to hack. There are a ton of benefits to hack days, including: […].

66

66

Advertisement

Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.

Cloud

Teradata

JUNE 7, 2020

New Teradata President and CEO Steve McMillan on the reasons why he joined the company as chief executive and the opportunities he sees for the future.

69

69