Confluent
APRIL 9, 2020
Many companies have recently started to take cybersecurity and data protection even more seriously, particularly driven by the recent General Data Protection Regulation (GDPR) legislation. They are increasing their investment […].
Start Data Engineering
APRIL 24, 2020
In this article we aim to go over the reasoning behind why someone might want to use dbt. If you are interested in learning dbt checkout this article.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Teradata
APRIL 16, 2020
How can Teradata's data analytics platform help communities stop the spread of COVID-19? Find out more.
Data Engineering Podcast
APRIL 27, 2020
Summary Data is a critical element to every role in an organization, which is also what makes managing it so challenging. With so many different opinions about which pieces of information are most important, how it needs to be accessed, and what to do with it, many data projects are doomed to failure. In this episode Chris Bergh explains how taking an agile approach to delivering value can drive down the complexity that grows out of the varied needs of the business.
Speaker: Tamara Fingerlin, Developer Advocate
Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.
Netflix Tech
APRIL 20, 2020
How Netflix brings safer and faster streaming experience to the living room on crowded networks using TLS 1.3 By Sekwon Choi At Netflix, we are obsessed with the best streaming experiences. We want playback to start instantly and to never stop unexpectedly in any network environment. We are also committed to protecting users’ privacy and service security without sacrificing any part of the playback experience.
U-Next
APRIL 17, 2020
Workforce analytics – what sounds like a complex technical term is relatively easy and inevitable when you understand its importance and how to leverage its full potential to your benefit. Some of the elite HR analysts out there have implemented workforce analytics in their businesses and have seen results that their competitors couldn’t dream of. As an HR manager, you need to be aware of these analytics concepts and theories to pave way for organizational and departmental success and curb any b
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
Start Data Engineering
APRIL 17, 2020
When getting started with Apache Airflow , data engineers have questions similar to the two below “What are people’s opinions of Airflow?
Teradata
APRIL 6, 2020
By fully utilizing the data for telco operators in China, Teradata helped communities battle the COVID-19 epidemic through ongoing public health communication, travel updates and inquiries.
Data Engineering Podcast
APRIL 19, 2020
Summary Modern applications frequently require access to real-time data, but building and maintaining the systems that make that possible is a complex and time consuming endeavor. Eventador is a managed platform designed to let you focus on using the data that you collect, without worrying about how to make it reliable. In this episode Eventador Founder and CEO Kenny Gorman describes how the platform is architected, the challenges inherent to managing reliable streams of data, the simplicity off
Netflix Tech
APRIL 2, 2020
By Haruka Miyagawa & Kylee Peña Continue reading on Netflix TechBlog ».
Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage
There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.
Rock the JVM
APRIL 29, 2020
Master file reading in Scala with ease: compare it to other languages and discover how our simple API approach is almost as straightforward as Python's read()
Confluent
APRIL 21, 2020
Confluent Raises $250M and Kicks Off Project Metamorphosis It’s an exciting day for Confluent, in the middle of a very unusual and difficult time in the larger world. Nonetheless, I […].
Start Data Engineering
APRIL 10, 2020
Many data engineers coming from traditional batch processing frameworks have questions about real time data processing systems, like “What kind of data model did you implement, for real-time processing?
Teradata
APRIL 9, 2020
Teradata participated in the MIT COVID Challenge Hackathon to design approaches and mentor teams focused on stamping out the pandemic. Learn more.
Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives
Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri
Data Engineering Podcast
APRIL 13, 2020
Summary The software applications that we build for our businesses are a rich source of data, but accessing and extracting that data is often a slow and error-prone process. Rookout has built a platform to separate the data collection process from the lifecycle of your code. In this episode, CTO Liran Haimovitch discusses the benefits of shortening the iteration cycle and bringing non-engineers into the process of identifying useful data.
Advancing Analytics: Data Engineering
APRIL 23, 2020
The thought of learning Scala fills many with fear, its very name often causes feelings of terror. This suggests it’s either doing something very good, or very bad! The truth is Scala can be used for many things; from a simple web application to complex ML (Machine Learning). Moreover, it unusually fully incorporates two programming paradigms: OOP (Object Orientated Programming) and FP (Functional programming).
Rock the JVM
APRIL 27, 2020
Maximize call-by-name semantics in Scala: control when your results are evaluated
Confluent
APRIL 24, 2020
We are pleased to announce the release of Confluent Platform 5.5. With this release, Confluent makes event streaming more broadly accessible to developers of all backgrounds, enhancing three categories of […].
Advertisement
With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you
Start Data Engineering
APRIL 4, 2020
One of the most common issues when ingesting and processing user generated events is, how to deal with late arriving events. Yet this topic is not extensively discussed. Some of the general issues that data engineers usually have are “What should be considered the event time?
Teradata
APRIL 8, 2020
There is no ROI in technology - specifically in data analytics, AI & Machine Learning - until we deploy in production and change the way we do business.
Data Engineering Podcast
APRIL 6, 2020
Summary Knowledge graphs are a data resource that can answer questions beyond the scope of traditional data analytics. By organizing and storing data to emphasize the relationship between entities, we can discover the complex connections between multiple sources of information. In this episode John Maiden talks about how Cherre builds knowledge graphs that provide powerful insights for their customers and the engineering challenges of building a scalable graph.
Hepta Analytics
APRIL 17, 2020
Last year in September, Hepta Analytics was amongst the few startup companies invited to participate in the World Economic Forum for Africa in Cape Town, South Africa. Such a rare opportunity for a young company like ours given the high profile individuals, such as heads of states invited to attend these types of events. It also included a great mix of local, regional and international companies execs, academic and civil society leaders, all coming together to discuss one thing: Shaping inclusi
Advertisement
Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.
Rock the JVM
APRIL 22, 2020
Discover the essential skill for optimizing Spark performance: mastering the Spark UI and understanding the job execution graph
Confluent
APRIL 27, 2020
When Confluent Schema Registry was first introduced, Apache Avro™ was initially chosen as the default format. While Avro has worked well for many users, over the years, we’ve received many […].
U-Next
APRIL 17, 2020
There’s always a sense of apprehension when someone walks down to the HR desk to put down their papers. More so if it is a key employee whose loss is going to be a definite setback. Then people wonder – the upper management, the line manager, the HR department – how it is that they never saw this coming. There used to be a time when employee retention processes would kick in only after an employee resigned.
Teradata
APRIL 29, 2020
Lots of smart people have created many predictive analytics models to help us manage the COVID-19 pandemic. But many of these models use different inputs, different heuristics, and come to different conclusions.
Advertisement
Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?
Grouparoo
APRIL 14, 2020
Grouparoo is the Reverse ETL platform to connect Google Sheets data to your SaaS tools. This enables all of those crazy sheets out there to be the source of truth for your profiles and be fed into your marketing tools. Don't forget: with great power comes great responsibility! Google setup In Grouparoo, apps make the connection to facilitate data movement in the form of sources and destinations.
Rockset
APRIL 28, 2020
Recently, InfoWorld’s Martin Heller described Rockset as a "one-of-a-kind database for operational analytics." After testing Rockset with a variety of queries on a large collection, Heller rated Rockset 4.5 out of 5 stars. Heller’s review of Rockset can be found here. Only one of the test queries timed out: SELECT * FROM commons."twitter-firehose" ORDER BY "twitter-firehose".favorite_count DESC LIMIT 10 For context, twitter-firehose is one of Rockset’s demo collections.
Rock the JVM
APRIL 22, 2020
Discover the essential skill for optimizing Spark performance: mastering the Spark UI and understanding the job execution graph
Confluent
APRIL 28, 2020
Mainframes are still ubiquitous, used for almost every financial transaction around the world—credit card transactions, billing, payroll, etc. You might think that working on mainframe software would be dull, requiring […].
Speaker: Tamara Fingerlin, Developer Advocate
In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!
Expert insights. Personalized for you.
145 
Let's personalize your content