Uber Engineering
MAY 19, 2021
API gateways are an integral part of microservices architecture in recent years. An API gateway provides a single point of entry for all our apps and provides an interface to access data, logic, or functionality from back-end microservices. It also … The post The Architecture of Uber’s API gateway appeared first on Uber Engineering Blog.
Cloudera
MAY 19, 2021
Introduction. In the previous blog post in this series, we walked through the steps for leveraging Deep Learning in your Cloudera Machine Learning (CML) projects. This year, we expanded our partnership with NVIDIA , enabling your data teams to dramatically speed up compute processes for data engineering and data science workloads with no code changes using RAPIDS AI.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Confluent
MAY 18, 2021
With so many technologies in the modern development ecosystem, a common complaint is having to go through the mental gymnastics of adopting new products and keeping up with ever-expanding feature […].
Data Engineering Podcast
MAY 20, 2021
Summary Data governance is a phrase that means many different things to many different people. This is because it is actually a concept that encompasses the entire lifecycle of data, across all of the people in an organization who interact with it. Stijn Christiaens co-founded Collibra with the goal of addressing the wide variety of technological aspects that are necessary to realize such an important and expansive process.
Speaker: Tamara Fingerlin, Developer Advocate
Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.
Teradata
MAY 15, 2021
The concept of Data Mesh is abuzz in the industry right now. Find out why we're so enthusiastic about it.
Cloudera
MAY 18, 2021
This article is the second in a multipart series to showcase the power and expressibility of FlinkSQL applied to market data. In case you missed it, part I starts with a simple case of calculating streaming VWAP. Code and data for this series are available on github. Speed matters in financial markets. Whether the goal is to maximize alpha or minimize exposure, financial technologists invest heavily in having the most up-to-date insights on the state of the market and where it is going.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
|
Data Engineering Podcast
MAY 18, 2021
Summary Data lineage is the common thread that ties together all of your data pipelines, workflows, and systems. In order to get a holistic understanding of your data quality, where errors are occurring, or how a report was constructed you need to track the lineage of the data from beginning to end. The complicating factor is that every framework, platform, and product has its own concepts of how to store, represent, and expose that information.
Teradata
MAY 15, 2021
The concept of Data Mesh is abuzz in the industry right now. Find out why we're so enthusiastic about it.
Cloudera
MAY 20, 2021
Meet Kathleen Merto. To her colleagues, she’s Kat. . She works on our Emerging Talent team managing the hiring process for Interns and entry level roles. It’s a job she feels passionately about, so much so that she was eager to give her whole team a shout out! . Kat fell into the perfect career path for her. Growing up, she witnessed her mom, a nurse of 41 years now, dedicate so much of herself to helping others.
Cloud Academy
MAY 20, 2021
This is part 4 of a 5-part series on best practices for enterprise cloud migration. Released weekly from the end of April to the end of May 2021, each article will cover a new phase of a business’s transition to the cloud, what to be on the lookout for, and how to ensure the journey is a success. Be sure to subscribe to our blog to be notified when new content goes live!
Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage
There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.
A Cloud Guru: Data Engineering
MAY 20, 2021
A while back we published the Visual Guide to Azure Fundamentals on A Cloud Guru. The post got a lot of positive feedback so we thought we’d do another one — this time focused on Azure Data Factory! What is a visual guide? Visual guides are hi-resolution “sketchnotes.” They summarize a given topic or content […] The post A visual guide to Azure Data Factory appeared first on A Cloud Guru.
Grouparoo
MAY 20, 2021
The rise in popularity of frontend libraries and frameworks like React, Vue and Angular make it easier than ever before to build rich and interactive web apps. Pair these powerful libraries with a nice API to pull some data, and you can pretty quickly build out complex use cases. However, the ability to do so many things on the client side doesn't always mean you should.
Cloudera
MAY 17, 2021
Making a decision on a cloud data warehouse is a big deal. Beyond there being a number of choices each with very different strengths, the parameters for your decision have also changed. Modernizing your data warehousing experience with the cloud means moving from dedicated, on-premises hardware focused on traditional relational analytics on structured data to a modern platform.
Teradata
MAY 20, 2021
CEOs need a new relationship with data if they are to successfully transition to the hyper-personalized, hyper-localized future most recognize as today’s immediate imperative.
Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives
Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri
Confluent
MAY 20, 2021
When it comes to launching your next app with data in motion, few things pose the same risk to going live as meeting requirements for data security and compliance. Doing […].
Retail Insight
MAY 18, 2021
Elite sport is a results business. T he difference between winning and losing often com es down to the finest of margins. As Al Pacino said in Any Given Sunday , it is all about the ‘inches’.
Cloudera
MAY 17, 2021
CDP (Cloudera Data Platform) Private Cloud 1.2 was recently released and builds on the success of CDP Private Cloud Base (see the 7.1.6 release blog ). While Private Cloud Base is the ideal modernization of both CDH and HDP deployments for traditional workloads, Private Cloud adds cloud-native capabilities. In this blog, we’ll cover the complete range of new capabilities and updates for CDP Private Cloud as a whole (the platform) as well as for both the CDW (Cloudera Data Warehouse) and CML (Clo
Teradata
MAY 20, 2021
CEOs need a new relationship with data if they are to successfully transition to the hyper-personalized, hyper-localized future most recognize as today’s immediate imperative.
Advertisement
With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you
Grouparoo
MAY 18, 2021
When I was in charge of Product/Engineering at TaskRabbit, it was always challenging to prioritize integrations being requested by our Marketing, Sales, and Customer Success teams. First and foremost, most engineers just hate working on these kinds of integrations. Often, this preference alone is the deciding factor in organizations for what gets prioritized or not.
Booking.com Engineering
MAY 18, 2021
Introduction Threads are essential for responsive UI applications. When programming in Android, we make sure that any kind of work that could cause the slightest lagging is scheduled to a separate thread, other than the one responsible for the UI updates. And even though there are various high level constructs available for the developer’s convenience, how threading works at a very low level leaks from all these abstractions nonetheless.
Cloudera
MAY 18, 2021
Introduction. Prior the introduction of CDP Public Cloud, many organizations that wanted to leverage CDH, HDP or any other on-prem Hadoop runtime in the public cloud had to deploy the platform in a lift-and-shift fashion, commonly known as “Hadoop-on-IaaS” or simply the IaaS model. Even though that approach addressed the short-term need of moving to the cloud, it has had three significant disadvantages: .
Scribd Technology
MAY 18, 2021
Streaming data from Apache Kafka into Delta Lake is an integral part of Scribd’s data platform, but has been challenging to manage and scale. We use Spark Structured Streaming jobs to read data from Kafka topics and write that data into Delta Lake tables. This approach gets the job done but in production our experience has convinced us that a different approach is necessary to efficiently bring data from Kafka to Delta Lake.
Speaker: Tamara Fingerlin, Developer Advocate
In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!
Teradata
MAY 18, 2021
The data-driven, digital-first era has multiplied the complexity of customer conversations – but it has also provided the means to generate and act on real insight.
DataKitchen
MAY 17, 2021
The post How XOps Is Hoping to Unite All the Disparate Ops Disciplines Under One Banner first appeared on DataKitchen.
Cloudera
MAY 21, 2021
Enterprises are adopting a hybrid cloud approach. While more Cloudera customers want to move apps and data to the cloud, they also want to continue using their data centers for security and governance. By having both on-premises and cloud environments, organizations increase their agility, and hybrid model is gaining momentum. A hybrid approach benefits many organizations as it allows them to make best use of on-premises infrastructure while taking advantage of additional compute capacity and l
Rock the JVM
MAY 16, 2021
In this article, we explore the main testing styles in Scala and ScalaTest: understanding what terms like 'FunSuite' and 'FlatSpec' really mean
Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage
When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m
Rockset
MAY 21, 2021
In 2008, Dominos Pizza released its pizza tracker so that fans could monitor in real time if their pizza was in the oven or out for delivery. By 2019, 65% of Dominos’ sales came through digital channels including home devices and emoji texts, reimagining the brand for the digital era. The Dominos’ Pizza Tracker is the quintessential example of real-time analytics.
RudderStack
MAY 21, 2021
Our latest sprint focused on improving data reporting in multiple views of the product and adding several marketing integrations.
Monte Carlo
MAY 19, 2021
Monte Carlo today announced Berlin-based microlearning app Blinkist has selected Monte Carlo to achieve more reliable data through data observability. As a high-growth company with over 16 million users worldwide, Blinkist leverages paid performance marketing to fuel customer acquisition — and those channels rely on accurate behavioral data to optimize campaign spend.
DataKitchen
MAY 17, 2021
The post Connecting Your Data Mesh with DataOps first appeared on DataKitchen.
Advertisement
Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.
Expert insights. Personalized for you.
145 
Let's personalize your content