Uber Engineering
NOVEMBER 10, 2021
Introduction. Each day, Uber moves millions of people around the world and delivers tens of millions of food and grocery orders. This generates a large number of financial transactions that need to be stored with provable completeness, consistency, and compliance. … The post How Uber Migrated Financial Data from DynamoDB to Docstore appeared first on Uber Engineering Blog.
Confluent
NOVEMBER 8, 2021
How does Confluent provide fine-grained operational visibility to our customers throughout all of the multi-tenant services that we run in the cloud? At Confluent Cloud, we manage a large number […].
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Start Data Engineering
NOVEMBER 11, 2021
1. Introduction 2. Setting up services locally 3. Writing an end-to-end data pipeline test 4. Conclusion 5. Further reading 6. References 1. Introduction Data pipelines can have multiple software components. This makes testing all of them together difficult. If you are wondering What is the best way to end-to-end test data pipelines? Are end-to-end tests worth the effort?
Azure Data Engineering
NOVEMBER 10, 2021
In the previous post, we discussed the Switch Activity , which is useful for branching the control flow based on some condition. We will discuss about the Filter Activity in this post. The purpose of Filter Activity is to process array items based on some condition. Consider a scenario where we would like to set the value of a variable to the current array item that satisfies some business rule or condition.
Advertisement
In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate
KDnuggets
NOVEMBER 12, 2021
Check out these repositories to help you improve your data science skills.
DataKitchen
NOVEMBER 12, 2021
The post The Benefits and Drawbacks of DataOps in Practice first appeared on DataKitchen.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
|
Data Engineering Podcast
NOVEMBER 10, 2021
Summary A significant source of friction and wasted effort in building and integrating data management systems is the fragmentation of metadata across various tools. After experiencing the impacts of fragmented metadata and previous attempts at building a solution Suresh Srinivas and Sriharsha Chintalapani created the OpenMetadata project. In this episode they share the lessons that they have learned through their previous attempts and the positive impact that a unified metadata layer had during
KDnuggets
NOVEMBER 11, 2021
From the mind to the computer, make websites using your imagination!
Netflix Tech
NOVEMBER 9, 2021
by Liwei Guo , Ashwin Kumar Gopi Valliammal , Raymond Tam , Chris Pham , Agata Opalach , Weibo Ni AV1 is the first high-efficiency video codec format with a royalty-free license from Alliance of Open Media (AOMedia), made possible by wide-ranging industry commitment of expertise and resources. Netflix is proud to be a founding member of AOMedia and a key contributor to the development of AV1.
Cloudera
NOVEMBER 12, 2021
Arjun (my son) sat next to me at my desk. He was a bit nervous but we had practiced 3 times before he was ‘on stage’ in front of hundreds of people and the zoom meeting turned to him. My ten year old began to demonstrate how to deploy an Operational Database in AWS, showcasing how auto-scaling worked and how to set up replication. All of the sales team and my colleagues were quite impressed with him, and I am very proud of him.
Speaker: Tamara Fingerlin, Developer Advocate
Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.
Data Engineering Podcast
NOVEMBER 6, 2021
Summary Business intelligence is often equated with a collection of dashboards that show various charts and graphs representing data for an organization. What is overlooked in that characterization is the level of complexity and effort that are required to collect and present that information, and the opportunities for providing those insights in other contexts.
KDnuggets
NOVEMBER 12, 2021
The PyTorch Deep Learning framework has a C++ API for use on mobile platforms. This article shows an end-to-end demo of how to write a simple C++ application with Deep Learning capabilities using the PyTorch C++ API such that the same code can be built for use on mobile platforms (both Android and iOS).
Confluent
NOVEMBER 10, 2021
Schema Linking is a new feature that’s available in preview for both Confluent Cloud and Confluent Platform 7.0 and can be used to complement Cluster Linking, in order to keep […].
Cloudera
NOVEMBER 10, 2021
Cloudera Data Platform (CDP) brings many improvements to customers by merging technologies from the two legacy platforms, Cloudera Enterprise Data Hub (CDH) and Hortonworks Data Platform (HDP). CDP includes new functionalities as well as superior alternatives to some previously existing functionalities in security and governance. One such major change for CDH users is the replacement of Sentry with Ranger for authorization and access control. .
Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage
There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.
ProjectPro
NOVEMBER 11, 2021
Are you willing to learn computer vision but cannot find the perfect guide to start your learning? If yes, then here’s a blog that will help you attain your goal in the easiest way possible! Read this blog until the end to learn how you can learn computer vision from scratch as a beginner. Before diving into the blog further, let us first look at why now is the best time to hone some computer vision skills.
KDnuggets
NOVEMBER 8, 2021
With a lot of excitement and research around NLP, there are growing opportunities to apply these technologies to real-world scenarios. It's not trivial to become familiar with NLP and these open-source data sets can help you increase your skills.
Confluent
NOVEMBER 9, 2021
Companies are increasingly moving to the cloud, undergoing a transition that is often a multi-year journey across incremental stages. Along this journey, many companies embrace hybrid cloud architectures, either temporarily […].
Monte Carlo
NOVEMBER 10, 2021
Monte Carlo’s field-level lineage helps data teams track column-level dependencies from ingestion in the data warehouse or lake to dashboards and reports in the BI layer. It’s Friday evening, and you are wrapping up after a long week of work. Before logging off, you make a schema change to a table in your warehouse – and don’t think twice as you close your laptop and get started on your weekend.
Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives
Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri
Teradata
NOVEMBER 10, 2021
When it comes to making progress on climate change, banks have a critical role in translating commitments into actions by influencing where, how & when money is spent.
KDnuggets
NOVEMBER 10, 2021
A machine can only understand numbers. As a result, converting text to numbers, called embedding text, is an actively researched topic. In this article, we review different word embedding techniques for converting text into vectors.
AltexSoft
NOVEMBER 10, 2021
In 1901, a woman by the name of Julia Davis Chandler published the recipe that changed the world for good. People saw the very first recipe of a peanut butter and jelly sandwich. While both a peanut butter sandwich and a jelly sandwich are great individually, it’s hard to argue that together they make the most epic combo complementing each other’s best flavoring qualities.
Pipeline Data Engineering
NOVEMBER 9, 2021
Ambitious data analysts, data scientists trying to take their careers to the next level, product owners aiming to build next-generation data products, software engineers dealing with legacy data stacks. they all are facing the same challenge: how do I get the data engineering skills that enable me to achieve my goals? This frustration is very real, and it is indeed very common!
Advertisement
With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you
Rockset
NOVEMBER 9, 2021
There are multiple change data capture methods available when using a MySQL or Postgres database. Some of these methods overlap and are very similar regardless of which database technology you are using, others are different. Ultimately, we require a way to specify and detect what has changed and a method of sending those changes to a target system.
KDnuggets
NOVEMBER 9, 2021
Beginners in the field can often have many misconceptions about machine learning that sometimes can be a make-it-or-break-it moment for the individual switching careers or starting fresh. This article clearly describes the ground truth realities about learning new ML skills and eventually working professionally as a machine learning engineer.
Big Data Tools
NOVEMBER 8, 2021
The lockdowns are back again in Moscow, which means that conferences are again out of the question for me for some time. The good news is that I had time to put together this new installment of our Data Engineering Annotated! Hi, I’m Pasha Finkelshteyn , and I’ll be your guide through this month’s news. I’ll offer my impressions of recent developments in the data engineering space and highlight new ideas from the wider community.
ProjectPro
NOVEMBER 8, 2021
Computer vision is one of the most trending and compelling subfields of artificial intelligence. You must have encountered and used the applications of computer vision without even knowing it. Whether it is quality control of crops through image classification or image processing for electronic deposits, computer vision techniques are transforming industries across the globe.
Speaker: Tamara Fingerlin, Developer Advocate
In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!
RudderStack
NOVEMBER 9, 2021
This post details the implementation of graceful shutdown on Rudder Server. You'll find a number of anti-patterns & learn how to make exiting graceful in Go.
KDnuggets
NOVEMBER 12, 2021
The skills needed to create good data are also the skills needed for good leadership.
Big Data Tools
NOVEMBER 8, 2021
The lockdowns are back again in Moscow, which means that conferences are again out of the question for me for some time. The good news is that I had time to put together this new installment of our Data Engineering Annotated! Hi, I’m Pasha Finkelshteyn , and I’ll be your guide through this month’s news. I’ll offer my impressions of recent developments in the data engineering space and highlight new ideas from the wider community.
Cloudera
NOVEMBER 9, 2021
SANTA CLARA, Calif., Nov. 9, 2021 – Cloudera, the enterprise data cloud company, today announced Cloudera Data Platform capabilities available to help federal agencies meet requirements of the Biden Administration’s Executive Order on improving the Nation’s cybersecurity. Cloudera is committed to supporting the federal government in adhering to this executive order with the company’s technology and special government rates. .
Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage
When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m
Expert insights. Personalized for you.
136 
Let's personalize your content