Confluent
FEBRUARY 22, 2021
Apache Kafka® is at the heart of the data transportation layer at Pinterest. The amount of data that runs through Kafka has constantly grown over the years. This growth sometimes […].
Team Data Science
FEBRUARY 23, 2021
Companies can now make data useful to elevate decision making and to optimise products and processes. But what organizational capabilities are necessary and how to get started? It's currently easy to acquire data strategically. First, consider that smartphones function like questionnaires that customers are frequently filling out in a passive or active manner [ , 1 ].
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
François Nguyen
FEBRUARY 21, 2021
If you are working in the tech field, I think you have already heard of Squads, Tribes, Chapters or Guild. It comes from Spotify, a swedish audio streaming company.If you are organizing #datateams, it could be tempting to copy/paste. You should really not ! The Spotify Model and Engineering Culture If you want to go back to the original article, it his here.
Cloudera
FEBRUARY 23, 2021
Meet Kevin Smith, a Staff Customer Operations Engineer within the US Public Sector support team. He sums up his day-to-day by saying he works directly with clients on technical cases and provides support and guidance as they troubleshoot unexpected behavior. He also serves as a member of several project teams focusing on upgrade experiences, internal tools, product testing, training, and documentation.
Advertisement
In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate
Data Engineering Podcast
FEBRUARY 22, 2021
Summary Data integration is a critical piece of every data pipeline, yet it is still far from being a solved problem. There are a number of managed platforms available, but the list of options for an open source system that supports a large variety of sources and destinations is still embarrasingly short. The team at Airbyte is adding a new entry to that list with the goal of making robust and easy to use data integration more accessible to teams who want or need to maintain full control of thei
Confluent
FEBRUARY 24, 2021
SAP is a German multinational software corporation that develops and markets enterprise software to manage business operations and customer relations. SAP is most famous for its enterprise resource planning (ERP) […].
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
|
Cloudera
FEBRUARY 25, 2021
Be the light – Accepting the call to become the change we seek. As Black History Month comes to a close, global communities and companies alike are left reflecting on recent historical events with shock, awe and a commitment to drive change. We find ourselves faced with the unhealed wounds of our past, a defining moment for our future and an opportunity to become the change we seek as citizens and professionals. .
Teradata
FEBRUARY 22, 2021
Teradata has again been recognized as one of the World’s Most Ethical Companies, for 12th consecutive year! Read more.
Confluent
FEBRUARY 25, 2021
This blog post walks through an end-to-end demo that uses the Confluent Operator to deploy Confluent Platform to Kubernetes. We will deploy a connector that watches for commits to a […].
Netflix Tech
FEBRUARY 25, 2021
By Cyril Concolato Introduction In previous blog posts, our colleagues at Netflix have explained how 4K video streams are optimized , how even legacy video streams are improved and more recently how new audio codecs can provide better aural experiences to our members. In all these cases, prior to being delivered through our content delivery network Open Connect , our award-winning TV shows, movies and documentaries like The Crown need to be packaged to enable crucial features for our members.
Speaker: Tamara Fingerlin, Developer Advocate
Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.
Cloudera
FEBRUARY 22, 2021
In 1969, my aunt graduated from university and joined IBM, the dominant player in the nascent tech industry at the time. She remained at “Big Blue” where she met and married my uncle, and rose up through the management ranks, until their joint semi-retirement exactly 30 years later. She recently told me, “the only way you could get fired in those days was to murder someone, embezzle or steal”.
Teradata
FEBRUARY 24, 2021
Today's retailers need to focus on using data to create scenarios that encourage the customer to engage with them, and then ensure that they act appropriately when they do.
Retail Insight
FEBRUARY 26, 2021
When I left school to start a professional football career, I understood very little about data – I did keep a note of the goals I scored, the assists I made and, most likely, the keepie-ups I could perform, but that was about it.
Confluent
FEBRUARY 26, 2021
Kerberos authentication is widely used in today’s client/server applications; however getting started with Kerberos may be a daunting task if you don’t have prior experience. Information on setting up Kerberos […].
Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage
There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.
Cloudera
FEBRUARY 25, 2021
Today’s enterprise data science teams have one of the most challenging, yet most important roles to play in your business’s ML strategy. In our current landscape, businesses that have adopted a successful ML strategy are outperforming their competitors by over 9%. The implications of ML on the future of business are clear. However, only 4% of enterprise executives today report seeing success from their ML investment.
Grouparoo
FEBRUARY 25, 2021
Developers have been using the Grouparoo UI to set up automated data movement from their databases to Mailchimp, Marketo, Salesforce, and more. While having these integrations already written for them saved plenty of time, there was something they missed: their normal developer workflow. Grouparoo now supports declarative data models and integrations to continuously sync your data to all of your cloud-based tools.
Silectis
FEBRUARY 25, 2021
Data lineage can be a tremendously useful tool for data engineering and analytics, but is often treated as an afterthought both because of the challenges in implementation and the fact that it has not been broadly available within organizations. Many practitioners have never had access to data lineage information and may not know what they are missing.
Ripple Engineering
FEBRUARY 25, 2021
In 2020, Ripple accelerated our efforts to enhance diversity and inclusion throughout the company. As part of this commitment, we are sharing RippleNet Engineering 's initiative to replace language in our codebase that does not align with the reality in which we collectively want to live. This project was inspired especially by the protests last summer denouncing police brutality against Black citizens and the long fight against systemic racism in the United States.
Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives
Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri
Cloudera
FEBRUARY 22, 2021
The Cloudera Data Platform (CDP) represents a paradigm shift in modern data architecture by addressing all existing and future analytical needs. It builds on a foundation of technologies from CDH (Cloudera Data Hub) and HDP (Hortonworks Data Platform) technologies and delivers a holistic, integrated data platform from Edge to AI helping clients to accelerate complex data pipelines and democratize data assets.
Teradata
FEBRUARY 24, 2021
Today's retailers need to focus on using data to create scenarios that encourage the customer to engage with them, and then ensure that they act appropriately when they do.
A Cloud Guru: Data Engineering
FEBRUARY 23, 2021
How do you get started with an Azure database? As a database novice or someone new to Microsoft Azure, there are so many options it can be hard to know where to begin. Which is right for you as you get started on the path to becoming a data engineer? Let’s turn the question around […] The post Intro to databases on Azure: Basics for aspiring data engineers appeared first on A Cloud Guru.
Monte Carlo
FEBRUARY 22, 2021
Companies spend upwards of $15 million an nually tackling data downtime , in other words, periods of time where data is missing, broken, or otherwise erroneous, and over 88 percent of U.S. bu sinesses have lost money as a result of data quality issues. Fortunately, there’s hope in the n ext frontier of data engineering: observability. Here’s how the data engineering team at Blinkist, a book-summarizing subscription service, increases cost savings, collaboration, and productivity with data observ
Advertisement
With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you
Cloudera
FEBRUARY 26, 2021
Cloudera Operational Database is an operational database-as-a-service that brings ease of use and flexibility to Apache HBase. Cloudera Operational Database enables developers to quickly build future-proof applications that are architected to handle data evolution. In the previous blog posts, we looked at application development concepts and how Cloudera Operational Database (COD) interacts with other CDP services.
ProjectPro
FEBRUARY 26, 2021
Wondering how much is the machine learning engineer salary? Well, we have got you covered. In this article, you’ll get some insider expert advice, including helpful resources, to help determine the machine learning engineer's average salary for your location, skills, and experience level. So, let’s get started! Table of Contents Machine Learning Engineer Salary – How much can you earn in 2023?
Rockset
FEBRUARY 25, 2021
It’s difficult to create data analytics systems that can easily query across your various data sources while maintaining fast performance and real-time capabilities. In an attempt to mitigate these challenges, many companies are turning to more modern database solutions. Two of these real-time analytics solutions are Elasticsearch and Rockset. Elasticsearch , originally developed for text search, has recently tried to push into the data analytics space.
Preset
FEBRUARY 24, 2021
Semantic versioning is a common strategy for handling releases. We discuss why the Apache Superset™ community is adopting this approach. | Apache Superset™ 1.
Speaker: Tamara Fingerlin, Developer Advocate
In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!
Monte Carlo
FEBRUARY 24, 2021
We’re excited to share that Monte Carlo was today named to the third-annual 2021 Enterprise Tech 30 , a prestigious list of early-stage, mid-stage, and late-stage private companies identified by top VC and analysts as the most promising in enterprise tech. Organized by Wing Venture Capital, the Enterprise Tech 30 considers over 15,000 companies each year for this celebrated award, narrowing it down to just 30.
Zalando Engineering
FEBRUARY 24, 2021
In this article, I will show how teams at Zalando Marketing Services are using integration tests in Java-based backend applications. We will follow the idea of integration tests: the main concept and the attributes of a good integration test. Then, we will discuss an example based on the TestContainers library used in the Spring environment. Integration tests There are many definitions of integration testing.
Cloudera
FEBRUARY 24, 2021
With a new year ahead, it’s time for financial services to pause, take stock of the “new normal,” and plan a path forward. COVID-19 forced nearly every industry to adapt to a new reality, and the financial services industry was no exception. Consumer habits shifted drastically. Suddenly, many people started working from home. Employee and customer needs changed.
Expert insights. Personalized for you.
145 
Let's personalize your content