KDnuggets
SEPTEMBER 5, 2022
People assume that NoSQL is a counterpart to SQL. Instead, it’s a different type of database designed for use-cases where SQL is not ideal. The differences between the two are many, although some are so crucial that they define both databases at their cores.
Cloudera
SEPTEMBER 7, 2022
We are now well into 2022 and the megatrends that drove the last decade in data — The Apache Software Foundation as a primary innovation vehicle for big data, the arrival of cloud computing, and the debut of cheap distributed storage — have now converged and offer clear patterns for competitive advantage for vendors and value for customers. Cloudera has been parlaying those patterns into clear wins for the community at large and, more importantly, streamlining the benefits of that innovation to
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Data Engineering Podcast
SEPTEMBER 4, 2022
Summary Data observability is a product category that has seen massive growth and adoption in recent years. Monte Carlo is in the vanguard of companies who have been enabling data teams to observe and understand their complex data systems. In this episode founders Barr Moses and Lior Gavish rejoin the show to reflect on the evolution and adoption of data observability technologies and the capabilities that are being introduced as the broader ecosystem adopts the practices.
dbt Developer Hub
SEPTEMBER 7, 2022
If you’ve ever heard of Marie Kondo, you’ll know she has an incredibly soothing and meditative method to tidying up physical spaces. Her KonMari Method is about categorizing, discarding unnecessary items, and building a sustainable system for keeping stuff. As an analytics engineer at your company, doesn’t that last sentence describe your job perfectly?!
Advertisement
In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate
KDnuggets
SEPTEMBER 5, 2022
Ready to learn how to use Python for data science? This free course has got you covered!
Cloudera
SEPTEMBER 9, 2022
The promise of a modern data lakehouse architecture. Imagine having self-service access to all business data, anywhere it may be, and being able to explore it all at once. Imagine quickly answering burning business questions nearly instantly, without waiting for data to be found, shared, and ingested. Imagine independently discovering rich new business insights from both structured and unstructured data working together, without having to beg for data sets to be made available.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
|
Confluent
SEPTEMBER 9, 2022
Data streaming, analytics, and integration are at the backbone of every real-time application. Here are 3 reasons to attend Current this Oct. 2022.
KDnuggets
SEPTEMBER 8, 2022
Learn everything you need to know about data lakehouses.
Cloudera
SEPTEMBER 6, 2022
In the wake of the disruption caused by the world’s turbulence over the past few years , the telecommunications industry has come out reasonably unscathed. There remain challenges in workforce management, particularly in call centers, and order backlogs for fiber broadband and other physical infrastructure are being worked through. But digital transformation programs are accelerating, services innovation around 5G is continuing apace, and results to the stock market have been robust. .
Databand.ai
SEPTEMBER 9, 2022
How to analyze and resolve data pipeline incidents in Databand Niv Sluzki 2022-09-09 13:00:12 A data pipeline failure can cripple your downstream data flows. Whether it failed to start or quit unexpectedly, you need to know immediately if there is a pipeline incident. In this blog, we’re going to walk through how to analyze a failed Airflow pipeline and pinpoint the root cause of your data incidents.
Speaker: Tamara Fingerlin, Developer Advocate
Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.
U-Next
SEPTEMBER 9, 2022
Introduction . We all know that a Product Manager is one of the well-reputed, in-demand, and high-paying jobs in today’s world. However, the Product Manager’s detailed role description s are not always well-defined despite being a vital position in a company. Many of us know basic details, such as Product Manager salary range, skills, etc., but are unaware of real-life work experience.
KDnuggets
SEPTEMBER 6, 2022
Defining model evaluation metrics is crucial in ensuring that the model performs precisely for the purpose it is built. Confusion Matrix is one of the most popular and effective tools to evaluate the performance of the trained ML model. In this post, you will learn how to visualize the confusion matrix and interpret its output.
Cloudera
SEPTEMBER 8, 2022
“There are some unique challenges introduced by the requirement to govern data across a mixture of public cloud and on-premise data resources, ” according to the latest whitepaper published by the TM Forum , as “ their different characteristics require an awareness at the governance level in order to maintain cost, residency, performance, accessibility, and other objectives.” .
Confluent
SEPTEMBER 8, 2022
Aurora’s modern relational database and Confluent’s database streaming services offer real-time hybrid/multicloud data pipelines and streaming ETL for cloud-native agility, elasticity, and cost efficiency.
Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage
There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.
U-Next
SEPTEMBER 9, 2022
Introduction . Willing to know how to leverage the sales strategy program for your own business? Whether a business is involved in a B2B sales strategy, an inbound or outbound strategy, a small to medium business (SMB), or an enterprise, a reliable source of revenue is essential for the company to survive. A reliable revenue stream is achieved by aligning specific sales activities with solid, thoughtful, and data-supported objectives that are in line with the company’s long-term goals.
KDnuggets
SEPTEMBER 7, 2022
This post explains why and when you need machine learning and concludes by listing the key considerations for choosing the correct machine learning algorithm.
Monte Carlo
SEPTEMBER 7, 2022
You know that cartoon trope where a leak springs in the dike and the character quickly plugs it with a finger, only to find another leak has sprung that needs to be plugged, and so on until there are no more fingers or the entire dam bursts? Data engineers know that feeling all too well. Anomalies spring up, a member of the data team is assigned to resolve it, but the root cause analysis process takes so long that by the time everything is fixed, another three leaks have sprung and there are no
Confluent
SEPTEMBER 7, 2022
As mission-critical data infrastructure, Apache Kafka’s resiliency is non-negotiable. Learn how Confluent Cloud builds 10x higher resilience into its cloud-native services.
Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives
Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri
U-Next
SEPTEMBER 7, 2022
Introduction to Product Backlog . A properly-prioritized product backlog was created to facilitate planning for iterations and releases as well as to announce all of the projects that product teams plan to work on. In product management, a rational list of potential requirements for the finished product is known as the product backlog. Scrum and Agile development methodologies generally include product backlogs as a crucial element.
KDnuggets
SEPTEMBER 9, 2022
Putting the fun in fundamentals! A collection of short videos to amuse beginners and experts alike.
Afterpay Tech
SEPTEMBER 6, 2022
Photo by Leon S on Unsplash By: Jing Li Summary This article articulates the challenges, innovation and success of the Kafka implementation in Afterpay’s Global Payments Platform in the PCI zone. To satisfy the PCI DSS requirements, we decided to use AWS PrivateLink together with custom Kafka client libraries (producer & consumer) to form the solutions for the Payments Platform.
Elder Research
SEPTEMBER 6, 2022
The post Arranging a Suite of Analytics for Hotel Data appeared first on Elder Research.
Advertisement
With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you
U-Next
SEPTEMBER 7, 2022
Introduction . As data generation and consumption continue to soar, Business Intelligence (BI) has become more relevant in this digital world. With the data generation of more than 2.5 quintillion bytes daily , the significance of Big Data and Data Analytics can be recognized. Most business organizations are motivated to transform into data-driven companies irrespective of their size.
KDnuggets
SEPTEMBER 6, 2022
Here’s everything you need to know about A/B testing interview questions in data science interviews.
Monte Carlo
SEPTEMBER 6, 2022
Say it with me: bad data is inevitable. It doesn’t care about how proactive you are at writing dbt tests, how perfectly your data is modeled, or how robust your architecture is. The possibility of a major data incident (Null value? Errant schema change? Failed model?) that reverberates across the company is always lurking around the corner. That’s not to say things like data testing, validation, data contracts , domain-driven data ownership, and data diffing don’t play a role in reducing data in
dbt Developer Hub
SEPTEMBER 6, 2022
Analyzing financial data is rarely ever “fun.” In particular, generating and analyzing financial statement data can be extremely difficult and leaves little room for error. If you've ever had the misfortune of having to generate financial reports for multiple systems, then you will understand how incredibly frustrating it is to reinvent the wheel each time.
Speaker: Tamara Fingerlin, Developer Advocate
In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!
U-Next
SEPTEMBER 7, 2022
Introduction . Employee database software is quickly becoming a must-have for HR and people managers. After all, a company’s biggest asset is its workforce. Companies can achieve new heights with the right management and administration tools. A firm can benefit immensely from maintaining accurate and clean employee data, as it is still a difficult task. .
KDnuggets
SEPTEMBER 7, 2022
In this how-to, we’ll build a model to uncover which paths in user journeys have the biggest impact on product goals (e.g. conversion). You can use it to improve products or optimize marketing campaigns, or as a base for deeper user behavior analyses.
Monte Carlo
SEPTEMBER 6, 2022
Our biggest priority at Monte Carlo is to make the lives of our customers easier by reducing data downtime and helping them accelerate the adoption of reliable data at their companies. As part of this mission, Monte Carlo’s product, engineering, design, and data science teams are constantly releasing new product functionalities and features to improve the user experience and reduce time to detection, resolution, and prevention of broken data pipelines.
Picnic Engineering
SEPTEMBER 8, 2022
We continue our story on the Analytics Platform setup in Picnic. In the “Picnic Analytics Platform: Migration from AWS Kinesis to Confluent Cloud” we described why and how we migrated from AWS Kinesis to Confluent Cloud. This time we will dive into how we configure our internal services pipeline. Quick re-cap: the purpose of the internal pipeline is to deliver data from dozens of Picnic back-end services such as warehousing, machine learning models, customers and order status updates.
Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage
When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m
Expert insights. Personalized for you.
160 
Let's personalize your content