Start Data Engineering
JUNE 10, 2025
1. Introduction 2. Joins & Group bys are two of the most commonly used operations in data warehousing 2.1. Joins are used to create denormalized dimension tables & to enrich fact tables with dimensions for reporting 2.1.1. When to use joins 2.1.2. How to use joins 2.1.3. Things to watch out for when joining 2.2. Group bys are the cornerstone of reporting 2.
KDnuggets
JUNE 11, 2025
Blog Top Posts About Topics AI Career Advice Computer Vision Data Engineering Data Science Language Models Machine Learning MLOps NLP Programming Python SQL Datasets Events Resources Cheat Sheets Recommendations Tech Briefs Advertise Join Newsletter Bridging the Gap: New Datasets Push Recommender Research Toward Real-World Scale Publicly available datasets in recommender research currently shaping the field.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Start Data Engineering
JUNE 7, 2025
1. Introduction 2. CTE for short clean code & temp tables for re-usability 2.1. CTEs make medium-complex SQL easy to understand 2.2. Temp table enables you to reuse logic multiple times in a session 2.3. Performance depends on the execution engine 3. Conclusion 4. Recommended reading 1. Introduction As a data engineer, CTEs are one of the best techniques you can use to improve query readability.
databricks
JUNE 11, 2025
Skip to main content Login Why Databricks Discover For Executives For Startups Lakehouse Architecture Mosaic Research Customers Customer Stories Partners Cloud Providers Databricks on AWS, Azure, GCP, and SAP Consulting & System Integrators Experts to build, deploy and migrate to Databricks Technology Partners Connect your existing tools to your Lakehouse C&SI Partner Program Build, deploy or migrate to the Lakehouse Data Partners Access the ecosystem of data consumers Partner Solutions
Advertisement
In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate
Snowflake
JUNE 10, 2025
The Apache Iceberg™ project exemplifies the spirit of open source and shows what’s possible when a community comes together with a common goal: to drive a technology forward. With a mission to bring reliability, performance and openness to large-scale analytics, the Iceberg project continues to evolve and offer many benefits thanks to the diverse voices and efforts of its contributors.
ProjectPro
JUNE 6, 2025
Data Engineering is gradually becoming a popular career option for young enthusiasts. However, with so many tools and technologies available, it can be challenging to know where to start. That's why we've created a comprehensive data engineering roadmap for 2023 to guide you through the essential skills and tools needed to become a successful data engineer.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
The Pragmatic Engineer
JUNE 12, 2025
Originally published in The Pragmatic Engineer Newsletter. An eye-catching detail widely reported by media and on social media about the bankrupt business Builder.ai last week, was that the company faked AI with 700 engineers in India: “Microsoft-backed AI startup chatbots revealed to be human employees” – Mashable “Builder.ai used 700 engineers in India for coding work it marketed as AI-powered” – MSN “Builder.ai faked AI with 700 engineers, now
Snowflake
JUNE 10, 2025
Data transformations are the engine room of modern data operations — powering innovations in AI, analytics and applications. As the core building blocks of any effective data strategy, these transformations are crucial for constructing robust and scalable data pipelines. Today, we're excited to announce the latest product advancements in Snowflake to build and orchestrate data pipelines.
ProjectPro
JUNE 6, 2025
‘Man and machine together can be better than the human’ All thanks to deep learning frameworks like PyTorch, Tensorflow, Keras, Caffe, and DeepLearning4j for making machines learn like humans with special brain-like architectures known as Neural Networks. The war of deep learning frameworks has two prominent competitors- PyTorch vs Tensorflow because the other frameworks have not yet been adopted widely.
databricks
JUNE 11, 2025
Skip to main content Login Why Databricks Discover For Executives For Startups Lakehouse Architecture Mosaic Research Customers Customer Stories Partners Cloud Providers Databricks on AWS, Azure, GCP, and SAP Consulting & System Integrators Experts to build, deploy and migrate to Databricks Technology Partners Connect your existing tools to your Lakehouse C&SI Partner Program Build, deploy or migrate to the Lakehouse Data Partners Access the ecosystem of data consumers Partner Solutions
Speaker: Tamara Fingerlin, Developer Advocate
Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.
KDnuggets
JUNE 11, 2025
Blog Top Posts About Topics AI Career Advice Computer Vision Data Engineering Data Science Language Models Machine Learning MLOps NLP Programming Python SQL Datasets Events Resources Cheat Sheets Recommendations Tech Briefs Advertise Join Newsletter Why You Need RAG to Stay Relevant as a Data Scientist How retrieval-augmented generation (RAG) reduces LLM costs, minimises hallucinations, and keeps you employable in the age of AI.
ArcGIS
JUNE 9, 2025
The Anomaly Detection Wizard in ArcGIS Pro makes your image anomaly detection workflow easier.
Snowflake
JUNE 8, 2025
Snow is coming to Cannes, France! Snowflake is back again at the Cannes Lions International Festival of Creativity on June 16-20, 2025. As the premiere media and entertainment industry event of the year, Cannes brings together creative legends, marketing luminaries and cutting-edge content creators from around the world to shine a light on the latest trends and bring to the forefront ideas and critical topics shaping the future of the industry.
ProjectPro
JUNE 6, 2025
As a big data architect or a big data developer, when working with Microservices-based systems, you might often end up in a dilemma whether to use Apache Kafka or RabbitMQ for messaging. Rabbit MQ vs. Kafka - Which one is a better message broker? You might find some articles across the web that conclude that Apache Kafka is better than RabbitMQ and few others that mention RabbitMQ to be more reliable than Kafka.
Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage
There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.
databricks
JUNE 11, 2025
MLflow has become the foundation for MLOps at scale, with over 30 million monthly downloads and contributions from over 850 developers worldwide powering ML and
KDnuggets
JUNE 10, 2025
Blog Top Posts About Topics AI Career Advice Computer Vision Data Engineering Data Science Language Models Machine Learning MLOps NLP Programming Python SQL Datasets Events Resources Cheat Sheets Recommendations Tech Briefs Advertise Join Newsletter 7 Python Errors That Are Actually Features You never expected these Python errors to help your work, but they do!
Pinterest Engineering
JUNE 6, 2025
Xue Xia | Machine Learning Engineer, Home Feed Ranking; Saurabh Vishwas Joshi | Principal Engineer, ML Platform; Kousik Rajesh | Machine Learning Engineer, Applied Science; Kangnan Li | Machine Learning Engineer, Core ML Infrastructure; Yangyi Lu | Machine Learning Engineer, Home Feed Ranking; Nikil Pancha | (formerly) Machine Learning Engineer, Applied Science; Dhruvil Deven Badani | Engineering Manager, Home Feed Ranking; Jiajing Xu | Engineering Manager, Applied Science; Pong Eksombatchai | P
Snowflake
JUNE 10, 2025
PostgreSQL has become the undisputed choice for developers worldwide, celebrated for its open source flexibility, vibrant ecosystem and growing AI capabilities like vector support. But as companies race to build the next generation of AI agents and scale their critical operational systems, a fundamental question emerges: Is your Postgres truly ready for the enterprise, or does it come with hidden compromises?
Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives
Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri
ProjectPro
JUNE 6, 2025
With over 54 repositories and 20k stars, Streamlit is an open-source Python framework for developing and distributing web apps for data science and machine learning projects. Engineers can easily create highly dynamic online applications based on their data, machine learning models, etc., using Streamlit. Let us explore a few exciting Streamlit python project ideas for data scientists and data engineers. 5 Streamlit Python Project Ideas to Try Your Hands-On Here are five streamlit projects in Py
databricks
JUNE 11, 2025
At the Data and AI Summit, we introduced a new category of operational databases called lakebases for building intelligent applications.
KDnuggets
JUNE 11, 2025
Blog Top Posts About Topics AI Career Advice Computer Vision Data Engineering Data Science Language Models Machine Learning MLOps NLP Programming Python SQL Datasets Events Resources Cheat Sheets Recommendations Tech Briefs Advertise Join Newsletter Top 7 MCP Clients for AI Tooling The most popular clients that seamlessly and reliably work with MCP servers ranging from IDEs to chatbots and plugins.
Monte Carlo
JUNE 10, 2025
Data engineering is full of buzzwords—data mesh, reverse ETL, lakehouse, you name it. It’s easy to tune them out. So when someone drops “data observability,” it’s fair to ask: what’s data observability vs. monitoring? If you’ve ever wrestled with broken dashboards, missing data, or a pipeline that quietly failed overnight, you know how frustrating it is to figure out what went wrong.
Advertisement
Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.
Precisely
JUNE 10, 2025
Key takeaways: New Data Integrity Suite innovations include AI-powered data quality, and new data observability, lineage, location intelligence, and enrichment capabilities. These enhancements help you scale data quality for AI, boost visibility across hybrid data environments, and embed trusted location data into critical workflows. The Suite ensures you’re able to reduce risk, drive innovation, and maintain a competitive edge.
ProjectPro
JUNE 6, 2025
Early in 2020, when Myntra launched its visual product search for the first time, it created waves in e-commerce. With this new feature, the customers no longer had to spend hours searching for a dress similar to the one they came across randomly in an advertisement. All they had to do was take a picture/screenshot and upload it on Myntra; the app would automatically fetch outfits similar to the picture.
databricks
JUNE 11, 2025
Skip to main content Login Why Databricks Discover For Executives For Startups Lakehouse Architecture Mosaic Research Customers Customer Stories Partners Cloud Providers Databricks on AWS, Azure, GCP, and SAP Consulting & System Integrators Experts to build, deploy and migrate to Databricks Technology Partners Connect your existing tools to your Lakehouse C&SI Partner Program Build, deploy or migrate to the Lakehouse Data Partners Access the ecosystem of data consumers Partner Solutions
KDnuggets
JUNE 12, 2025
Blog Top Posts About Topics AI Career Advice Computer Vision Data Engineering Data Science Language Models Machine Learning MLOps NLP Programming Python SQL Datasets Events Resources Cheat Sheets Recommendations Tech Briefs Advertise Join Newsletter Building a Custom PDF Parser with PyPDF and LangChain PDFs look simple — until you try to parse one.
Advertisement
With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you
Monte Carlo
JUNE 10, 2025
Monte Carlo, the leader in data + AI observability, today announced extended support for the Databricks Data Intelligence Platform through new integrations with Databricks AI/BI and Unity Catalog Metrics. These enhancements, unveiled ahead of the Databricks Data + AI Summit 2025 , represent a major milestone in enabling AI-ready data at scale for joint customers of Databricks and Monte Carlo.
DataKitchen
JUNE 6, 2025
Data Quality Testing: A Shared Resource for Modern Data Teams In today’s AI-driven landscape, where data is king, every role in the modern data and analytics ecosystem shares one fundamental responsibility: ensuring that incorrect data never reaches business customers. Whether you’re a Data Engineer building ETL pipelines, a Data Scientist developing predictive models, or a Data Steward ensuring compliance, we all want the same outcome: data that is trustworthy, accurate, and underst
ProjectPro
JUNE 6, 2025
The demand for data-related roles has increased massively in the past few years. Companies are actively seeking talent in these areas, and there is a huge market for individuals who can manipulate data, work with large databases and build machine learning algorithms. While data science is the most hyped-up career path in the data industry, it certainly isn't the only one.
databricks
JUNE 11, 2025
We’re excited to announce the General Availability of Databricks Apps, enabling customers to securely build, deploy, and scale interactive data and AI-powered applications natively on
Speaker: Tamara Fingerlin, Developer Advocate
In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!
Expert insights. Personalized for you.
130 
Let's personalize your content