databricks
MAY 29, 2025
Introduction Marketing teams frequently encounter challenges in accessing their data, often depending on technical teams to translate that data into actionable insights.
Data Engineering Weekly
JUNE 1, 2025
Dagster for MLOps: Deep Dive into AI Orchestration Learn what it really takes to run production-grade ML systems—without breaking your architecture or compliance efforts. Join Dagster and Neurospace to learn: - How to build AI pipelines with orchestration baked in - How to track data lineage for audits and traceability - Tips for designing compliant workflows under the EU AI Act Register for the technical session DuckDB: DuckLake - SQL as a Lakehouse Format DuckDB announced a new open tabl
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
KDnuggets
MAY 27, 2025
You dont need to be a Python pro to write fast, clean code. Just a few smart coding habits can go a long way.
ArcGIS
MAY 27, 2025
In ArcGIS Pro 3.5, you can choose between three options to overwrite existing tool data, including appending and replacing data.
Advertisement
In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate
databricks
MAY 27, 2025
Until now, sharing data between enterprise systems often meant complex pipelines, duplication, and lock-in. With Oracles support for Delta Sharing, thats no longer the case.
Precisely
MAY 27, 2025
In a rapidly evolving financial landscape, one thing is clear: banks that prioritize agility and data-driven customer-centricity are not just staying afloattheyre thriving. During the recent American Banker webinar, Smart Banking in 2025: Intelligent Technologies Defining CX and Operations, I had the pleasure of speaking alongside Sarah Howell about the big shifts seen in bankingparticularly around digital transformation, compliance, and customer experience (CX).
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
KDnuggets
MAY 27, 2025
These ten compact and pythonic shortcuts will boost your time data analysis and processing workflows. See how and why.
Edureka
MAY 27, 2025
Microsoft Fabric is a next-generation data platform that combines business intelligence, data warehousing, real-time analytics, and data engineering into a single integrated SaaS framework. Microsoft Fabric, which is based on the principles of governance, scalability, and simplicity, enables companies to handle their whole analytics lifecycle in one location.
databricks
MAY 27, 2025
In a sign of how quickly enterprises are moving to embrace AI, 70% have moved past the pilot stage and are preparing to release new
ArcGIS
MAY 27, 2025
Learn how to analyze the relationships between your network features and structures using ArcGIS Utility Network.
Speaker: Tamara Fingerlin, Developer Advocate
Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.
KDnuggets
MAY 27, 2025
Develop a data system that every business user wants to use.
Edureka
MAY 27, 2025
Imagine entering a control room with complete control over your data ecosystem. You won’t have to deal with siloed systems, jump between tools, or write endless lines of code to make data useful. That’s how Microsoft Fabric works. With its ability to seamlessly integrate data engineering, analytics, and business intelligence, Microsoft Fabric stands out as the all-in-one superhero in a world where data is abundant but insights are scarce.
Confluent
MAY 29, 2025
Learn how to easily extract the data you need from Apache Kafka by generating Apache Flink SQL commands with natural language prompts or questions in this step-by-step demo.
Snowflake
MAY 27, 2025
At Snowflake, our mission is to empower every enterprise to achieve its full potential through data and AI. We actively support innovative companies within our ecosystem that demonstrate clear value for our customers, which is why we're excited to invest in Honeydew , a former Snowflake Startup Challenge finalist. Honeydews Semantic Layer revolutionizes the way data teams collaborate on business intelligence and deliver impactful data-driven insights.
Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage
There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.
ArcGIS
MAY 27, 2025
Discusses the enhancements introduced in ArcGIS Pro 3.5 to assist system administrators in optimizing performance settings.
Edureka
MAY 27, 2025
Docker completely changed the development, packaging, and deployment of applications. Docker provides consistent environments from development to production by isolating applications in containers. The Dockerfile is the foundation of this ecosystem since it serves as a guide for creating Docker images. This blog covers everything from the fundamentals to more complex subjects like comparisons, troubleshooting, and best practices.
databricks
MAY 27, 2025
Dimensional modeling is a time-tested approach to building analytics-ready data warehouses.
Confluent
MAY 27, 2025
Discover how Account Executive Jason helps customers turn roadblocks into winspowered by Confluents collaborative, one-team culture.
Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives
Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri
Snowflake
MAY 27, 2025
Enterprises are navigating a complex landscape marked by evolving challenges in privacy, economics and the rapid advancement of AI. Consumer data privacy is no longer just an expectation its a nonnegotiable, the foundation of consumer trust. Economic volatility has pushed companies to do more with less, demanding greater efficiency amid ever-changing regulations.
dbt Developer Hub
MAY 27, 2025
Today, we announced the dbt Fusion engine. Fusion isn't just one thing it's a set of interconnected components working together to power the next generation of analytics engineering. This post maps out each piece of the Fusion architecture, explains how they fit together, and clarifies what's available to you whether you're compiling from source, using our pre-built binaries, or developing within a dbt Fusion powered product experience.
Edureka
MAY 27, 2025
In the world of data analytics, Microsoft Fabric and Tableau stand out as powerful tools, but they have very different strengths. While Microsoft Fabric offers an all-in-one data platform for enterprises deeply integrated with Azure, Tableau focuses on intuitive, high-quality data visualization for users at all levels. This guide compares their features, architecture, pricing, and use cases to help you decide which is the best fit for your data strategy.
Precisely
MAY 27, 2025
In a rapidly evolving financial landscape, one thing is clear: banks that prioritize agility and data-driven customer-centricity are not just staying afloattheyre thriving. During the recent American Banker webinar, Smart Banking in 2025: Intelligent Technologies Defining CX and Operations, I had the pleasure of speaking alongside Sarah Howell about the big shifts seen in bankingparticularly around digital transformation, compliance, and customer experience (CX).
Advertisement
With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you
databricks
MAY 28, 2025
Apache Spark 4.0 marks a major milestone in the evolution of the Spark analytics engine.
Snowflake
MAY 27, 2025
It sounds like a cliche to say it's a transformative time in telecommunications. But thats never been more accurate. Companies across the entire ecosystem are undergoing unprecedented change and incredible innovation across every aspect of the business. Fueled by efficiency, cost and customer experience pressures, telecoms must ensure that networks are not only highly reliable but easily adaptable to the rapidly changing needs of modern businesses and customers.
dbt Developer Hub
MAY 27, 2025
Today, we announced that the dbt Fusion engine is available in beta. If Fusion works with your project today, great! You're in for a treat If it's your first day using dbt, welcome! You should start on Fusion you're in for a treat too. Today is Launch Day the first day of a new era: the Age of Fusion. We expect many teams with existing projects will encounter at least one issue that will prevent them from adopting the dbt Fusion engine in production environments.
Edureka
MAY 27, 2025
In today’s data-driven world, the role of an AWS Data Engineer is more important than ever! Organizations are on the lookout for talented professionals who can design, build, and maintain strong data pipelines and infrastructure on the Amazon Web Services (AWS) platform. If you’re eager to kickstart your career in AWS data engineering or ready to take it to the next level, mastering the interview process is essential.
Speaker: Tamara Fingerlin, Developer Advocate
In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!
Waitingforcode
MAY 29, 2025
When I was learning about watermarks in Apache Flink, I saw they were taking the smallest event times instead of the biggest ones in Apache Spark Structured Streaming. From that I was puzzled. How is it possible the pipeline doesn't go back to the past? The answer came when I reread the Streaming Systems book. There was one keyword I had missed that clarified everything.
KDnuggets
MAY 26, 2025
Want recruiters and collaborators to find you? Fix your LinkedIn, even if you hate self-promotion.
Snowflake
MAY 30, 2025
Earlier this year, we announced a multiphased plan to block single-factor password sign-ins. Starting in May, multi-factor authentication (MFA) will be enforced on all password sign-ins to Snowsight UI for human users as part of BCR 2025_04. Password sign-ins outside of Snowsight, such as those in BI tools like PowerBI, will be exempted from this policy.
databricks
MAY 28, 2025
As data and AI workloads scale, organizations need a platform that does more than just connect servicesit must unify them.
Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage
When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m
Expert insights. Personalized for you.
66 
Let's personalize your content