Start Data Engineering
JUNE 14, 2024
1. Introduction 2. Run Data Pipelines 2.1. Run on codespaces 2.2. Run locally 3. Projects 3.1. Projects from least to most complex 3.2. Batch pipelines 3.3. Stream pipelines 3.4. Event-driven pipelines 3.5. LLM RAG pipelines 4. Conclusion 1. Introduction Whether you are new to data engineering or have been in the data field for a few years, one of the most challenging parts of learning new frameworks is setting them up!
KDnuggets
JUNE 12, 2024
A simple tutorial on how to connect to databases, execute SQL queries, and analyze and visualize data.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Jesse Anderson
JUNE 12, 2024
This video covers the latest announcements from StreamNative, Confluent, and WarpStream. We discuss communication protocols, how they’re used, and what they mean for you. We also discuss the various systems using Kafka’s protocol. Finally, we discuss the announcements about writing to Iceberg and DeltaLake directly from the broker and what that means for costs and operational ease.
Data Engineering Podcast
JUNE 9, 2024
Summary Streaming data processing enables new categories of data products and analytics. Unfortunately, reasoning about stream processing engines is complex and lacks sufficient tooling. To address this shortcoming Datorios created an observability platform for Flink that brings visibility to the internals of this popular stream processing system. In this episode Ronen Korman and Stav Elkayam discuss how the increased understanding provided by purpose built observability improves the usefulness
Speaker: Tamara Fingerlin, Developer Advocate
Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.
databricks
JUNE 13, 2024
We are excited to announce that we are open sourcing Unity Catalog, the industry’s first open source catalog for data and AI governance.
KDnuggets
JUNE 12, 2024
Learn programming for free from top-tier universities like Harvard and MIT.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
|
ArcGIS
JUNE 11, 2024
Revolutionizing GIS: Streamlining Change Detection for Mapping Agencies.
databricks
JUNE 12, 2024
Over the last year, we have seen a surge of commercial and open-source foundation models showing strong reasoning abilities on general knowledge tasks.
KDnuggets
JUNE 13, 2024
Want to build APIs with Python? Learn how to do so using FastAPI with this step-by-step tutorial.
Engineering at Meta
JUNE 12, 2024
Meta is currently operating many data centers with GPU training clusters across the world. Our data centers are the backbone of our operations, meticulously designed to support the scaling demands of compute and storage. A year ago, however, as the industry reached a critical inflection point due to the rise of artificial intelligence (AI), we recognized that to lead in the generative AI space we’d need to transform our fleet.
Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage
There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.
Snowflake
JUNE 10, 2024
Discovering and surfacing telemetry traditionally can be a tedious and challenging process, especially when it comes to pinpointing specific issues for debugging. However, as applications and pipelines grow in complexity, understanding what’s happening beneath the surface becomes increasingly crucial. A lack of visibility hinders the development and maintenance of high-quality applications and pipelines, ultimately impacting customer experience.
databricks
JUNE 11, 2024
Today, we are excited to announce Databricks AI/BI , a new type of business intelligence product built from the ground up to deeply.
KDnuggets
JUNE 10, 2024
Machine Learning model is an exciting project. Learn how to develop your first model that the company would want to use.
ArcGIS
JUNE 11, 2024
The Living Atlas team publishes a lot of web layers. Here's some of our favorite tips and tricks for customizing your layers and maps.
Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives
Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri
Snowflake
JUNE 10, 2024
Today’s data-driven world requires an agile approach. Modern data teams are constantly under pressure to deliver innovative solutions faster than ever before. Fragmented tooling across data engineering, application development and AI/ML development creates a significant bottleneck, hindering the speed of value delivery required to stay competitive.
databricks
JUNE 13, 2024
Today, we are excited to announce Databricks LakeFlow, a new solution that contains everything you need to build and operate production data pipelines.
KDnuggets
JUNE 10, 2024
Learn SQL and databases through free courses, tutorials, tools, guides, books, practice exercises, projects, awesome lists, and other resources.
ArcGIS
JUNE 10, 2024
Sketch an extent on your map and choose between more new features with the Processing Extent control in ArcGIS Pro 3.3!
Advertisement
With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you
Snowflake
JUNE 13, 2024
The journey toward achieving a robust data platform that secures all your data in one place can seem like a daunting one. But at Snowflake, we’re committed to making the first step the easiest — with seamless, cost-effective data ingestion to help bring your workloads into the AI Data Cloud with ease. Snowflake is launching native integrations with some of the most popular databases, including PostgreSQL and MySQL.
databricks
JUNE 12, 2024
In an era marked by rapid advancements in artificial intelligence and an explosion of data and Gen AI tools, enterprises face fragmented data.
KDnuggets
JUNE 13, 2024
Data privacy has been a long-standing issue that continues to challenge the data industry. Let’s understand how rapid developments in the world of AI have elevated data privacy concerns.
Engineering at Meta
JUNE 10, 2024
At Meta, Bento , our internal Jupyter notebooks platform, is a popular tool that allows our engineers to mix code, text, and multimedia in a single document. Use cases run the entire spectrum from what we call “lite” workloads that involve simple prototyping to heavier and more complex machine learning workflows. However, even though the lite workflows require limited compute, users still have to go through the same process of reserving and provisioning remote compute – a process that takes time
Speaker: Tamara Fingerlin, Developer Advocate
In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!
Snowflake
JUNE 11, 2024
Bringing machine learning (ML) models into production is often hindered by fragmented MLOps processes that are difficult to scale with the underlying data. Many enterprises stitch together a complex mix of various MLOps tools to build an end-to-end ML pipeline. The friction of having to set up and manage separate environments for features and models creates operational complexity that can be costly to maintain and difficult to use.
databricks
JUNE 12, 2024
“FactSet’s mission is to empower clients to make data-driven decisions and supercharge their workflows and productivity. To deliver AI-driven solutions across our entire.
KDnuggets
JUNE 10, 2024
This short tutorial will guide you through the process of converting JSON data into a Pandas DataFrame.
Data Engineering Weekly
JUNE 10, 2024
Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. Learn More → Cube Research: Crystallizing Snowflake Summit 2024 We should officially call the first week of June the data engineering week, as two major data companies are running their developer conference.
Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage
When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m
Snowflake
JUNE 12, 2024
It is estimated that between 80% and 90% of the world’s data is unstructured 1 , with text files and documents making up a significant portion. Every day, countless text-based documents, like contracts and insurance claims, are stored for safekeeping. Despite containing a wealth of insights, this vast trove of information often remains untapped, as the process of extracting relevant data from these documents is challenging, tedious and time-consuming.
databricks
JUNE 13, 2024
At Databricks, our mission is to democratize data + AI. An open approach to sharing and collaboration is critical to maximize reach and.
KDnuggets
JUNE 12, 2024
A detailed comparative analysis of the differences between LangChain and LlamaIndex for building and working with LLM applications.
Cloudera
JUNE 13, 2024
Hadoop. The first time that I really became familiar with this term was at Hadoop World in New York City some ten or so years ago. There were thousands of attendees at the event – lining up for book signings and meetings with recruiters to fill the endless job openings for developers experienced with MapReduce and managing Big Data. This was the gold rush of the 21st century, except the gold was data.
Advertisement
Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.
Expert insights. Personalized for you.
239 
Let's personalize your content