Start Data Engineering
JUNE 14, 2024
1. Introduction 2. Run Data Pipelines 2.1. Run on codespaces 2.2. Run locally 3. Projects 3.1. Projects from least to most complex 3.2. Batch pipelines 3.3. Stream pipelines 3.4. Event-driven pipelines 3.5. LLM RAG pipelines 4. Conclusion 1. Introduction Whether you are new to data engineering or have been in the data field for a few years, one of the most challenging parts of learning new frameworks is setting them up!
KDnuggets
JUNE 12, 2024
A simple tutorial on how to connect to databases, execute SQL queries, and analyze and visualize data.
Jesse Anderson
JUNE 12, 2024
This video covers the latest announcements from StreamNative, Confluent, and WarpStream. We discuss communication protocols, how they’re used, and what they mean for you. We also discuss the various systems using Kafka’s protocol. Finally, we discuss the announcements about writing to Iceberg and DeltaLake directly from the broker and what that means for costs and operational ease.
Data Engineering Podcast
JUNE 9, 2024
Summary Streaming data processing enables new categories of data products and analytics. Unfortunately, reasoning about stream processing engines is complex and lacks sufficient tooling. To address this shortcoming Datorios created an observability platform for Flink that brings visibility to the internals of this popular stream processing system. In this episode Ronen Korman and Stav Elkayam discuss how the increased understanding provided by purpose built observability improves the usefulness
Advertisement
Whether you’re creating complex dashboards or fine-tuning large language models, your data must be extracted, transformed, and loaded. ETL and ELT pipelines form the foundation of any data product, and Airflow is the open-source data orchestrator specifically designed for moving and transforming data in ETL and ELT pipelines. This eBook covers: An overview of ETL vs.
databricks
JUNE 13, 2024
We are excited to announce that we are open sourcing Unity Catalog, the industry’s first open source catalog for data and AI governance.
KDnuggets
JUNE 12, 2024
Learn programming for free from top-tier universities like Harvard and MIT.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
ArcGIS
JUNE 11, 2024
Revolutionizing GIS: Streamlining Change Detection for Mapping Agencies.
databricks
JUNE 12, 2024
Over the last year, we have seen a surge of commercial and open-source foundation models showing strong reasoning abilities on general knowledge tasks.
KDnuggets
JUNE 13, 2024
Want to build APIs with Python? Learn how to do so using FastAPI with this step-by-step tutorial.
Engineering at Meta
JUNE 12, 2024
Meta is currently operating many data centers with GPU training clusters across the world. Our data centers are the backbone of our operations, meticulously designed to support the scaling demands of compute and storage. A year ago, however, as the industry reached a critical inflection point due to the rise of artificial intelligence (AI), we recognized that to lead in the generative AI space we’d need to transform our fleet.
Speaker: Tamara Fingerlin, Developer Advocate
In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!
ArcGIS
JUNE 11, 2024
The Living Atlas team publishes a lot of web layers. Here's some of our favorite tips and tricks for customizing your layers and maps.
databricks
JUNE 11, 2024
Today, we are excited to announce Databricks AI/BI , a new type of business intelligence product built from the ground up to deeply.
KDnuggets
JUNE 10, 2024
Machine Learning model is an exciting project. Learn how to develop your first model that the company would want to use.
Engineering at Meta
JUNE 10, 2024
At Meta, Bento , our internal Jupyter notebooks platform, is a popular tool that allows our engineers to mix code, text, and multimedia in a single document. Use cases run the entire spectrum from what we call “lite” workloads that involve simple prototyping to heavier and more complex machine learning workflows. However, even though the lite workflows require limited compute, users still have to go through the same process of reserving and provisioning remote compute – a process that takes time
Advertisement
Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.
ArcGIS
JUNE 10, 2024
Sketch an extent on your map and choose between more new features with the Processing Extent control in ArcGIS Pro 3.3!
databricks
JUNE 13, 2024
Today, we are excited to announce Databricks LakeFlow, a new solution that contains everything you need to build and operate production data pipelines.
KDnuggets
JUNE 10, 2024
Learn SQL and databases through free courses, tutorials, tools, guides, books, practice exercises, projects, awesome lists, and other resources.
Snowflake
JUNE 10, 2024
Today’s data-driven world requires an agile approach. Modern data teams are constantly under pressure to deliver innovative solutions faster than ever before. Fragmented tooling across data engineering, application development and AI/ML development creates a significant bottleneck, hindering the speed of value delivery required to stay competitive.
Advertisement
Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?
Cloudera
JUNE 13, 2024
Hadoop. The first time that I really became familiar with this term was at Hadoop World in New York City some ten or so years ago. There were thousands of attendees at the event – lining up for book signings and meetings with recruiters to fill the endless job openings for developers experienced with MapReduce and managing Big Data. This was the gold rush of the 21st century, except the gold was data.
databricks
JUNE 12, 2024
In an era marked by rapid advancements in artificial intelligence and an explosion of data and Gen AI tools, enterprises face fragmented data.
KDnuggets
JUNE 13, 2024
Data privacy has been a long-standing issue that continues to challenge the data industry. Let’s understand how rapid developments in the world of AI have elevated data privacy concerns.
Snowflake
JUNE 10, 2024
Discovering and surfacing telemetry traditionally can be a tedious and challenging process, especially when it comes to pinpointing specific issues for debugging. However, as applications and pipelines grow in complexity, understanding what’s happening beneath the surface becomes increasingly crucial. A lack of visibility hinders the development and maintenance of high-quality applications and pipelines, ultimately impacting customer experience.
Speaker: Jay Allardyce, Deepak Vittal, Terrence Sheflin, and Mahyar Ghasemali
As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.
Data Engineering Weekly
JUNE 10, 2024
Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. Learn More → Cube Research: Crystallizing Snowflake Summit 2024 We should officially call the first week of June the data engineering week, as two major data companies are running their developer conference.
databricks
JUNE 12, 2024
“FactSet’s mission is to empower clients to make data-driven decisions and supercharge their workflows and productivity. To deliver AI-driven solutions across our entire.
KDnuggets
JUNE 10, 2024
This short tutorial will guide you through the process of converting JSON data into a Pandas DataFrame.
ArcGIS
JUNE 13, 2024
Explore the Global Wildland-Urban Interface (WUI) dataset to find out how close you are to areas prone to wildfires. Understand the intermixing of human activity with wildland vegetation.
Speaker: Nikhil Joshi, Founder & President of Snic Solutions
Is your manufacturing operation reaching its efficiency potential? A Manufacturing Execution System (MES) could be the game-changer, helping you reduce waste, cut costs, and lower your carbon footprint. Join Nikhil Joshi, Founder & President of Snic Solutions, in this value-packed webinar as he breaks down how MES can drive operational excellence and sustainability.
Engineering at Meta
JUNE 11, 2024
MobileConfig enables developers to centrally manage a mobile app’s configuration parameters in our data centers. Once a parameter value is changed on our central server, billions of app devices automatically fetch and apply the new value without app updates. These remotely managed configuration parameters serve various purposes such as A/B testing, feature rollout, and app personalization.
databricks
JUNE 10, 2024
In the dynamic, innovative landscape of the San Francisco Bay Area, Databricks stands out not just for our groundbreaking data and AI solutions.
KDnuggets
JUNE 12, 2024
A detailed comparative analysis of the differences between LangChain and LlamaIndex for building and working with LLM applications.
Confessions of a Data Guy
JUNE 11, 2024
One of the things I love about Python is its flexibility and huge community, a community that puts out a never-ending stream of useful packages for the average Software Engineer. In a show of solidarity to the open-source community, I thought I would publish a PYPI package that will probably be used by 5 people […] The post Building Open-Source Python Packages – SparklePop appeared first on Confessions of a Data Guy.
Advertisement
Cloud Development Environments (CDEs) are changing how software teams work by moving development to the cloud. Our Cloud Development Environment Adoption Report gathers insights from 223 developers and business leaders, uncovering key trends in CDE adoption. With 66% of large organizations already using CDEs, these platforms are quickly becoming essential to modern development practices.
Expert insights. Personalized for you.
239 
Let's personalize your content