The Pragmatic Engineer
FEBRUARY 6, 2025
The past six months has been something of a Doomsday scenario-esque countdown for TikTok, as the start date of its ban in the US crept ever closer. In the event, TikTok did indeed go offline for a few hours on 19 January, before President Trump gave the social network a stay of execution lasting 75 days. How has this uncertainty affected software engineers at the Chinese-owned social network?
Snowflake
FEBRUARY 3, 2025
Generative AIs potential to drive innovation, improve efficiency and create competitive advantages is enormous. However, the ability to fully realize the benefits of generative AI hinges on one crucial factor: data strategy. Data Strategies for AI Leaders , a report co-written by MIT and Snowflake, underscores how organizations must invest in robust data foundations to succeed in the AI era.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Engineering at Meta
FEBRUARY 4, 2025
Were sharing how Meta built support for data logs, which provide people with additional data about how they use our products. Here we explore initial system designs we considered, an overview of the current architecture, and some important principles Meta takes into account in making data accessible and easy to understand. Users have a variety of tools they can use to manage and access their information on Meta platforms.
Precisely
FEBRUARY 3, 2025
Key Takeaways A unified customer communication management (CCM) solution eliminates reliance on IT for communication updates, which empowers business users to create and deploy content quickly. Fast, personalized, and seamless customer communications help you build customer trust and drive loyalty. Save time and money with streamlined processes and automation that increase operational efficiency and improve the customer experience.
Advertisement
In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate
Simon Späti
FEBRUARY 3, 2025
While everyone’s talking about AI revolutionizing business, there’s a quiet renaissance happening with one of the most influential business tools created: the pivot table. In 2025, we’re witnessing something remarkable - modern data tools are bringing pivot tables back to the forefront. But why would cutting-edge platforms invest in a decades-old spreadsheet feature?
Snowflake
FEBRUARY 4, 2025
AI is here to stay. While 2023 brought wonder and 2024 ushered in widespread experimentation, 2025 will mark the year that retailers get serious about AI's real-world applications. But its complicated: AI proofs of concept are graduating from the sandbox to production even as major AI innovators face competition from newer upstarts. At this point, the pace of AI evolution is outstripping the news cycle.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
|
Start Data Engineering
FEBRUARY 3, 2025
1. Introduction 2. Split your SQL into smaller parts 2.1. Start with a baseline validation to ensure that your changes do not change the output too much 2.2. Split your CTAs/Subquery into separate functions (or models if using dbt) 2.3. Unit test your functions for maintainability and evolution of logic 3. Conclusion 4. Required reading 1. Introduction If you’ve been in the data space long enough, you would have come across really long SQL scripts that someone had written years ago.
KDnuggets
FEBRUARY 3, 2025
Check out this comparison of 5 AI frameworks to determine which you should choose.
Precisely
FEBRUARY 6, 2025
As businesses evolve and delivery speeds increase, IT operations teams face environments where downtime isn’t an option. The traditional ways of operations management are over modernization and holistic approaches are now essential. For IT operations (ITOps) teams, 2025 means reassessing technology stacks, processes, and people. Success in tackling modernization of IT operations management starts with assessing where your team is.
Data Engineering Weekly
FEBRUARY 2, 2025
Automate Airflow deploys with built-in CI/CD. Streamline code deployment, enhance collaboration, and ensure DevOps best practices with Astro's robust CI/CD capabilities. Try Astro Free → Jay Alammar: The Illustrated DeepSeek-R1 DeepSeek triggered quite a conversation and had an economic impact last week. Many articles explain how DeepSeek works, and I found the illustrated example much simpler to understand.
Speaker: Tamara Fingerlin, Developer Advocate
Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.
Robinhood
FEBRUARY 3, 2025
Robinhood is launching event contracts for the upcoming Kansas City vs. Philadelphia championship game through Robinhood Derivatives, LLC Today, Robinhood Derivatives, LLC (RHD) is launching event contracts for the Pro Football Championship, allowing eligible customers to place trades on the outcome of the upcoming showdown between Kansas City and Philadelphia.
KDnuggets
FEBRUARY 6, 2025
Interact with the advanced multimodal Llama model locally through its intuitive user interface or powerful endpoint service.
databricks
FEBRUARY 3, 2025
In the rapidly evolving landscape of data engineering and analytics, speed, scalability, and simplicity are invaluable. Serverless compute addresses these needs by eliminating.
Pinterest Engineering
FEBRUARY 3, 2025
Zhibo Fan | Machine Learning Engineer, Homefeed Candidate Generation; Bowen Deng | Machine Learning Engineer, Homefeed Candidate Generation; Hedi Xia | Machine Learning Engineer, Homefeed Candidate Generation; Yuke Yan | Machine Learning Engineer, Homefeed Candidate Generation; Hongtao Lin | Machine Learning Engineer, ATG Applied Science; Haoyu Chen | Machine Learning Engineer, ATG Applied Science; Dafang He | Machine Learning Engineer, Homefeed Relevance; Jay Adams | Principal Engineer, Pinner
Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage
There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.
Picnic Engineering
FEBRUARY 6, 2025
In our previous blogs, we explored how Picnics Page Platform transformed the way we build new featuresenabling faster iteration, tighter collaboration and less feature-specific complexity. Advances we made there have been a major boost for productivity and allow us to focus on the things that actually deliver customer value. However, while focussing on our technical architecture, we havent yet discussed one major opportunity: bringing our commercial and customer teams into themix.
KDnuggets
FEBRUARY 7, 2025
As tech layoffs increase, data scientists must adapt. Here's how to safeguard your data science job in 2025.
databricks
FEBRUARY 6, 2025
FOX Sports has a long history of driving the evolution of broadcast technology, from its high-definition coverage to experiments with virtual reality. Eventually.
Striim
FEBRUARY 5, 2025
AI thrives on real-time data. In a world where businesses generate massive volumes of data every second, success hinges on the ability to process, analyze, and act on that data instantly. Change Data Capture (CDC) and streaming technologies form the foundation for AI-driven analytics, ensuring data is always fresh, accurate, and actionable. Together, CDC and streaming empower businesses to: Supercharge AI models with real-time data: Provide AI with up-to-the-second insights to improve prediction
Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives
Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri
Sync Computing
FEBRUARY 4, 2025
As data engineers, understanding the intricacies of your Databricks environment is important. You cant optimize performance, budget or ensure efficient resource allocations without it. Thankfully, Databricks gives you a behind-the-scenes look at how your workspace is running in system tables. Everything from query performance to job execution and cluster activity is in those tables.
KDnuggets
FEBRUARY 3, 2025
Fine-tune the DeepSeek model step by step. even if you're new to LLMs!
databricks
FEBRUARY 6, 2025
Introducing Serverless Support for AWS Instance Profiles: Uniform Data Access At Databricks, we continuously strive to simplify data access and drive innovation across.
Striim
FEBRUARY 4, 2025
Introduction At Striim, we use our Salesforce Reader to read from our Salesforce account and write into Google BigQuery where we join data from HubSpot to create Looker reports that multiple internal teams (Sales, Customer Success and Finance) use for reporting, analysis and drive action items for their departments. This recipe shows how you can build a data pipeline to read data from Salesforce and write to BigQuery.
Advertisement
With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you
WeCloudData
FEBRUARY 6, 2025
We live in a highly data oriented world, thus it’s important to understand the key roles of the data ecosystem. Data scientists and engineers are two of the most important data professions and it is important to understand the difference between data engineering vs data science. At WeCloudData we specialize in preparing professionals for careers […] The post Data Engineering vs.
KDnuggets
FEBRUARY 7, 2025
Tools I use for coding, writing, grammar improvement, research, machine learning experiments, and organizing projects.
databricks
FEBRUARY 4, 2025
Databricks welcomes BladeBridge, a proven provider of AI-powered migration solutions for enterprise data warehouses. Together, Databricks and BladeBridge will help enterprises accelerate the.
ArcGIS
FEBRUARY 3, 2025
Explore how Dar used ArcGIS CityEngine for procedural urban design to transform a brownfield into a vibrant waterfront destination in Jeddah.
Speaker: Tamara Fingerlin, Developer Advocate
In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!
WeCloudData
FEBRUARY 3, 2025
Data is the new Gold. Everyday we use and generate data more than we often realize. Data is shaping our decisions, from scrolling through personalized social media feeds to checking weather forecasts before leaving home. Behind the scenes, Data Science powers banking apps to detect suspicious activity or when you get personalized recommendations on […] The post What is Data Science appeared first on WeCloudData.
KDnuggets
FEBRUARY 3, 2025
Learn to create a document-based chatbot with memory, powered by one of the world's top-performing LLMs.
databricks
FEBRUARY 5, 2025
Were thrilled to announce the General Availability (GA) of Databricks Clean Rooms on AWS and Azure, a significant step forward in enabling secure.
Confluent
FEBRUARY 3, 2025
Confluent Cloud Freight clusters are now Generally Available on AWS.
Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage
When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m
Expert insights. Personalized for you.
175 
Let's personalize your content