The Pragmatic Engineer
MARCH 12, 2024
👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. In this article, we cover one out of five topics from today’s subscriber-only article What Changed in 50 Years of Computing.
Christophe Blefari
MARCH 15, 2024
Mountains I hope this e-mail finds you well, wherever you are. I'd like to thank you for the excellent comments you sent me last week after the publication of the first version of the Recommendations. This is just the beginning! This week I've added a subscribe button in the Recommendations page in order for you to opt-in for the weekly recommendation email—every Tuesday.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
KDnuggets
MARCH 13, 2024
If you are a data scientist who works with large amounts of data and hasn’t learned SQL yet - now might be the time.
Data Engineering Podcast
MARCH 10, 2024
Summary Data lakehouse architectures are gaining popularity due to the flexibility and cost effectiveness that they offer. The link that bridges the gap between data lake and warehouse capabilities is the catalog. The primary purpose of the catalog is to inform the query engine of what data exists and where, but the Nessie project aims to go beyond that simple utility.
Advertisement
In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate
Engineering at Meta
MARCH 12, 2024
Marking a major investment in Meta’s AI future, we are announcing two 24k GPU clusters. We are sharing details on the hardware, network, storage, design, performance, and software that help us extract high throughput and reliability for various AI workloads. We use this cluster design for Llama 3 training. We are strongly committed to open compute and open source.
ArcGIS
MARCH 11, 2024
A new R package created by the R-ArcGIS Bridge team enables integration with ArcGIS location services, enhancing their combined powers.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
|
databricks
MARCH 13, 2024
Sharing a belief that open source solutions will foster innovation and transparency in generative AI development, Databricks has announced a partnership and participation.
Waitingforcode
MARCH 12, 2024
That's the question. The lack of the processing time trigger means more a reactive micro-batch triggering but it cannot be considered as the single true best practice. Let's see why.
Snowflake
MARCH 14, 2024
As a cohesive ERP solution, SAP is often one of the largest data resources in an organization, containing everything from financial and transactional data to master information about customers, vendors, materials, facilities, planning and even HR. But SAP has limited analytics capabilities, and directly ingesting SAP data into Snowflake can present a challenge.
KDnuggets
MARCH 14, 2024
Explore the fundamental steps for creating a successful AI Application with Python and other tools.
Speaker: Tamara Fingerlin, Developer Advocate
Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.
databricks
MARCH 11, 2024
Today, we are excited to announce the general availability of Feature Serving. Features play a pivotal role in AI Applications, typically requiring considerable.
Seattle Data Guy
MARCH 11, 2024
Recently, I wrote an article diving into what Druid is and which companies are using it. Now I wanted to do a deeper dive into Apache Druid’s architecture. Apache Druid has several unique features that allow it to be used as a real-time OLAP. Everything from its various nodes and processes that each have unique… Read more The post Apache Druid’s Architecture – How Druid Processes Data In Real Time At Scale appeared first on Seattle Data Guy.
Snowflake
MARCH 12, 2024
Geospatial data can give a business a competitive edge — especially when it’s combined with the company’s own data resources. Considering a new store location? You’ll want to analyze not just where your nearest competitors and potential customers are, but also retail footfall numbers, historical traffic patterns, distance from distribution centers, environmental factors, potential delivery times to customers and more.
KDnuggets
MARCH 13, 2024
Want to move into the data science field? Or advance your career in the data? Don’t miss these must-have skills.
Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage
There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.
databricks
MARCH 13, 2024
Policymakers around the world are paying increased attention to artificial intelligence. The world’s most comprehensive AI regulation to date was just passed by.
Yelp Engineering
MARCH 12, 2024
What is Engineering Effectiveness Metrics (EE Metrics)? EE Metrics was envisioned as a hub that helps teams manage their technical debt. EE Metrics provides every team with a detailed web page that contains information about technical debt that needs to be addressed. It also serves as a platform to highlight top engineering initiatives at the organization level.
Cloudera
MARCH 14, 2024
This year I had the pleasure of joining world leaders, business titans, and changemakers at the 54th World Economic Forum in Davos to grapple with complex challenges that demand collective action. It was validating to see that AI wasn’t just a fringe topic – it was the protagonist. We clearly saw how much potential AI holds to help create and foster new business opportunities, forge stronger partnerships, find new avenues for dialogue, and achieve more efficient use of emerging technologi
KDnuggets
MARCH 12, 2024
Want to switch to a tech career? Make it happen with these free computer science courses.
Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives
Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri
databricks
MARCH 13, 2024
Introduction Let’s explore a common scenario – your team is eager to leverage open source LLMs to build chatbots for customer support interactions.
ArcGIS
MARCH 11, 2024
The ArcGIS Well-Architected Framework and ArcGIS Architecture Center provides guidance for implementing systems with ArcGIS.
Snowflake
MARCH 11, 2024
Last year, we held our first Accelerate event , to explore industry trends, data and technology innovations, and data strategy case studies in financial services. This year, we are expanding to five industry events, featuring leaders in financial services; retail and consumer goods; manufacturing; media, advertising and entertainment; and healthcare and life sciences.
KDnuggets
MARCH 15, 2024
Data science is not the only career path you could take, even if you have already learned to be one.
Advertisement
With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you
databricks
MARCH 13, 2024
Artificial Intelligence is top-of-mind with every C-suite in Retail & Consumer Goods. Companies see the potential to deliver better customer service, derive faster.
ThoughtSpot
MARCH 15, 2024
This is part of our ongoing spotlight series which highlights ThougthSpot’s quarterly Selfless Excellence champion. ThoughtSpot's culture is rooted in our core value of Selfless Excellence. This means we consider our teammates, customers, and society at large ahead of our own personal wins without the distraction of office politics. Our common ground ensures that we are moving together with intention and integrity in everything we do—when we run the business, plan our go-to-market strategy,
Confessions of a Data Guy
MARCH 14, 2024
Most Software Engineers think of themselves as too smart. They think they are the best and brightest coder alive or that has ever lived. Doing so, they stunt themselves from becoming Senior Engineers and become hard to work with, the nightmare of the PR process. You don’t need to be the smartest person in the […] The post Don’t Be So Smart appeared first on Confessions of a Data Guy.
KDnuggets
MARCH 11, 2024
From learning to earning: 4 essential DataCamp certifications to land your dream job.
Speaker: Tamara Fingerlin, Developer Advocate
In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!
Towards Data Science
MARCH 15, 2024
Career advice for aspiring data practitioners Continue reading on Towards Data Science »
Knowledge Hut
MARCH 12, 2024
Project managers are in high demand, and companies are willing to pay top dollar for qualified individuals. In today's fast-paced business world, having a Project Management Professional (PMP) certification can give you a significant advantage over other job candidates. As per the PMI statistics, a certified PMP is entitled to a pay increase of 20%.
Confessions of a Data Guy
MARCH 14, 2024
When you’ve been data modeling as long as I have, it gets to be the same old … same old. People make data modeling harder than it has to be. There is a lot of jargon that gets thrown around … third-normal-form, OLAP, OLTP … I give you the 3-4 basics that are at the […] The post Data Modeling Is Easy appeared first on Confessions of a Data Guy.
KDnuggets
MARCH 14, 2024
Ready to become a SAS Certified Specialist in Statistics for Machine Learning? Here’s everything you need to know about the recently released certification from SAS.
Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage
When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m
Expert insights. Personalized for you.
329 
Let's personalize your content