The Pragmatic Engineer

JULY 6, 2023

Originally published 6 July 2023 👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. We cover one out of six topics in today’s subscriber-only The Scoop issue. If you’re not yet a full subscriber, you missed this week’s deep-dive on What a senior engineer is at Big Tech. To get the full issues twice a week, subscribe here.

Google Cloud 260

Google Cloud Media Cloud Utilities 260

Analytics Vidhya

JULY 6, 2023

Introduction In this era of Generative Al, data generation is at its peak. Building an accurate machine learning and AI model requires a high-quality dataset. The quality assurance of the dataset is the most critical task, as poor data causes inaccurate analytics and unidentified predictions that can affect the entire repo of any business and […] The post Getting Started with Amazon SageMaker Ground Truth appeared first on Analytics Vidhya.

Datasets 243

Datasets Machine Learning Building Algorithm 243

Tweag

JULY 5, 2023

Meta recently announced they have made Buck2 open-source. Buck2 is a from-scratch rewrite of Buck , a polyglot, monorepo build system that was developed and used at Meta (Facebook), and shares a few similarities with Bazel. As you may know, the Scalable Builds Group at Tweag has a strong interest in such scalable build systems. We were thrilled to have the opportunity to work with Meta on Buck2 to help make the tool useful and successful in the open-source use case.

Systems 141

Systems Building Java Programming Language 141

Confessions of a Data Guy

JULY 7, 2023

Sometimes it seems like the Data Engineering landscape is starting to shoot off into infinity. With the rise of Rust, new tools like DuckDB, Polars, and whatever else, things do seem to shifting at a fundamental level. It seems like there is someone at the base of a titering rock with a crowbar, picking and […] The post Ballista (Rust) vs Apache Spark.

Data Engineer 130

Data Engineer Data Engineering Engineering IT 130

Advertisement

In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate

Data Pipeline

Waitingforcode

JULY 6, 2023

That's often a dilemma, whether we should put multiple sinks working on the same data source in the same or in different Apache Spark Structured Streaming applications? Both solutions may be valid depending on your use case but let's focus here on the former one including multiple sinks together.

Data 130

Data 130

Christophe Blefari

JULY 3, 2023

2 summits ( credits I cropped the image) Hey, since I said I should try to send the newsletter at a specific schedule I did not. Haha. Still here the newsletter for last week. This is a small wrap-up from the Snowflake and Databricks Data + AI summits which have taken place last week. There are so many sessions at both summits that this is impossible to watch everything, more Databricks and Snowflake do not put in free access online everything so I can't wait everything.

SQL 130

SQL Data Kafka AWS 130

KDnuggets

JULY 7, 2023

Reinforcement learning basics to get your feet wet. Learn the components and key concepts in the reinforcement loading framework: from agents and rewards to value functions, policy, and more.

Machine Learning 123

Machine Learning 123

databricks

JULY 6, 2023

This is part six of a multi-part series to share key insights and tactics with Senior Executives leading data and AI transformation initiatives.

Data Analytics 105

Data Analytics Data 105

ArcGIS

JULY 7, 2023

ArcGIS methods to maintain measure attributes on LRS routes along with samples and linear referencing use cases.

Transportation 98

Transportation Utilities 98

Towards Data Science

JULY 5, 2023

And how to ingest valuable data for free Photo by Tobias Fischer on Unsplash Data modeling can be a challenging task for analytics teams. With unique business entities in every organization, finding the right structure and granularity for each table becomes open-ended. But fear not! Some of the data you need is simplistic, free, and occupies minimal storage.

Data Warehouse 98

Data Warehouse BI Python Datasets 98

Speaker: Tamara Fingerlin, Developer Advocate

Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.

Data

KDnuggets

JULY 5, 2023

Predicting Movie Status Based on Review Sentiment.

Data Science 121

Data Science Project Data Machine Learning 121

Knowledge Hut

JULY 4, 2023

Pattern recognition is a field of computer science that deals with the automatic identification of patterns in data. This can be done by finding regularities in the data, such as correlations or trends, or by identifying specific features in the data. Pattern recognition is used in a wide variety of applications, including Image processing, Speech recognition, Biometrics, Medical diagnosis, and Fraud detection.

Machine Learning 98

Machine Learning Medical Algorithm Deep Learning 98

ArcGIS

JULY 3, 2023

Learn more about 3D GIS and Digital Twins at the 2023 Esri User Conference, which takes place on July 11-14, 2023.

98

98

databricks

JULY 5, 2023

Get started and build a credit data platform for your business by visiting the demo at dbdemos.ai. Introduction According to the World Bank's.

Building 98

Building Banking Data 98

Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage

There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.

Systems

KDnuggets

JULY 1, 2023

Building a datalake for semi-structured data or json has always been challenging. Imagine if the json documents are streaming or continuously flowing from healthcare vendors then we need a robust modern architecture that can deal with such a high volume. At the same time analytics layer also needs to be created so as to generate value from it.

Building 119

Building Healthcare Structured Data Architecture 119

Knowledge Hut

JULY 6, 2023

Lean in project management, where the word ‘lean’ is associated with less wastage and more value addition. Lean is an Agile methodology that helps industries to improve productivity, increase customer value, eliminate problems, enhance the organization’s processes, reduce waste, and encourage continuous improvement. Historically, it was first introduced in the manufacturing industry, but today it is prevalent in almost every industry, including healthcare, education, software d

Project 98

Project Management Manufacturing Transportation 98

Towards Data Science

JULY 3, 2023

What matters is not the quantity, but the quality.

Data Science 97

Data Science Data Programming Data Engineering 97

databricks

JULY 5, 2023

This blog authored post by Jaison Dominic, Senior Manager, Information Systems at Amgen, and Lakhan Prajapati, Director of Architecture and Engineering at ZS.

Data Governance 95

Data Governance Government Architecture Engineering 95

Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives

Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri

Manufacturing

KDnuggets

JULY 6, 2023

This article delves into the concept of Chain-of-Thought (CoT) prompting, a technique that enhances the reasoning capabilities of large language models (LLMs). It discusses the principles behind CoT prompting, its application, and its impact on the performance of LLMs.

IT 116

IT 116

Knowledge Hut

JULY 3, 2023

In a world of limitless possibilities driven by cutting-edge technology, innovations, and artificial intelligence, businesses can no longer rely on traditional models for opportunities and expansion. While traditional KPIs may still be important to certain aspects of business and economics, current times demand more enduring efforts to match up with the fast-paced environment and business tactics.

Project 98

Project Management Algorithm Consulting 98

Towards Data Science

JULY 5, 2023

Understanding ACID properties in the context of database transactions Continue reading on Towards Data Science »

Data Science 97

Data Science Database Data SQL 97

Lyft Engineering

JULY 5, 2023

By Yuko Yamazaki a Senior Director of Engineering on Lyft’s Customer Platform Team & the Founder of Lyft’s Equitable Development Initiative (EDI). Lyft’s Tech Diversity Over the last three years, Lyft has increased the representation of Underrepresented Minorities (URM) in technical leadership roles by more than three times. At Lyft, URM is defined as team members from Women, Black, and Latinx communities, and technical leadership roles are defined as Staff+ IC and M1+ manager roles.

Programming 81

Programming Engineering Management Designing 81

Advertisement

With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you

Data Engineering

KDnuggets

JULY 6, 2023

Techniques to address imbalanced data in the context of classification, while keeping the data distribution in mind.

Data 116

Data Machine Learning 116

Knowledge Hut

JULY 4, 2023

Project Management Professional (PMP) is an internationally recognized professional designation offered by the Project Management Institute. The PMP official credential designates competencies in the areas of core project management, principles, practices, processes, and tools outlined in the Project Management Body of Knowledge (PMBOK) and is hence globally regarded as the gold standard in the field of project management.

Programming 98

Programming Education Certification Project 98

Towards Data Science

JULY 6, 2023

Maximize the quality and productivity of your data pipelines Continue reading on Towards Data Science »

Data Pipeline 96

Data Pipeline Data Science Data Data Engineering 96

ArcGIS

JULY 3, 2023

Landsat imagery provides a great way to assess damage to regions from wildfires and multidimensional imagery allows you to see it as it happens.

IT 69

IT 69

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

Data

KDnuggets

JULY 6, 2023

Project management can be one of the biggest challenges in data science projects. Learn how you can ensure your project management methods are down-packed and effective.

Data Science 115

Data Science Project Management Data 115

Knowledge Hut

JULY 7, 2023

For over 2 decades, Java has been the mainstay of app development. It is one of the most versatile web development tools today and hence popular among app developers. Another reason for its popularity is its cross-platform and cross-browser compatibility, making applications written in Java highly portable. These very qualities gave rise to the need for reusability of code, version control, and other tools for Java developers.

Java 97

Java Technology Programming Language Scala 97

Towards Data Science

JULY 4, 2023

How Hamilton can help you write more maintainable Airflow DAGs An abstract representation of how Airflow & Hamilton relate. Airflow helps bring it all together, while Hamilton has make the innards manageable. Image from Pixabay. This post is written in collaboration with Thierry Jean and originally appeared here. This post walks you through the benefits of having two open source projects, Hamilton and Airflow , and their directed acyclic graphs (DAGs) work in tandem.

Python 95

Python Data Science Machine Learning Coding 95

Workfall

JULY 4, 2023

Reading Time: 8 minutes In the world of data engineering, a mighty tool called DBT (Data Build Tool) comes to the rescue of modern data workflows. Imagine a team of skilled data engineers on an exciting quest to transform raw data into a treasure trove of insights. With DBT, they weave powerful SQL spells to create data models that capture the essence of their organization’s information.

Data Warehouse 69

Data Warehouse SQL Database PostgreSQL 69

Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage

When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m

Software Engineer