The Pragmatic Engineer

MARCH 23, 2023

👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. We cover one out of five topics in today’s subscriber-only The Scoop issue. To get full issues twice a week, subscribe here. There’s plenty of news and anecdotal evidence suggesting the jobs market for software engineers is cooling. In October 2022, I wrote about the start of a Big Tech hiring slowdown.

Software Engineer 279

Software Engineer Software Engineering Engineering Recruitment 279

Analytics Vidhya

MARCH 21, 2023

Introduction In today’s world, data is growing exponentially with time with digitalization. Organizations are using various cloud platforms like Azure, GCP, etc., to store and analyze this data to get valuable business insights from it. You will study top 11 azure interview questions in this article which will discuss different data services like Azure Cosmos […] The post Top 11 Azure Data Services Interview Questions in 2023 appeared first on Analytics Vidhya.

Data 247

Data Cloud Data Lake Database 247

KDnuggets

MARCH 21, 2023

The first part covers the list of Programming, Web scraping, Statistics & Probability, Data Analytics, SQL, and Business Intelligence free courses.

Data Science 160

Data Science Business Intelligence SQL Data Analytics 160

Waitingforcode

MARCH 23, 2023

Welcome to the 2nd part of the series with great streaming and project organization blog posts summaries!

Data Engineering 147

Data Engineering Data Engineer Engineering Project 147

Advertisement

In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate

Data Pipeline

databricks

MARCH 23, 2023

Summary We show that anyone can take a dated off-the-shelf open source large language model (LLM) and give it magical ChatGPT-like instruction following.

IT 145

IT 145

Analytics Vidhya

MARCH 24, 2023

Introduction Containerization is becoming more popular and widely used by developers in the software industry in recent years. Docker is still considered one of the top tools for creating containers by building Images between containerization platforms or cloud platforms. Containerizing is all about bundling up a software application/service and isolating it from the host environment […] The post Top 4 Cloud Platforms to Host or Run Docker Containers for Free appeared first on Analytics Vi

Cloud 225

Cloud Building Data Warehouse IT 225

DoorDash Engineering

MARCH 21, 2023

While building a feature store to handle the massive growth of our machine-learning (“ML”) platform, we learned that using a mix of different databases can yield significant gains in efficiency and operational simplicity. We saw that using Redis for our online machine-learning storage was not efficient from a maintenance and cost perspective.

Machine Learning 132

Machine Learning AWS Database Utilities 132

Christophe Blefari

MARCH 24, 2023

The Earth can also generate great images ( credits ) Dear readers, I hope this new edition finds you well. It seems that you really liked the recent editions, which is perfect because it was fun to write. I feel that this week all the articles I found relevant for the newsletter are either AI related or technical. I really don't know how to deal with news overflow about the Gen AI landscape.

Lambda Architecture 130

Lambda Architecture Data Pipeline Data SQL 130

Analytics Vidhya

MARCH 24, 2023

Introduction With the world of data science constantly evolving, it is important to stay up-to-date with the latest trends and techniques for aspiring and established professionals alike. That’s why we at Analytics Vidhya host a series of informative and interactive webinars designed to help you enhance your skills and expand your knowledge of data tech […] The post Don’t Miss Out: Last Few and Exciting DataHour of March appeared first on Analytics Vidhya.

Data Science 222

Data Science Designing Data Machine Learning 222

Confessions of a Data Guy

MARCH 22, 2023

The post Future Proof Yourself Against AI. appeared first on Confessions of a Data Guy.

Data 130

Data Data Engineering Data Engineer Engineering 130

Speaker: Tamara Fingerlin, Developer Advocate

Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.

Data

Waitingforcode

MARCH 20, 2023

Have you missed any cloud data engineering-related news in the last 3 months? No worries, I got you covered with the new part of the "What's new on the cloud for data engineers." series.

Data Engineering 130

Data Engineering Data Engineer Cloud Engineering 130

KDnuggets

MARCH 24, 2023

Accelerate your data-cleaning process without a hassle.

Python 129

Python Data Process Data Science 129

Data Engineering Podcast

MARCH 18, 2023

Summary As with all aspects of technology, security is a critical element of data applications, and the different controls can be at cross purposes with productivity. In this episode Yoav Cohen from Satori shares his experiences as a practitioner in the space of data security and how to align with the needs of engineers and business users. He also explains why data security is distinct from application security and some methods for reducing the challenge of working across different data systems.

Data Security 130

Data Security Machine Learning Data Science Metadata 130

Confessions of a Data Guy

MARCH 20, 2023

Are lambdas one of those tools that everyone uses and no one talks about? I guess I’ve taken them for granted over the years, even though they are incredibly useful. For a lot of my Data Engineering career I didn’t really think about or use AWS lambdas, I just saw them as little annoying flies […] The post AWS Lambdas. Useful for Data Engineering?

AWS 130

AWS Data Engineering Data Engineer Engineering 130

Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage

There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.

Systems

databricks

MARCH 20, 2023

Large language models (LLMs) are currently in the spotlight following the sensational release of ChatGPT. Many are wondering how to take advantage of.

Data Science 128

Data Science Engineering Data 128

Lyft Engineering

MARCH 22, 2023

lyft2vec — Embeddings at Lyft Co-authors: Javen Xu , Hakan Baba and Adriana Deneault Intro Graph learning methods can reveal interesting insights that capture the underlying relational structures. Graph learning methods have many industry applications in areas such as product or content recommender systems and network analysis. In this post, we discuss how we use graph learning methods at Lyft to generate embeddings — compact vector representation of high-dimensional information.

Algorithm 127

Algorithm Machine Learning Building Structured Data 127

KDnuggets

MARCH 22, 2023

SQL and Python Interview Questions for Data Analysts • 5 SQL Visualization Tools for Data Engineers • 5 Free Tools For Detecting ChatGPT, GPT3, and GPT2 • Top Free Resources To Learn ChatGPT • Free TensorFlow 2.

SQL 126

SQL Python Data Engineering Data Engineer 126

ArcGIS

MARCH 20, 2023

Using ArcGIS Pro to build 'Human Geography' style vector basemaps in different projections, for use in ArcGIS Online

Project 104

Project Building 104

Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives

Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri

Manufacturing

databricks

MARCH 21, 2023

We are thrilled to announce that Databricks Unity Catalog is now generally available on Google Cloud Platform (GCP). Unity Catalog provides a unified.

Google Cloud 105

Google Cloud Cloud 105

Uber Engineering

MARCH 23, 2023

In this post we will dive into the algorithm, data modeling, and system design that go into estimating the length of time drivers would have to wait for a trip request at a given location, empowering them to strategically remain or reposition.

Algorithm 98

Algorithm Designing Systems Data 98

KDnuggets

MARCH 24, 2023

Adding embedding signals into generated text can help mitigate potential risks of plagiarism, misinformation, and abuse in large language models.

Process 123

Process 123

Knowledge Hut

MARCH 23, 2023

Project management is vital to the success of any company. It is responsible for keeping all project details organized, prioritized, and on track to meet deadlines and ensure quality. It also has a lot of influence over whether or not a project is completed successfully. If you're an entrepreneur looking to build your business, you'll want to ensure your project management has the skills necessary to keep things on track.

Project 98

Project Management Systems Process 98

Advertisement

With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you

Data Engineering

databricks

MARCH 21, 2023

This blog is authored by Mohamed Afifi Ibrahim, Principal Machine Learning Engineer at Barracuda Networks. 74% of organizations globally have fallen victim to.

Machine Learning 105

Machine Learning Engineering 105

Towards Data Science

MARCH 20, 2023

Infrastructure as Code for database practitioners Continue reading on Towards Data Science »

MySQL 98

MySQL AWS Data Science Database 98

KDnuggets

MARCH 21, 2023

In this course, we'll dive into the world of prompt design and learn how to create AI products like auto-generated podcasts.

Designing 120

Designing Programming Building 120

ThoughtSpot

MARCH 21, 2023

This is part of our ongoing spotlight series which highlights ThougthSpot’s quarterly Selfless Excellence champion. At ThoughtSpot, Selfless Excellence is the guiding principle for our culture. It means we strive for excellence in everything we do, while always putting the customer and team ahead of ourselves. We prioritize humility and actively discourage office politics of any kind.

Recruitment 98

Recruitment Education Programming Building 98

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

Data

databricks

MARCH 22, 2023

We are excited to announce that PrivateLink and using customer-managed keys (CMK) for encryption are now Generally Available (GA) for Databricks on AWS.

AWS 98

AWS Management 98

Towards Data Science

MARCH 23, 2023

The difference between tables and views and how to use them Continue reading on Towards Data Science »

Data Warehouse 98

Data Warehouse Data Science Data Cloud Computing 98

KDnuggets

MARCH 22, 2023

Our latest cheat sheet is a handy reference for Plotly Express, a high-level data visualization library in Python built on top of Plotly.

Python 120

Python Data Machine Learning Data Science 120

Tweag

MARCH 22, 2023

This is the second of two companion blog posts to the paper Linearly Qualified Types , published at ICFP 2021 (there is also a long version, with appendices ). These blog posts will dive into some subjects that were touched, but not elaborated on, in the paper. For more introductory content, you may be interested in my talk at ICFP. The problem with O(1) freeze The problem with scopes In the example API for pure mutable arrays, the original Linear Haskell paper ( Arxiv version ) featured the fun

Coding 98

Coding IT Designing Management 98

Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage

When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m

Software Engineer