Python - Data Engineering Digest

AWS Lambdas – Python vs Rust. Performance and Cost Savings.

Confessions of a Data Guy

FEBRUARY 26, 2023

Someone on Linkedin recently brought up the point that companies could save gobs of money by swapping out AWS Python lambdas for Rust ones. While it raised the ire of many a Python Data Engineer, I thought it sounded like a great idea. At least it’s an excuse to […] The post AWS Lambdas – Python vs Rust.

AWS

AWS Python Data Engineering Data Engineer

Isolated Python Environments using Docker

Analytics Vidhya

FEBRUARY 6, 2023

Introduction While working with multiple projects, there are chances of issues with versions of packages in python; for example, a project needs a new version of a package, and another requires a different version. Sometimes the python version itself changes from project to project.

Python

Python Project Management Data Engineering

Building cost effective data pipelines with Python & DuckDB

Start Data Engineering

MAY 28, 2024

KISS: DuckDB + Python = easy to debug and quick to develop 4. Cost calculation: DuckDB + Ephemeral VMs = dirt cheap data processing 4.3. Processing data less than 100GB? Use DuckDB 4.4. Distributed systems are scalable, resilient to failures, & designed for high availability 4.5.

Data Pipeline

Data Pipeline Python Building Data

What are Data Access Object and Data Transfer Object in Python?

Analytics Vidhya

FEBRUARY 6, 2023

This ensures easy […] The post What are Data Access Object and Data Transfer Object in Python? Especially while working with databases, it is often considered a good practice to follow a design pattern. appeared first on Analytics Vidhya.

Accessible

Accessible Accessibility Python Database

Apache Airflow® Crash Course: From 0 to Running your Pipeline in the Cloud

Airflow enables you to define workflows as Python code, allowing for dynamic and scalable pipelines suitable to any use case from ETL/ELT to running ML/AI operations in production. With over 30 million monthly downloads, Apache Airflow is the tool of choice for programmatically authoring, scheduling, and monitoring data pipelines.

Cloud

10 GitHub Repositories to Master Python

KDnuggets

APRIL 10, 2024

Learn Python through tutorials, blogs, books, project work, and exercises. Access all of it on GitHub for free and join a supportive open-source community.

Python

Python Accessible Accessibility Project

7 Python Libraries Every Data Engineer Should Know

KDnuggets

APRIL 25, 2024

Here’s a list of Python libraries you’ll find super helpful. Interested in switching to data engineering?

Python

Python Data Engineering Data Engineer Engineering

Seamless SQL And Python Transformations For Data Engineers And Analysts With SQLMesh

Data Engineering Podcast

JUNE 25, 2023

__init__ covers the Python language, its community, and the innovative ways it is being used. __init__ covers the Python language, its community, and the innovative ways it is being used. Closing Announcements Thank you for listening! Don't forget to check out our other shows. Closing Announcements Thank you for listening!

Data Engineering

Data Engineering Data Engineer Python Engineering

5 Free Python Courses for Data Science Beginners

KDnuggets

MAY 31, 2024

Are you a data science beginner looking to learn Python? Start learning today with these 5 free courses.

Data Science

Data Science Python Data

7 Free Online Python REPLs

KDnuggets

SEPTEMBER 24, 2024

Running Python code directly in your browser is incredibly convenient, eliminating the need for Python environment setup and allowing instant code execution without dependency or hardware concerns. I am a strong advocate of using a cloud-based IDE for working with data, machine learning, and learning Python as a beginner.

Python

Python Machine Learning Coding Programming

5 Simple Steps to Automate Data Cleaning with Python

KDnuggets

MAY 3, 2024

Automate your data cleaning process with a practical 5-step pipeline in Python, ideal for beginners.

Python

Python Data Process

FastAPI Tutorial: Build APIs with Python in Minutes

KDnuggets

JUNE 13, 2024

Want to build APIs with Python? Learn how to do so using FastAPI with this step-by-step tutorial.

Python

Python Building

10 Python Libraries Every Data Scientist Should Know

KDnuggets

AUGUST 12, 2024

Check out these Python libraries for data science that you can't do without. Want to take the next step in your journey to becoming a data scientist?

Python

Python Data Science Data

Don’t Manage Your Python Environments, Just Use Docker Containers

KDnuggets

JANUARY 28, 2025

Python environment management can sometimes give you that awful feeling in the pit of your stomach. So don't do it: just use Docker containers.

Python

Python Management AWS IT

11 Python Magic Methods Every Programmer Should Know

KDnuggets

NOVEMBER 28, 2023

Want to support the behavior of built-in functions and method calls in your Python classes? Magic methods in Python let you do just that! So let’s uncover the method behind the magic.

Python

10 Built-In Python Modules Every Data Engineer Should Know

KDnuggets

SEPTEMBER 2, 2024

Check out this round-up of built-in Python modules that'll come in handy for data engineering tasks. Interested in data engineering?

Python

Python Data Engineering Data Engineer Engineering

Mastering Python for Data Science: Beyond the Basics

KDnuggets

MARCH 28, 2024

This article serves as a detailed guide on how to master advanced Python techniques for data science. It covers topics such as efficient data manipulation with Pandas, parallel processing with Python, and how to turn models into web services.

Data Science

Data Science Python Data Process

Challenges You Will Face When Parsing PDFs With Python – How To Parse PDFs With Python

Seattle Data Guy

NOVEMBER 19, 2024

Most of us will turn to Python and our trusty list of Python libraries and start plugging away. Of course, there are many challenges… Read more The post Challenges You Will Face When Parsing PDFs With Python – How To Parse PDFs With Python appeared first on Seattle Data Guy.

Python

Python Data Data Science Data Engineering

Is Python OOP the Devil? Or Savior?

Confessions of a Data Guy

JUNE 3, 2024

Nothing will raise the hackles on the backs of hairy and pale programmers who’ve been stuck in their mom’s basement for a decade like bringing up OOP (Object Oriented Programming), especially in the context of Python. appeared first on Confessions of a Data Guy.

Python

Python Programming IT Data

3 Ways of Building Python Projects using GPT-4o

KDnuggets

AUGUST 7, 2024

Learn about essential AI tools that can help you develop Python projects faster and with fewer bugs using natural language.

Python

Python Project Building

Build An AI Application with Python in 10 Easy Steps

KDnuggets

MARCH 14, 2024

Explore the fundamental steps for creating a successful AI Application with Python and other tools.

Python

Python Building

10 Python Statistical Functions

KDnuggets

AUGUST 14, 2024

This guide will go over 10 essential statistical functions in Python using commonly-used libraries.

Python

Tips for Handling Large Datasets in Python

KDnuggets

NOVEMBER 29, 2024

Here are some tips to make working with such large datasets in Python simpler. Working with large datasets is common but challenging.

Datasets

Datasets Python

Free Harvard Course: Introduction to AI with Python

KDnuggets

DECEMBER 18, 2023

Looking for a great course to learn Artificial Intelligence with Python? Check out this free course from Harvard University.

Python

5 Free University Courses to Learn Python

KDnuggets

DECEMBER 14, 2023

Looking for the best resources to learn Python programming? Check out these free university courses.

Python

Python Programming

A New Python Package Manager

KDnuggets

NOVEMBER 14, 2024

Manage Python projects, run scripts and tools, handle dependencies, and install packages—all with the uv tool.

Python

Python Management Project

5 Python Best Practices for Data Science

KDnuggets

MAY 29, 2024

Level up your Python skills for data science with these by following these best practices.

Data Science

Data Science Python Data

10 Python Libraries Every Developer Should Know

KDnuggets

DECEMBER 2, 2024

In this article, we’ll go over Python libraries for tasks like logging, unit testing, data handling, and more — each with features that can simplify your application development.

Python

Python Data

Mastering f-strings in Python

KDnuggets

NOVEMBER 7, 2024

Discover how to leverage Python's f-strings (formatted string literals) to write cleaner, more efficient, and more readable code.

Python

Python Coding

Mastering Python: 7 Strategies for Writing Clear, Organized, and Efficient Code

KDnuggets

MAY 17, 2024

Optimize Your Python Workflow: Proven Techniques for Crafting Production-Ready Code

Python

Python Coding

Partial Functions in Python: A Guide for Developers

KDnuggets

SEPTEMBER 19, 2024

In Python, functions often require multiple arguments, and you may find yourself repeatedly passing the same values for certain parameters. Python’s built-in functools module allows you to create partial functions. This is where partial functions can help.

Python

Python in Finance: Real Time Data Streaming within Jupyter Notebook

KDnuggets

FEBRUARY 20, 2024

This guide covers dynamic visualizations, a Python for quant finance use case, and Bollinger Bands analysis with live data. Learn a modern approach to stream real-time data in Jupyter Notebook.

Finance

Finance Python Data

How to Learn Python the Lazy Way

KDnuggets

OCTOBER 18, 2024

It is a guide for lazy people who want to learn Python and earn dollars. The title says everything.

Python

Python IT

5 Free Courses to Master Python for Data Science

KDnuggets

FEBRUARY 7, 2024

Want to learn Python to kickstart your career in data? Here are five free courses to help you master Python for data science.

Data Science

Data Science Python Data

Beginner’s Guide to Building LLM Apps with Python

KDnuggets

JUNE 6, 2024

In this article, you will be impacted by the knowledge you need to start building LLM apps with Python programming language.

Python

Python Building Programming Language Programming

5 Free Advanced Python Programming Courses

KDnuggets

APRIL 24, 2024

Looking to level up your Python skills without spending a dime? Check out this article featuring 5 advanced Python courses that you can take for free!

Python

Python Programming

10 Python Libraries Every Data Analyst Should Know

KDnuggets

NOVEMBER 19, 2024

Here's a list of Python libraries you cannot do without. Interested in data analytics?

Python

Python Data Analytics Data

Convert Bytes to String in Python: A Tutorial for Beginners

KDnuggets

JULY 15, 2024

Strings are common built-in data types in Python. Let’s learn how to convert bytes to string in Python. But sometimes, you may need to work with bytes instead.

Bytes

Bytes Python Data

Introduction to Multithreading and Multiprocessing in Python

KDnuggets

DECEMBER 4, 2023

Learn about Multithreading and Multiprocessing environments using Python with their implementation and limitations.

Python

10 Useful Python One-Liners for Data Cleaning

KDnuggets

OCTOBER 29, 2024

Here are some useful Python one-liners for common data cleaning tasks.

Python

Python Data

Replacing Pandas with Polars. A Practical Guide.

Confessions of a Data Guy

JANUARY 19, 2023

For those using Python, it’s probably one of the […] The post Replacing Pandas with Polars. I haven’t used Pandas in many a year, decades, or whatever. We’ve all been there, done that. Pandas I mean. I would dare say it’s a rite of passage for most data folk. A Practical Guide.

Python

Python IT Data Big Data

Pip Install YOU: A Beginner’s Guide to Creating Your Python Library

KDnuggets

AUGUST 22, 2024

Have you ever wanted to create your library in Python? Well, it’s achievable and surprisingly quite easy!

Python

Beginner’s Guide to Machine Learning with Python

KDnuggets

JUNE 4, 2024

Master the Fundamentals of Predictive Modeling with Python: An In-Depth Guide to Machine Learning Algorithms and Sci-kit Learn Implementation.

Machine Learning

Machine Learning Python Algorithm

Building Interactive Data Science Applications with Python

KDnuggets

OCTOBER 25, 2024

Using Python to build engaging and interactive applications where users can pass in an input, get and feedback and make use of multimedia elements such as images, videos, and audio.

Python

Python Building Data Science Data

AWS Lambdas – Python vs Rust. Performance and Cost Savings.

Isolated Python Environments using Docker

Trending Sources

Building cost effective data pipelines with Python & DuckDB

What are Data Access Object and Data Transfer Object in Python?

Apache Airflow® Crash Course: From 0 to Running your Pipeline in the Cloud

10 GitHub Repositories to Master Python

7 Python Libraries Every Data Engineer Should Know

Seamless SQL And Python Transformations For Data Engineers And Analysts With SQLMesh

5 Free Python Courses for Data Science Beginners

7 Free Online Python REPLs

5 Simple Steps to Automate Data Cleaning with Python

FastAPI Tutorial: Build APIs with Python in Minutes

10 Python Libraries Every Data Scientist Should Know

Don’t Manage Your Python Environments, Just Use Docker Containers

11 Python Magic Methods Every Programmer Should Know

10 Built-In Python Modules Every Data Engineer Should Know

Mastering Python for Data Science: Beyond the Basics

Challenges You Will Face When Parsing PDFs With Python – How To Parse PDFs With Python

Is Python OOP the Devil? Or Savior?

3 Ways of Building Python Projects using GPT-4o

Build An AI Application with Python in 10 Easy Steps

10 Python Statistical Functions

Tips for Handling Large Datasets in Python

Free Harvard Course: Introduction to AI with Python

5 Free University Courses to Learn Python

A New Python Package Manager

5 Python Best Practices for Data Science

10 Python Libraries Every Developer Should Know

Mastering f-strings in Python

Mastering Python: 7 Strategies for Writing Clear, Organized, and Efficient Code

Partial Functions in Python: A Guide for Developers

Python in Finance: Real Time Data Streaming within Jupyter Notebook

How to Learn Python the Lazy Way

5 Free Courses to Master Python for Data Science

Beginner’s Guide to Building LLM Apps with Python

5 Free Advanced Python Programming Courses

10 Python Libraries Every Data Analyst Should Know

Convert Bytes to String in Python: A Tutorial for Beginners

Introduction to Multithreading and Multiprocessing in Python

10 Useful Python One-Liners for Data Cleaning

Replacing Pandas with Polars. A Practical Guide.

Pip Install YOU: A Beginner’s Guide to Creating Your Python Library

Beginner’s Guide to Machine Learning with Python

Building Interactive Data Science Applications with Python

Stay Connected