Python - Data Engineering Digest

AWS Lambdas – Python vs Rust. Performance and Cost Savings.

Confessions of a Data Guy

FEBRUARY 26, 2023

Someone on Linkedin recently brought up the point that companies could save gobs of money by swapping out AWS Python lambdas for Rust ones. While it raised the ire of many a Python Data Engineer, I thought it sounded like a great idea. At least it’s an excuse to […] The post AWS Lambdas – Python vs Rust.

AWS

AWS Python Data Engineer Data Engineering

Isolated Python Environments using Docker

Analytics Vidhya

FEBRUARY 6, 2023

Introduction While working with multiple projects, there are chances of issues with versions of packages in python; for example, a project needs a new version of a package, and another requires a different version. Sometimes the python version itself changes from project to project.

Python

Python Project Management Data Engineer

Building cost effective data pipelines with Python & DuckDB

Start Data Engineering

MAY 28, 2024

KISS: DuckDB + Python = easy to debug and quick to develop 4. Cost calculation: DuckDB + Ephemeral VMs = dirt cheap data processing 4.3. Processing data less than 100GB? Use DuckDB 4.4. Distributed systems are scalable, resilient to failures, & designed for high availability 4.5.

Data Pipeline

Data Pipeline Python Building Data

Webinars

Apache Airflow®: The Ultimate Guide to DAG Writing

MORE WEBINARS

What are Data Access Object and Data Transfer Object in Python?

Analytics Vidhya

FEBRUARY 6, 2023

This ensures easy […] The post What are Data Access Object and Data Transfer Object in Python? Especially while working with databases, it is often considered a good practice to follow a design pattern. appeared first on Analytics Vidhya.

Accessibility

Accessibility Accessible Python Database

Python Essentials for Data Engineers

Start Data Engineering

MAY 30, 2024

Introduction Data is stored on disk and processed in memory Running the code Run on Codespaces Run on your laptop Using python REPL Python basics Python is used for extracting data from sources, transforming it, & loading it into a destination [Extract & Load] Read and write data to any system [Transform] Process data in Python or instruct (..)

Python

Python Data Engineer Data Engineering Engineering

10 GitHub Repositories to Master Python

KDnuggets

APRIL 10, 2024

Learn Python through tutorials, blogs, books, project work, and exercises. Access all of it on GitHub for free and join a supportive open-source community.

Python

Python Accessibility Accessible Project

Tips for Handling Large Datasets in Python

KDnuggets

NOVEMBER 29, 2024

Here are some tips to make working with such large datasets in Python simpler. Working with large datasets is common but challenging.

Datasets

Datasets Python

7 Free Online Python REPLs

KDnuggets

SEPTEMBER 24, 2024

Running Python code directly in your browser is incredibly convenient, eliminating the need for Python environment setup and allowing instant code execution without dependency or hardware concerns. I am a strong advocate of using a cloud-based IDE for working with data, machine learning, and learning Python as a beginner.

Python

Python Machine Learning Coding Programming

10 Python Statistical Functions

KDnuggets

AUGUST 14, 2024

This guide will go over 10 essential statistical functions in Python using commonly-used libraries.

Python

10 Python Libraries Every Developer Should Know

KDnuggets

DECEMBER 2, 2024

In this article, we’ll go over Python libraries for tasks like logging, unit testing, data handling, and more — each with features that can simplify your application development.

Python

Python Data

7 Python Libraries Every Data Engineer Should Know

KDnuggets

APRIL 25, 2024

Here’s a list of Python libraries you’ll find super helpful. Interested in switching to data engineering?

Python

Python Data Engineer Data Engineering Engineering

5 Free Advanced Python Programming Courses

KDnuggets

APRIL 24, 2024

Looking to level up your Python skills without spending a dime? Check out this article featuring 5 advanced Python courses that you can take for free!

Python

Python Programming

7 Best Platforms to Practice Python

KDnuggets

APRIL 23, 2024

Looking to level up your Python skills and ace coding interviews? Start practicing today on these platforms.

Python

Python Coding

Mastering Python for Data Science: Beyond the Basics

KDnuggets

MARCH 28, 2024

This article serves as a detailed guide on how to master advanced Python techniques for data science. It covers topics such as efficient data manipulation with Pandas, parallel processing with Python, and how to turn models into web services.

Data Science

Data Science Python Data Process

5 Free Python Courses for Data Science Beginners

KDnuggets

MAY 31, 2024

Are you a data science beginner looking to learn Python? Start learning today with these 5 free courses.

Data Science

Data Science Python Data

FastAPI Tutorial: Build APIs with Python in Minutes

KDnuggets

JUNE 13, 2024

Want to build APIs with Python? Learn how to do so using FastAPI with this step-by-step tutorial.

Python

Python Building

Seamless SQL And Python Transformations For Data Engineers And Analysts With SQLMesh

Data Engineering Podcast

JUNE 25, 2023

__init__ covers the Python language, its community, and the innovative ways it is being used. __init__ covers the Python language, its community, and the innovative ways it is being used. Closing Announcements Thank you for listening! Don't forget to check out our other shows. Closing Announcements Thank you for listening!

Data Engineer

Data Engineer Data Engineering Python Engineering

Python Typer Tutorial: Build CLIs with Python in Minutes

KDnuggets

OCTOBER 28, 2024

Learn how to build CLIs with Python using Typer in a few simple steps.

Python

Python Building

A New Python Package Manager

KDnuggets

NOVEMBER 14, 2024

Manage Python projects, run scripts and tools, handle dependencies, and install packages—all with the uv tool.

Python

Python Management Project

5 Simple Steps to Automate Data Cleaning with Python

KDnuggets

MAY 3, 2024

Automate your data cleaning process with a practical 5-step pipeline in Python, ideal for beginners.

Python

Python Data Process

Introduction to Memory Profiling in Python

KDnuggets

FEBRUARY 19, 2024

To figure out, learn how to profile your Python code for memory usage using the memory-profiler package. So where did all the memory go?

Python

Python Coding

11 Python Magic Methods Every Programmer Should Know

KDnuggets

NOVEMBER 28, 2023

Want to support the behavior of built-in functions and method calls in your Python classes? Magic methods in Python let you do just that! So let’s uncover the method behind the magic.

Python

5 Free Courses to Master Python for Data Science

KDnuggets

FEBRUARY 7, 2024

Want to learn Python to kickstart your career in data? Here are five free courses to help you master Python for data science.

Data Science

Data Science Python Data

Mastering f-strings in Python

KDnuggets

NOVEMBER 7, 2024

Discover how to leverage Python's f-strings (formatted string literals) to write cleaner, more efficient, and more readable code.

Python

Python Coding

10 Python Libraries Every Data Scientist Should Know

KDnuggets

AUGUST 12, 2024

Check out these Python libraries for data science that you can't do without. Want to take the next step in your journey to becoming a data scientist?

Python

Python Data Science Data

3 Ways of Building Python Projects using GPT-4o

KDnuggets

AUGUST 7, 2024

Learn about essential AI tools that can help you develop Python projects faster and with fewer bugs using natural language.

Python

Python Project Building

10 Built-In Python Modules Every Data Engineer Should Know

KDnuggets

SEPTEMBER 2, 2024

Check out this round-up of built-in Python modules that'll come in handy for data engineering tasks. Interested in data engineering?

Python

Python Data Engineer Data Engineering Engineering

5 Python Tips for Data Efficiency and Speed

KDnuggets

AUGUST 7, 2024

Want to write better Python code? Get one step closer with this tutorial on writing maintainable, faster, and memory-efficient Python code.

Python

Python Coding Data

Understanding and Implementing Genetic Algorithms in Python

KDnuggets

JUNE 24, 2024

Understanding what genetic algorithms are and how they can be implemented in Python.

Algorithm

Algorithm Python

Convert Bytes to String in Python: A Tutorial for Beginners

KDnuggets

JULY 15, 2024

Strings are common built-in data types in Python. Let’s learn how to convert bytes to string in Python. But sometimes, you may need to work with bytes instead.

Bytes

Bytes Python Data

Speeding Up Your Python Code with NumPy

KDnuggets

AUGUST 12, 2024

Why NumPy is significantly faster than standard Python code execution.

Python

Python Coding

Build An AI Application with Python in 10 Easy Steps

KDnuggets

MARCH 14, 2024

Explore the fundamental steps for creating a successful AI Application with Python and other tools.

Python

Python Building

5 Tips for Writing Better Python Functions

KDnuggets

JUNE 4, 2024

This tutorial covers five simple yet effective practices for writing better and maintainable Python functions.

Python

Build a Data Science App with Python in 10 Easy Steps

KDnuggets

NOVEMBER 25, 2024

Learn how to build a data science app with Python, using Scikit-Learn and FastAPI, one step at a time.

Data Science

Data Science Python Building Data

5 Python Best Practices for Data Science

KDnuggets

MAY 29, 2024

Level up your Python skills for data science with these by following these best practices.

Data Science

Data Science Python Data

Beginner’s Guide to Machine Learning with Python

KDnuggets

JUNE 4, 2024

Master the Fundamentals of Predictive Modeling with Python: An In-Depth Guide to Machine Learning Algorithms and Sci-kit Learn Implementation.

Machine Learning

Machine Learning Python Algorithm

How To Create Custom Context Managers in Python

KDnuggets

JUNE 3, 2024

Context managers in Python help you manage resources efficiently. Learn how to write your own custom context managers.

Python

Python Management

Pydantic Tutorial: Data Validation in Python Made Simple

KDnuggets

MARCH 25, 2024

Want to write more robust Python applications? Learn how to use Pydantic, a popular data validation library, to model and validate your data.

Data Validation

Data Validation Python Data

Challenges You Will Face When Parsing PDFs With Python – How To Parse PDFs With Python

Seattle Data Guy

NOVEMBER 19, 2024

Most of us will turn to Python and our trusty list of Python libraries and start plugging away. Of course, there are many challenges… Read more The post Challenges You Will Face When Parsing PDFs With Python – How To Parse PDFs With Python appeared first on Seattle Data Guy.

Python

Python Data Data Science Data Engineer

5 Free University Courses to Learn Python

KDnuggets

DECEMBER 14, 2023

Looking for the best resources to learn Python programming? Check out these free university courses.

Python

Python Programming

Python in Finance: Real Time Data Streaming within Jupyter Notebook

KDnuggets

FEBRUARY 20, 2024

This guide covers dynamic visualizations, a Python for quant finance use case, and Bollinger Bands analysis with live data. Learn a modern approach to stream real-time data in Jupyter Notebook.

Finance

Finance Python Data

Introduction to Multithreading and Multiprocessing in Python

KDnuggets

DECEMBER 4, 2023

Learn about Multithreading and Multiprocessing environments using Python with their implementation and limitations.

Python

Mastering Python: 7 Strategies for Writing Clear, Organized, and Efficient Code

KDnuggets

MAY 17, 2024

Optimize Your Python Workflow: Proven Techniques for Crafting Production-Ready Code

Python

Python Coding

Partial Functions in Python: A Guide for Developers

KDnuggets

SEPTEMBER 19, 2024

In Python, functions often require multiple arguments, and you may find yourself repeatedly passing the same values for certain parameters. Python’s built-in functools module allows you to create partial functions. This is where partial functions can help.

Python

AWS Lambdas – Python vs Rust. Performance and Cost Savings.

Isolated Python Environments using Docker

Building cost effective data pipelines with Python & DuckDB

Webinars

What are Data Access Object and Data Transfer Object in Python?

Python Essentials for Data Engineers

10 GitHub Repositories to Master Python

Tips for Handling Large Datasets in Python

7 Free Online Python REPLs

10 Python Statistical Functions

10 Python Libraries Every Developer Should Know

7 Python Libraries Every Data Engineer Should Know

5 Free Advanced Python Programming Courses

7 Best Platforms to Practice Python

Mastering Python for Data Science: Beyond the Basics

5 Free Python Courses for Data Science Beginners

FastAPI Tutorial: Build APIs with Python in Minutes

Seamless SQL And Python Transformations For Data Engineers And Analysts With SQLMesh

Python Typer Tutorial: Build CLIs with Python in Minutes

A New Python Package Manager

5 Simple Steps to Automate Data Cleaning with Python

Introduction to Memory Profiling in Python

11 Python Magic Methods Every Programmer Should Know

5 Free Courses to Master Python for Data Science

Mastering f-strings in Python

10 Python Libraries Every Data Scientist Should Know

3 Ways of Building Python Projects using GPT-4o

10 Built-In Python Modules Every Data Engineer Should Know

5 Python Tips for Data Efficiency and Speed

Understanding and Implementing Genetic Algorithms in Python

Convert Bytes to String in Python: A Tutorial for Beginners

Speeding Up Your Python Code with NumPy

Build An AI Application with Python in 10 Easy Steps

5 Tips for Writing Better Python Functions

Build a Data Science App with Python in 10 Easy Steps

5 Python Best Practices for Data Science

Beginner’s Guide to Machine Learning with Python

How To Create Custom Context Managers in Python

Pydantic Tutorial: Data Validation in Python Made Simple

Challenges You Will Face When Parsing PDFs With Python – How To Parse PDFs With Python

5 Free University Courses to Learn Python

Python in Finance: Real Time Data Streaming within Jupyter Notebook

Introduction to Multithreading and Multiprocessing in Python

Mastering Python: 7 Strategies for Writing Clear, Organized, and Efficient Code

Partial Functions in Python: A Guide for Developers

Stay Connected