Coding and Python - Data Engineering Digest

How to Write Efficient Python Code Even If You’re a Beginner

KDnuggets

MAY 27, 2025

You dont need to be a Python pro to write fast, clean code. Just a few smart coding habits can go a long way.

Coding

Coding Python

Managing Your Reusable Python Code as a Data Scientist

KDnuggets

FEBRUARY 11, 2022

Here are a few approaches that I have settled on for managing my own reusable Python code as a data scientist, presented from most to least general code use, and aimed at beginners.

Python

Python Coding Management Data

How to test PySpark code with pytest

Start Data Engineering

APRIL 22, 2024

Ensure the code’s logic is working as expected with tests 2.1. pytest: A powerful Python library for testing 2.2.1. Set context, run code, check results & clean up 2.2.2. Introduction 2. Test types for data pipelines 2.2. Tests are identified by their name 2.2.3. Use fixture to create fake data for testing 2.2.4.

Coding

Coding Data Pipeline Python Data

Webinars

Mastering Apache Airflow® 3.0: What’s New (and What’s Next) for Data Orchestration

MORE WEBINARS

Announcing FawltyDeps - a dependency checker for your Python code

Tweag

MARCH 13, 2023

It is a truth universally acknowledged that the Python packaging ecosystem is in need of a good dependency checker. If you work with Python, and care about keeping your projects lean and repeatable, then this is for you. The dependency is now installed in your Python virtual environment or on your system. 3rd-party imports).

Python

Python Coding Project Systems

Apache Airflow® Crash Course: From 0 to Running your Pipeline in the Cloud

Airflow enables you to define workflows as Python code, allowing for dynamic and scalable pipelines suitable to any use case from ETL/ELT to running ML/AI operations in production. With over 30 million monthly downloads, Apache Airflow is the tool of choice for programmatically authoring, scheduling, and monitoring data pipelines.

Cloud

Data Pipeline Design Patterns - #2. Coding patterns in Python

Start Data Engineering

JANUARY 12, 2023

Introduction Sample project Code design patterns 1. Singleton, & Object pool patterns Python helpers 1. Functional design 2. Factory pattern 3. Strategy pattern 4. Dataclass 3. Context Managers 4. Testing with pytest 5.

Designing

Designing Coding Python Data Pipeline

15 Python Coding Interview Questions You Must Know For Data Science

KDnuggets

APRIL 27, 2022

Solving the Python coding interview questions is the best way to get ready for an interview. That’s why we’ll lead you through 15 examples and five concepts these questions cover.

Python

Python Coding Data Science Data

What are Data Access Object and Data Transfer Object in Python?

Analytics Vidhya

FEBRUARY 6, 2023

The pattern is not an actual code but a template that can be used to solve problems in different situations. This ensures easy […] The post What are Data Access Object and Data Transfer Object in Python? Introduction A design pattern is simply a repeatable solution for problems that keep on reoccurring.

Accessible

Accessible Accessibility Python Database

Enhancing the Python ecosystem with type checking and free threading

Engineering at Meta

MAY 5, 2025

Meta and Quantsight have improved key libraries in the Python Ecosystem. Enhancing typed Python in the Python scientific stack Type hints, introduced in Python 3.5 with PEP-484 , allow developers to specify variable types, enhancing code understanding without affecting runtime behavior.

Python

Python Coding Programming Project

How To Write Efficient Python Code: A Tutorial for Beginners

KDnuggets

NOVEMBER 21, 2023

Are you a programmer looking to get better at Python? Learn some of Python’s features that’ll help you write more elegant and Pythonic code.

Python

Python Coding

Speeding Up Your Python Code with NumPy

KDnuggets

AUGUST 12, 2024

Why NumPy is significantly faster than standard Python code execution.

Python

Python Coding

How to Develop Serverless Code Using Azure Functions?

Analytics Vidhya

JANUARY 30, 2023

Introduction Azure Functions is a serverless computing service provided by Azure that provides users a platform to write code without having to provision or manage infrastructure in response to a variety of events. Azure functions allow developers […] The post How to Develop Serverless Code Using Azure Functions?

Coding

Coding Database Management Process

Mastering Python: 7 Strategies for Writing Clear, Organized, and Efficient Code

KDnuggets

MAY 17, 2024

Optimize Your Python Workflow: Proven Techniques for Crafting Production-Ready Code

Python

Python Coding

Enhance Your Python Coding Style with Ruff

KDnuggets

NOVEMBER 20, 2023

Ruff's 700+ built-in lint rules, reimplemented in Rust for speed, provide comprehensive linting and formatting to enforce clean and consistent Python code.

Coding

Coding Python Programming

10 Python One-Liners That Will Change Your Coding Game

KDnuggets

JANUARY 17, 2025

A not-to-be-missed list of elegant Python solutions to perform common programming and processing tasks in a single line of code.

Coding

Coding Python Programming Process

How to Write Clean Python Code as a Beginner

KDnuggets

DECEMBER 18, 2024

Writing Python code thats clean and easy to understand isnt just for experts — learn how to avoid common pitfalls and write like a pro from the start!

Coding

Coding Python

How To Speed Up Python Code with Caching

KDnuggets

JUNE 27, 2024

This tutorial will teach you how to make Python function calls faster using cache decorators: functools.cache and functools.lru_cache.

Python

Python Coding

Beginner’s Guide to Unit Testing Python Code with PyTest

KDnuggets

DECEMBER 13, 2024

Learn how to write and run effective unit tests in Python using PyTest, ensuring your code is reliable and bug-free.

Coding

Coding Python

Stop Creating Bad DAGs — Optimize Your Airflow Environment By Improving Your Python Code

Towards Data Science

JANUARY 30, 2025

That said, this tutorial aims to introduce airflow-parse-bench , an open-source tool I developed to help data engineers monitor and optimize their Airflow environments, providing insights to reduce code complexity and parsetime. Parsing occurs every time Airflow processes your Python files to build the DAGs dynamically.

Python

Python Coding Google Cloud Database

7 Tools To Help Write Better Python Code

KDnuggets

JANUARY 31, 2025

Want to focus on writing useful Python applications without worrying about code quality? Let these tools do the heavy lifting for you!

Coding

Coding Python

8 Built-in Python Decorators to Write Elegant Code

KDnuggets

FEBRUARY 26, 2024

Developers can modify a function's behavior using decorators, without changing its source code. This provides a concise and flexible way to enhance and extend the functionality of functions.

Coding

Coding Python IT

7 Free Online Python REPLs

KDnuggets

SEPTEMBER 24, 2024

Running Python code directly in your browser is incredibly convenient, eliminating the need for Python environment setup and allowing instant code execution without dependency or hardware concerns. I am a strong advocate of using a cloud-based IDE for working with data, machine learning, and learning Python as a beginner.

Python

Python Machine Learning Coding Programming

3 Simple Ways to Speed Up Your Python Code

KDnuggets

OCTOBER 11, 2022

The post explains three popular frameworks, PySpark, Dask, and Ray, and discusses various factors to select the most appropriate one for your project.

Coding

Coding Python Project

Data News — Week 25.02

Christophe Blefari

JANUARY 11, 2025

I have a 15% discount code if you're interested BLEF_AIProductDay25. Actually a modern Kaggle for Agentic AI, in the end it's a mechanism to lower human labor cost, because spoiler human will code to create these agents. Agents write python code to call tools and orchestrate other agents.

Data

Data Data Warehouse Coding Programming Language

5 Different Ways to Load Data in Python

KDnuggets

APRIL 15, 2022

Here, five Python techniques to bring in your data are reviewed with code examples for you to follow. Data is the bread and butter of a Data Scientist, so knowing many approaches to loading data for analysis is crucial.

Python

Python Data Coding

Python: The programming language of machine learning

KDnuggets

JUNE 6, 2022

You can't avoid learning Python if you work on machine learning problems. You need to know what other people's code means and you need to convey your ideas to them too.

Programming Language

Programming Language Machine Learning Python Programming

Python in Excel: This Will Change Data Science Forever

KDnuggets

SEPTEMBER 18, 2023

You can now run Python code in Excel to analyze data, build machine learning models, and create visualizations.

Python

Python Data Science Machine Learning Data

7 Powerful Python Decorators to Level Up Your Coding Game

KDnuggets

MARCH 3, 2025

Want to write cleaner, efficient, and more Pythonic code? Learn to use these built-in Python decorators today.

Python

Python Coding

Visual Studio Code (VSCode) extensions for data engineers

Start Data Engineering

FEBRUARY 16, 2025

Python environment setup 3. Python test & debug 3. Introduction Whether you are setting up visual studio code for your colleagues or want to improve your workflow, tons of extensions are available. Introduction 2. VSCode Primer 4. Extensions overview 1. SQL Tools 5. Data Wrangler 7. autoDocstring 8. Rainbow csv 9.

Coding

Coding Data Engineering Data Engineer Engineering

Interesting startup idea: benchmarking cloud platform pricing

The Pragmatic Engineer

OCTOBER 17, 2024

Python, Angular, SSR, SQLite, DuckDB, Cockroach DB, and many others. Other infrastructure: Primarily AWS ( S3 for cloud object storage, Parameter Store for hierarchical storage, Elastic Container Service ( ECS ) for container deployment and orchestration) The team manages AWS via infrastructure-as-a-code Pulumi. Tech stack.

Cloud

Cloud AWS Metadata Cloud Computing

Duck, Duck, Code: An Introduction to Python’s Duck Typing

KDnuggets

JULY 3, 2024

Explore the simplicity and flexibility of duck typing in Python — where code adapts based on behavior, not rigid types!

Coding

Coding Python

Mastering f-strings in Python

KDnuggets

NOVEMBER 7, 2024

Discover how to leverage Python's f-strings (formatted string literals) to write cleaner, more efficient, and more readable code.

Python

Python Coding

No Python, No SQL Templates, No YAML: Why Your Open Source Data Quality Tool Should Generate 80% Of Your Data Quality Tests Automatically

DataKitchen

FEBRUARY 17, 2025

No Python, No SQL Templates, No YAML: Why Your Open Source Data Quality Tool Should Generate 80% Of Your Data Quality Tests Automatically As a data engineer, ensuring data quality is both essential and overwhelming. The reality is that 80% of data quality tests can be generated automatically , eliminating the need for tedious manual coding.

SQL

SQL Python Government Data Engineering

How to Stand Out in a Python Coding Interview - Functions, Data Structures & Libraries

Knowledge Hut

MAY 3, 2024

Any coding interview is a test that primarily focuses on your technical skills and algorithm knowledge. However, if you want to stand out among the hundreds of interviewees, you should know how to use the common functionalities of Python in a convenient manner. Before moving ahead, read about Self in Python and what is markdown !

Python

Python Coding Data Programming

7 Best Platforms to Practice Python

KDnuggets

APRIL 23, 2024

Looking to level up your Python skills and ace coding interviews? Start practicing today on these platforms.

Python

Python Coding

Python Files within Snowflake Python Procedures

Cloudyard

SEPTEMBER 2, 2024

Read Time: 1 Minute, 36 Second Snowflake’s support for Python stored procedures allows data engineers and scientists to leverage Python’s vast ecosystem directly within Snowflake. This capability enables advanced analytics, custom data processing, and seamless integration of Python libraries.

Python

Python Utilities Coding Data Engineering

5 Python Tips for Data Efficiency and Speed

KDnuggets

AUGUST 7, 2024

Want to write better Python code? Get one step closer with this tutorial on writing maintainable, faster, and memory-efficient Python code.

Python

Python Coding Data

Snowflake To Acquire Ponder, Boosting Python Capabilities In the Data Cloud

Snowflake

OCTOBER 23, 2023

One of our goals at Snowflake is to ensure we continue to deliver a best-in-class platform for Python developers. Snowflake customers are already harnessing the power of Python through Snowpark , a set of runtimes and libraries that securely deploy and process non-SQL code directly in Snowflake.

Python

Python Cloud Data Science Machine Learning

Streamlit in Snowflake: Build Python data apps on the Data Cloud

Snowflake

SEPTEMBER 18, 2023

Enter Streamlit Streamlit is an open-source library that turns Python scripts into shareable web apps in minutes. No front-end experience is needed and apps are written in pure Python. The left hand side shows the app code and the right hand side shows the app’s output. and become a part of the Streamlit Open source community.

Python

Python Building Cloud Amazon Web Services

Flask Python: A Comprehensive Guide to Building Web Applications

Edureka

JANUARY 21, 2025

One of the most popular choices among developers is Flask, a Python framework that is both lightweight and flexible. This blog will explain a core web framework, go over the basics of Python and Flask, discuss its uses, show how popular it is, compare it to Django, and give you a general idea of the pros and cons of using Flask.

Python

Python Building Certification Database

How Meta discovers data flows via lineage at scale

Engineering at Meta

JANUARY 22, 2025

In order to build high-quality data lineage, we developed different techniques to collect data flow signals across different technology stacks: static code analysis for different languages, runtime instrumentation, and input and output data matching, etc. Hack, C++, Python, etc.) web endpoints, data tables, AI models) used across Meta.

Data Warehouse

Data Warehouse SQL Programming Language Data

How to Speed Up Python Pandas by Over 300x

KDnuggets

JULY 5, 2024

In this blog, we will define Pandas and provide an example of how you can vectorize your Python code to optimize dataset analysis using Pandas to speed up your code over 300x times faster.

Python

Python Datasets Coding

Introduction to Memory Profiling in Python

KDnuggets

FEBRUARY 19, 2024

To figure out, learn how to profile your Python code for memory usage using the memory-profiler package. So where did all the memory go?

Python

Python Coding

Why You Should Not Overuse List Comprehensions in Python

KDnuggets

NOVEMBER 20, 2023

List comprehensions in Python are super helpful one-liners. But if overused, they can make your code a pain to maintain. Here’s why.

Python

Python Coding

How to Write Efficient Python Code Even If You’re a Beginner

Managing Your Reusable Python Code as a Data Scientist

Webinars

Trending Sources

How to test PySpark code with pytest

Webinars

Announcing FawltyDeps - a dependency checker for your Python code

Apache Airflow® Crash Course: From 0 to Running your Pipeline in the Cloud

Data Pipeline Design Patterns - #2. Coding patterns in Python

15 Python Coding Interview Questions You Must Know For Data Science

What are Data Access Object and Data Transfer Object in Python?

Enhancing the Python ecosystem with type checking and free threading

How To Write Efficient Python Code: A Tutorial for Beginners

Speeding Up Your Python Code with NumPy

How to Develop Serverless Code Using Azure Functions?

Mastering Python: 7 Strategies for Writing Clear, Organized, and Efficient Code

Enhance Your Python Coding Style with Ruff

10 Python One-Liners That Will Change Your Coding Game

How to Write Clean Python Code as a Beginner

How To Speed Up Python Code with Caching

Beginner’s Guide to Unit Testing Python Code with PyTest

Stop Creating Bad DAGs — Optimize Your Airflow Environment By Improving Your Python Code

7 Tools To Help Write Better Python Code

8 Built-in Python Decorators to Write Elegant Code

7 Free Online Python REPLs

3 Simple Ways to Speed Up Your Python Code

Data News — Week 25.02

5 Different Ways to Load Data in Python

Python: The programming language of machine learning

Python in Excel: This Will Change Data Science Forever

7 Powerful Python Decorators to Level Up Your Coding Game

Visual Studio Code (VSCode) extensions for data engineers

Interesting startup idea: benchmarking cloud platform pricing

Duck, Duck, Code: An Introduction to Python’s Duck Typing

Mastering f-strings in Python

No Python, No SQL Templates, No YAML: Why Your Open Source Data Quality Tool Should Generate 80% Of Your Data Quality Tests Automatically

How to Stand Out in a Python Coding Interview - Functions, Data Structures & Libraries

7 Best Platforms to Practice Python

Python Files within Snowflake Python Procedures

5 Python Tips for Data Efficiency and Speed

Snowflake To Acquire Ponder, Boosting Python Capabilities In the Data Cloud

Streamlit in Snowflake: Build Python data apps on the Data Cloud

Flask Python: A Comprehensive Guide to Building Web Applications

How Meta discovers data flows via lineage at scale

How to Speed Up Python Pandas by Over 300x

Introduction to Memory Profiling in Python

Why You Should Not Overuse List Comprehensions in Python

Stay Connected