Coding and Python - Data Engineering Digest

How to test PySpark code with pytest

Start Data Engineering

APRIL 22, 2024

Ensure the code’s logic is working as expected with tests 2.1. pytest: A powerful Python library for testing 2.2.1. Set context, run code, check results & clean up 2.2.2. Introduction 2. Test types for data pipelines 2.2. Tests are identified by their name 2.2.3. Use fixture to create fake data for testing 2.2.4.

Coding

Coding Data Pipeline Python Data

Managing Your Reusable Python Code as a Data Scientist

KDnuggets

FEBRUARY 11, 2022

Here are a few approaches that I have settled on for managing my own reusable Python code as a data scientist, presented from most to least general code use, and aimed at beginners.

Python

Python Coding Management Data

Mastering Python: 7 Strategies for Writing Clear, Organized, and Efficient Code

KDnuggets

MAY 17, 2024

Optimize Your Python Workflow: Proven Techniques for Crafting Production-Ready Code

Python

Python Coding

Announcing FawltyDeps - a dependency checker for your Python code

Tweag

MARCH 13, 2023

It is a truth universally acknowledged that the Python packaging ecosystem is in need of a good dependency checker. If you work with Python, and care about keeping your projects lean and repeatable, then this is for you. The dependency is now installed in your Python virtual environment or on your system. 3rd-party imports).

Python

Python Coding Project Systems

Apache Airflow® Crash Course: From 0 to Running your Pipeline in the Cloud

Airflow enables you to define workflows as Python code, allowing for dynamic and scalable pipelines suitable to any use case from ETL/ELT to running ML/AI operations in production. With over 30 million monthly downloads, Apache Airflow is the tool of choice for programmatically authoring, scheduling, and monitoring data pipelines.

Cloud

Data Pipeline Design Patterns - #2. Coding patterns in Python

Start Data Engineering

JANUARY 12, 2023

Introduction Sample project Code design patterns 1. Singleton, & Object pool patterns Python helpers 1. Functional design 2. Factory pattern 3. Strategy pattern 4. Dataclass 3. Context Managers 4. Testing with pytest 5.

Designing

Designing Coding Python Data Pipeline

15 Python Coding Interview Questions You Must Know For Data Science

KDnuggets

APRIL 27, 2022

Solving the Python coding interview questions is the best way to get ready for an interview. That’s why we’ll lead you through 15 examples and five concepts these questions cover.

Coding

Coding Python Data Science Data

What are Data Access Object and Data Transfer Object in Python?

Analytics Vidhya

FEBRUARY 6, 2023

The pattern is not an actual code but a template that can be used to solve problems in different situations. This ensures easy […] The post What are Data Access Object and Data Transfer Object in Python? Introduction A design pattern is simply a repeatable solution for problems that keep on reoccurring.

Accessible

Accessible Accessibility Python Database

How To Write Efficient Python Code: A Tutorial for Beginners

KDnuggets

NOVEMBER 21, 2023

Are you a programmer looking to get better at Python? Learn some of Python’s features that’ll help you write more elegant and Pythonic code.

Python

Python Coding

Speeding Up Your Python Code with NumPy

KDnuggets

AUGUST 12, 2024

Why NumPy is significantly faster than standard Python code execution.

Python

Python Coding

Enhance Your Python Coding Style with Ruff

KDnuggets

NOVEMBER 20, 2023

Ruff's 700+ built-in lint rules, reimplemented in Rust for speed, provide comprehensive linting and formatting to enforce clean and consistent Python code.

Coding

Coding Python Programming

10 Python One-Liners That Will Change Your Coding Game

KDnuggets

JANUARY 17, 2025

A not-to-be-missed list of elegant Python solutions to perform common programming and processing tasks in a single line of code.

Coding

Coding Python Programming Process

How to Develop Serverless Code Using Azure Functions?

Analytics Vidhya

JANUARY 30, 2023

Introduction Azure Functions is a serverless computing service provided by Azure that provides users a platform to write code without having to provision or manage infrastructure in response to a variety of events. Azure functions allow developers […] The post How to Develop Serverless Code Using Azure Functions?

Coding

Coding Database Management Process

7 Tools To Help Write Better Python Code

KDnuggets

JANUARY 31, 2025

Want to focus on writing useful Python applications without worrying about code quality? Let these tools do the heavy lifting for you!

Coding

Coding Python

8 Built-in Python Decorators to Write Elegant Code

KDnuggets

FEBRUARY 26, 2024

Developers can modify a function's behavior using decorators, without changing its source code. This provides a concise and flexible way to enhance and extend the functionality of functions.

Coding

Coding Python IT

How to Write Clean Python Code as a Beginner

KDnuggets

DECEMBER 18, 2024

Writing Python code thats clean and easy to understand isnt just for experts — learn how to avoid common pitfalls and write like a pro from the start!

Coding

Coding Python

How To Speed Up Python Code with Caching

KDnuggets

JUNE 27, 2024

This tutorial will teach you how to make Python function calls faster using cache decorators: functools.cache and functools.lru_cache.

Python

Python Coding

Beginner’s Guide to Unit Testing Python Code with PyTest

KDnuggets

DECEMBER 13, 2024

Learn how to write and run effective unit tests in Python using PyTest, ensuring your code is reliable and bug-free.

Coding

Coding Python

Automating dead code cleanup

Engineering at Meta

OCTOBER 24, 2023

Meta’s Systematic Code and Asset Removal Framework (SCARF) has a subsystem for identifying and removing dead code. SCARF combines static and dynamic analysis of programs to detect dead code from both a business and programming language perspective. These are combined and form an augmented dependency graph.

Coding

Coding Programming Language Python MySQL

7 Free Online Python REPLs

KDnuggets

SEPTEMBER 24, 2024

Running Python code directly in your browser is incredibly convenient, eliminating the need for Python environment setup and allowing instant code execution without dependency or hardware concerns. I am a strong advocate of using a cloud-based IDE for working with data, machine learning, and learning Python as a beginner.

Python

Python Machine Learning Coding Programming

3 Simple Ways to Speed Up Your Python Code

KDnuggets

OCTOBER 11, 2022

The post explains three popular frameworks, PySpark, Dask, and Ray, and discusses various factors to select the most appropriate one for your project.

Coding

Coding Python Project

5 Different Ways to Load Data in Python

KDnuggets

APRIL 15, 2022

Here, five Python techniques to bring in your data are reviewed with code examples for you to follow. Data is the bread and butter of a Data Scientist, so knowing many approaches to loading data for analysis is crucial.

Python

Python Data Coding

7 Best Platforms to Practice Python

KDnuggets

APRIL 23, 2024

Looking to level up your Python skills and ace coding interviews? Start practicing today on these platforms.

Python

Python Coding

Python in Excel: This Will Change Data Science Forever

KDnuggets

SEPTEMBER 18, 2023

You can now run Python code in Excel to analyze data, build machine learning models, and create visualizations.

Python

Python Data Science Machine Learning Data

Python: The programming language of machine learning

KDnuggets

JUNE 6, 2022

You can't avoid learning Python if you work on machine learning problems. You need to know what other people's code means and you need to convey your ideas to them too.

Programming Language

Programming Language Machine Learning Python Programming

How to Stand Out in a Python Coding Interview - Functions, Data Structures & Libraries

Knowledge Hut

MAY 3, 2024

Any coding interview is a test that primarily focuses on your technical skills and algorithm knowledge. However, if you want to stand out among the hundreds of interviewees, you should know how to use the common functionalities of Python in a convenient manner. Before moving ahead, read about Self in Python and what is markdown !

Python

Python Coding Data Programming

Duck, Duck, Code: An Introduction to Python’s Duck Typing

KDnuggets

JULY 3, 2024

Explore the simplicity and flexibility of duck typing in Python — where code adapts based on behavior, not rigid types!

Coding

Coding Python

Mastering f-strings in Python

KDnuggets

NOVEMBER 7, 2024

Discover how to leverage Python's f-strings (formatted string literals) to write cleaner, more efficient, and more readable code.

Python

Python Coding

Introduction to Memory Profiling in Python

KDnuggets

FEBRUARY 19, 2024

To figure out, learn how to profile your Python code for memory usage using the memory-profiler package. So where did all the memory go?

Python

Python Coding

Python Files within Snowflake Python Procedures

Cloudyard

SEPTEMBER 2, 2024

Read Time: 1 Minute, 36 Second Snowflake’s support for Python stored procedures allows data engineers and scientists to leverage Python’s vast ecosystem directly within Snowflake. This capability enables advanced analytics, custom data processing, and seamless integration of Python libraries.

Python

Python Utilities Coding Data Engineering

5 Python Tips for Data Efficiency and Speed

KDnuggets

AUGUST 7, 2024

Want to write better Python code? Get one step closer with this tutorial on writing maintainable, faster, and memory-efficient Python code.

Python

Python Coding Data

Meta contributes new features to Python 3.12

Engineering at Meta

OCTOBER 5, 2023

Python 3.12 It includes new features and performance improvements – some contributed by Meta – that we believe will benefit all Python users. We’re sharing details about these new features that we worked closely with the Python community to develop. This week’s release of Python 3.12 This week’s release of Python 3.12

Python

Python Coding Programming Language Programming

Snowflake To Acquire Ponder, Boosting Python Capabilities In the Data Cloud

Snowflake

OCTOBER 23, 2023

One of our goals at Snowflake is to ensure we continue to deliver a best-in-class platform for Python developers. Snowflake customers are already harnessing the power of Python through Snowpark , a set of runtimes and libraries that securely deploy and process non-SQL code directly in Snowflake.

Python

Python Cloud Data Science Machine Learning

ChatGPT for Coding: Unleash the Power of ChatGPT

Edureka

FEBRUARY 8, 2023

One of them is Chat GPT, a conversational model of AI that is a powerful chatbot that answers follow-up questions and writes code for the users. In this blog we will get to know about the perks of ChatGPT for coding. 4 Step 6: Receive a code through SMS or WhatsApp. 5 Step 7: After entering the code, select “New Chat.”

Coding

Coding Deep Learning Programming Java

Zenlytic Is Building You A Better Coworker With AI Agents

Data Engineering Podcast

MAY 18, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management This episode is supported by Code Comments, an original podcast from Red Hat. Search for "Code Commentst" in your podcast player or go to dataengineeringpodcast.com/codecomments today to subscribe.

Building

Building Data Lake High Quality Data Business Intelligence

How To Comment Your Python Code as a Data Scientist

KDnuggets

FEBRUARY 13, 2024

Don’t overlook these essential aspects of programming activity.

Coding

Coding Python Programming Data

How To Run Your Python Scripts

Knowledge Hut

APRIL 19, 2024

If you are planning to enter the world of Python programming, the first and the most essential skill you should learn is knowing how to run Python script and code. Once you grab a seat in the show, it will be easier for you to understand whether the code will actually work or not. The user can directly execute the code.

Python

Python Programming Language Programming Coding

Why You Should Not Overuse List Comprehensions in Python

KDnuggets

NOVEMBER 20, 2023

List comprehensions in Python are super helpful one-liners. But if overused, they can make your code a pain to maintain. Here’s why.

Python

Python Coding

How to Speed Up Python Pandas by Over 300x

KDnuggets

JULY 5, 2024

In this blog, we will define Pandas and provide an example of how you can vectorize your Python code to optimize dataset analysis using Pandas to speed up your code over 300x times faster.

Python

Python Datasets Coding

Novice to Ninja: Why Your Python Skills Matter in Data Science

KDnuggets

OCTOBER 31, 2023

As a data scientist, is it worthwhile leveling up your Python skills? Dive into code comparisons across expertise levels & discover if "good enough" is really enough.

Python

Python Data Science Coding Data

How (and Why) To Create Custom Exceptions in Python

KDnuggets

SEPTEMBER 24, 2024

In Python, you can use exceptions to anticipate and handle errors that disrupt the normal flow of a program. While Python offers many built-in exceptions—common exceptions like ValueError, TypeError, and KeyError—there are cases where custom exceptions are necessary for handling unique error conditions specific to your application.

Python

Python Programming Coding

Building Applications With Data As Code On The DataOS

Data Engineering Podcast

JANUARY 15, 2023

At the Modern Data Company they created the DataOS platform as a means of driving your full analytics lifecycle through code, while providing automatic knowledge graphs and data discovery. Listeners can save $375 off standard rates with code GARTNERDA. Listeners can save $375 off standard rates with code GARTNERDA. Missing data?

Coding

Coding Building PostgreSQL Data Lake

What are the Basics of Python 3

Knowledge Hut

APRIL 26, 2024

What is Python 3? Python 3 is an interpreted language, which means that anyone can read and execute the code. Python is used to create websites, perform scientific research, data analysis etc. Python 3.9 is the latest version of Python. Why Learn Python 3? So, variable can be declared as below.

Python

Python Programming Language Programming Certification

KDnuggets News, August 30: 7 Projects Built with Generative AI • Beyond Numpy and Pandas: Lesser-Known Python Libraries

KDnuggets

AUGUST 30, 2023

7 Projects Built with Generative AI • Beyond Numpy and Pandas: Unlocking the Potential of Lesser-Known Python Libraries • 5 Ways You Can Use ChatGPT’s Code Interpreter For Data Science • GPT-4: 8 Models in One; The Secret is Out

Python

Python Project Data Science Coding

How to test PySpark code with pytest

Managing Your Reusable Python Code as a Data Scientist

Trending Sources

Mastering Python: 7 Strategies for Writing Clear, Organized, and Efficient Code

Announcing FawltyDeps - a dependency checker for your Python code

Apache Airflow® Crash Course: From 0 to Running your Pipeline in the Cloud

Data Pipeline Design Patterns - #2. Coding patterns in Python

15 Python Coding Interview Questions You Must Know For Data Science

What are Data Access Object and Data Transfer Object in Python?

How To Write Efficient Python Code: A Tutorial for Beginners

Speeding Up Your Python Code with NumPy

Enhance Your Python Coding Style with Ruff

10 Python One-Liners That Will Change Your Coding Game

How to Develop Serverless Code Using Azure Functions?

7 Tools To Help Write Better Python Code

8 Built-in Python Decorators to Write Elegant Code

How to Write Clean Python Code as a Beginner

How To Speed Up Python Code with Caching

Beginner’s Guide to Unit Testing Python Code with PyTest

Automating dead code cleanup

7 Free Online Python REPLs

3 Simple Ways to Speed Up Your Python Code

5 Different Ways to Load Data in Python

7 Best Platforms to Practice Python

Python in Excel: This Will Change Data Science Forever

Python: The programming language of machine learning

How to Stand Out in a Python Coding Interview - Functions, Data Structures & Libraries

Duck, Duck, Code: An Introduction to Python’s Duck Typing

Mastering f-strings in Python

Introduction to Memory Profiling in Python

Python Files within Snowflake Python Procedures

5 Python Tips for Data Efficiency and Speed

Meta contributes new features to Python 3.12

Snowflake To Acquire Ponder, Boosting Python Capabilities In the Data Cloud

ChatGPT for Coding: Unleash the Power of ChatGPT

Zenlytic Is Building You A Better Coworker With AI Agents

How To Comment Your Python Code as a Data Scientist

How To Run Your Python Scripts

Why You Should Not Overuse List Comprehensions in Python

How to Speed Up Python Pandas by Over 300x

Novice to Ninja: Why Your Python Skills Matter in Data Science

How (and Why) To Create Custom Exceptions in Python

Building Applications With Data As Code On The DataOS

What are the Basics of Python 3

KDnuggets News, August 30: 7 Projects Built with Generative AI • Beyond Numpy and Pandas: Lesser-Known Python Libraries

Stay Connected