ken

Python generate QR code, Python read QR code, Photo by Lukas on Unsplash
 by ken

Read and Generate QR Code With 5 Lines of Python Code

 Introduction

QR Code is the most popular 2 dimensional barcodes that widely used for document management, track and trace in supply chain and logistics industry, mobile payment,  and even the “touchless” health declaration and contact tracing during the COVID-19 pandemic. Comparing to 1D barcode, QR code can be very small in size but hold more information, and also easier for scanning as you can scan it from any direction.

In this article, I would be sharing with you how to use some pure Python packages to generate QR code and read QR code from images.

Generate QR code with Python

To generate QR code, we will use a Python package called qrcode. Below is the pip command to install this package:

#install qrcode together with pillow
pip install qrcode[pil]

#or install qrcode if you already have pillow installed
pip install qrcode

As it has dependency to Pillow package, you will need to have this package installed as well. Once you have these packages ready, let’s import the modules at the beginning of our code:

import qrcode
from PIL import Image

Generating a QR code with this qrcode library can be easily done with 1 line of code:

img = qrcode.make('QR Code')

If you check the “img” object from Jupyter Notebook, you can see the below image:

Generate QR code Python, Read QR code Python

This make function provides a quick way to generate QR code with all the default parameters. To specify the parameters like the size, style or border of the boxes, you can use the QRCode class. For instance:

qr = qrcode.QRCode(
    version=1,
    error_correction=qrcode.constants.ERROR_CORRECT_L,
    box_size=10,
    border=4,
)

Here is the explanations for these parameters:

version – QR code has 40 different sizes which indicated as the version parameter in above, version 1 represents a 21×21 matrix.  You can use (v-1)*4 + 21 to calculate the size of the matrix for each of the version number.

error_correction – specifies error correction level which controls how many error correction code blocks to be inserted in order to achieve the error correction capability. In another words, if you want your barcode to be readable even when it’s damaged (or have a logo/image onto it) , you may increase the error correction level, but this would also make your barcode more compact.

box_size – the number of pixels of the square box

border – the thickness of the square box border

Once you have a QRCode instance, you can use the below code to specify the barcode data, color and generate a image:

#barcode content
qr.add_data('codeforests.com')

#auto adjust the size
qr.make(fit=True)

#specifying barcode color
img = qr.make_image(fill_color="#040359", back_color="#f7f7fa")

If you check the “img” object from Jupyter Notebook again, you shall see something similar to below:

 

Generate QR code Python, Read QR code Python

 

To use the same barcode style to generate new barcode, you can just clear the data and then re-generate a new image object:

 

qr.clear()
qr.add_data('Python Tutorials')

img2 = qr.make_image(fill_color="#015b82", back_color="TransParent")

When inspecting the “img2” in Jupyter Notebook, you shall see below:

Generate QR code Python, Read QR code Python, Python QR code with different color

 

You can simply use the “save” method to save it into an image file since it is a Pillow Image object:

img2.save("qr_code.png")

The qrcode package cannot directly generate multiple QR codes into one image, if you need that, you may use the Pillow package to combine the images. For instance:

#create a blank image
new_img = Image.new("RGBA", (600, 350), "#fcfcfc")

new_img.paste(img, (0, 0))

new_img.paste(img2, (300, 0))

new_img.save("multi-QR-code.png")

The above will create a new image and combine the two barcode images into one. If you check the saved image file, you shall see:

 

Generate QR code Python, Read QR code Python, generate multiple QR codes on one page

With this package, you can also generate styled QR code e.g.: rounded corners, radial gradient, embedded image or different color masks. You can take a look at the samples from it’s office site.

Read QR Code in Python

To read QR code, we will use another Python package called pyzbar. You can use below pip command to install it:

pip install pyzbar

This library is also a very easy to use, you can directly pass in a Pillow Image object, numpy.ndarray or raw bytes to the decode method to detect the barcode. For instance:

import pyzbar.pyzbar as pyzbar
from pyzbar.pyzbar import ZBarSymbol

input_image = Image.open("multi-QR-code.png")

decoded_objects = pyzbar.decode(input_image, symbols=[ZBarSymbol.QRCODE])

The decode method returns a list of barcode objects detected from the image with their position info. You can use the symbols parameter to restrict what type of barcodes you want to detect. When this parameter is not specified, all its supported barcode types will be checked.

From the above, you can further loop through the list to get the actual content data of the barcodes:

for obj in decoded_objects:
    zbarData = obj.data.decode("utf-8")
    print(zbarData)

You shall see the below result:

Generate QR code Python, Read QR code Python

In your real-world project, if you need to read one barcode among the multiple barcodes from a document, you may try to use the symbols to restrict the barcode types, or use regular expression to validate the detected barcode data in order to find the correct one you need.

If you need to do a lot of image pre-processing or even read barcode from video or webcam, you may install OpenCV and use the detectAndDecodeMulti method to read the QR code.

Conclusion

In this article, we have reviewed through two simple but useful packages – qrcode for generating QR code, and pyzbar for reading the content from a QR code. There are quite many other Python packages for generating all sorts of one or two dimensional barcodes, some are in pure Python packages and some are Python wrappers, you may take a look at the summary table from this blog if any specific barcode type you need is not supported by these two packages.

 

in Python / Resources 0 comments
20 Useful Tips for Using Python Pip
 by ken

20 Tips for Using Python Pip

Introduction

Python has become one of the most popular programming languages due to the easy to use syntax as well as the thousands of open-source libraries developed by the Python community. Almost every problem you want to solve, you can find a solution with these third-party libraries, so that you do not need to reinvent the wheels. Majority of these libraries are hosted in the repository called Pypi and you can install these libraries with the Python pip command.

Python pip module helps you to manage the downloading, installation of the packages, and solving the dependency requirements. Although you probably have used pip for some time, you may not spend much time to read through it’s user guide for some of the useful operations. In this article, we have summarize the 20 useful tips for managing Python third party packages with Python pip.

Check the current pip version

Since Python version 3.4, the pip module has been included by default within the Python binary installer, so you do not need to install it separately once you have Python program installed. To check the version of the pip package, you can use the below:

pip --version

Sample output:

Python Pip version

Install package from Pypi

Installing package is very simple with pip command, you can use “install” option followed by one or multiple package names:

pip install requests

By default, pip looks for the latest release and install the latest version for you together with the dependency packages. Sample output as per below:

Python Pip install package

You can also specify the version number of the package to be installed:

py -m pip install pip==21.1.1

Sample output:

Python Pip install package with version number

Pip also supports a list of version specifier such as >=1.2, <2.0, ~=2.0, !=2.0 or ==1.9.* for matching the correct version of the package to be installed.

When you are not in a virtual environment, the package will be installed into the global folder (system-site) by default, you can use the “–user” option to specify the installation folder in user-site in case of any permission issue. E.g.:

pip install --user requests

Output as per below:

Python Pip install package to user-site

Although you can specify your own customized installation path for your different projects, using virtual environment is still the best way to manage dependencies and conflicts.

Show package version and installation location

To check the basic information such as version number or installation location for an existing package, you can use the “show” option:

pip show colorama

You can see the below information about the package:

Python Pip show package version and installation location

And you can also use the “–verbose” mode to display the additional meta info.

List all the packages installed

To list out all the packages installed, you can use the “list” option:

py -m pip list

You shall see the output format similar to below:

Python Pip list packages

You can add a “–user” option to list all packages installed in your user-site, e.g.:

py -m pip list --user

When you are using virtual environment with “–system-site-packages” (allowing virtual environment to access system-site packages), you can use the “list –local” option to show only the packages installed in your virtual environment:

py -m pip list --local

List all the outdated packages

To check if any installed packages are outdated, you can use the “–outdated” option:

py -m pip list -o
# or
py -m pip list --outdated

Below is the sample output:

Python Pip list outdated packages

Upgrade package to the latest version

Once identified the outdated packages, you can manually use the “–upgrade” option to upgrade the package to the latest version. Multiple package name can be specified with whitespaces:

py -m pip install --upgrade pip
#or 
py -m pip install --U pip setuptools

Sample output as per below:

Python Pip upgrade package

Auto upgrade packages to the latest version

Pip does not have an option to auto upgrade the outdated packages, but you can make use of the result from “list -o” and create a simple script to achieve it, e.g.:

#in Windows command line
for /F "skip=2 delims= " %i in ('pip list --o --local') do pip install -U %i

#in linux
pip list --o --local | grep -v '^\-e' | cut -d = -f 1  | xargs -n1 pip install -U

Export installed packages

You can use “freeze” option to export all your installed package names into a text file, so that you can re-create exactly the same project environment in another PC. For instance:

py -m pip freeze -l > requirements_demo.txt

Result in the output text file:

Python Pip install requirement file

Install multiple packages from requirement file

For the packages you’ve exported with “freeze” option, you can re-install all the packages in another environment with the below “-r” option:

py -m pip install -r requirements.txt

You may see the below output when you have package name “numpy” in your requirements.txt file:

Python Pip install package with requirement file

The requirements.txt also allows to include other requirement files. This may be useful when you have a sub module requires extra packages and can be run independently as a separate application. So you may put the common packages in the requirements.txt and the additional packages in the requirements_module1.txt file,  the include the requirements.txt file in your module file.

E.g. the content in the requirements_module1.txt:

#opencv-python
#comment out some packages
python-dateutil

-r requirements.txt

When you run the “install” command:

py -m pip install -r requirements_module1.txt

You shall the sample output as per below:

Python Pip install package with multiple requirement files

Uninstall packages

Uninstalling an existing package can be done with below command:

pip uninstall numpy

Output as per below:

Python Pip uninstall package

Install package from wheel file

When you have a binary wheel file downloaded in your local folder, you can also use the “install” option to install the wheel file directly:

py -m pip install --force-reinstall C:\Users\codef\Downloads\python_dateutil-2.8.2-py2.py3-none-any.whl

Output as per below:

pip install whl

Install package from non-Pypi index

If the package is not hosted in Pypi index, you can manually specify the index url with “–index-url” or simply “-i” :

py -m pip install -i https://mirrors.huaweicloud.com/repository/pypi/simple/ colorama

Above command would download and install the package from huawei cloud repository (a PyPi mirror):

Python Pip install package from Pypi mirrors

This would be also helpful when you are not able to access the Pypi directly due to the firewall or proxy issue in your network, you can find a Pypi mirror repository and download the packages from there. Usually these mirrors synchronize with Pypi in a few minutes interval which should not cause any issue for your development work.

Configure global index url

To permanently save the index url so that you do not have to key in the url for every package installation, you can use the “config” option to set the url globally. e.g:

pip config set global.index-url https://mirrors.aliyun.com/pypi/simple

With the above setting, you can install package from the mirror repository as per normal without specifying the url option.

Check package compatibility

When you manually install the packages, sometimes you may encounter issues that some dependency packages

having incompatible version installed. To check if you have any such issue, you can use the “check” option :

python -m pip check

You may see something similar to below when there is any conflict:

Python Pip check package compatibility

Download package into local folder

You can download the package wheel files into your local folder when you need:

pip download requests -d .\requests

The “-d” option allows you specify the target folder where you want to save the wheel files. You may get multiple wheel files if the package has any dependency packages. (you can use “–no-deps” when you do not want to download the dependency files)

Below is the sample result:

 

Python Pip download wheels file

Install package into local folder

To install the package from a folder, you can use the “-f” with the file path:

pip install requests -f .\requests

This is the same as installing the package from Pypi:

Python Pip install package offline

Conclusion

In this article we have summarized some useful tips for using Python pip to manage the installation and upgrading of the third party packages for your Python projects. For more advanced usage of this module, you may refer to it’s official document.

in Python / Resources 0 comments
tips for handle python exception, raise and catch multiple exception, user defined exception
 by ken

7 Tips for Handling Python Exception

Introduction

Exception handling is commonly seen in Java, C++ or any other modern programming languages. Similarly, in Python, it has two types of errors – syntax errors and exceptions. Syntax errors refer to the syntactic problems which are detected by interpreter when translating the source code into byte code, while the exceptions are only detected during the runtime when the particular line is evaluated and executed. Python exceptions can be handled by application code but not the syntax errors.

In this article, we will discuss about the different ways to handle Python exceptions and when shall we use them.

Raise Exception

Python has a list of built-in exceptions which can be generated by interpreter or the built-in functions. Before we touch on the exception handling, let’s first look at how to raise a built-in exception.

Without implementing any new exception, you can use the raise keyword to throw out a built-in exception from your code. For instance, the below code checks the data type and value for mode variable, and raises built-in exceptions when wrong data type or value specified:

if not isinstance(mode, int):
    raise TypeError("mode must be numeric values")
        
if not mode in [0, 1]:
    raise ValueError("mode has to be 0 or 1")

When the mode variable is passed as a string “1”, you shall see exception raised as per below:

tips for handle python exception, raise and catch multiple exception, user defined exception

Catch Exception

To catch an exception, Python uses try-except which is similar to the try-catch syntax in Java/C++. In the except block, you can either write your own logic to handle the exception, or re-raise the exception to the callers.

For example, the below code would catch the KeyError exception and re-raise it as ValueError to the caller:

def read_config(config):
    
    try:
        mode = config["mode"]
    except KeyError as err:
        raise ValueError(f"No mode has been specified.")        
                
    return mode

When calling the read_config function with an empty dictionary:

read_config(config={})

You shall see the below exception raised:

tips for handle python exception, raise and catch multiple exception, user defined exception

The traceback shows both the cause of the exception and the last exception raised. If you want to hide the details of the cause (e.g. KeyError exception), you can use the raise exception from None to disable the chaining of the exceptions:

raise ValueError(f"No mode has been specified.") from None

When re-run the code, you shall see that only the last exception was showing in the trackback:

tips for handle python exception, raise and catch multiple exception, user defined exception

You can also specify a different exception when you think a explicit cause is needed rather than the original exception, e.g. instead of raise from KeyError, you can raise from AttributeError for better clarity to the caller.

Catch Multiple Exceptions

Catching multiple exceptions are as simple as adding multiple else if statements. Be cautious about the sequence of the exception classes as the derived exception classes must be placed before the base class, otherwise all your exceptions would be caught by the base exception class.

Below is an example to catch all the file operation exceptions, since the OSError is the base class, it shall be put as the last except block:

file_path = r"c://config.txt"

try:
    with open(file_path, "w") as f:
        config = f.write("password")
except FileNotFoundError:
    print(f"file {file_path} is not found")
except PermissionError:
    print(f"you have no permission to write to {file_path}")
except OSError as e:
    print("os error {e}")

When you run the above code, you shall see the output similar to below:

you have no permission to write to c://config.txt

You can also put multiple exceptions with parentheses to handle them within the same except block when there is no need to distinguish them:

except (OSError, FileNotFoundError, PermissionError) as e: 
    print(e)

This is similar to an IN operator, the exception will be caught as as long as it is an instance of any of these classes.

Suppress Exception with Pass Keyword

There are cases that you cannot do anything when the exception happens, but you do not wish your program to halt due to these exceptions. In this case, you shall just suppress the exception with pass keyword in the except block.

Below is an example to perform some clean-up in a class destructor method for a selenium automation project. When exception happens, you cannot do anything but letting the program exit peacefully:

def __del__(self):

    if self.driver:
        try:
            self.driver.quit()
        except:
            #suppress any exceptions
            pass

Use of the else block

The Python try-except statement also has a optional else clause, which allows you to execute some code when no exception is raised. This probably useful when you do not want some exceptions to be caught by the try-except which is not intended for them. For instance, when you put a few lines of codes in the try block, and intend to catch the exception for the first line, you may accidently catch some exceptions raised in the subsequent lines. To avoid that, you can split your code and put the rest of the code in the else block where it only get executed when no exception raised for the first line.

Another useful example would be, if you have a for-loop inside the try-except block, and you need to check if all iterations are successful without exception, instead of having a flag to keep track it, you can use the else block:

import math
from random import randint

def check_all_successful(num):
       
    while(True):
        result = {}
        try:
            #some dummy logic for demo purpose
            #exception would not raise if all random numbers generated are positive
            for i in [randint(-num, num) for _ in range(6)]:
                result[i] = math.sqrt(i)
                
        except Exception as e:
            print(f"failed to process: {i}, error: {e}")
        else:
            #when all iterations are successful
            print(result)
            break

When calling this function:

check_all_successful(10)

You may see the function ended when no exception raised, and below is the output:

tips for handle python exception, raise and catch multiple exception, user defined exception

Perform Clean-up action with Finally Keyword

If there is any operation you must perform regardless of whether the exception is raised or not, such as closing an IO object or network connection, you shall consider to use the finally statement. It is designed for any clean-up logic after the exception is raised, and make sure the code is executed before the exception raised to the caller.

For instance, below is some code snippet to connect to SQLite and update the data for a user table. When exception happens, the finally block shall close the connection properly before program exits:

import sqlite3
con = sqlite3.connect('example.db')

try:
    cur = con.cursor()
    cur.execute("UPDATE USER SET PASSWORD = 'password' WHERE NAME = 'admin'")
    con.commit()
finally:
    con.close()

One thing to be noted, if you have return statements in both try/except and finally block, only the result from finally block will be returned to the caller no matter the exception happens or not. For instance, the below will always return 9:

def test():
    try:
        return 1/0
    except:
        return 2
    finally:
        return 9

Create User Defined Exceptions

When writing your own application, you may want to define your own exceptions based on your business scenario. To create custom exceptions, you shall first create your base exception class which is derived from the Python base exception – Exception, then you can create your other exceptions deriving from this base exception class.

For instance, you may create below exception classes:

# exceptions.py

class AppException(Exception):
    pass

class CurrencyNotSupported(AppException):
    pass

And in your application code, you can raise your own exceptions whenever appropriate. E.g. :

def convert_to_usd(local_currency, local_amount):
    
    if local_currency.lower() not in ["sgd"]:
        raise CurrencyNotSupported(f"The currency {local_currency} is not supported")
        
    return local_amount * 1.32

When you passed in some invalid input to the function:

convert_to_usd("eur", 100)

You shall see the exception raised in the traceback:

tips for handle python exception, raise and catch multiple exception, user defined exception

Conclusion

In this article, we have reviewed through the different scenarios for raising and handling Python exceptions. As a general rule, you shall not suppress any exception when you are not 100% sure what to do with it. Because this may need to other system bugs or troubleshooting issues later on. The exceptions shall be handled by parties who has the best knowledge to handle it, so raise rather than suppress it when you are not sure how to handle it.

in Python 0 comments
reading email from outlook with python pywin32
 by ken

5 Useful Tips for Reading Email From Outlook In Python

Introduction

Pywin32 is one of the most popular packages for automating your daily work for Microsoft outlook/excel etc. In my previous post, we discussed about how to use this package to read emails and save attachments from outlook. As there were quite many questions raised in the comments which were not covered in the original post, this article is intended to review through some of the advanced topic for reading emails from outlook via Python Pywin32 package.

If you have not yet read through the previous post, you may check it out from here.

Prerequisites:

Assuming you have already installed the latest Pywin32 package and imported below necessary packages in your script, and you shall not encounter any error after executing the GetNamespace method to establish the outlook connection:

import win32com.client

#other libraries to be used in this script 
import os 
from datetime import datetime, timedelta

outlook = win32com.client.Dispatch('outlook.application') 
mapi = outlook.GetNamespace('MAPI')

When using below code to iterate the Accounts property, you shall see whichever accounts you have configured in your outlook:

for account in mapi.Accounts: 
    print(account.DeliveryStore.DisplayName)

#Assuming below accounts have been configured:
#abc@hotmail.com
#def@gmail.com

Now let’s move on to the topics we are going to discuss in this article.

Reading Email from Multiple Outlook Accounts

If you have multiple accounts configured in your outlook application, to access one of the accounts, you can use the Folders method and specify the account name or index of the account, e.g.:

for idx, folder in enumerate(mapi.Folders):
    #index starts from 1
    print(idx+1, folder)

#Assuming below output:
# 1  abc@hotmail.com
# 2  def@gmail.com

And to access the sub folders under a particular email account, you can continue to use the folders method to specify the sub folder name or index of the folder. Before that, you may want to check what are the available sub folders and it’s index value as per below:

for idx, folder in enumerate(mapi.Folders("abc@hotmail.com").Folders):
    print(idx+1, folder)
# or using index to access the folder
for idx, folder in enumerate(mapi.Folders(1).Folders): 
    print(idx+1, folder)

You shall see something similar to the below:

reading email from outlook with Python pywin32

With the above folder index and name, you shall be able to access the email messages as per below:

messages = mapi.Folders("abc@company.com").Folders("Inbox").Items
# or
messages = mapi.Folders(1).Folders(2).Items
for msg in list(messages):
    print(msg.Subject)

Although the index would not get changed when you move up/down of your folders in outlook, obviously using folder name still is much better than index in terms of readability of the code.

Filter Email Based on Receiving Time Window

When reading emails from outlook inbox, you may want to zoom into the emails within a specific receiving time window rather than scanning through thousands of emails you have received in the inbox. To filter emails based on certain conditions, you can use restrict method together with the logical operators.

For instance, to filter the emails received from 1st day of the current month until today 12am:

today = datetime.today()

# first day of the month
start_time = today.replace(month=1, hour=0, minute=0, second=0).strftime('%Y-%m-%d %H:%M %p')

#today 12am
end_time = today.replace(hour=0, minute=0, second=0).strftime('%Y-%m-%d %H:%M %p')

messages = messages.Restrict("[ReceivedTime] >= '" + start_time
+ "' And [ReceivedTime] <= '" + end_time + "'")

With logical operators like AND, OR and NOT, you are able to combine multiple criteria together. For instance, to check the email with certain subject but not from a particular sender email:

messages = messages.Restrict("[Subject] = 'Sample Report'" 
                             + " And Not ([SenderEmailAddress] = 'abc@company.com')")

And you can also use the Restrict method as many times as you wish if it makes your code more readable than combining all conditions in one filter, e.g.:

messages = messages.Restrict("[Subject] = 'Sample Report'")
messages = messages.Restrict("Not ([SenderEmailAddress] = 'abc@company.com')")

Getting First N emails

When using Restrict method for filtering email messages, you would not be able to specify max number of emails you want to read. If you wish to get the first/last N emails based on the receiving time, you can use the Sort method to sort the messages based on certain email properties before you slice the list. Below is the sample code to get the latest 10 email messages based on the receiving time:

messages.Sort("[ReceivedTime]", Descending=True)

#read only the first 10 messages
for message in list(messages)[:10]:
    print(message.Subject, message.ReceivedTime, message.SenderEmailAddress)

Wildcard Matching for Filtering

With the Restrict method, you cannot do wildcard matching such as searching whether the email subject or body contains certain keywords. To be able to achieve that, you will need to use the DASL query.

For instance, with the below DASL query syntax, you can filter email subject which contains “Sample Report” keyword:

messages = messages.Restrict("@SQL=(urn:schemas:httpmail:subject LIKE '%Sample Report%')")

You may want to check here to see what are the fields supported in ADSL query and the correct namespace to be used.

Include/Exclude Multiple Email Domains

To filter the emails only from a particular domain, you can use the ADSL query similar to the previous example:

messages = messages.Restrict("@SQL=(urn:schemas:httpmail:SenderEmailAddress LIKE '%company.com')")

And to exclude the emails from a few domains, you can use multiple conditions with logical operators:

messages = messages.Restrict("@SQL=(Not(urn:schemas:httpmail:senderemail LIKE '%@abc%') \
And Not(urn:schemas:httpmail:senderemail LIKE '%@123%') \
And Not(urn:schemas:httpmail:senderemail LIKE '%@xyz%'))")

Conclusion

In this article, we have reviewed through some advanced usage of the Pywin32 package for filtering emails. You may not find many Python tutorials for this package from online directly, but you shall be able to see the equivalent VBA code from its official website for most of the code you have seen in this article. In the event that you cannot find a solution for you problem, you may check and see whether there is something implemented in VBA code that you can convert it into Python syntax.

Link for the previous post Reading Email From Outlook In Python.  Follow me on twitter for more updates.

in Python / Resources 16 comments
python virtual environment, isolated environment
 by ken

3 Ways for Managing Python Virtual Environment

 Introduction

Python virtual environment refers to an isolated execution environment for managing Python versions, dependencies, and indirectly permissions. When you have multiple projects working on and there are potential conflicting requirements such as different Python versions or libraries to be used in these projects, you need to consider using a virtual environment so that installing packages for one project will not impact another.

In this project, we will discuss about the different ways to create Python virtual environment for your multiple projects.

Using venv or virtualenv

Since Python 3.3, it introduced a lightweight venv module for you to create virtual environment, so that you do not need to install any additional tools for it. But if you are working on some projects with older Python versions, you will need to install another popular package called virtualenv for managing the virtual environment. Using Windows as an example, the steps to create a virtual environment are as simple as below:

  • Go to your command line window (Win- R)
  • Use “cd” command to switch to a writable folder where you want your Python virtual environment to be created e.g.: a dedicated folder called “py_venv”
  • Use below command to create a virtual environment for your project
python -m venv project_name
  • It takes a few seconds for the above to complete, and you shall be able to see below folder structure created under “project_name”:Python virtual environment, Python isolated environment

The pyvenv.cfg file describes the home directory, Python version etc. If you are using virtualenv, it will also indicate the version of this package you used.

  • Under the “Scripts” folder, you can see the following files:

Python virtual environment, Python isolated environment

Now the virtual environment has been created successfully. From the command line window, you can go to the “Scripts” folder and type “activate” or “activate.bat” and hit enter. You shall see below on your command line:

Python virtual environment, Python isolated environment

When seeing project name prefix at the beginning of the command, it means you are already in the virtual environment mode for this project, now you can proceed to install all the necessary packages, build and debug your code in this isolated environment. There is a system-site-packages parameter which you can specify whether you want to inherit the packages from the global environment, this might be useful when you have some heavy packages that you do not want to re-install them in each of your virtual environment.

To exit from the current virtual environment, you can run the “deactivate.bat”:

venv deactivate

Or if you need to switch from one virtual environment to another, for instance another virtual environment called “test”, you can activate the “test”, then it will exit the “project_name” automatically and switch to “test” environment:

Python virtual environment, Python isolated environment

From the above steps, you may wonder how to specify the different Python version when creating your virtual environment.

Assuming you have already installed Python 3.7 and Python 3.8, and during the installation, you have added the Python installation directories to your PATH variable. When you use “where python”:

Python virtual environment, Python isolated environment

You can see all the different Python versions you have installed. As “venv” does not support to specify the Python version, you will need to use virtualenv for this case, for instance:

virtualenv test1 -p python3.7
virtualenv test2 -p python3.8

This shall create the virtual environments based on the Python version you’ve specified, you can verify the version_info from the pyvenv.cfg file in the folder.

Many people may have question on whether the virtual environment folder shall be created separately or put it under the same place where your source code is placed.

Generally there is no right or wrong where you shall create your virtual environment folder, but you shall exclude this folder when you submit your code to the repository since other people may not be able to re-use whatever packages you’ve installed due to the different OS they are using.

My personal preference is to have a dedicated folder for all the virtual environment setup, and use the below command to export the installed packages when submitting to the repository, so that your source code folders will be cleaner:

python3 -m pip freeze > requirements.txt

Sometimes you may find tedious to switch between your existing virtual environments by using the activate/deactivate script, the virtualenvwrapper provides easier way to switch environment by names. You can read more from its official documents

Managing Python Virtual Environment with Conda

Conda is an open-source package and environment management tool which you can easily use it for handling your Python with conflicting dependency requirements. If you have never used it before, I would suggest you to do a quick review on the user guide from their official website, and down the miniconda based on which OS you are using.

To create a virtual environment with Conda, you can run the below from your command line:

conda create -n test python=3.9

You can see that Conda allows to specify the Python version, and it will check and download the Python version you’ve specified if it is not installed yet, and set up the virtual environment in the below default folder.

Python virtual environment, Python isolated environment

To activate your virtual environment:

conda activate test

Once it’s activated, you shall see the prefix added to your command line. And you can use below command to check what are the packages available in your current environment:

Python virtual environment, Python isolated environment

To deactivate your virtual environment, you can use the below:

conda deactivate
# or
conda activate other_project

Note that you can use both Conda or pip to install new packages, e.g.:

pip install numpy

You can see that when it is installed from pip, the channel will be indicated as pypi:

Python virtual environment, Python isolated environment

For more usage of Conda, you may refer to their official documentation here.

Managing Python Virtual Environment with IDE

If you are using IDE for Python programming, such as PyCharm, Visual Studio Code or Sublime Text, they usually provide an option for you to specify whether you want to set up a virtual environment when creating the new project. This would save you some effort for manually setting up the virtual environment and you do not have to worry about any potential conflict among your multiple projects.

Using PyCharm as an example, when create/import a new project, you are able to use the virtualenv tool to create a project-specific isolated virtual environment. The virtualenv tool comes bundled with PyCharm, so you do not need to install it separately.

Python virtual environment, Python isolated environment

For the detailed steps, you can refer to the PyCharm official document. Similar guide you can find for the other IDEs.

Conclusion

In this article, we have discussed about the purpose of using Python virtual environment and the different ways you can use to set up an isolated environment for your Python project. If you are new to topic, you may get confused as you would see many variant of the virtual environment tools people talked about, such as pyenv, pyvenv, pipenv, pyenv-virtualenv etc. Some of them are already deprecated in the later Python versions, so for a start, you shall concentrate on the built-in module venv, and then explore virtualenv and virtualenvwrapper for more advanced features.

in Python / Tutorials 0 comments