Ctrl+K

4.3 Dependency Management and Virtual Environments

4.3 Dependency Management and Virtual Environments#

Dependency Management#

We already discussed why managing dependencies is an important aspect of our software development process. Poetry simplifies this task through an intuitive set of commands and a pyproject.toml file for configuration.

Adding Dependencies#

To see how we declare dependencies in a Poetry project. Let’s first add a first simple module to our my-package project to simulate a common use case. To do so, let’s create a new file called analysis.py in the my-package/my-package directory with the following content:

import pandas as pd
df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})

Pandas is a popular library for data manipulation and analysis in Python very frequently used in Data Science projects.

Now we want, to use Poetry to keep track that our project requires pandas as a dependency. To add pandas as a dependency, run the following command in your project directory:

poetry add pandas

What to notice

  • We see that if the project was setup with python 3.8, Poetry will not add pandas as a dependency and let us know that pandas requires python 3.9 or higher.

We can then go to the pyproject.toml file and bump the python version to 3.9 and run poetry add pandas again.

[tool.poetry.dependencies]
python = "^3.9"

poetry add pandas

The currently activated Python version 3.8.2 is not supported by the project (^3.9).
Trying to find and use a compatible version. 

Poetry was unable to find a compatible version. If you have one, you can explicitly use it via the "env use" command.

What to notice

  • We see that pandas was added as a dependency.

  • We also see that a poetry.lock file was created.

Adding Dev Dependencies#

For development dependencies, which are not required in production, use poetry add <package> --group <group_name>:

poetry add pytest --group dev

The currently activated Python version 3.8.2 is not supported by the project (^3.9).
Trying to find and use a compatible version. 

Poetry was unable to find a compatible version. If you have one, you can explicitly use it via the "env use" command.

Version Constraints#

Poetry keeps track of the dependencies in the pyproject.toml file under the [tool.poetry.dependencies] section. You can define the version constraints to ensure compatibility and stability of your project. For example, we look now at the pyproject.toml file of our my-package project we see that the pandas version is specified as ^2.2.1, which means that any version of pandas greater than or equal to 2.2.1 and less than 3.0.0 is acceptable. This is a common practice to ensure that your project remains compatible with future versions of its dependencies.

[tool.poetry.dependencies]
python = "^3.9"
pandas = "^2.2.1"


[tool.poetry.group.dev.dependencies]
pytest = "^8.1.1"

Virtual Environment Management with Poetry#

If we now want to run our package, we need to install it in a virtual environment along with the dependencies that we defined. Poetry makes it easy to create and install packages in virtual environments.

Configuring Poetry to Store Virtual Environments Within the Project#

By default, Poetry stores virtual environments in a centralized location. However, you can configure Poetry to create the virtual environment within your project’s directory by setting the virtualenvs.in-project configuration to true:

poetry config virtualenvs.in-project true

This configuration makes it easier to manage project environments, especially when working with version control systems or when you need to share the project with others.

Creating and Using Your Virtual Environment#

This is now when the clean conda environment that we created in the previous section comes in handy. Although Poetry can help us managing package versions, it cannot help us (yet) to manage the Python versions. So in order to create the environment, we will need to pass the path to the python executable of the conda environment that we created with the appropiate python version. Otherwise, it will use the system python version.

We use the command conda env list to list all the environments and their paths. We then use the path to the python executable of the environment that we created to create the virtual environment for our project.

So in my case the path is

/home/callaram/.conda/envs/py3.10.4_clean/bin/python

now, to create the environment I can use the command:

poetry env use /home/callaram/.conda/envs/py3.10.4_clean/bin/python

What to notice

  • We also see that a .venv directory was created in the project directory.

and we can use the command poetry install to install the dependencies in the virtual environment.

poetry install

What to notice

  • We see that the dependencies are installed in the virtual environment.

  • We see that also the dependencies of our dependencies are installed.

  • If a poetry.lock file exists, Poetry will install dependencies according to the versions listed there to ensure consistency across environments.

Running Commands Within the Virtual Environment#

If you need to run a single command within the context of your virtual environment without activating it, you can use poetry run. For example, to run a Python script:

poetry run python my_package/analysis.py

Activating the Virtual Environment#

To activate the project’s virtual environment, you can use the poetry shell command. This command spawns a new shell instance with the virtual environment activated, allowing you to run Python and other commands within the context of your project’s dependencies.

Switching Between and Managing Multiple Environments#

Poetry allows you to manage multiple virtual environments for different Python versions or dependency configurations. Use the poetry env use command to switch between different Python interpreters or virtual environments associated with your project.

Displaying and Listing Environment Information#

To view details about the current environment, including its location and Python version, you can use poetry env info. To list all environments associated with the project, use poetry env list.

Deleting Environments Associated with Your Project#

If you need to remove a virtual environment managed by Poetry, you can use poetry env remove followed by the environment’s name or Python executable path. This is useful for cleaning up or resetting your project’s environments.

Differences Between Installing With and Without poetry.lock#

  • With poetry.lock: Ensures that the exact versions of dependencies are installed, as specified in the lock file. This is critical for consistency and reproducibility across development, testing, and production environments.

  • Without poetry.lock: Poetry resolves dependencies based on the version constraints in pyproject.toml and generates a new poetry.lock file. This might lead to installing newer versions of dependencies than previously used if they still satisfy the specified constraints.

Committing the poetry.lock File to Version Control#

Committing the poetry.lock file to your version control system is recommended. It locks the project to specific versions of dependencies, ensuring that other developers and environments use the exact same versions, thereby avoiding “it works on my machine” problems.

Updating Dependencies to Their Latest Versions#

To update dependencies to their latest versions that still comply with the version constraints defined in pyproject.toml, use:

poetry update

Managing Dependencies Only Installations#

The current project is installed in editable mode by default. If you want to install the dependencies without installing the package itself, you can use the poetry install --no-root command. This is particularly useful for CI/CD environments where you might need the dependencies for testing or building the project but not the project package itself.

Poetry Virtual Environments with Visual Studio Code#

Visual Studio Code (VS Code) has built-in support for Python development, including the detection and use of virtual environments. When you create a virtual environment in the root folder of your project, VS Code can automatically detect this environment and suggest it for use in the workspace. This seamless integration allows you to run and debug your Python code within the context of the project’s virtual environment directly from the IDE.

To use the detected virtual environment in VS Code, follow these steps:

  1. Open the Command Palette (Ctrl+Shift+P on Windows/Linux, Command+Shift+P on macOS).

  2. Type and select “Python: Select Interpreter.”

  3. Choose the virtual environment from the list. This environment will typically be located in your project’s root directory and prefixed with ./.venv or similar, indicating that it is the local environment created by Poetry.

Once the virtual environment is selected, VS Code will use it for running and debugging your project’s Python files. This ensures that you are working with the specific dependencies defined for your project, enhancing consistency and reliability during development.

Resources and Further Reading#

Contents