A First Look at Poetry

Learn how Poetry simplifies Python development by consolidating common tools for managing dependencies and environments efficiently.

Outline

• Overview
  • Python Tools Replaced by Poetry
  • What About Setuptools and Wheel?
• Getting Started
  • Install Poetry
  • Create and Build a New Project with Poetry
  • Adding, Installing, and Updating Dependencies
• Managing Environments
  • Creating and Activating Virtual Environments
  • Environment Isolation and Default Python Version
  • Switching and Deleting Environments
  • Environment Information and Listing Environments
• Poetry Configuration Files
  • Global Configuration with config.toml
  • Project Specific Local Configuration with poetry.toml
  • Project Specific Version Controlled Configuration with pyproject.toml

Overview

Poetry is a comprehensive tool for managing Python dependencies and packaging. It is designed to handle project dependencies and package management by integrating several pieces of functionality you might otherwise seek from separate tools.

Python Tools Replaced by Poetry

Have you ever struggled to understand how and when to use tools like virtualenv, venv, pipenv, pyenv, pip, PDM, conda, and miniconda? If you have, I’ve got some good news. By integrating these separate concerns around Python development, Poetry significantly reduces the risk of descending into a nihilistic fit of rage, despair, and eventual insanity that typically accompanies the task of managing virtual environments in Python.

Here’s a breakdown of how Poetry interacts with or replaces the need for other commonly used tools in Python development:

• Poetry automatically creates/manages virtual environments eliminating the need for manually using venv or virtualenv to create isolated environments.
• If you aren’t working in a specific domain that benefits from Conda’s features (like scientific computing), you may not need conda or miniconda.
• There is no need to use Pipenv if you are using Poetry, as both serve similar purposes in dependency management and virtual environment creation.
• Like Pipenv, PDM is another tool for dependency management and project setup that you wouldn’t need if you’re using Poetry.
• If you work with multiple versions of Python, pyenv can still be useful alongside Poetry.
  • pyenv helps manage multiple Python versions on your system.
  • Poetry can be configured to use different Python versions managed by pyenv for different projects.
• Poetry uses pip internally to install dependencies.
  • Understanding how pip works can be helpful, but you generally won’t need to use pip directly when managing packages within a Poetry-managed project.

To summarize, when using Poetry it can be beneficial to understand pip but not necessary for direct use. You may also find pyenv helpful if you need to switch between multiple Python versions. However, other tools like venv, virtualenv, Pipenv, PDM, and conda can be safely replaced by Poetry for most standard Python project workflows. If specific features of these tools are required for specialized tasks, consider that an opportunity to reevaluate whether those tasks are actually worth keeping around.

What About Setuptools and Wheel?

setuptools and wheel are essential tools in the Python ecosystem, especially for packaging and distribution of Python projects. For most development work where you are using Poetry, you do not need to directly interact with setuptools or wheel, as Poetry abstracts much of the functionality of these tools.

However, understanding how they work is beneficial, especially if you’re dealing with Python packaging beyond the scope of typical application development, like creating libraries intended for public distribution. Basically, even if you are using Poetry, it’s useful to understand what these tools do because they are foundational to how Python packages are built and distributed.

Setuptools is a library designed to help you create and distribute Python packages.

• It extends the distutils (distribution utilities) provided by the Python standard library.
• It includes commands for building and installing packages, handling metadata like dependencies, and more.
• Essentially, it provides the tools to package your Python project and manage its dependencies.
• While Poetry handles project dependencies and can package projects on its own, setuptools remains a fundamental part of the Python packaging ecosystem that Poetry interacts with under the hood.
• When you build a package with Poetry for distribution (e.g., creating a source distribution), it creates a setup.py dynamically if needed for compatibility, though this is abstracted away from the user.

Wheel is a package format designed as a faster and more reliable method of installing Python software than re-building from source code each time.

• A wheel is a pre-built distribution format that can speed up the installation process for Python packages.
• Unlike a source distribution, which requires building and compiling at the time of installation, a wheel is a binary distribution that can be quickly installed.
• Poetry can build wheel files directly when you prepare your project for distribution.
• This means that when you publish a package using Poetry, it can create .whl files, which are then uploaded to package indices like PyPI.

Getting Started

This guide covers the essential aspects of Poetry for someone just learning the tool. It will walk you through the basics of using Poetry for your Python projects including installation, project setup, python version management, working with dependencies, using virtual environments, and configuring your projects and local machine.

Install Poetry

Install Poetry with the official install script:

curl -sSL https://install.python-poetry.org | python3 -

On MacOS, this will install poetry in .local/bin inside your home directory (/Users/ajcwebdev/.local/bin for example). If this directory is not already in your PATH, add the following to your .zshrc file:

export PATH="$HOME/.local/bin:$PATH"

Verify your installation completed successfully by checking your poetry version:

poetry --version

At the time of publication for this blog post, I was using Poetry version 1.8.3. poetry self manages the Poetry installation itself with subcommands such as:

• add for installing plugins
• show plugins for listing installed plugins
• update for updating Poetry

Running the update command like so:

poetry self update

Gives the following output:

Updating Poetry version ...

Using version ^1.8.3 for poetry

Updating dependencies
Resolving dependencies... (5.1s)

No dependencies to install or update

Writing lock file

Create and Build a New Project with Poetry

To create a new project with Poetry, use the poetry new command:

poetry new ajcwebdev-poetry

This creates a directory structure for your project, including a pyproject.toml file, which is the heart of your Poetry project. It contains all the project configurations and orchestrates your project and its dependencies. Later in this post, we’ll dive deeper into the various configuration files used by Poetry. For now, here’s a basic example:

[tool.poetry]
  name = "ajcwebdev-poetry"
  version = "0.1.0"
  description = "A getting started example repo for Poetry created by Anthony Campolo"
  authors = ["Anthony Campolo <12433465+ajcwebdev@users.noreply.github.com>"]
  readme = "README.md"

[tool.poetry.dependencies]
  python = "^3.12"

[build-system]
  requires = ["poetry-core"]
  build-backend = "poetry.core.masonry.api"

Under tool.poetry.dependencies in the pyproject.toml file, you can specify the Python version your project supports. In the above example, python is set to ^3.12 which allows any Python version greater than or equal to 3.12.0 and less than 4.0.0.

You can show the project version by running:

poetry version
# ajcwebdev-poetry 0.1.0

For managing project versions, the project’s version can be incremented with version patch:

poetry version patch
# Bumping version from 0.1.0 to 0.1.1

poetry build builds the source and wheels archives and can be used for specifying the output directory for build artifacts (poetry build --output dist).

poetry build

Output:

Creating virtualenv ajcwebdev-poetry-RnIQm9t6-py3.12 in /Users/ajc/Library/Caches/pypoetry/virtualenvs
Building ajcwebdev-poetry (0.1.1)
  - Building sdist
  - Built ajcwebdev_poetry-0.1.1.tar.gz
  - Building wheel
  - Built ajcwebdev_poetry-0.1.1-py3-none-any.whl

Adding, Installing, and Updating Dependencies

To add a dependency to your project, use the poetry add command. This will add the package to your pyproject.toml file and install it:

poetry add cowsay

Output:

Using version ^6.1 for cowsay

Updating dependencies
Resolving dependencies... (0.3s)

Package operations: 1 install, 0 updates, 0 removals

  - Installing cowsay (6.1)

Writing lock file

The command can be used for numerous types of dependencies including but not limited to:

• Adding packages with specific version constraints (e.g., poetry add pendulum@^2.0.5)
• Adding git dependencies (e.g., poetry add git+https://github.com/sdispater/pendulum.git)
• Adding local directories or files as dependencies
• Adding packages with extras (e.g., poetry add "requests[security,socks]")

To install all dependencies specified in your pyproject.toml file, use the poetry install command.

poetry install

This creates a poetry.lock file if it doesn’t exist, ensuring consistent installations across different environments. poetry export exports the lock file to other formats, like requirements.txt which is important for compatibility with other tools or deployment processes. The poetry install command can also be used for:

• Installing only specific dependency groups (e.g., poetry install --only test,docs)
• Synchronizing the environment with the lock file (poetry install --sync)
• Installing with or without compilation of Python bytecode (poetry install --compile)

To update your dependencies to their latest versions (within the constraints specified in pyproject.toml), use the poetry update command to update the poetry.lock file with the new versions:

poetry update

This command can be used to update only specific packages (poetry update requests toml) or update dependencies without installing them (poetry update --lock).

Managing Environments

Poetry ensures each project works in an isolated environment so you can easily switch between different Python versions for your project.

Creating and Activating Virtual Environments

Poetry creates a virtual environment for your project. To run commands within this environment, use the poetry run command for executing scripts defined in pyproject.toml (e.g., poetry run my-script). Add the following to your __init__.py file in your project’s root (ajcwebdev_poetry in my case):

import cowsay

cowsay.cow("Hello, I'm a cow!")

Run this script:

poetry run python3 ajcwebdev_poetry/__init__.py

You did a cow.

  _________________
| Hello, I'm a cow! |
  =================
                 \
                  \
                    ^__^
                    (oo)\_______
                    (__)\       )\/\
                        ||----w |
                        ||     ||

To activate the virtual environment in your shell, use the poetry shell command to spawn a shell within the project’s virtual environment (useful for working directly within the project environment):

poetry shell

If you’re a naive, noob Python developer like me, you might try immediately running print("hello") and get a weird error message related to zsh. This is because you haven’t actually started a Python REPL yet. To explicitly use the Python interpreter to run Python code in the shell, use python -c:

python -c "print('hello')"
# hello

Alternatively, you can start the actual Python interactive shell by running python and then executing your code:

>>> print("hello")
# hello

If you want to run Python commands directly in your shell without typing python each time, you might need to adjust your shell configuration. However, Gibbity (my good buddy and trusted scholar) told me it’s generally considered good practice to use the Python interpreter explicitly to avoid confusion with shell commands.

Environment Isolation and Default Python Version

Poetry prioritizes project environment isolation as a core feature. This means Poetry always works isolated from your global Python installation. It first checks if it’s running inside a virtual environment.

• If yes, it uses that environment directly.
• If no, it uses a previously created environment or creates a new one.

By default, Poetry uses the Python version used during its installation to create virtual environments.

• If this version is incompatible with the project’s Python range, Poetry will try to find a compatible version.
• If unable to find a compatible version, you’ll be prompted to activate one explicitly.
• If you use pyenv to manage Python versions, you can set virtualenvs.prefer-active-python to true which tells Poetry to use the current Python version of your shell.

Example workflow:

pyenv install 3.9.8
pyenv local 3.9.8  # Activate Python 3.9 for the current project
poetry install

Switching and Deleting Environments

The env commands provide full control over environment management. Always ensure your project’s python dependency in pyproject.toml matches the environment you’re using. By mastering these environment management techniques, you can maintain clean, isolated, and version-specific environments for each of your Poetry projects.

• Use the env use command to specify which Python version to use:
  • Full path: poetry env use /full/path/to/python
  • From PATH: poetry env use python3.7
  • Minor version: poetry env use 3.7
  • Revert to default: poetry env use system
• Use env remove to delete environments:
  • By path: poetry env remove /full/path/to/python
  • By version: poetry env remove python3.7 or poetry env remove 3.7
  • By name: poetry env remove test-O3eWbxRl-py3.7
  • Multiple at once: poetry env remove python3.6 python3.7 python3.8
  • All environments: poetry env remove --all
  • Removing the active environment will automatically deactivate it.

Environment Information and Listing Environments

• Display environment info: poetry env info
  • Show only environment path: poetry env info --path
  • Show only Python executable path: poetry env info --executable
• List all virtual environments: poetry env list
  • Show full paths: poetry env list --full-path

Poetry Configuration Files

Poetry uses various configuration files that serve different purposes and are used in different contexts. These files include pyproject.toml, poetry.toml, and config.toml.

• All three use the TOML format for configuration and all influence Poetry’s behavior in some way. However, they differ in their scope and whether or not they should be publicly shared through version control.
• By understanding the purpose and scope of each configuration file, you can effectively manage your Poetry projects and maintain consistency across different environments and team members.

Here’s a quick summary of when to use one or the other based on their scope and content plus whether they should be committed to version control or not:

• Scope:
  • pyproject.toml is project-wide.
  • poetry.toml is project-specific but local.
  • config.toml is system-wide.
• Version control:
  • pyproject.toml should be committed.
  • poetry.toml and config.toml typically should not be committed.
• Content:
  • pyproject.toml contains project metadata and dependencies.
  • poetry.toml and config.toml contain Poetry-specific settings.

poetry config allows editing Poetry configuration settings and repositories. Run poetry config --list to show current config variables.

Global Configuration with config.toml

The config.toml file contains global configuration settings for Poetry across all projects on your system.

• It sets your personal defaults for Poetry across all projects and is created and modified using the poetry config command without the --local flag.
• It’s located in system-wide configuration directories that vary by OS.
• Since it’s globally scoped, it affects all projects on your system and sets default behaviors for Poetry.
• It can be overridden by project-specific settings and is not shared between different machines or users.

Project Specific Local Configuration with poetry.toml

The poetry.toml file is used for local, project-specific Poetry configurations that shouldn’t be shared with other developers.

• It’s created when you use the poetry config command with the --local flag and is located at the root of the project directory (alongside pyproject.toml).
• It’s scoped per-project but for local settings and overrides global Poetry settings for the specific project.
• The file is useful for developer-specific settings that shouldn’t affect other team members and typically should not be committed to version control.
• Settings in poetry.toml override those in config.toml for a specific project.
• To ensure you don’t accidentally change global settings, always use poetry config --local to modify poetry.toml.

Project Specific Version Controlled Configuration with pyproject.toml

The pyproject.toml file is the main configuration file for a Poetry project and is essential for defining your project and its dependencies.

• It’s located in the root of project directories and is scoped per-project.
• The file is meant to be version-controlled and shared among all developers working on the project.
• Its contents include project metadata (name, version, description, authors, etc.), dependencies, and dev-dependencies.
• It’s used for build system requirements and tool-specific configurations (including Poetry-specific settings).

poetry check validates the content of pyproject.toml and its consistency with poetry.lock which is important for ensuring project configuration is correct.