Mastering Python Packaging and Distribution: From Local Scripts to PyPI

Python’s strength lies in its vast ecosystem of third-party libraries. However, for many developers, the journey from writing a script to distributing a robust package remains a black box. Understanding how to properly package, build, and distribute Python projects is essential for any intermediate to advanced developer who wants to share code, collaborate on open-source projects, or publish internal tools to a private index.

In this guide, we will demystify the modern Python packaging landscape. We will move beyond the outdated setup.py monolith and embrace the newer, more robust standards recommended by the Python Packaging Authority (PyPA).

The Shift from Setuptools to PEP 517/518

Historically, Python packaging relied heavily on setuptools and a verbose setup.py file. While functional, this approach often led to brittle builds and configuration sprawl. The community has since moved towards standardized build backends defined in PEP 517 and PEP 518.

The new standard introduces two key files:

1. pyproject.toml: Declares the build requirements and project metadata.
2. build backend: A tool like setuptools, poetry, or flit that performs the actual build.

Defining Your Project Structure

A clean project structure is the foundation of a maintainable package. While there are many variations, a common modern layout looks like this:

my_package/
├── pyproject.toml
├── README.md
├── LICENSE
└── src/
    └── my_package/
        ├── __init__.py
        ├── core.py
        └── utils.py

Using a src/ layout is highly recommended. It prevents accidental imports of local files during testing and ensures that the installed package is always the one being tested, not the source code in your current directory.

Configuring pyproject.toml

The pyproject.toml file is the single source of truth for your project’s metadata. Below is an example configuration using setuptools as the build backend, which is the most widely adopted standard.

[build-system]
requires = ["setuptools>=61.0", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "my-awesome-package"
version = "0.1.0"
description = "A sample Python package for demonstration."
readme = "README.md"
license = {text = "MIT"}
requires-python = ">=3.8"
authors = [
  {name = "Jane Developer", email = "jane@example.com"},
]
dependencies = [
  "requests>=2.28.0",
  "click>=8.0.0",
]

[project.optional-dependencies]
dev = [
  "pytest>=7.0.0",
  "black>=22.0.0",
]

Key features to note in this configuration:

• Dynamic metadata: You can keep version numbers in separate files to avoid manual updates.
• Optional dependencies: Users can install dev tools via pip install my-package[dev].
• Python version constraint: Explicitly defining requires-python ensures compatibility checks.

pip install build twine

python -m build

Creating sdist archive: dist/my_awesome_package-0.1.0.tar.gz
Creating  wheel archive: dist/my_awesome_package-0.1.0-py3-none-any.whl

twine upload --repository testpypi dist/*

pip install --index-url https://test.pypi.org/simple/ --no-deps my-awesome-package