The Chaos of Floating Dependencies
In the Python ecosystem, the phrase "it works on my machine" is a notorious symptom of dependency management failures. When you rely on unversioned constraints like
>=1.0, your environment is subject to the whims of upstream package updates. A minor version bump in a transitive dependency can introduce breaking changes, leading to cryptic errors in production that are impossible to reproduce locally. This phenomenon, widely known as "Dependency Hell," is the primary reason why reproducibility is the cornerstone of professional software engineering.
To combat this, the industry standard has shifted toward lock files—static snapshots of all resolved dependencies. While tools like
pip have long lacked native lock file support, the community has developed robust solutions. Two of the most powerful tools for achieving deterministic builds are Poetry and pip-tools (specifically
pip-compile). This guide explores how to use both to create bulletproof, reproducible Python environments.
Strategy A: Poetry's Integrated Ecosystem
Poetry approaches dependency management by treating it as a core feature rather than an afterthought. It combines dependency resolution, virtual environment management, and packaging into a single tool. The heart of this system is the
poetry.lock file.
When you run
poetry install, Poetry does not just install packages listed in
pyproject.toml; it reads the lock file to ensure every single transitive dependency is pinned to an exact version hash. This ensures that the environment on your laptop is identical to the one in CI/CD and production.
Here is how to set up a basic Poetry project:
# Initialize a new project
poetry new my-project
# Add a dependency
poetry add requests
# This generates poetry.lock, pinning requests and all its sub-dependencies
To install dependencies in production, always use the lock file. Never run
poetry update in a production pipeline, as this will regenerate the lock file with potentially newer versions.
# Strict installation from lock file
poetry install --no-dev
The advantage of Poetry is its seamless integration. However, if you are tied to standard
pip workflows or require strict separation of build-time and run-time dependencies, Poetry's all-in-one approach might feel too monolithic.
Strategy B: The pip-tools Approach
For developers who prefer the simplicity of
pip but need deterministic builds,
pip-tools is the ideal solution. The
pip-compile tool takes a
requirements.in file (which lists your direct dependencies) and resolves them to a
requirements.txt file with full version pinning.
This method mimics the behavior of Go modules or Cargo, providing a clear separation between intent (what you want) and implementation (what you get).
# Create a requirements.in file
echo "requests==2.28.0" > requirements.in
# Resolve and compile dependencies
pip-compile requirements.in
# This generates a fully pinned requirements.txt
The generated
requirements.txt will look like this:
#
# This file is autogenerated by pip-compile with python 3.10
# To update, run:
#
# pip-compile requirements.in
#
certifi==2022.12.7
charset-normalizer==2.1.1
idna==3.4
requests==2.28.0
urllib3==1.26.13
Notice how
requests pulls in specific versions of
certifi,
urllib3, etc. This file is safe to commit to your repository and to install in production using
pip install -r requirements.txt.
Choosing the Right Tool
Both tools solve the same fundamental problem: reproducibility. Poetry is better suited for greenfield projects or teams that want a unified workflow for building and publishing packages. It handles virtual environments automatically and offers a rich CLI for managing scripts and plugins.
On the other hand, pip-tools is lighter and integrates seamlessly with existing Dockerfiles and CI/CD pipelines that already use
pip. It is also preferable for projects that have complex build dependencies (like C-extensions) where Poetry's resolver occasionally struggles.
Conclusion
Dependency hell is no longer an excuse for production instability. By adopting either Poetry's
poetry.lock or pip-tools'
requirements.txt, you guarantee that your software behaves consistently across all environments. The key takeaway is simple: never commit to a floating version constraint. Pin your dependencies, trust your lock files, and sleep easier at night.