fix: let fewer conflicts occur across requirements

**tl;dr** Introduce a common umbrella constraints file (that works
across requirement extras) to avoid package version conflicts and
to reduce the need for manual pinning in `*.in` files.

Previously, spurious package version conflicts could sometimes occur
across requirements for `pip install -e .`, `pip install -e .[help]`,
`pip install -e .[playwright]`, and so on. Here’s why:

- There are five different requirement configs: the set of base
  requirements (`requirements.txt`) and four additional requirement sets\
  (aka "extras"): `dev`, `help`, `browser`, and `playwright`.

- Each of those five configurations spans its own tree of dependencies
  [1]. Those five trees can slightly overlap. (For example, `greenlet`
  is a transitive requirement for both the `help` and `playwright`
  trees, respectively.)

- If you want to resolve those dependency trees so you get concrete
  version numbers, you can’t just look at each tree independently.
  This is because when trees overlap, they sometimes pull in the same
  package for different reasons, respectively, and maybe with different
  version constraints.
  For example, the `help` tree pulls in `greenlet`, because `sqlalchemy`
  requires it. At the same time, the `playwright` tree also pulls in
  `greenlet` because it’s needed by the `playwright` package.
  Resolving those constraints strictly individually (i.e., per tree) is
  usually a mistake. It may work for a while, but occasionally you’re
  going to end up with two conflicting versions of the same package.

To prevent those version conflicts from occurring, the five
`pip-compile` invocations were designed as a chain.
The process starts at the smallest tree (i.e., the base
`requirements.in` file). It calculates the version numbers for the tree,
remembers the result, and feeds it into the calculation of the next
tree.

The chain design somewhat helped mitigate conflicts, but not always.
The reason for that is that the chain works like a greedy algorithm:
once a decision has been made for a given package in a tree, that
decision is immediately final, and the compilation process isn’t allowed
to go back and change that decision if it learns new information.
New information comes in all the time, because larger trees usually have
more complex constraints than smaller trees, and the process visits
larger trees later, facing additional constraints as it hops from tree
to tree. Sometimes it bumps into a new constraint against a package for
which it has already made a decision earlier (i.e., it has pinned the
concrete version number in the `requirements*.txt` file of an earlier
tree).

That’s why the greedy chain-based method, even though it mostly works
just fine, can never avoid spurious conflicts entirely.
To help mitigate those conflicts, pinning entries were manually added to
`requirements.in` files on a case-by-case basis as conflicts occurred.
Those entries can make the file difficult to reason about, and they must
be kept in sync manually as packages get upgraded. That’s a maintenance
burden.

Turning the chain into an umbrella may help. Instead of hopping from
tree to tree, look at the entire forest at once, calculate all the
concrete version numbers for all trees in one fell swoop, and save the
results in a common, all-encompassing umbrella file.

Armed with the umbrella file (called `common-constraints.txt`), visit
each tree (in any order – it no longer matters) and feed it just the
umbrella file as a constraint, along with its own `*.in` file as the
input.
Chaining is no longer necessary, because the umbrella file already
contains all version constraints for all the packages one tree could
possibly need, and then some.

This technique should reduce manual pinning inside `*.in` files, and
makes sure that computed version numbers no longer contradict each other
across trees.

[1]: From a graph theory point of view, I’m being blatantly incorrect
here; those dependency graphs are usually not trees, because they have
cycles. I’m still going to call them "trees" for the sake of this
discussion, because the word "tree" feels less abstract and intimidating
and hopefully more relatable.
This commit is contained in:
Claudia Pellegrino 2025-03-02 01:35:48 +01:00
parent a94c4b4ce4
commit 31c4198cee
No known key found for this signature in database
GPG key ID: 7AA67DE7B73139CE
12 changed files with 1133 additions and 412 deletions

View file

@ -2,234 +2,312 @@
# This file is autogenerated by pip-compile with Python 3.12
# by the following command:
#
# pip-compile --allow-unsafe --constraint=requirements.txt --output-file=requirements/requirements-dev.txt requirements/requirements-dev.in
# pip-compile --allow-unsafe --constraint=requirements/common-constraints.txt --output-file=requirements/requirements-dev.txt requirements/requirements-dev.in
#
alabaster==1.0.0
# via sphinx
# via
# -c requirements/common-constraints.txt
# sphinx
babel==2.17.0
# via sphinx
# via
# -c requirements/common-constraints.txt
# sphinx
build==1.2.2.post1
# via pip-tools
# via
# -c requirements/common-constraints.txt
# pip-tools
certifi==2025.1.31
# via
# -c /Users/gauthier/Projects/aider/requirements.txt
# -c requirements.txt
# -c requirements/common-constraints.txt
# requests
cfgv==3.4.0
# via pre-commit
# via
# -c requirements/common-constraints.txt
# pre-commit
charset-normalizer==3.4.1
# via
# -c /Users/gauthier/Projects/aider/requirements.txt
# -c requirements.txt
# -c requirements/common-constraints.txt
# requests
click==8.1.8
# via
# -c /Users/gauthier/Projects/aider/requirements.txt
# -c requirements.txt
# -c requirements/common-constraints.txt
# pip-tools
# typer
codespell==2.4.1
# via -r requirements/requirements-dev.in
# via
# -c requirements/common-constraints.txt
# -r requirements/requirements-dev.in
cogapp==3.4.1
# via -r requirements/requirements-dev.in
# via
# -c requirements/common-constraints.txt
# -r requirements/requirements-dev.in
contourpy==1.3.1
# via matplotlib
# via
# -c requirements/common-constraints.txt
# matplotlib
cycler==0.12.1
# via matplotlib
# via
# -c requirements/common-constraints.txt
# matplotlib
dill==0.3.9
# via
# -c requirements/common-constraints.txt
# multiprocess
# pathos
distlib==0.3.9
# via virtualenv
# via
# -c requirements/common-constraints.txt
# virtualenv
docutils==0.21.2
# via
# -c requirements/common-constraints.txt
# sphinx
# sphinx-rtd-theme
filelock==3.17.0
# via
# -c /Users/gauthier/Projects/aider/requirements.txt
# -c requirements.txt
# -c requirements/common-constraints.txt
# virtualenv
fonttools==4.56.0
# via matplotlib
# via
# -c requirements/common-constraints.txt
# matplotlib
identify==2.6.8
# via pre-commit
# via
# -c requirements/common-constraints.txt
# pre-commit
idna==3.10
# via
# -c /Users/gauthier/Projects/aider/requirements.txt
# -c requirements.txt
# -c requirements/common-constraints.txt
# requests
imagesize==1.4.1
# via sphinx
# via
# -c requirements/common-constraints.txt
# sphinx
imgcat==0.6.0
# via -r requirements/requirements-dev.in
# via
# -c requirements/common-constraints.txt
# -r requirements/requirements-dev.in
iniconfig==2.0.0
# via pytest
# via
# -c requirements/common-constraints.txt
# pytest
jinja2==3.1.5
# via
# -c /Users/gauthier/Projects/aider/requirements.txt
# -c requirements.txt
# -c requirements/common-constraints.txt
# sphinx
kiwisolver==1.4.8
# via matplotlib
# via
# -c requirements/common-constraints.txt
# matplotlib
lox==0.12.0
# via -r requirements/requirements-dev.in
# via
# -c requirements/common-constraints.txt
# -r requirements/requirements-dev.in
markdown-it-py==3.0.0
# via
# -c /Users/gauthier/Projects/aider/requirements.txt
# -c requirements.txt
# -c requirements/common-constraints.txt
# rich
markupsafe==3.0.2
# via
# -c /Users/gauthier/Projects/aider/requirements.txt
# -c requirements.txt
# -c requirements/common-constraints.txt
# jinja2
matplotlib==3.10.0
# via -r requirements/requirements-dev.in
# via
# -c requirements/common-constraints.txt
# -r requirements/requirements-dev.in
mdurl==0.1.2
# via
# -c /Users/gauthier/Projects/aider/requirements.txt
# -c requirements.txt
# -c requirements/common-constraints.txt
# markdown-it-py
multiprocess==0.70.17
# via pathos
# via
# -c requirements/common-constraints.txt
# pathos
nodeenv==1.9.1
# via pre-commit
# via
# -c requirements/common-constraints.txt
# pre-commit
numpy==1.26.4
# via
# -c /Users/gauthier/Projects/aider/requirements.txt
# -c requirements.txt
# -c requirements/common-constraints.txt
# contourpy
# matplotlib
# pandas
packaging==24.2
# via
# -c /Users/gauthier/Projects/aider/requirements.txt
# -c requirements.txt
# -c requirements/common-constraints.txt
# build
# matplotlib
# pytest
# sphinx
pandas==2.2.3
# via -r requirements/requirements-dev.in
# via
# -c requirements/common-constraints.txt
# -r requirements/requirements-dev.in
pathos==0.3.3
# via lox
# via
# -c requirements/common-constraints.txt
# lox
pillow==10.4.0
# via
# -c /Users/gauthier/Projects/aider/requirements.txt
# -c requirements.txt
# -c requirements/common-constraints.txt
# matplotlib
pip-tools==7.4.1
# via -r requirements/requirements-dev.in
# via
# -c requirements/common-constraints.txt
# -r requirements/requirements-dev.in
platformdirs==4.3.6
# via virtualenv
# via
# -c requirements/common-constraints.txt
# virtualenv
pluggy==1.5.0
# via pytest
# via
# -c requirements/common-constraints.txt
# pytest
pox==0.3.5
# via pathos
# via
# -c requirements/common-constraints.txt
# pathos
ppft==1.7.6.9
# via pathos
# via
# -c requirements/common-constraints.txt
# pathos
pre-commit==4.1.0
# via -r requirements/requirements-dev.in
# via
# -c requirements/common-constraints.txt
# -r requirements/requirements-dev.in
pygments==2.19.1
# via
# -c /Users/gauthier/Projects/aider/requirements.txt
# -c requirements.txt
# -c requirements/common-constraints.txt
# rich
# sphinx
pyparsing==3.2.1
# via matplotlib
# via
# -c requirements/common-constraints.txt
# matplotlib
pyproject-hooks==1.2.0
# via
# -c requirements/common-constraints.txt
# build
# pip-tools
pytest==8.3.4
# via
# -c requirements/common-constraints.txt
# -r requirements/requirements-dev.in
# pytest-env
pytest-env==1.1.5
# via -r requirements/requirements-dev.in
# via
# -c requirements/common-constraints.txt
# -r requirements/requirements-dev.in
python-dateutil==2.9.0.post0
# via
# -c /Users/gauthier/Projects/aider/requirements.txt
# -c requirements.txt
# -c requirements/common-constraints.txt
# matplotlib
# pandas
pytz==2025.1
# via pandas
# via
# -c requirements/common-constraints.txt
# pandas
pyyaml==6.0.2
# via
# -c /Users/gauthier/Projects/aider/requirements.txt
# -c requirements.txt
# -c requirements/common-constraints.txt
# pre-commit
requests==2.32.3
# via
# -c /Users/gauthier/Projects/aider/requirements.txt
# -c requirements.txt
# -c requirements/common-constraints.txt
# sphinx
rich==13.9.4
# via
# -c /Users/gauthier/Projects/aider/requirements.txt
# -c requirements.txt
# -c requirements/common-constraints.txt
# typer
roman-numerals-py==3.1.0
# via sphinx
# via
# -c requirements/common-constraints.txt
# sphinx
semver==3.0.4
# via -r requirements/requirements-dev.in
# via
# -c requirements/common-constraints.txt
# -r requirements/requirements-dev.in
shellingham==1.5.4
# via typer
# via
# -c requirements/common-constraints.txt
# typer
six==1.17.0
# via
# -c /Users/gauthier/Projects/aider/requirements.txt
# -c requirements.txt
# -c requirements/common-constraints.txt
# python-dateutil
snowballstemmer==2.2.0
# via sphinx
# via
# -c requirements/common-constraints.txt
# sphinx
sphinx==8.2.1
# via
# -c requirements/common-constraints.txt
# sphinx-rtd-theme
# sphinxcontrib-jquery
sphinx-rtd-theme==3.0.2
# via lox
# via
# -c requirements/common-constraints.txt
# lox
sphinxcontrib-applehelp==2.0.0
# via sphinx
# via
# -c requirements/common-constraints.txt
# sphinx
sphinxcontrib-devhelp==2.0.0
# via sphinx
# via
# -c requirements/common-constraints.txt
# sphinx
sphinxcontrib-htmlhelp==2.1.0
# via sphinx
# via
# -c requirements/common-constraints.txt
# sphinx
sphinxcontrib-jquery==4.1
# via sphinx-rtd-theme
# via
# -c requirements/common-constraints.txt
# sphinx-rtd-theme
sphinxcontrib-jsmath==1.0.1
# via sphinx
# via
# -c requirements/common-constraints.txt
# sphinx
sphinxcontrib-qthelp==2.0.0
# via sphinx
# via
# -c requirements/common-constraints.txt
# sphinx
sphinxcontrib-serializinghtml==2.0.0
# via sphinx
# via
# -c requirements/common-constraints.txt
# sphinx
typer==0.15.1
# via -r requirements/requirements-dev.in
# via
# -c requirements/common-constraints.txt
# -r requirements/requirements-dev.in
typing-extensions==4.12.2
# via
# -c /Users/gauthier/Projects/aider/requirements.txt
# -c requirements.txt
# -c requirements/common-constraints.txt
# typer
tzdata==2025.1
# via pandas
# via
# -c requirements/common-constraints.txt
# pandas
urllib3==2.3.0
# via
# -c /Users/gauthier/Projects/aider/requirements.txt
# -c requirements.txt
# -c requirements/common-constraints.txt
# requests
virtualenv==20.29.2
# via pre-commit
# via
# -c requirements/common-constraints.txt
# pre-commit
wheel==0.45.1
# via pip-tools
# via
# -c requirements/common-constraints.txt
# pip-tools
# The following packages are considered to be unsafe in a requirements file:
pip==25.0.1
# via
# -c /Users/gauthier/Projects/aider/requirements.txt
# -c requirements.txt
# -c requirements/common-constraints.txt
# pip-tools
setuptools==75.8.1
# via pip-tools
# via
# -c requirements/common-constraints.txt
# pip-tools