This PEP proposes a backward-compatible two-phase transition process to speed up, simplify and robustify installing from the pypi.python.org (PyPI) package index. To ease the transition and minimize client-side friction, no changes to distutils or existing installation tools are required in order to benefit from the first transition phase, which will result in faster, more reliable installs for most existing packages.
The first transition phase implements easy and explicit means for a package maintainer to control which release file links are served to present-day installation tools. The first phase also includes the implementation of analysis tools for present-day packages, to support communication with package maintainers and the automated setting of default modes for controlling release file links. The first phase also will default newly-registered projects on PyPI to only serve links to release files which were uploaded to PyPI.
The second transition phase concerns end-user installation tools, which shall default to only install release files that are hosted on PyPI and tell the user if external release files exist, offering a choice to automatically use those external files. External release files shall in the future be registered together with a checksum hash so that installation tools can verify the integrity of the eventual download (PyPI-hosted release files always carry such a checksum).
Alternative PyPI server implementations should implement the new simple index serving behaviour of transition phase 1 to avoid installation tools treating their release links as external ones in phase 2.
When PyPI went online, it offered release registration but had no facility to host release files itself. When hosting was added, no automated downloading tool existed yet. When Phillip Eby implemented automated downloading (through setuptools), he made the choice to allow people to use download hosts of their choice. The finding of externally-hosted packages was implemented as follows:
See the easy_install documentation for a complete description of this behavior. [1]
Today, most packages indexed on PyPI host their release files on PyPI. Out of 29,117 total projects on PyPI, only 2,581 (less than 10%) include any links to installable files that are available only off-PyPI. [2]
There are many reasons [3] why people have chosen external hosting. To cite just a few:
Irrespective of the present-day validity of these reasons, there clearly is a history why people choose to host files externally and it even was for some time the only way you could do things. This PEP takes the position that there remain some valid reasons for external hosting even today.
Today, python package installers (pip, easy_install, buildout, and others) often need to query many non-PyPI URLs even if there are no externally hosted files. Apart from querying pypi.python.org’s simple index pages, also all homepages and download pages ever specified with any release of a package are crawled by an installer. The need for installers to crawl external sites slows down installation and makes for a brittle and unreliable installation process. Those sites and packages also don’t take part in the PEP 381 mirroring infrastructure, further decreasing reliability and speed of automated installation processes around the world.
Most packages are hosted directly on pypi.python.org [2]. Even for these packages, installers still crawl their homepage and download-url, if specified. Many package uploaders are not aware that specifying the “homepage” or “download-url” in their package metadata will needlessly slow down the installation process for all users.
Relying on third party sites also opens up more attack vectors for injecting malicious packages into sites using automated installs. A simple attack might just involve getting hold of an old now-unused homepage domain and placing malicious packages there. Moreover, performing a Man-in-The-Middle (MITM) attack between an installation site and any of the download sites can inject malicious packages on the installation site. As many homepages and download locations are using HTTP and not HTTPS, such attacks are not hard to launch. Such MITM attacks can easily happen even for packages which never intended to host files externally as their homepages are contacted by installers anyway.
There is currently no way for package maintainers to avoid external-link crawling, other than removing all homepage/download url metadata for all historic releases. While a script [4] has been written to perform this action, it is not a good general solution because it removes useful metadata from PyPI releases.
Even if the sites referenced by “Homepage” and “Download-URL” links were not scraped for further links, there is no obvious way under the current system for a package owner to link to an installable file from a long_description metadata field (which is shown as package documentation on /pypi/PKG) without installation tools automatically considering that file a candidate for installation. Conversely, there is no way to explicitly register multiple external release files without putting them in metadata fields.
These are the goals to be achieved by implementation of this PEP:
The first transition phase introduces a “hosting-mode” field for each project on PyPI, allowing package owners explicit control of which release file links are served to present-day installation tools in the machine-readable simple/ index. The first transition will, after successful hosting-mode manipulations by individual early-adopters, set a default hosting mode for existing packages, based on automated analysis. Maintainers will be notified one month ahead of any such automated change. At completion of the first transition phase, all present-day existing release and installation processes and tools are expected to continue working. Any remaining errors or problems are expected to only relate to installation of individual packages and can be easily corrected by package maintainers or PyPI admins if maintainers are not reachable.
Also in the first phase, each link served in the simple/ index will be explicitly marked as rel="internal" if it is hosted by the index itself (even if on a separate domain, which may be the case if the index uses a CDN for file-serving). Any link not so marked will be considered an external link.
In the second transition phase, PyPI client installation tools shall be updated to default to only install rel="internal" packages unless a user specifies option(s) to permit installing from external links. See second transition phase for details on how installers should behave.
Maintainers of packages which currently host release files on non-PyPI sites shall receive instructions and tools to ease “re-hosting” of their historic and future package release files. This re-hosting tool MUST be available before automated hosting-mode changes are announced to package maintainers.
The foundation of the first transition phase is the introduction of three “modes” of PyPI hosting for a package, affecting which links are generated for the simple/ index. These modes are implemented without requiring changes to installation tools via changes to the algorithm for generating the machine-readable simple/ index.
The modes are:
Thus the hope is that eventually all projects on PyPI can be migrated to the pypi-explicit mode, while preserving the ability to install release files hosted externally via installer tools. Deprecation of hosting modes to eventually only allow the pypi-explicit mode is NOT REGULATED by this PEP but is expected to become feasible some time after successful implementation of the transition phases described in this PEP. It is expected that deprecation requires a new process to deal with abandoned packages because of unreachable maintainers for still popular packages.
The proposed solution consists of multiple implementation and communication steps:
For the second transition phase, maintainers of installation tools are asked to release two updates.
The first update shall provide clear warnings if externally-hosted release files (that is, files whose link does not include rel="internal") are selected for download, for which projects and URLs exactly this happens, and warn that in future versions externally-hosted downloads will be disabled by default.
The second update should change the default mode to allow only installation of rel="internal" package files, and allow installation of externally-hosted packages only when the user supplies an option.
The installer should distinguish between verifiable and non-verifiable external links. A verifiable external link is a direct link to an installable file from the PyPI simple/ index that includes a hash in the URL fragment (“#hashtype=hashvalue”) which can be used to verify the integrity of the downloaded file. A non-verifiable external link is any link (other than those explicitly supplied by the user of an installer tool) without a hash, scraped from external HTML, or injected into the search via some other non-PyPI source (e.g. setuptools’ dependency_links feature).
Installers should provide a blanket option to allow installing any verifiable external link. Non-verifiable external links should only be installed if the user-provided option specifies exactly which external domains can be used or for which specific package names external links can be used.
When download of an externally-hosted package is disallowed by the default configuration, the user should be notified, with instructions for how to make the install succeed and warnings about the implication (that a file will be downloaded from a site that is not part of the package index). The warning given for non-verifiable links should clearly state that the installer cannot verify the integrity of the downloaded file. The warning given for verifiable external links should simply note that the file will be downloaded from an external URL, but that the file integrity can be verified by checksum.
Alternative PyPI-compatible index implementations should upgrade to begin providing the rel="internal" metadata and the <meta name="api-version" value="2"> tag as soon as possible. For alternative indexes which do not yet provide the meta tag in their simple/ pages, installation tools should provide backwards-compatible fallback behavior (treat links as internal as in pre-PEP times and provide a warning).
New distribution URLs may be submitted by performing a HTTP POST to the URL:
With the following form-encoded data:
| Name | Value |
| :action | The string “urls” |
| name | The package name as a string |
| version | The release version as a string |
| new-url | The new URL to store |
| submit_new_url | The string “yes” |
The POST must be accompanied by an HTTP Basic Auth header encoding the username and password of the user authorized to maintain the package on PyPI.
The HTTP response to this request will be one of:
| Code | Meaning | URL submission implications |
| 200 | OK | Everything worked just fine |
| 400 | Bad request | Data provided for submission was malformed |
| 401 | Unauthorised | The username or password supplied were incorrect |
| 403 | Forbidden | User does not have permission to update the package information (not Owner or Maintainer) |
Phillip Eby for precise information and the basic ideas to implement the transition via server-side changes only.
Donald Stufft for pushing away from external hosting and offering to implement both a Pull Request for the necessary PyPI changes and the analysis tool to drive the transition phase 1.
Marc-Andre Lemburg, Alyssa Coghlan and catalog-sig in general for thinking through issues regarding getting rid of “external hosting”.
This document has been placed in the public domain.
Source: https://github.com/python/peps/blob/main/peps/pep-0438.rst
Last modified: 2025-02-01 08:59:27 UTC