The Python Software Foundation, with support from the Open Source Security Foundation, has published a comprehensive white paper addressing the long-standing issue of ‘phantom dependencies’ in open source software. These dependencies, which are not tracked through standard packaging metadata, manifest in a way that makes them invisible to vulnerability scanners, compliance tools, and policy enforcement mechanisms. The solution proposed by the foundation is PEP 770, which introduces an approach for packages to provide Software Bill-of-Materials (SBOM) metadata through a directory of files, ensuring backward compatibility and minimal disruption for existing systems.
According to the white paper, PEP 770 is designed to be backward-compatible and can be enabled by default by tools, meaning most projects won’t need to manually opt in to begin generating valid SBOM metadata. This approach is not limited to Python, as other software ecosystems face similar challenges with phantom dependencies. The foundation is eager to share its solution with other ecosystems, and is open to providing guidance on how to implement and adapt this methodology. The white paper concludes with a helpful note, emphasizing that other open source packaging ecosystem maintainers have raised similar concerns, and the foundation is willing to assist with the adoption and implementation of this approach.
Endor Labs’ 2023 dependencies report further highlights the severity of the issue, naming Python as one of the most affected ecosystems. Several factors contributed to Python’s susceptibility to phantom dependencies, including its ability to interface with non-Python software through methods like the C-API and FFI. Python’s role as the premier language for scientific computing and AI means many high-performance libraries written in system languages need to be accessed from Python code. Additionally, the use of ‘wheel’ for distributing packages, which involves unzipping a zip file into a directory without a compilation step, requires pre-compilation of all languages into binaries before installation.