Skip to main content Brad's PyNotes

PEP 420 - Implicit Namespace Packages

TL;DR

PEP 420 introduced implicit namespace packages, allowing Python packages to be split across multiple directories without requiring an __init__.py file. The import machinery automatically discovers and combines all portions of the package, enabling flexible distribution and avoiding file conflicts.

Interesting!

Namespace packages have a dynamically computed __path__ that automatically updates when sys.path changes. This means you can add new directories containing package portions after the package has already been imported, and Python will seamlessly incorporate them.

What Problem Did This Solve?

Before PEP 420, creating namespace packages required manual __path__ manipulation using pkgutil.extend_path() or setuptools’ proprietary declare_namespace(). Each approach had limitations:

  • Every package portion needed its own __init__.py file
  • Multiple vendors installing portions into the same directory would create file conflicts
  • The mechanisms were inconsistent and required boilerplate code

PEP 420 eliminated these issues by making namespace packages a native feature of the import system.

How Namespace Packages Work

When Python searches for a package, it follows this process:

  1. Check each directory in sys.path for package_name/__init__.py (regular package)
  2. Check for a module file like package_name.py
  3. If neither exists but a package_name/ directory exists, record it
  4. If at least one directory was found, create a namespace package

Here’s an example directory structure:

code snippet start

/usr/lib/python3/site-packages/
    company/
        project_a/
            __init__.py
            module_a.py

/home/user/.local/lib/python3/site-packages/
    company/
        project_b/
            __init__.py
            module_b.py

code snippet end

With no __init__.py in the company directories, Python treats company as a namespace package spanning both locations:

python code snippet start

import company.project_a.module_a  # Works
import company.project_b.module_b  # Also works
print(company.__path__)
# ['usr/lib/python3/site-packages/company', '/home/user/.local/lib/python3/site-packages/company']

python code snippet end

Key Differences from Regular Packages

Namespace packages have unique characteristics:

  • No __init__.py file: The defining feature
  • No __file__ attribute: Since there’s no physical file representing the package
  • Dynamic __path__: An iterable that’s recomputed when accessing submodules
  • Multiple locations: Can span directories across different file systems

python code snippet start

import company

# Namespace packages lack __file__
try:
    print(company.__file__)
except AttributeError:
    print("No __file__ attribute")  # This runs

# But they have __path__
print(type(company.__path__))  # <class '_NamespacePath'>

python code snippet end

When to Use Namespace Packages

Namespace packages excel in these scenarios:

Plugin architectures: Different packages can contribute to the same namespace

code snippet start

myapp-core/
    myapp/
        __init__.py  # Regular package
        core.py

myapp-plugin-auth/
    myapp/
        plugins/     # Namespace package (no __init__.py)
            auth.py

myapp-plugin-db/
    myapp/
        plugins/     # Namespace package
            database.py

code snippet end

Large organizations: Teams can develop separate portions independently

Distribution flexibility: Users can install only the portions they need

Performance Considerations

Regular packages (with __init__.py) are still preferable when you control the entire package. The PEP notes: “If it is known that a package will never be split across directories, a regular package is preferred for performance.”

Namespace packages involve additional filesystem scanning during import, though this overhead is typically negligible for most applications.

Migration from Legacy Namespace Packages

If you’re using older namespace package implementations, PEP 420 provides a smoother path:

python code snippet start

# Old style (pkgutil)
# company/__init__.py
from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

# PEP 420 style
# No company/__init__.py needed at all!

python code snippet end

Just remove the __init__.py files from namespace package directories. However, be aware that mixing legacy and PEP 420 portions has limitations on dynamic path computation.

Namespace packages transformed Python’s packaging ecosystem by making distributed packages simple and conflict-free. Whether you’re building plugin systems or managing large codebases across teams, PEP 420 provides the flexibility to structure code without artificial constraints.

Python modules tutorial | PEP 621 packaging metadata

Reference: PEP 420 - Implicit Namespace Packages