Create python package toolbox using poetry + namespace

1Toolbox for data projects

Motivation

A (internal) python data-processing package is created because the high re-use rate for the data process modules, especially when the format of input are similar across different projects. This situation happens quite often within the organization.

Namespace

In order to avoid the dependency conflict between sub-packages, namespace is a common practice to split a big single big package across multiple directories. Namespace packages allow you to split the sub-packages and modules within a single package across multiple, separate distribution packages.

For example: if you have folder structure as below

toolbox/
    __init__.py
    subpackage_1/
        __init__.py
        ...
    subpackage_2/
        __init__.py
        ...
setup.py

you can split to 2 different repositories:

Repo1

toolbox-subpackage-1/
    setup.py
    toolbox/
        subpackage_1/
            __init__.py

Repo2

toolbox-subpackage-2/
    setup.py
    toolbox/
        subpackage_2/
            __init__.py

In this case, each sub-package can be separately installed, used, and versioned.

There are three ways to create namespace package:

1. native namespace packages.
2. pkgutil-style namespace packages.
3. pkg_resources-style namespace packages.

In my opinion, native namespace packages is the straight forward one, you just need to add the info in the setup.py like below

from setuptools import setup, find_namespace_packages
setup(
    name='toolbox-subpackage-1',
    ...
    packages=find_namespace_packages(include=['toolbox.*'])
)

It looks easy, right? BUT…there is a very big BUT when you want to use poetry for packaging. Because there is NO setup.py in packaging!!! There is only project.toml! The question then is, how to add namespace information in toml?

How to use poetry to develop the namespace

For example, I want to have 2 sub-packages : hello and goodbye within my BioInfoToolBox. I need to design the folder structure like below

Toolbox_subpackage1

Toolbox_subpackage2

And the most important, add this into the poject.toml file

And it will work as below

It looks like from the same package BioInfoToolBox! Isn’t it?

and you can see in site-packages, it is actually belong to the same package!

Have fun using poetry+namespace!