CourtBouillon

💕💕💕

This article is part of a series of tearful articles about Python packaging:

1. The Can of Worms
2. The Roots of Evil
3. Delusions of Formats
4. Files Everywhere
5. The Toolbox
6. The Expression of Needs
7. The Minimal Solution

Before starting, we would like to send a lot of love to the members of the PyPA team. We complain a lot in this series, but we have a lot of respect for the sisyphean work already done.

That being said, let’s start (again) the whining 😭.

💕💕💕

Where Do We Put What?

We saw in the previous article what we can put in our packages. That’s a lot of stuff to include, but we haven’t really defined where or how we can include it.

As this article is just an example among others, we won’t have any qualms to propose purely subjective opinions based on a peculiar sensitivity. It’s not the absolute truth, it’s just an opinion. But, of course, as it’s our own, it’s a good one, otherwise you would be reading something else.

The Wheel Package

The wheel package is the level 1 of packaging. It’s not only the simplest to create, but also the one that’ll be the most used.

The goal of this package is to provide the functional code already ready for the target. The pip command that will install it will have nothing to do apart from decompressing the archive, retrieving some information (like dependencies) and putting the decompressed folder on the disk. No compilation, no arbitrary Python code execution, no special case for a particular platform.

Incidentally, this means that if your code isn’t the same depending on the Python version or the operating system, you’ll have to create a different package for each combination on which your creation will be installed. So, you’ll have to work a little bit.

It doesn’t matter: we have all the tools we need to do that. What is important is to agree on what we want to put inside. On this subject, our opinion is simple: code, metadata, and maybe one or two files like the README or the license.

Nothing else.

No need to cry. Who installs a wheel to look inside at the CHANGELOG? Who installs a wheel to launch unit tests? Who installs a wheel to rebuild the documentation of the module? If you already did that, the time has come for you to discover an amazing tool called Internet. It’s full of nice pages presenting the documentation with colors, pictures and even links.

More seriously: the only purpose of wheels is to be installed. No matter how they work, what they contain, the most important thing is that their installation works properly with pip. Everything else is secondary. Really. Goodbye tests, goodbye documentation, goodbye configuration files.

The Source Package

Sources are the basis, the alternative, the last resort, the ultimate comprehensiveness, and that’s why wheels will never send into oblivion this venerable format that has been supported by Python users since the beginning of package sharing.

Before defining what we’ll put inside, let’s settle a crucial issue right away: what will our source package be really used for? The answer is simple:

1. reading sources,
2. creating packages for other package managers (for example, for Linux distributions),
3. installing packages, for hopeless cases.

For these cases, it’s often useful to have more information than what’s included in the wheel. We like to have some files to discover changes between versions, some tests to see how it works, some documentation that can be read with a text editor… We may want to see how the package is configured, to play with the setup.py file, or even to try to modify the code.

So, the source package will contain more or less the same content as the versioned repository. We have, on broad terms, to retrieve what we’d retrieved with git clone (or the equivalent command of your favorite version control software, no offense). Of course, we’ll remove the versioning data and some details like the CI configuration.

Let’s Go

With all these details, we’re now able to venture into package creation. We won’t explain each line of configuration or code to write, but we’ll try, at least, to lay the necessary foundation for the creation of your packages.

Simple Package, Simple Solution

We’ll get rid, with only a few words, of a huge unspoken fact that clutters us. You want to build a package? Let’s say that your code contains only Python, and that you want to follow the rules we’ve settled before. There, it’s already better.

Do we agree? Otherwise you can console yourself with another article, somewhere on Internet, about setuptools. We leave that to you.

The Tool

Without further suspense, the tool we’ll use is Flit.

Flit is the hammer of package creation. It’s limited, it only does one thing, but it’s clear, limpid, efficient. We want to create and share packages with simple and foolish rules, that’s all.

Flit is also one of the tools behind PEP 517 and PEP 518. Yes, it’s thanks to its creator Thomas Kluyver that now we can be free from setuptools and setup.py. Respect.

Flit is simplicity above all. If you don’t want to ask questions about package creation, if you don’t want to write more code than your module, it’s the choice of reason.

The Architecture

Forget the tons of files and endless configurations. Here, we aim simplicity, we’ll have to eradicate the obesity of root folders. We’re going to drastically lighten the home page of your repository.

In the past, we took examples of staggering overdoses. Without going back into all the projects we’ve already talked about, let’s take only one example of what we don’t want: Requests.

(Requests isn’t the ultimate evil, don’t make us say what we didn’t write. It’s just a good example of want we don’t want.)

The root folder contains 22 files and folders. Among them, we find the usual suspects of package creation: setup.py, setup.cfg, MANIFEST.in, Pipfile… We also find nice configuration files for third-party tools: Tox, Coverage, Pytest. Some metadata, some folders, here’s something to impress a person who would like to use this famous project as an example to understand how Python modules work.

In contrast to this project, we suggest 6 basic folders and files to include at the root of your folder:

1. the folder containing the code of your project,
2. a doc folder,
3. a tests folder,
4. a LICENSE file,
5. a README file,
6. a pyproject.toml file.

Of course, this is only a basis that you can adapt to your needs. But restricting ourselves to keep a light and clean root folder is also a good reason to think about the hygiene and the structure of projects. Let’s see what we can put in these little boxes…

The Folders

In the folder containing the code of your project, you’ll first put… the code of your project. This seems to be obvious, but if we follow our idea of having a minimal wheel, we quickly understand that this folder will be the one ready to be decompressed. For the simple goal of installing the module, the rest is just decoration.

A consequence of this division is that this folder has to include the additional files required for the module. Pictures for your game? Include them in this folder. A lists of famous passwords for your NSA hacking tool? Include them in this folder.

That also explains why we don’t include tests or documentation in this folder. Everybody knows that tests and documentation are useless when the code is limpid and bug-free. However, in doubt, until all humans are omniscient, we could keep these relics, but only in the source package.

Tests, if they follow the naming conventions of your favorite tool, will be automatically discovered. For this purpose, tests seems to be an appropriate name, for humans and for tools (Flit or Pytest, for example). Then, up to you to organize your tests as you want, but you’ll help everyone by putting them in the same folder, at the root of your project.

The documentation has, also, good reasons to be put in the source package. You give to the curious people the possibility to dig into the clear explanations about your project, next to your code, without Internet access requirement. You give to the Linux distributions the possibility to include an appreciable introduction to your tool, but also to optional manual pages. In fact, you give the possibility to anyone to do anything with content that helps people, and in this case you can never be safe from pleasant surprises.

This documentation is also the perfect place to store some information that we may want to put in the root, like a CHANGELOG or configuration samples. So, they’ll be available in a pleasant format, in addition to text files. Code hosting platforms also offer dedicated pages to some of this information, so that a lot of root files become unnecessary. Nothing stops you, if you really want to, to put links into your README file to guide people who only look at the root folder.

The Files

The README file is the basis of your project. In pure text format or with a light markup, it’s the entry point of the majority of people interested in code. There’s also a good chance that the file is brought to the fore by your software forge and by PyPI.

That’s a good reason to work on your README file. Beyond the project description, you have to point all the necessary information we like to quickly have when we discover a project: the license, the supported versions of Python, the way to contact people taking care of the project…

By the way, putting the license at the root of the project, in a specific file, is very classic but questionable. After all, couldn’t this legal information be in the documentation? Wouldn’t a line in the README be enough to indicate which license applies to the project?

Yes, probably. But many tools expect to find this file at the root, and some of them even read it to deduct the license of the project. If it’s easy to change the habits of people, who’ll be happy with an indication in the documentation, it’s more complex to change the habits of machines. So… Let’s say this choice is a small agreement between ideals and reality. Let’s work to ensure that in some years we can get rid of this more simply.

At last, the main dish: pyproject.toml. This file allows you to indicate everything required to create a package. The default choices of Flit being really good (objectively), you shouldn’t need to change a lot of things to the proposed values. But you should know that if you want to, you’ll find a long list of options that will satisfy all your crazy ideas.

With Flit, pyproject.toml is going to replace what you can do with (at least) setup.py, setup.cfg, requirements.txt and MANIFEST.in. Of course, possibilities are limited, just because you can’t write Python code to commit atrocities executed during the creation or the installation of a package. But it’s not a limitation, it’s a feature: stop playing with this idea, interesting at the beginning but becoming more than filthy. It may be more useful for the posteriority to write your module.

This file will also allow you to configure most of the third-party tools you use: Tox, Black, Pytest, Coverage.py, isort, Pylint… Yes, it means that you can say goodbye to this pile of configuration files, each using its own naming conventions and formats! The list of supported tools is getting bigger, don’t hesitate to have a look time to time to see if your favorite project dared to take the plunge.

From Creation To Deployment

Don’t expect a tutorial where we hold your hand, write your configuration files, and give you all the tricks to use Flit. The title of the article isn’t "7 things you don’t know about Flit—the 5^th one will surprise you."

Why? Simply because the Flit’s documentation is amazing. You’ll find all you need to install and use Flit (almost) blindly. It’s limpid, it’s quick, and mainly it works.

init, install, build, publish. It’s all you need to model you little package with love. No more need to suffer our derogatory prose, we leave you among the delicate words of a delicate tool.

Enjoy flying on your own, let the wind .

Finally…

Don’t lie to ourselves: Flit doesn’t solve everything. We already saw its limitations and its small wicknesses, but unfortunately there’s more.

Do you want another tool? Poetry can do the same as Flit, but it does more: managing virtual environments, solving dependencies, installing packages, numbering versions… If you like all-in-one tools that avoid the pitfall of becoming a tentacled and megalomaniac behemoths impossible to maintain (we won’t give names), you can find in Poetry an elegant substitute to Pipenv (oops, sorry).

But… whether it’s on Flit or Poetry, there’s a big shadow: abandonment. Flit and Poetry are widely used, but they stay third-party tools that aren’t as supported as setuptools is. As well as many open-source projects, they already go through some problems, and there will be other ones.

Fortunately, PEPs are now widely adopted, leaving the door open to other future tools. Out of the shackles of setuptools, we can use other tools based on pyproject.toml. The keys and the values of options will change, but at least we won’t need to depend on a unique implementation stuck by the weight of legacy and the need of backwards compatibility.

Wheels are wheels, sources are sources, and they lived happily ever after.

Tools change, but formats remain.