CourtBouillon

💕💕💕

This article is part of a series of tearful articles about Python packaging:

1. The Can of Worms
2. The Roots of Evil
3. Delusions of Formats
4. Files Everywhere
5. The Toolbox
6. The Expression of Needs
7. The Minimal Solution

Before starting, we would like to send a lot of love to the members of the PyPA team. We complain a lot in this series, but we have a lot of respect for the sisyphean work already done.

That being said, let’s start (again) the whining 😭.

💕💕💕

What’s the Problem?

If you are here, we can assume that you’ve already written some lines of Python. You’ve already installed a Python project, with dependencies. And for that, you may have had to do some stuff. A lot of stuff. Way too much stuff. Not so much? We bet that you’ve already encountered things called pip, pipenv, poetry, setuptools, distutils, requirements.txt, setup.py, Pipfile, setup.cfg, pyproject.toml, venv, virtualenv, wheel… We can stop here, but you know many more, without even realizing.

That’s too much, and that’s not normal.

A sentence from The Zen of Python says "There should be one—and preferably only one—obvious way to do it". You see the problem now, right?

The Horror of the Beginnings

You may not remember, but at some point you didn’t know about Python. Yes, it’s true.

Try to explain to someone at ease behind a keyboard (but who doesn’t know Python) how to install a Python program. In his or her place, as an average neophyte of the XXI^e century, we would expect something like:

1. Install Python by following the super short tutorial from the super documentation, or with the command download-python-and-install-it.
2. Install the program by clicking here, or with the command python-install my-program.

Sorry to crush your dreams: reality is not like that. Not at all. The "super short tutorial from the super documentation" to install Python doesn’t exist. And too many python-install programs exist, all incompatible with each other, and not included in Python. That’s why installing a program is unpleasant.

Yeah: 🤦. We agree.

Wow, Such Tutorial

Speaking about tutorials, one of the best sources is, indubitably, The Hitchhiker’s Guide to Python. A lot of things can always be said about documentation, but let’s bring a touch of optimism: the guide is translated in several languages, and some parts are really well written in a rather didactic way.

We won’t talk about the fact that it’s not the official documentation. We won’t talk about the fact that the whole guide is a not-so-well-hidden advertisement for the glory of the projects of its arguable author. No, don’t insist, we won’t do that.

Problem: this tutorial is not short, and not simple. You’re using macOS? Here is the first paragraph:

Before installing Python, you’ll need to install GCC. GCC can be obtained by downloading Xcode, the smaller Command Line Tools (must have an Apple account) or the even smaller OSX-GCC-Installer package.

We save you from the hell 🔥 of the installation on Windows, where more or less wobbly solutions are listed. Python is in the Store (that’s what Microsoft says), in Chocolatey (that’s what the Hitchhiker’s Guide says in English, but not in French), on python.org (that’s what python.org says, but not its documentation).

What’s the best solution? We don’t know. In fact there’s none. And that’s very bad.

Wow, Such Installer

OK, the first point was kind of a disaster. But… Maybe we can do better for the second one? Installing a package after installing Python should be a child’s play. We do pip install my-package, don’t we?

We don’t.

Maybe sudo pip install my-package?

No. No. No. No. No. No. No. No.

To install a package, you need to create a virtualenv, but, well, not always. To create a virtualenv, you’re going to use a module integrated in Python, but not always. And after that, we’re going to use a program integrated in Python, but not always. And after that, the package can be installed, but not always.

If you don’t want to complicate things, you can follow what the Hitchhiker’s Guide says and use Pipenv (we let you guess who created this wonderful tool). Of course you have to disregard the fact that Pipenv has been deserted during a year and a half in 2019 with a broken version and no update. Of course, you have to turn a blind eye to the fact that the project is sprawling, includes about forty (40!) duplicated (and sometimes modified) projects, and contains more than 200.000 lines of code 😱…

Hahahahahahaha. Haha. No.

The Loneliness of Package Creation

And we’ve just installed a package. If you want to create and share a package, summon up your patience, expect other surprises.

Let’s be honest: a tutorial and a guide exist to learn how to create packages. These documents are rather well written and should quickly help you to share your library. In theory, we’re taken by the hand. In theory.

In reality, things are way more chaotic. If we look at only three projects (Flask, Requests and NumPy), we have a long list of files more or less related to package managment, without counting the classic README, LICENSE, and so on:

• setup.py,
• setup.cfg,
• MANIFEST.in,
• Pipfile,
• Pipfile.lock,
• tox.ini,
• pytest.ini,
• .coveragerc,
• requirements.txt,
• pyproject.toml…

Bad news: Python doesn’t provide tools to create a new project ready to be shared. You’re going to be lost in third-party projects that create packages their own way (we’re still looking askance at you, Pipenv), or you’re going to learn the hard way what these weird files are used for.

(Small funny note: setup.py, setup.cfg, requirements.txt and pyproject.toml allow, among others, to list the dependencies of a project. They use 4 different file formats, and one of them is not even managed by the Python standard library. True story.)

The different tutorials and online forums will only give you fragmented (and often obsolete) information. And there’s a reason: good practice is always changing. These tools have often been developed with no specifications, with documentation at best incomplete (we are really thinking about you, setuptools). Everybody suggests ideas, new tools regularly appear, everything quickly looks like a fashion moment that becomes outmoded after a few years (even a few months if you’re unlucky)…

We ear you defending your beloved language: "Yes, but this stuff isn’t that complicated, blah blah blah blah…". Do you want to cry? Go on the Cargo documentation, Rust’s package manager. The first page is the installation page, it contains 136 words (according to wc), including commands. It allows you to install Rust, with Cargo. 136 words, it’s one less than the Zen of Python. For real.

The second page gives you everything needed to create a simple package, with a tool that creates all the files. It contains the source code, the metadata file, the commands used and their results, a tree of files and folders in ASCII-art. Everything in 183 words.

There we are. It is possible. You know that, now.

And After That?

With Python, you’re going to learrn things randomly from your online reading, on doubtful blogs like this one. You’re going to find absolutely outdated solutions on StackOverflow, unspecified tricks using implementation details which will change next week. Do we include tests in packages, or not? Do we put them in the module, or not? Do we supply a source package, or not? Do we indicate exact versions of the dependencies, or not? Do we put the documentation in a folder aside, or not? So many questions, and many more, for which you’re in the deep loneliness of packages creation.

You probably think that everything is lost, that the people who developed everything are incompetent, that Python is in fact a language rotten beyond repair… Actually, no. We’ll try in the next episodes to understand how we got there, and how things changed during all those years, often for the best. Will we, one day, succeed in creating a nice package? Of course, we will.

To be continued…