It’s been some time since my last (and also first) status report about Toolbox. It is time to make another one. This time I’ll try to structure this blog post more by commenting on every release to give it a bit of a backstory. I will not comment on every single change in the changelog as that would be too long and too boring.

Toolbox is currently sitting at version 0.0.96 and it’s steadily getting to a 0.1.0 release. The updates include mainly bugfixes but at times a little feature, we thought is important enough, got in, too.

v0.0.92

Overview of changes:

This was another quick bugfix following v0.0.91.

Embed the version from Meson into the binary

This one was important for us to have a single place for tracking the version of Toolbox. Toolbox uses a build system called Meson even though Go does not really require one. It is mainly due to the fact that Toolbox requires several files to be present on the target system (e.g., man pages, bash-completion,..) and it’s easier to use a build system than just rely on distributors to “get it right”. Meson has support for specifying the version number of a software, but there’s no native support for Go in Meson. Therefore, we had to make use of Go’s functionality for embedding values into the source files at build time.

v0.0.93

Overview of changes:

v0.0.93 was a quite exciting release due to the long list of changes. Most of the changes were the “under the hood” type but few user-facing changes also got in (highlighting of containers or the fixed hint).

Ensure reproducible builds by using the -trimpath build flag

This is a nice thing for ensuring reproducible builds of Toolbox. Added in Go 1.13, text from the changelog: The new go build flag -trimpath removes all file system paths from the compiled executable, to improve build reproducibility.

Make listing of container and images more robust against changes in the JSON returned by Podman

This work was initiated by the fact that a set of small changes in Podman’s JSON output (e.g., field ‘Id’ renamed to ‘ID’) completely broke Toolbox (causing the program to “panic”). The new implementation should withstand now not just these subtle changes in format but also changes in the type of data stored in the JSON fields. Maybe a field in the output of ’toolbox list’ will be empty but it will not crash!

To prevent such bugs from happening, I recommending carefully reading the documentation and blog posts for the library.

Re-enable highlighting of running containers

This one was quite interesting to work on. Toolbox uses the text/tabwriter package for printing the tables of containers and images to the console. It’s shortcoming is it doesn’t support ANSI escape code. The solution was a hack that puts these codes on every line and uses a non-standard length for one of the sequences.

v0.0.94

Overview of changes:

I’d call v0.0.94 a “routine release” but I can’t because even though most changes are either test-related, project/repo-related or bugfixes, some of them are quite significant and important.

Add more information to errors from creating symbolic links when setting up the toolbox container in the entry point

Doesn’t sound like much but actually can significantly help when debugging problems during a toolbox’s start-up. Toolbox did not to print the underlining error and just printed the linking failed. No longer. And in Go it’s just a matter of updating fmt.Println() to look like this: fmt.Println("my error: %v", err) instead of fmt.Println("my error"). I still know of several places in the code where we don’t do this.

Ensure binaries build on Fedora 33 run on Fedoras 32 & 31

Toolbox is now written in Go, which is a compiled language. The final product is a binary called toolbox. Go by design statically links all the dependencies. There is but a single exception to this rule. Use of some libraries (e.g. net) will make the go build command use cgo. First line from docs of cgo: Cgo enables the creation of Go packages that call C code. The use of cgo cause the final binary to by dynamically linked to the system’s C library (aka libc). On most distributions is used glibc.

This normally is not a problem at all because when a distribution builds a package, it is used only on that version of the distribution for which the package was built (e.g. Fedora 32 packages are only used on Fedora 32 systems). Toolbox behaves differently. It mounts the host system’s Toolbox binary inside of a toolbox and then uses it as it’s entry-point. Inside of a toolbox the Toolbox binary links to the libc library and tries to operate. The problem is that glibc only guarantees backwards compatibility (e.g., Toolbox built on Fedora 31 will work on Fedora 32, 33) but not forward compatibility (e.g., Toolbox built on Fedora 33 will work on Fedora 32, 31). To fix this we could do 3 things:

  1. Not to use any C code
  2. Statically link the C library
  3. Mask out the new symbols

First option was rejected because the C code is much more powerful than it’s Go equivalent. Second option would inflate the final binary size. So, in the end we went with the third option of masking the new symbols. It is probably the hardest option as it involved playing with options for go build and playing a bit with assembler.

glibc uses “symbol versioning”

This may need to be revisited when a new version of glibc is released but until then we’re good :).

Mount a tmpfs at /tmp to match the host

Podman does not mount a tmpfs to /tmp in a container by default. So, we did it manually using the toolbox’s entry-point.

v.0.0.95

Overview of changes:

Very exciting release! The log holds only 4 items but all of them are quite significant.

Try to handle configuration files that’re absolute symlinks when the entry points sets up the container

Toolbox’s use command toolbox init-container as their entry-point. This command has several tasks. One of them is to create symlinks for some files (for directories we usually use bind-mounts). Until now init-container simply created a symlink that lead to it’s target but did not handle in any way cases when the target did not exist. This is quite common when the target of a symlink is another symlink that is absolute. A real example from a toolbox container:

/etc/resolv.conf is a symlink to /run/host/etc/resolv.conf
/run/host/etc/resolv.conf is a symlink to /run/systemd/resolve/stub-resolv.conf
/run/systemd/resolve/stub-resolv.conf does not exist

Toolbox now notices if the target is a symlink and checks if it is an absolute symlink. If it is, it tries to prepend /run/host to the target and links to that. This usually works unless there’s something more funky going on.

Overall, our recommendation for users/admins is to use relative symlinks. Then, usually, there is no intervention needed.

Unbreak ’enter’ on Fedora CoreOS

Toolbox used to work on Fedora CoreOS but at some point (I don’t know when exactly) it stopped. It was due to a discrepancy when ensuring a sudoers group is available in a toolbox. Toolbox used to explicitly add a group with --add-group option for podman create and the added group was taken from the host’s environment (e.g., on Fedora the group is wheel, on Debian/Ubuntu that group is sudo). For some reason Fedora CoreOS has both of these groups and due to the way the function is written, it first checks for the existence of the sudo group, while the important group was wheel.

Solution to this was to completely this explicit addition of the group and instead just set-up the group using the entry-point.

Unbreak ‘sudo’ inside toolbox containers with Podman 2.0.5

Podman v2.0.5 change slightly the behaviour of option --userns=keep-id. It started creating an entry for the user invoking podman create in /etc/passwd. Toolbox was not prepared for this because it was used to adding the user itself. The change involved adopting use of the usermod(8) utility alongside useradd(8).

Still, a bug was found in Podman itself. The option --userns=keep-id added an entry in /etc/passwd but did not add entry to /etc/group for the user’s group. This was fixed by Podman developers.

Warn if $TERM has no terminfo entry in the container

Not all Linux user’s use the same terminal emulator. Some use GNOME Terminal, some Tilix (both rely on VTE), some Alacritty and some something else. Point being that these emulators differ. Important thing for terminals is terminfo. Toolbox now throws an error if an appropriate terminfo is not available in a toolbox.

v.0.0.96

Overview of changes:

Not as exciting as v0.0.95 but it is very important nonetheless.

Fix containers with missing /media possibly due to a failed RPM transaction

Sometimes it would happen that an already existing toolbox broke after running sudo dnf upgrade, stopping it (or simply restarting your machine) and trying to enter it again. This is caused by a fail during an upgrade of the filesystem package. This was fixed by making Toolbox forgive non-existence of target directories.

Unbreak X11 applications with GNOME 3.38

GNOME Shell 3.38 stopped using an abstract unix socket for launching apps over XWayland. It now searches for them in their “traditional” location - /tmp. But /tmp is now a blank tmpfs in toolboxes since v.0.0.94. There seemed to be two solutions:

  1. Cherry-pick X11 related socket from the host and link/bind-mount them in /tmp in a toolbox
  2. Mount the host’s /tmp inside of a toolbox

In the end we went with the second option as it seemed to be the most beneficial and required the least amount of changes.

Testing

Few days ago my pull request, adding a number of tests for Podman testing functionality needed by Toolbox, was merged! This hopefully will prevent at least some breakage that could be caused by a change in Podman’s behaviour.

There’s also an upcoming effort to include Toolbox’s system tests in Podman’s gating tests in Fedora that prevent pushing an update when an error is detected. I don’t know when this will become a final thing but it definitely is on the radar.

Last thing is that there’s an excellent PR made by Juanje Ojeda that refactors Toolbox’s system tests. When this is merged, the tests could be made more advanced and reliable than they are now!

Community highlights

Shout out to all the contributors who helped us push Toolbox forward! Bug reports, pull requests - from documentation to code changes are all hugely appreciated!

Some names involved in these releases: Ryan McGuire, Leonard König, Juanje Ojeda, Mario Sebastian Chacon, Morten Linderud, Martin Pitt

I apologize if I left somebody out. Let me know in the comments :).

Summary

There’s is some upcoming work on the CLI UX to make it more intuitive, welcoming for new users and less limiting for experienced users. I’d like to write a post about that in the future, too, and get your opinions on our thoughts.

Toolbox is slowly getting to a v0.1.0 release! Few more iterations and I think we’ll be there. Thank you for using the tool and helping us with it!