It’s been some time since my last (and also first) status report about Toolbox. It is time to make another one. This time I’ll try to structure this blog post more by commenting on every release to give it a bit of a backstory. I will not comment on every single change in the changelog as that would be too long and too boring.
Toolbox is currently sitting at version 0.0.96 and it’s steadily getting to a 0.1.0 release. The updates include mainly bugfixes but at times a little feature, we thought is important enough, got in, too.
Overview of changes:
- Embed the version from Meson into the binary
- Make it build on aarch64
- Fedora-specific fix
This was another quick bugfix following v0.0.91.
Embed the version from Meson into the binary
This one was important for us to have a single place for tracking the version of Toolbox. Toolbox uses a build system called Meson even though Go does not really require one. It is mainly due to the fact that Toolbox requires several files to be present on the target system (e.g., man pages, bash-completion,..) and it’s easier to use a build system than just rely on distributors to “get it right”. Meson has support for specifying the version number of a software, but there’s no native support for Go in Meson. Therefore, we had to make use of Go’s functionality for embedding values into the source files at build time.
Overview of changes:
- Ensure reproducible builds by using the -trimpath build flag
- Fix the test suite to work with the Go implementation
- Make listing of container and images more robust against changes in the JSON returned by Podman
- List out dependencies and installation instructions
- Re-enable highlighting of running containers
- Show the spinner only when connected to a terminal
- Speed things up by caching the Podman version at runtime
- Update hint after creating a container to use the new syntax
- Use the correct verb format for string
v0.0.93 was a quite exciting release due to the long list of changes. Most of the changes were the “under the hood” type but few user-facing changes also got in (highlighting of containers or the fixed hint).
Ensure reproducible builds by using the -trimpath build flag
This is a nice thing for ensuring reproducible builds of Toolbox. Added in Go 1.13, text from the changelog:
The new go build flag -trimpath removes all file system paths from the compiled executable, to improve build reproducibility.
Make listing of container and images more robust against changes in the JSON returned by Podman
This work was initiated by the fact that a set of small changes in Podman’s JSON output (e.g., field ‘Id’ renamed to ‘ID’) completely broke Toolbox (causing the program to “panic"). The new implementation should withstand now not just these subtle changes in format but also changes in the type of data stored in the JSON fields. Maybe a field in the output of ‘toolbox list’ will be empty but it will not crash!
Re-enable highlighting of running containers
This one was quite interesting to work on. Toolbox uses the text/tabwriter package for printing the tables of containers and images to the console. It’s shortcoming is it doesn’t support ANSI escape code. The solution was a hack that puts these codes on every line and uses a non-standard length for one of the sequences.
Overview of changes:
- Add contribution guidelines
- Add fedora-toolbox image definition for Fedora 34
- Add more information to errors from creating symbolic links when setting up the toolbox container in the entry point
- Ensure binaries build on Fedora 33 run on Fedoras 32 & 31
- Install the tests
- Make it more obvious when falling back to /bin/bash
- Document that sudo(8) should work without a password
- Mount a tmpfs at /tmp to match the host
- Update issue templates
I’d call v0.0.94 a “routine release” but I can’t because even though most changes are either test-related, project/repo-related or bugfixes, some of them are quite significant and important.
Add more information to errors from creating symbolic links when setting up the toolbox container in the entry point
Doesn’t sound like much but actually can significantly help when debugging problems during a toolbox’s start-up. Toolbox did not to print the underlining error and just printed the linking failed. No longer. And in Go it’s just a matter of updating
fmt.Println() to look like this:
fmt.Println("my error: %v", err) instead of
fmt.Println("my error"). I still know of several places in the code where we don’t do this.
Ensure binaries build on Fedora 33 run on Fedoras 32 & 31
Toolbox is now written in Go, which is a compiled language. The final product is a binary called
toolbox. Go by design statically links all the dependencies. There is but a single exception to this rule. Use of some libraries (e.g.
net) will make the
go build command use
cgo. First line from docs of cgo:
Cgo enables the creation of Go packages that call C code. The use of
cgo cause the final binary to by dynamically linked to the system’s C library (aka
libc). On most distributions is used glibc.
This normally is not a problem at all because when a distribution builds a package, it is used only on that version of the distribution for which the package was built (e.g. Fedora 32 packages are only used on Fedora 32 systems). Toolbox behaves differently. It mounts the host system’s Toolbox binary inside of a toolbox and then uses it as it’s entry-point. Inside of a toolbox the Toolbox binary links to the libc library and tries to operate. The problem is that
glibc only guarantees backwards compatibility (e.g., Toolbox built on Fedora 31 will work on Fedora 32, 33) but not forward compatibility (e.g., Toolbox built on Fedora 33 will work on Fedora 32, 31). To fix this we could do 3 things:
- Not to use any C code
- Statically link the C library
- Mask out the new symbols
First option was rejected because the C code is much more powerful than it’s Go equivalent. Second option would inflate the final binary size. So, in the end we went with the third option of masking the new symbols. It is probably the hardest option as it involved playing with options for
go build and playing a bit with assembler.
glibc uses “symbol versioning”
This may need to be revisited when a new version of
glibc is released but until then we’re good :).
Mount a tmpfs at /tmp to match the host
Podman does not mount a tmpfs to /tmp in a container by default. So, we did it manually using the toolbox’s entry-point.
Overview of changes:
- Try to handle configuration files that’re absolute symlinks when the entry points sets up the container
- Unbreak ‘enter’ on Fedora CoreOS
- Unbreak ‘sudo’ inside toolbox containers with Podman 2.0.5
- Warn if $TERM has no terminfo entry in the container
Very exciting release! The log holds only 4 items but all of them are quite significant.
Try to handle configuration files that’re absolute symlinks when the entry points sets up the container
Toolbox’s use command
toolbox init-container as their entry-point. This command has several tasks. One of them is to create symlinks for some files (for directories we usually use bind-mounts). Until now
init-container simply created a symlink that lead to it’s target but did not handle in any way cases when the target did not exist. This is quite common when the target of a symlink is another symlink that is absolute. A real example from a toolbox container:
/etc/resolv.conf is a symlink to /run/host/etc/resolv.conf /run/host/etc/resolv.conf is a symlink to /run/systemd/resolve/stub-resolv.conf /run/systemd/resolve/stub-resolv.conf does not exist
Toolbox now notices if the target is a symlink and checks if it is an absolute symlink. If it is, it tries to prepend
/run/host to the target and links to that. This usually works unless there’s something more funky going on.
Overall, our recommendation for users/admins is to use relative symlinks. Then, usually, there is no intervention needed.
Unbreak ‘enter’ on Fedora CoreOS
Toolbox used to work on Fedora CoreOS but at some point (I don’t know when exactly) it stopped. It was due to a discrepancy when ensuring a sudoers group is available in a toolbox. Toolbox used to explicitly add a group with
--add-group option for
podman create and the added group was taken from the host’s environment (e.g., on Fedora the group is
wheel, on Debian/Ubuntu that group is
sudo). For some reason Fedora CoreOS has both of these groups and due to the way the function is written, it first checks for the existence of the
sudo group, while the important group was
Solution to this was to completely this explicit addition of the group and instead just set-up the group using the entry-point.
Unbreak ‘sudo’ inside toolbox containers with Podman 2.0.5
Podman v2.0.5 change slightly the behaviour of option
--userns=keep-id. It started creating an entry for the user invoking
podman create in
/etc/passwd. Toolbox was not prepared for this because it was used to adding the user itself. The change involved adopting use of the
usermod(8) utility alongside
Still, a bug was found in Podman itself. The option
--userns=keep-id added an entry in
/etc/passwd but did not add entry to
/etc/group for the user’s group. This was fixed by Podman developers.
Warn if $TERM has no terminfo entry in the container
Not all Linux user’s use the same terminal emulator. Some use GNOME Terminal, some Tilix (both rely on VTE), some Alacritty and some something else. Point being that these emulators differ. Important thing for terminals is terminfo. Toolbox now throws an error if an appropriate
terminfo is not available in a toolbox.
Overview of changes:
- Don’t break GNU Readline’s ctrl-p shortcut
- Enable system tests on Fedora 33
- Fix containers with missing /media possibly due to a failed RPM transaction
- Give access to the udev database
- Unbreak X11 applications with GNOME 3.38
- Update default release to 31 for non-Fedora hosts
Not as exciting as v0.0.95 but it is very important nonetheless.
Fix containers with missing /media possibly due to a failed RPM transaction
Sometimes it would happen that an already existing toolbox broke after running
sudo dnf upgrade, stopping it (or simply restarting your machine) and trying to enter it again. This is caused by a fail during an upgrade of the
filesystem package. This was fixed by making Toolbox forgive non-existence of target directories.
Unbreak X11 applications with GNOME 3.38
GNOME Shell 3.38 stopped using an abstract unix socket for launching apps over XWayland. It now searches for them in their “traditional” location -
/tmp is now a blank
tmpfs in toolboxes since v.0.0.94. There seemed to be two solutions:
- Cherry-pick X11 related socket from the host and link/bind-mount them in
/tmpin a toolbox
- Mount the host’s
/tmpinside of a toolbox
In the end we went with the second option as it seemed to be the most beneficial and required the least amount of changes.
Few days ago my pull request, adding a number of tests for Podman testing functionality needed by Toolbox, was merged! This hopefully will prevent at least some breakage that could be caused by a change in Podman’s behaviour.
There’s also an upcoming effort to include Toolbox’s system tests in Podman’s gating tests in Fedora that prevent pushing an update when an error is detected. I don’t know when this will become a final thing but it definitely is on the radar.
Last thing is that there’s an excellent PR made by Juanje Ojeda that refactors Toolbox’s system tests. When this is merged, the tests could be made more advanced and reliable than they are now!
Shout out to all the contributors who helped us push Toolbox forward! Bug reports, pull requests - from documentation to code changes are all hugely appreciated!
Some names involved in these releases: Ryan McGuire, Leonard König, Juanje Ojeda, Mario Sebastian Chacon, Morten Linderud, Martin Pitt
I apologize if I left somebody out. Let me know in the comments :).
There’s is some upcoming work on the CLI UX to make it more intuitive, welcoming for new users and less limiting for experienced users. I’d like to write a post about that in the future, too, and get your opinions on our thoughts.
Toolbox is slowly getting to a v0.1.0 release! Few more iterations and I think we’ll be there. Thank you for using the tool and helping us with it!