As part of the team that maintains several testing tools for Ubuntu, including checkbox, I sometimes find myself needing to build .deb packages from our source tree.
A simple way of achieving this is of course to run dpkg-buildpackage or even bzr-buildpackage. Assuming all build-deps are correctly installed in the host system, this will result in a nicely built set of .debs.
This approach has a few caveats, in that it’s different from the build process actually employed to create the packages that ultimately get uploaded to Ubuntu (or even the ones available in Launchpad PPAs).
The two main differences are that Launchpad builds the packages in a “clean” environment, installing build-deps from scratch, whereas dpkg-buildpackage will rely on what’s installed in the system. So if you miss specifying a build-dep, your local build may work because you have it installed, but the PPA build will fail because it will not be present.
The second big difference is that with the local approach, you’re “limited” to building packages for the “host” system. Sure, you can specify a different target release in your debian/changelog file, but some aspect of your build may be tied to your system’s tools, versions and layout, and if for some reason they don’t match the actual target at installation time, things will fail in interesting ways.
Clearly, one way to test what the Launchpad build process will spit out is to build a source package and dput that to be built directly on a PPA. The problem here is that the feedback loop becomes excruciatingly slow; PPAs are a shared resource and build times can go from minutes to many hours.
Based on all this, it makes sense to try to use a local build environment that more closely replicates what PPAs do to build your packages.
Fortunately, the PPA builders use free software, so it’s relatively easy to do local builds in a similar environment, completing quickly due to use of local resources, and only upload to Launchpad once you’re pretty sure your build will succeed.
The software in question is sbuild, and I already wrote a post detailing how to install sbuild and set up a build environment for any Ubuntu release you need.
This setup worked fine for the occasional package build when you know packaging is mostly correct. For a fast build such as checkbox, setting up the build environment with all needed packages and build-deps takes about 10 minutes (depending mostly on download speed for all the packages). Of course on a more complex package, compilation time may start to be a factor.
Anyway, the 10-minute time can be too slow if you’re trying to fix a tricky problem and need a fast feedback loop. Plus the process produces a lot of transient files and downloads a set of packages many times, so there’s plenty of room for improvement here.
Speeding up local package installation and build
A large part of the time spent doing the “local” part of the process is writing files to disk. One way to speed this up is to use a ramdisk to store the build. I’m too lazy and have too little RAM to use this approach, so the alternative was setting up eatmydata inside the chroot. Since these are mostly temporary files or throwaway packages, it’s OK to lose the safety of constant syncs in exchange for a huge boost in speed.
The setup for eatmydata inside the chroot is described here. This looks a bit hard to automate, but luckily we don’t have to, as recent versions of mk-sbuild simply support a –eatmydata parameter, if given this will install eatmydata inside the chroot and do the choot config file change to enable eatmydata.
Adding PPA
You can add a custom PPA to an image. Once the chroot image is built, enter the “golden master”:
sudo schroot -c source:saucy-amd64 -u root
You can add a deb line (get it from launchpad) to your sources:
cat >>/etc/apt/sources.list.d/something.list
# Copy line here
Then you need to get the GPG key for the PPA and add it manually with the very basic tools provided in the chroot (sorry, no apt-add-repository):
apt-key add -
# Paste GPG armored key here
Then exit the golden image. After this, your builds from this chroot will be able to fetch packages from the PPA.
Again, that’s a bit of work to do for each VM. Instead, what I did was create a file in /etc/schroot/setup.d to do this automatically. You can of course replace the PPAs you need in the echo lines at the end. Name the file something like 81add-ppas:
#!/bin/sh
set -e
. "$SETUP_DATA_DIR/common-data"
. "$SETUP_DATA_DIR/common-functions"
. "$SETUP_DATA_DIR/common-config"
echo "$STAGE" >>/tmp/stages
if [ $STAGE = "setup-start" ] || [ $STAGE = "setup-recover" ]; then
echo "APT::Get { AllowUnauthenticated "1"; };" > $CHROOT_PATH/etc/apt/apt.conf.d/80unauthenticate
info "ADDING PPAS"
SLD_PATH="${CHROOT_PATH}/etc/apt/sources.list.d/roadmr.list"
. $CHROOT_PATH/etc/lsb-release
MY_RELEASE=$DISTRIB_CODENAME
[ -n "$MY_RELEASE" ] || MY_RELEASE=trusty
echo "# Added by the schroot setup mechanism (roadmr)" > $SLD_PATH
echo "deb http://ppa.launchpad.net/checkbox-dev/ppa/ubuntu $MY_RELEASE main" > $SLD_PATH
echo "deb http://ppa.launchpad.net/ubuntu-sdk-team/ppa/ubuntu $MY_RELEASE main" > $SLD_PATH
fi
Notice that again, I was very lazy and instead of downloading the gpg keys as shown above (as for some reason trying to run gpg from the setup script didn’t work), I just configured apt to allow unauthenticated packages. Since this sbuild is mainly for testing purposes it’s not a big deal to skip this verification step. Also, there’s some logic to automatically detect the chroot release, so the same config file works equally well for any Ubuntu release.
Apt-cacher-ng
As the name suggests, this nifty utility will cache packages so the next time you need them they’ll be fetched from local storage rather than from the network. A bit of config is needed to have sbuild download packages from here.
First, install apt-cacher-ng on the host system. You can verify it’s listening on port 3142 by any means you like.
Then, to set it up automatically in chroots, add this to the host system’s /etc/schroot/setup.d/80apt-cacher-ng (rather, create that file; it doesn’t exist by default):
#!/bin/sh
set -e
. "$SETUP_DATA_DIR/common-data"
. "$SETUP_DATA_DIR/common-functions"
. "$SETUP_DATA_DIR/common-config"
if [ $STAGE = "setup-start" ] || [ $STAGE = "setup-recover" ]; then
echo "# Added by the schroot setup mechanism (roadmr)" > "${CHROOT_PATH}/etc/apt/apt.conf.d/80proxy"
echo "Acquire::http::Proxy \"http://127.0.0.1:3142\";" >> "${CHROOT_PATH}/etc/apt/apt.conf.d/80proxy"
fi
With these two setup.d scripts and the –eatmydata magic, it’s easy to create sbuild environments which will be much faster when building packages.
As a comparison, building msmtp (chosen because this tests mainly the speedup components, not needing any packages from a PPA) takes about 40 seconds with these suggested tweaks:
Build Architecture: amd64
Build-Space: 5948
Build-Time: 17
Distribution: trusty
Host Architecture: amd64
Install-Time: 12
Job: msmtp_1.4.31-1.dsc
Machine Architecture: amd64
Package: msmtp
Package-Time: 40
Source-Version: 1.4.31-1
Space: 5948
Status: successful
Version: 1.4.31-1
─────────────────────────────────────────────────
Finished at 20140320-1301
Build needed 00:00:40, 5948k disc space
Whereas on a non-tweaked chroot it takes about 1:38 minutes:
Build Architecture: amd64
Build-Space: 5568
Build-Time: 17
Distribution: trusty
Host Architecture: amd64
Install-Time: 31
Job: msmtp_1.4.31-1.dsc
Machine Architecture: amd64
Package: msmtp
Package-Time: 98
Source-Version: 1.4.31-1
Space: 5568
Status: successful
Version: 1.4.31-1
──────────────────────────────────────────────────Finished at 20140320-1310
Build needed 00:01:38, 5568k disc space
It looks like they’re about 3 times faster, but that’s misleading because I deliberately chose a small, quick-to-compile package. Still, you can at least reduce network and disk access very easily now. Note, also, that my test system has a fast SSD. Speedup on a traditional rotary magnetic hard-disk is likely to be much higher.