Distributed Cross-compiling using Icecream
This blog post is a guide on using Icecream for both distributed compiling and distributed cross compiling to speed up package compilation times on your linux machines. Even though the commands used are specific to Arch Linux based systems, this guide can be adopted to any Linux distribution.
I'm a developer for EndeavourOS ARM. I have to build some ARM SBC specific kernels to support those devices. But compiling Linux kernel on the Raspberry Pi takes about 2 hours. I wanted to reduce this compile time. A common way to do it is to use distributed cross-compiling. This means that we will use an x86_64 machine in addition to the ARM computer to build packages. Most people use distcc to perform this task. There are a few issues with distcc including my inability to make ccache work (ccache is a compiler cache which further improves build times), made me look for an alternative and thus I found
There are a few issues I have with
- You have to edit the configuration in all the computers in the cluster if even if you're adding a single new node. This makes the configurations hard to maintain and to scale. There is no need to do this in
icecream. We just have to configure the new node being added. The node automatically gets detected by other computers in the cluster.
- There is no load scheduler in
distcci.e. the jobs get distributed to all computers in the order decided by the configuration even if the computer listed first in the config is already under heavy load. In
icecreamthere is a centralized scheduler. This scheduler decides which computer to send the job to depending on several factors including node speed and availability. From
Icecream uses a central server that dynamically schedules the compile jobs to the fastest free server.
- I wasn't able to get ccache working with
distccwhen I'm using
makepkg(the tool used to build packages on Arch Linux based systems). I tried reading several wikis and tutorials and couldn't figure it out. Then Peter from CachyOS told me that
icecreamsupports ccache and it's easier to configure it. This was the main reason I started exploring
I hope that this guide will help you get started on your distributed compiling/cross-compiling endeavour with
Lets go through the initial setup of
- Install icecream on all the computers you want to include in the compilation cluster. On Arch Linux based systems
icecreamis available in the AUR.
yay -S icecream
Optionally install these monitoring tools (that are also available in the AUR):
icemon: a GUI monitor tool or
icecream-sundae: a TUI monitor tool.
2. Once you install icecream, choose a computer that you want to act as a scheduler and then start and enable
icecream-scheduler.service on that computer.
sudo systemctl enable --now icecream-scheduler.service
- After this, add the ip of the scheduler to the configuration file on the computers you want to be included in the cluster. The configuration file for
icecreamcan be found on
/etc/icecream.conf. In that file edit this line by adding the ip.
ICECREAM_SCHEDULER_HOST="scheduler ip here"
- In the same configuration file you can optionally configure the max number of jobs that can be allocated to that node parallelly. The default number used by
icecreamis the number of cores of the node.
- After finishing the configuration, now start and enable
sudo systemctl enable --now icecream.service
Now you have
icecream-scheduler running on your network. But you can't start using it for building packages before more configuration that I will show in the next section.
If you want to cross compile using
icecream use the instructions in the next subsection. If all the nodes in your cluster are of the same architecture, you can skip it.
Configure toolchain for cross compiling
The most common cross-compiling scenario is using
x64 computers to compile for
aarch64 since ARM computers are generally not powerful. To compile
x64 you first need to configure a cross-compiler toolchain. There is an AUR package that does most of it for you. You can install it by running
yay -S distccd-alarm-armv8
Creating icecream environment
Next you need to create an
icecream environment to tell
icecream that it has to use the cross-compiler instead of the normal x64 compiler. That can be done using
/usr/lib/icecream/bin/icecc-create-env --gcc /opt/x-tools8/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc /opt/x-tools8/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-g++
This will create a tarball with the environment, which you then have to copy over to each ARM computer where you will run jobs from. If you want to use your local node for compiling you need to also generate a icecream environment for that. This step is not given in the official documentation but the local node wouldn't take part in compiling otherwise. You can do that by using:
Next, you need to configure the
path in your computer to tell it to use icecream for compiling instead of the inbuilt compiler. This configuration will also setup
icecream to use
ccache. So first install
yay -S ccache
Then modify your path like so:
export CCACHE_PREFIX=icecc export PATH=~/scripts:$PATH export PATH=/usr/lib/icecream/bin:$PATH export PATH=/usr/lib/ccache/bin/:$PATH export ICECC_VERSION=<path to local icecream environment>,x86_64:<path to cross icecream environment>
Finally you need to configure
makepkg to run more job parallelly. This can be done by editing
/etc/makepkg.conf like so:
N is the number of jobs. A rule of thumb here is to set it as number of total threads available in the cluster.
Congratulations on getting this far. Now you can use
makepkg to build packages using
icecream. You can monitor it using either
icecream-sundae that you may have installed earlier.
icemon screen looks like this:
One of the common issues is the firewall blocking the ports for
icecream so make sure that this port is open on all nodes.
UDP Source Port 8765
I hope you enjoy your newfound power to build packages quickly even on less powerful computers. One advantage on the above setup is that even building of AUR packages using AUR helpers like
paru will now use distributed compiling which will make your normal updates faster.