Implement POSIX jobserver protocol to coordinate parallel builds

[tiran]

Abstract

The POSIX jobserver protocol, also known as GNU make jobserver protocol, is a system to coordinate parallel jobs. It is basically a semaphore. The job server writes X characters into a pipe. Each clients reads one character from the pipe, starts a build process, and finally writes the same character back into the pipe when the process has finished.

Proposal

Fromager could provide an optional POSIX jobserver and set up MAKEFLAGS accordingly. The modern FIFO-based implementation is easy to set up with a few lines of code:

def jobserver(sem: pathlib.Path, jobs: int) -> tuple[str, int, int]:
    sem.unlink(missing_ok=True)
    os.mkfifo(sem)
    rfd = os.open(sem, os.O_RDONLY | os.O_NONBLOCK)
    wfd = os.open(sem, os.O_WRONLY | os.O_NONBLOCK)
    os.write(wfd, jobs * b"+")
    makeflags =  f" -j{jobs} --jobserver-auth=fifo:{sem}"
    return makeflags, rfd, wfd

Open questions

• How to make sure that Gnu Make, Ninja, and CMake use the job server?
• How do deal with memory_per_job_gb build option? Should we treat jobs with the option as exclusive? Fromager could "steal" jobs from the job server before it starts a build with memory_per_job_gb, then give jobs back at the end of the build.
• Should every build task acquire a job from the job server, too?

Downstream changes

Make and Ninja in RHEL 9 are too old. We'd have to update to latest versions with custom builds:

• Ninja added job server support in 1.13, RHEL 9 has 1.10.2. According to the documentation, only FIFO is supported on POSIX.
• Make added FIFO job server in 4.44, RHEL 9 has make 4.43

References

• https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html
• https://ninja-build.org/manual.html#_gnu_jobserver_support
• NVCC added job server support with CUDA 13.0, https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/#jobserver-jobserver -
• https://github.com/ninja-build/ninja/blob/v1.13.1/misc/jobserver_pool.py
• https://cgit.git.savannah.gnu.org/cgit/make.git/tree/src/posixos.c