-
Notifications
You must be signed in to change notification settings - Fork 1.3k
Description
Context
#6447 switched syz-cluster to local git clones. When we do git clone ./repo1 ./repo2, git object database is not actually copied, only hardlinks are created. Thus, even though we get a completely separate repository, the operation is very fast and takes no extra space.
We could use a similar approach in syz-ci.
Why
- Reduce negative impact from errors like syzbot:
failed to symbolize reporterrors #6466 - Avoid having git repositories with tons of remote origins (like we do for the repositories that handle
#syz testjobs or bisections).
How
Currently, we have a fixed kernel checkout for each syz-manager and two kernel checkouts for two job processors. Instead, we could use disposable local checkouts:
- Keep a checkout C_i for each manager.
- When building a new kernel image:
- Do a local clone of C_i, poll it and build a new kernel there.
- If successful, switch to the new checkout and remove the old one.
- When doing jobs:
- Do a local clone of C_i (for the corresponding instance - jobs are always associated with a particular instance), fetch the target commit/tree.
- After job is finished, destroy the checkout.
For a kernel checkout,
.gitfolder takes 8-9 GB.- Checked out source code takes 1.5 GB.
- After a build, object files take ~ extra 13 GB in total.
We'll keep N per manager repositories (like we already do) and will have temporarily spikes of ~15GB during jobs or during building new kernel builds. Given that we serialize kernel builds, total disk usage should remain more or less the same.
Concerns
Fuchsia and Cuttlefish checkouts that are very massive and use submodules.