Skip to content

syz-ci: switch to local git clones #6467

@a-nogikh

Description

@a-nogikh

Context

#6447 switched syz-cluster to local git clones. When we do git clone ./repo1 ./repo2, git object database is not actually copied, only hardlinks are created. Thus, even though we get a completely separate repository, the operation is very fast and takes no extra space.

We could use a similar approach in syz-ci.

Why

How

Currently, we have a fixed kernel checkout for each syz-manager and two kernel checkouts for two job processors. Instead, we could use disposable local checkouts:

  • Keep a checkout C_i for each manager.
  • When building a new kernel image:
    • Do a local clone of C_i, poll it and build a new kernel there.
    • If successful, switch to the new checkout and remove the old one.
  • When doing jobs:
    • Do a local clone of C_i (for the corresponding instance - jobs are always associated with a particular instance), fetch the target commit/tree.
    • After job is finished, destroy the checkout.

For a kernel checkout,

  • .git folder takes 8-9 GB.
  • Checked out source code takes 1.5 GB.
  • After a build, object files take ~ extra 13 GB in total.

We'll keep N per manager repositories (like we already do) and will have temporarily spikes of ~15GB during jobs or during building new kernel builds. Given that we serialize kernel builds, total disk usage should remain more or less the same.

Concerns

Fuchsia and Cuttlefish checkouts that are very massive and use submodules.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions