Pipeline caching

[fnetX]

Clear and concise description of the problem

Caching is a issue often talked about. It would save a lot of resources if dependencies for projects are not built every time, but only run from a certain point on.

Suggested solution

There is already a feature request in #758 that might try to achieve something similar, but it sounds focused on results, not prepare steps.

My proposal is the following: For every pipeline stage, there is an optional "refresh: " parameter, that defines how often this needs to be rebuilt. For example, when dependencies change frequently, you can specifiy "refresh: 1d". If your prepare step is used seldom, you can rebuilt it every 14d.

If you install some system requirements (e.g. git, nodejs, your compiler, ...), you could specifiy to rebuild it every 30d, but the next pipeline that pulls your specific dependencies (npm, go packages etc) is refreshed every 12h.

The container state that is created at this point is than cached for the maximum specified time, maybe by simply tagging the docker image, just like Docker builds are cached when the context and the Dockerfile doesn't change.

The big question is how the clone is supposed to work there. Probably by defining prepare steps and clone the project afterwards. Or clone into a mount at the beginning and keep it even with different containers.

Alternative

The

Additional context

No response

Validations

• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

[reivilibre]

This proposal makes some kind of sense to me.

I find myself having to create special-purpose Docker containers with base dependencies for CI to run faster.
I've always felt a wish that I could just keep the instructions written in the CI YAML file but have the stages 'cached'.

There is another wish to have a way to cache package downloads/compilations (for e.g. Cargo), since that takes a long time on my CI and outweighs the build time of my project (also wasting energy/time/...). I tried setting up MinIO and sccache but it was pretty clunky and didn't work that well. I wish there was a nicely-integrated built-in method.

[anbraten]

Proposal

As caching needs to work for different backends (docker, kubernetes, local, ssh, ...) it would probably be the best to allow the backends to internally map the cache volume to their supported system.

config

steps:
  restore-cache:
    image: awesome-volume-cacher
    settings:
      cache_key: 123
    volumes:
      - name: cache
        path: /cache
  build:
    image: node
    commands:
      - npm install
    volumes:
      - name: cache
        path: /cache
  save-cache:
    image: awesome-volume-cacher
    settings:
      cache_key: 123
    volumes:
      - name: cache
        path: /cache
volumes:
  - name: cache

backend support

docker & kubernetes

• using container volumes (as suggested by Pipeline-scoped (temporary) volumes [solves caching, preparation steps, inter step communication, ...] #2272)
• with a prefix to restrict access to projects & prevent name colision with existing volumes

local & ssh

• maybe some ln (restricted to specific directories) pointing to a folder

problems & open questions

multiple agents support

There could be an issue that the volume doesn't exists on the agent. We could therefore add logic to the cache plugin or the backend directly to download non existing volume data from a s3 bucket.

space qutoas

There need to be a way to restrict volume sizes

cache cleanup / expiration

How can we easily define a volume expiration date

[zc-devs]

We could therefore add logic to the cache plugin or the backend directly to download non existing volume data from a s3 bucket.

To the cache plugin, please. It is deep implementation: S3, FTP, SCP, encryption (?) This is not Agent responsible for. Let Agent deal with volumes stuff.

[dvjn]

I have implemented a lot of caching for my woodpecker pipelines at work.

I feel, most of the cache requirements can be solved by a plugin, a volume mount, and a shared storage across agents (using an NFS server, or maybe S3FS?).

The only annoying thing that I encountered throughout the caching journey is that I have to mark all the pipelines trusted in which I want to mount the cache volume.

Just a thought: can the problem statement be reduced to "finding a better way to allow mounting cache volumes in caching plugins without needing to be trusted?"

[qwerty287]

#2272 might be a nice generic approach if you want something to be stored on the agent

How does this solve caching?

Caching is across all pipelines and should be kept, but this discussion is about stuff that's deleted after the pipeline finished if I get it right?

[zc-devs]

I have to mark all the pipelines privileged

Trusted?

caching plugins without needing to be privileged

Is it really privileged? Dont think Kubernetes requires privileged container.

[dvjn]

Trusted?

Yes, my bad. I meant trusted, not privileged.
I'll just update my original comment so that others don't get misinformed about the same.

[zc-devs]

Have you tried https://github.com/woodpecker-ci/plugin-s3 or https://github.com/drone-plugins/drone-s3?
What drawbacks did you experience?
Why do you need (separate) volume if plugin works (doesn't requires) special volume? I assume you can download/upload files/directories to/from workspace.

[dvjn]

What drawbacks did you experience?

Assuming you have 2 types of users for woodpecker, the administrators and the users.

This approach helps in off-loading caching infrastructure to the "administrators", by making the caching infrastructure a part of the agent config. Why? Because the administrators already have access to a platform (VMs, K8s, AWS, etc) which they used to deploy woodpecker, which can also be used to provision the caching infrastructure.

For the "users", it is much easier to use as they don't have to understand what infrastructure goes into making caching possible. They can directly use the caching plugin and voila everything works.

Another reason is that this makes the pipelines more portable across different woodpecker deployments.

These might not seem like big reasons, but they add up when working with multiple teams.

[zc-devs]

to install some system requirements
specific dependencies (npm, go packages etc)

I would suggest using as simple as a stick solution - pull through cache. It is completely independent from CI.

If you can't afford Artifactory or Nexus, you can set up Nginx proxy as well as specialized tools (goproxy, athens for go).

This is example config for Nginx and Go:

# /etc/nginx/conf.d/default.conf
proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=cache:10m max_size=10g inactive=7d use_temp_path=off;

server {
  listen        8080;
  listen        [::]:8080;
  server_name   go.proxy.local;
  resolver      1.1.1.1;

  # redirect server error pages to the static page /50x.html
  error_page   500 502 503 504  /50x.html;
  location = /50x.html {
    root /usr/share/nginx/html;
  }

  location / {
    proxy_pass                https://proxy.golang.org$request_uri;
    proxy_cache               cache;
    proxy_cache_valid         60m;
    proxy_cache_use_stale     error timeout http_500 http_502 http_503 http_504;
  }

  location ~* ".(tar|tgz|bz2|gz|lz|lz4|lzma|lzo|xz|zst|7z|jar|war|rar|zip)$"  {
    proxy_pass                https://proxy.golang.org$request_uri;
    proxy_cache               cache;
    proxy_cache_valid         7d;
    proxy_cache_lock          on;
    proxy_cache_lock_timeout  30s;
    proxy_cache_use_stale     error timeout http_500 http_502 http_503 http_504;
  }
}

This works for Node as well as for APK, DEB, RPM (with little adjustments). I believe it will work for other package managers like Cargo and Maven.

Woodpecker user DevOps have to set up proxy:

GOPROXY="http://go.proxy.local" # for golang
NPM_CONFIG_REGISTRY="http://node.proxy.local" # for Node

It would be even easier, when #3723 is merged.

[dvjn]

This is neat. Thanks.

[qwerty287]

That might work for network stuff, but not for e.g. a build cache.

[blm768]

A pattern that's worked really well for the (Jenkins-based) CI at my job is to build the Docker images for each CI step right there on the executor node as part of the pipeline. The local Docker cache then holds onto pretty much anything I might want to keep cached. That's probably not sufficiently sandboxed for Woodpecker, and it doesn't really solve multi-node caching, but for a setup without many nodes and a fairly high degree of trust, it seems to work pretty well.