build container images automatically #1593

NUZAT-TABASSUM · 2025-10-29T14:24:14Z

This pull request adds a GitHub Actions workflow that automatically builds and publishes Tobira’s Docker images.

The workflow does the following:

Builds Tobira’s Docker image for both linux/amd64 and linux/arm64
Uses Node and Rust versions from util/scripts/build-container-image.sh
Builds and pushes images to GitHub Container Registry (GHCR) under ghcr.io/elan-ev/tobira
Automatically generates appropriate tags based on the Git event:
- For pull requests, it only builds the image (no pushing).
- For pushes to main, it builds and pushes development tags:
  edge → represents the latest code from main
  sha-<commit> → a unique tag for each commit,
- For version tags (e.g., v3.9.0), it creates release images with multiple tags: v3.9.0, v3.9, v3, and latest
Each architecture (amd64 and arm64) is built separately and then combined into a multi-architecture image, so users can simply pull ghcr.io/elan-ev/tobira:<tag> and Docker automatically selects the correct architecture.

KatrinIhler · 2025-11-05T16:24:17Z

This is related to opencast/opencast-docker#243 which is supposed to be a first step for opencast/opencast-docker#240.

@NUZAT-TABASSUM In your description, could you elaborate a bit more on what your workflow does?

NUZAT-TABASSUM · 2025-11-10T12:13:33Z

This is related to opencast/opencast-docker#243 which is supposed to be a first step for opencast/opencast-docker#240.

@NUZAT-TABASSUM In your description, could you elaborate a bit more on what your workflow does?

I’ve updated the PR description and added more details about what the workflow does.

LukasKalbertodt

Hi! Sorry for the delay responding to your PR.

I don't know a ton about Docker, certainly not about best practices. This is my first review of it, but which is mostly just questions to you, not necessarily change requests. It would be nice if you could explain the reasoning behind your changes for me to understand.

Generally the script seems to be quite involved, covering many different cases. I would have expected something simpler first. Maybe all of this makes sense, it's just my intuition to start with something with fewer bells and whistles.
Did you test all of this on your own fork? I.e. also test the upload part?
Can you quickly explain why you decided for ghcr.io instead of say, quay.io, which is used in build-container-image.sh?

Also summoning @mtneug who knows a lot more about these things. I know you don't have a lot of time right now, but maybe you want to have a look here anyway.

LukasKalbertodt · 2025-11-24T11:41:21Z

.github/workflows/build-container-image.yml

+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@v3


What is this used for?

This creates an environment for cross-platform builds.

So it is required for building the ARM build? Because Rust doesn't need QEMU to cross-compile to ARM. Not sure what Docker (or buildx or whatever) requires.

Yes, rust cross-compiling doesn’t need QEMU, but the Docker build still needs it to run package installs/scripts during the arm64 stages.

LukasKalbertodt · 2025-11-24T11:45:42Z

.github/workflows/build-container-image.yml

+          - name: arm64
+            platform: linux/arm64
+            rust_target: aarch64-unknown-linux-musl


Why build for ARM? 🤔 Tobira never released for ARM, was never tested for ARM and is very unlikely to be used on an ARM machine.

Why not? I build Opencast container images for ARM. Tobira is probably even easier to build for this platform.

(a) bc it wastes compute and storage resources if no one uses it. And (b) we would have to at least clearly document somewhere that this is experimental/not officially supported or whatever. People should know that Tobira devs never compile nor test it. So I'd rather leave it out, until someone asks for it. Or am I missing something?

Users on macOS are on arm. While you can run other architectures, Docker (and probably any other container engine) defaults to downloading arm images. I would always build them if not only for local testing.

LukasKalbertodt · 2025-11-24T11:49:08Z

.github/workflows/build-container-image.yml

+on:
+  pull_request:
+    branches: [ main ]
+  push:
+    branches: [ main ]          
+    tags:     [ 'v*' ]          


I would run this workflow a lot less often, specifically: just for tags (i.e. releases). Is there any particular reason why you also build for PRs and build+upload for main pushes?

Probably debatable if you want this, but container images for Opencast have {version}-dev images that build from the r/{version}.x release branch (and dev that builds from develop) to ease testing of upcoming versions. There are no images resulting from Opencast PRs.

LukasKalbertodt · 2025-11-24T11:50:40Z

.github/workflows/build-container-image.yml

+      - name: Build & push single-arch image
+        uses: docker/build-push-action@v6


Is using this action better than using build-container-image.sh? Would be nice to not duplicate the logic I guess? But maybe there are good reasons for not reusing the script?

Yes, I can do that too.

LukasKalbertodt · 2025-11-24T11:53:31Z

.github/workflows/build-container-image.yml

+          cache-from: type=gha
+          cache-to: type=gha,mode=max


What exactly is cached here? Is it actually worth it using a cache? In particular if we switch to only building releases.

These are BuildKit caches. This could speed up builds if Docker can verify that a step in the Dockerfile does not need to be performed again. I don't think this will improve much if only releases are built. Also, I'm a bit afraid that some layers with external changes, e.g. OS package installs, are cached indefinitely.

Those lines just turn on Docker BuildKit caching. It saves already-built Docker layers in GitHub Actions so the next build can reuse them and run faster.
This is mainly useful if we build often (PRs/main). If we switch to release-only builds, the cache probably won’t help much. I can remove it then.

LukasKalbertodt · 2025-11-24T11:54:51Z

.github/workflows/build-container-image.yml

+  manifest:
+    name: Publish multi-arch tags
+    needs: [meta, build]
+    if: github.event_name != 'pull_request'
+    runs-on: ubuntu-latest
+    steps:
+      - name: Set up Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Generate and publish multi-arch Docker manifest
+        run: |
+          for TAG in ${{ needs.meta.outputs.tags }}; do
+            echo "→ ${TAG}"
+            docker buildx imagetools create \
+              --tag "${TAG}" \
+              "${TAG}-amd64" \
+              "${TAG}-arm64"
+          done


I'm not quite sure what all of this does

This is creating a multi-platform image, e.g. pulling ghcr.io/elan-ev/tobira:latest will automatically pull the arm64 or amd64 image based on the host (if not set otherwise during the pull). I never used the imagetools subcommands. That control is interesting, especially since I had some trouble with ghcr.io in the past when deleting some tags of multi-platform images.

@NUZAT-TABASSUM are you sure this works? Are the ${TAG}-amd64 and ${TAG}-arm64 variants created as tags?

yes, its working

mtneug · 2025-11-24T12:47:41Z

.github/workflows/build-container-image.yml

+          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
+          tags: |
+            type=sha,format=short,prefix=sha-
+            type=raw,value=edge,enable=${{ github.ref == format('refs/heads/{0}', github.event.repository.default_branch) }}


Use dev instead of edge?

mtneug · 2025-11-24T12:49:24Z

.github/workflows/build-container-image.yml

+        with:
+          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
+          tags: |
+            type=sha,format=short,prefix=sha-


Do we want that? Git hashes as container tags?

I added the sha mainly for traceability as it is easy to say use the image from this exact commit when debugging or testing. If you don’t want that, I can disable it.

mtneug · 2025-11-24T12:51:16Z

.github/workflows/build-container-image.yml

+            type=semver,pattern={{version}},enable=${{ startsWith(github.ref, 'refs/tags/') }}
+            type=semver,pattern={{major}}.{{minor}},enable=${{ startsWith(github.ref, 'refs/tags/') }}
+            type=semver,pattern={{major}},enable=${{ startsWith(github.ref, 'refs/tags/') }}


Is this working? Tobira tags don't follow semver.

mtneug · 2025-11-24T12:53:16Z

.github/workflows/build-container-image.yml

+            rust_target: aarch64-unknown-linux-musl
+
+    steps:
+      - uses: actions/checkout@v4


There is now v6. Please check all versions again for current releases.

mtneug · 2025-11-24T12:55:45Z

.github/workflows/build-container-image.yml

+      - name: Extract Node and Rust versions
+        id: tools
+        run: |
+          nv=$(grep -E '^NODE_VERSION=' util/scripts/build-container-image.sh | head -n1 | cut -d= -f2)
+          rv=$(grep -E '^RUST_VERSION=' util/scripts/build-container-image.sh | head -n1 | cut -d= -f2)
+          echo "node=${nv}" >> "$GITHUB_OUTPUT"
+          echo "rust=${rv}" >> "$GITHUB_OUTPUT"


Is this the canonical place to look for the current versions? Maybe add a new file that contains this info for all scripts in the repository? Not sure what is best for Rust projects. The Opencast container build repo simply has VERSION_OPENCAST, VERSION_FFMPEG etc.

Good point. Yes I already noticed that stuff is kind of out of date. There is util/scripts/check-system.sh and build-container-images.sh. There are probably better ways to do it, honestly. Cargo.toml has a field to specify the minimum Rust version required, for example. Will look into it.

Right now it's reading the Node/Rust versions from build-container-image.sh just because they already live there, but I agree that’s not a great way. Could I create versions.env fiile that hold both Node and Rust versions in one shared place for scripts? while Cargo.toml only covers Rust and not Node.

mtneug · 2025-11-24T12:59:07Z

.github/workflows/build-container-image.yml

+            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ steps.vars.outputs.version }}-${{ matrix.name }}
+            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:sha-${{ steps.vars.outputs.short_sha }}-${{ matrix.name }}


Why set other tags than selected with docker/metadata-action?

I use docker/metadata-action for the final tags people will pull (latest, edge, release tags, etc.).
The extra tags I set manually (…-amd64 / …-arm64) are just temporary helper tags so the manifest step can combine both architectures into one multi-arch tag.

mtneug · 2025-11-24T13:05:31Z

.github/workflows/build-container-image.yml

+          cache-from: type=gha
+          cache-to: type=gha,mode=max


These are BuildKit caches. This could speed up builds if Docker can verify that a step in the Dockerfile does not need to be performed again. I don't think this will improve much if only releases are built. Also, I'm a bit afraid that some layers with external changes, e.g. OS package installs, are cached indefinitely.

mtneug · 2025-11-24T13:19:27Z

.github/workflows/build-container-image.yml

+  manifest:
+    name: Publish multi-arch tags
+    needs: [meta, build]
+    if: github.event_name != 'pull_request'
+    runs-on: ubuntu-latest
+    steps:
+      - name: Set up Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Generate and publish multi-arch Docker manifest
+        run: |
+          for TAG in ${{ needs.meta.outputs.tags }}; do
+            echo "→ ${TAG}"
+            docker buildx imagetools create \
+              --tag "${TAG}" \
+              "${TAG}-amd64" \
+              "${TAG}-arm64"
+          done


This is creating a multi-platform image, e.g. pulling ghcr.io/elan-ev/tobira:latest will automatically pull the arm64 or amd64 image based on the host (if not set otherwise during the pull). I never used the imagetools subcommands. That control is interesting, especially since I had some trouble with ghcr.io in the past when deleting some tags of multi-platform images.

@NUZAT-TABASSUM are you sure this works? Are the ${TAG}-amd64 and ${TAG}-arm64 variants created as tags?

NUZAT-TABASSUM · 2025-12-01T16:58:44Z

Hi! Sorry for the delay responding to your PR.

I don't know a ton about Docker, certainly not about best practices. This is my first review of it, but which is mostly just questions to you, not necessarily change requests. It would be nice if you could explain the reasoning behind your changes for me to understand.
* Generally the script seems to be quite involved, covering many different cases. I would have expected something simpler first. Maybe all of this makes sense, it's just my intuition to start with something with fewer bells and whistles.

* Did you test all of this on your own fork? I.e. also test the upload part?

* Can you quickly explain why you decided for ghcr.io instead of say, quay.io, which is used in `build-container-image.sh`?
Also summoning @mtneug who knows a lot more about these things. I know you don't have a lot of time right now, but maybe you want to have a look here anyway.

Hi Lukas — thanks a lot for the careful review.
I tried to cover the common “full workflow” in one go ((build, tag, push, and multi-arch).If you’d prefer I can start simpler (e.g. build/push only on release tags first).
Yes — I tested this on my fork on a main push, it pushed the images and created the multi-arch tags.
I used ghcr.io instead of quay.io as it was suggested from tobira group.

NUZAT-TABASSUM · 2025-12-02T18:03:07Z

Hi! I’ve updated the workflow based on the feedback. Could you please take another look and re-review when you have a time? Thanks!

NUZAT-TABASSUM added 3 commits October 29, 2025 14:11

build image automatically

a4a866a

fix invalid value of workflow issue

fa7a19c

fix invalid format of workflow issue

00d0d57

LukasKalbertodt requested changes Nov 24, 2025

View reviewed changes

mtneug suggested changes Nov 24, 2025

View reviewed changes

NUZAT-TABASSUM added 5 commits December 1, 2025 11:33

test

365e480

test ci

aa4c71a

test workflow

9cfb9c3

correct workflow

09d5d8b

update tobira image workflow

a65b12a

update tobira image workflow, add versions.env file

3a6eb78

		- name: Build & push single-arch image
		uses: docker/build-push-action@v6

		${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ steps.vars.outputs.version }}-${{ matrix.name }}
		${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:sha-${{ steps.vars.outputs.short_sha }}-${{ matrix.name }}

build container images automatically #1593

Are you sure you want to change the base?

build container images automatically #1593

Conversation

NUZAT-TABASSUM commented Oct 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

KatrinIhler commented Nov 5, 2025

Uh oh!

NUZAT-TABASSUM commented Nov 10, 2025

Uh oh!

LukasKalbertodt left a comment

Choose a reason for hiding this comment

Uh oh!

LukasKalbertodt Nov 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

NUZAT-TABASSUM commented Dec 1, 2025

Uh oh!

NUZAT-TABASSUM commented Oct 29, 2025 •

edited

Loading

LukasKalbertodt Nov 24, 2025 •

edited

Loading