-
Notifications
You must be signed in to change notification settings - Fork 112
Description
We are running a custom Tekton Hub instance on OpenShift. After the api pod has been running for a while, it becomes unusable: running any command inside the container (e.g., ps, ls, or even spawning a shell) fails with the following error:
sh: can't fork: Resource temporarily unavailable
This strongly suggests a process leak inside the Tekton Hub api pod. These processes are likely being spawned but not properly reaped or terminated. In this state the Tekton Hub api pod can not fetch current tekton catalog git content.
Expected Behavior
The Tekton Hub api container should not leak processes over time. The number of active processes should remain stable, and basic shell operations should always work.
Actual Behavior
After the Tekton Hub pod has been running for an extended period of time, it becomes unresponsive due to an internal process leak. It looks like git process are spawned and not cleaned up:
/app $ ps -e
PID USER TIME COMMAND
1 10011100 0:00 /app/api-server
18 10011100 0:00 [git]
25 10011100 0:00 [git]
32 10011100 0:00 [git]
39 10011100 0:00 [git]
46 10011100 0:00 [git]
53 10011100 0:00 [git]
60 10011100 0:00 [git]
67 10011100 0:00 [git]
74 10011100 0:00 [git]
81 10011100 0:00 [git]
88 10011100 0:00 [git]
95 10011100 0:00 [git]
102 10011100 0:00 [git]
105 10011100 0:00 sh
111 10011100 0:00 ps -e
Steps to Reproduce the Problem
- Deploy Tekton Hub on OpenShift (or Kubernetes).
- Leave the pod running under typical load or idle state.
- After a period of time (e.g., hours to days), exec into the pod:
oc exec -it <tekton-hub-pod> -- sh - Attempt to run a command:
/app $ ps-> sh: can't fork: Resource temporarily unavailable
Additional Info
- UI Screenshots
