Access denied for rekor-trillian-createdb pod connecting to trillian-mysql

[kevchu3]

Description

I've deployed the Rekor Helm chart which also spins up the trillian-system chart.

The trillian-logserver, trillian-logsigner, and trillian-mysql pods come up and stay running, but the rekor-trillian-createdb job continually fails. Since it's a job, it keeps trying every few minutes. Here's a look into the goroutine panic that I'm getting:

$ kubectl logs rekor-trillian-createdb-5mhbc 
2024/12/23 19:43:57 failed to ping db: Error 1045: Access denied for user 'mysql'@'10.42.0.222' (using password: YES)
panic: failed to ping db: Error 1045: Access denied for user 'mysql'@'10.42.0.222' (using password: YES)

goroutine 1 [running]:
log.Panicf({0xb8602c?, 0xca28f0?}, {0xc0000b7d38?, 0x32?, 0x2?})
	log/log.go:395 +0x67
main.main()
	github.com/sigstore/scaffolding/cmd/trillian/createdb/main.go:238 +0x3b2

I restarted the trillian-mysql server pod to get some fresh logs, here's a look at that as well. In the logs below, the 10.42.0.222 IP address is the rekor-trillian-createdb pod:

$ kubectl logs trillian-mysql-776fc545d9-46gqj
2024-12-23T19:42:51.125238Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2024-12-23T19:42:51.127052Z 0 [Note] mysqld (mysqld 5.7.38) starting as process 1 ...
2024-12-23T19:42:51.130118Z 0 [Note] InnoDB: PUNCH HOLE support available
2024-12-23T19:42:51.130133Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2024-12-23T19:42:51.130139Z 0 [Note] InnoDB: Uses event mutexes
2024-12-23T19:42:51.130144Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2024-12-23T19:42:51.130149Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2024-12-23T19:42:51.130153Z 0 [Note] InnoDB: Using Linux native AIO
2024-12-23T19:42:51.130725Z 0 [Note] InnoDB: Number of pools: 1
2024-12-23T19:42:51.130822Z 0 [Note] InnoDB: Using CPU crc32 instructions
2024-12-23T19:42:51.133206Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2024-12-23T19:42:51.141631Z 0 [Note] InnoDB: Completed initialization of buffer pool
2024-12-23T19:42:51.144291Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2024-12-23T19:42:51.154314Z 0 [ERROR] InnoDB: Unable to lock ./ibdata1 error: 11
2024-12-23T19:42:51.154334Z 0 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2024-12-23T19:42:51.154340Z 0 [Note] InnoDB: Retrying to lock the first data file
2024-12-23T19:42:52.154441Z 0 [ERROR] InnoDB: Unable to lock ./ibdata1 error: 11
2024-12-23T19:42:52.154463Z 0 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2024-12-23T19:42:53.165679Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2024-12-23T19:42:53.189696Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2024-12-23T19:42:53.189715Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2024-12-23T19:42:53.189812Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2024-12-23T19:42:53.237178Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2024-12-23T19:42:53.237945Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
2024-12-23T19:42:53.237958Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
2024-12-23T19:42:53.238912Z 0 [Note] InnoDB: 5.7.38 started; log sequence number 12575253
2024-12-23T19:42:53.239402Z 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2024-12-23T19:42:53.239610Z 0 [Note] Plugin 'FEDERATED' is disabled.
2024-12-23T19:42:53.258446Z 0 [Note] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them.
2024-12-23T19:42:53.258666Z 0 [Note] Skipping generation of SSL certificates as certificate files are present in data directory.
2024-12-23T19:42:53.258673Z 0 [Warning] A deprecated TLS version TLSv1 is enabled. Please use TLSv1.2 or higher.
2024-12-23T19:42:53.258678Z 0 [Warning] A deprecated TLS version TLSv1.1 is enabled. Please use TLSv1.2 or higher.
2024-12-23T19:42:53.259352Z 0 [Warning] CA certificate ca.pem is self signed.
2024-12-23T19:42:53.259390Z 0 [Note] Skipping generation of RSA key pair as key files are present in data directory.
2024-12-23T19:42:53.260615Z 0 [Note] Server hostname (bind-address): '*'; port: 3306
2024-12-23T19:42:53.260669Z 0 [Note] IPv6 is available.
2024-12-23T19:42:53.260684Z 0 [Note]   - '::' resolves to '::';
2024-12-23T19:42:53.260710Z 0 [Note] Server socket created on IP: '::'.
2024-12-23T19:42:53.261395Z 0 [Warning] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
2024-12-23T19:42:53.262744Z 0 [Note] InnoDB: Buffer pool(s) load completed at 241223 19:42:53
2024-12-23T19:42:53.295771Z 0 [Note] Event Scheduler: Loaded 0 events
2024-12-23T19:42:53.296197Z 0 [Note] mysqld: ready for connections.
Version: '5.7.38'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Server (GPL)
2024-12-23T19:43:47.970157Z 4 [Note] Access denied for user 'mysql'@'10.42.0.222' (using password: YES)
2024-12-23T19:43:49.973826Z 5 [Note] Access denied for user 'mysql'@'10.42.0.222' (using password: YES)
2024-12-23T19:43:51.976939Z 6 [Note] Access denied for user 'mysql'@'10.42.0.222' (using password: YES)
2024-12-23T19:43:53.981175Z 7 [Note] Access denied for user 'mysql'@'10.42.0.222' (using password: YES)
2024-12-23T19:43:55.984607Z 8 [Note] Access denied for user 'mysql'@'10.42.0.222' (using password: YES)
2024-12-23T19:43:57.987249Z 9 [Note] Access denied for user 'mysql'@'10.42.0.222' (using password: YES)

Version

I'm deploying onto Red Hat Microshift, so some of the security best practices of using non-root users are more along the lines of OpenShift than vanilla Kubernetes
Rekor helm chart is version 1.3.7
I'm using a pretty vanilla values.yaml file:

server:
  ingress:
    enabled: false