Upgrade from 25.11.0 to 25.11.1 fails

[meenzen]

Self-Hosted Version

25.11.1

CPU Architecture

x86_64

Docker Version

27.5.1, build 9f9e405

Docker Compose Version

v2.32.4

Machine Specification

• My system meets the minimum system requirements of Sentry

Steps to Reproduce

On my existing install I just did the normal upgrade steps:

• git checkout 25.11.1
• vim -d sentry/sentry.conf.py sentry/sentry.conf.example.py
• vim -d sentry/config.yml sentry/config.example.yml
• ./install.sh

Note: The config is very close to the default, only mail, hostname, secrets and reverse proxy configuration have been changed.

Expected Result

The upgrade is successful

Actual Result

First attempt:

▶ Setting up / migrating database ...
 Container sentry-self-hosted-postgres-1  Running
 Container sentry-self-hosted-postgres-1  Waiting
 Container sentry-self-hosted-postgres-1  Healthy
 Container sentry-self-hosted-postgres-1  Running
 Container sentry-self-hosted-pgbouncer-1  Creating
 Container sentry-self-hosted-pgbouncer-1  Created
 Container sentry-self-hosted-postgres-1  Waiting
 Container sentry-self-hosted-postgres-1  Healthy
 Container sentry-self-hosted-pgbouncer-1  Starting
 Container sentry-self-hosted-pgbouncer-1  Started
 Container sentry-self-hosted-pgbouncer-1  Waiting
 Container sentry-self-hosted-postgres-1  Waiting
 Container sentry-self-hosted-postgres-1  Healthy
 Container sentry-self-hosted-pgbouncer-1  Healthy
 Container sentry-self-hosted-redis-1  Running
 Container sentry-self-hosted-clickhouse-1  Running
 Container sentry-self-hosted-memcached-1  Creating
 Container sentry-self-hosted-symbolicator-1  Creating
 Container sentry-self-hosted-kafka-1  Running
 Container sentry-self-hosted-seaweedfs-1  Running
 Container sentry-self-hosted-snuba-api-1  Creating
 Container sentry-self-hosted-postgres-1  Running
 Container sentry-self-hosted-smtp-1  Creating
 Container sentry-self-hosted-pgbouncer-1  Running
 Container sentry-self-hosted-memcached-1  Created
 Container sentry-self-hosted-smtp-1  Created
 Container sentry-self-hosted-snuba-api-1  Created
 Container sentry-self-hosted-symbolicator-1  Created
 Container sentry-self-hosted-memcached-1  Starting
 Container sentry-self-hosted-postgres-1  Waiting
 Container sentry-self-hosted-smtp-1  Starting
 Container sentry-self-hosted-symbolicator-1  Starting
 Container sentry-self-hosted-kafka-1  Waiting
 Container sentry-self-hosted-redis-1  Waiting
 Container sentry-self-hosted-clickhouse-1  Waiting
 Container sentry-self-hosted-smtp-1  Started
 Container sentry-self-hosted-memcached-1  Started
 Container sentry-self-hosted-symbolicator-1  Started
 Container sentry-self-hosted-postgres-1  Healthy
 Container sentry-self-hosted-redis-1  Healthy
 Container sentry-self-hosted-clickhouse-1  Healthy
 Container sentry-self-hosted-kafka-1  Healthy
 Container sentry-self-hosted-snuba-api-1  Starting
 Container sentry-self-hosted-snuba-api-1  Started
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
Traceback (most recent call last):
  File "/.venv/bin/sentry", line 4, in <module>
    raise SystemExit(main())
                     ~~~~^^
  File "/usr/src/sentry/src/sentry/runner/main.py", line 144, in main
    func(**kwargs)
    ~~~~^^^^^^^^^^
  File "/.venv/lib/python3.13/site-packages/click/core.py", line 1442, in __call__
    return self.main(*args, **kwargs)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.13/site-packages/click/core.py", line 1363, in main
    rv = self.invoke(ctx)
  File "/.venv/lib/python3.13/site-packages/click/core.py", line 1830, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/.venv/lib/python3.13/site-packages/click/core.py", line 1226, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.13/site-packages/click/core.py", line 794, in invoke
    return callback(*args, **kwargs)
  File "/.venv/lib/python3.13/site-packages/click/decorators.py", line 34, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/src/sentry/src/sentry/runner/decorators.py", line 33, in inner
    configure()
    ~~~~~~~~~^^
  File "/usr/src/sentry/src/sentry/runner/__init__.py", line 33, in configure
    _configure(ctx, py, yaml, skip_service_validation)
    ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/sentry/src/sentry/runner/settings.py", line 124, in configure
    initialize_app(
    ~~~~~~~~~~~~~~^
        {"config_path": py, "settings": settings, "options": yaml},
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        skip_service_validation=skip_service_validation,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/src/sentry/src/sentry/runner/initializer.py", line 360, in initialize_app
    validate_options(settings)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/usr/src/sentry/src/sentry/runner/initializer.py", line 436, in validate_options
    default_manager.validate(settings.SENTRY_OPTIONS, warn=True)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/sentry/src/sentry/options/manager.py", line 440, in validate
    self.validate_option(k, v)
    ~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/usr/src/sentry/src/sentry/options/manager.py", line 450, in validate_option
    raise TypeError(f"{key!r}: got {_type(value)!r}, expected {opt.type!r}")
TypeError: 'filestore.profiles-options': got <class 'NoneType'>, expected dictionary
Error in install/set-up-and-migrate-database.sh:23.
'$dcr web upgrade --create-kafka-topics' exited with status 1
-> ./install.sh:main:44
--> install/set-up-and-migrate-database.sh:source:23

install/error-handling.sh: line 81: /usr/bin/docker: Argument list too long

Second attempt:

▶ Bootstrapping seaweedfs (node store)...
 Network sentry-self-hosted_default  Creating
 Network sentry-self-hosted_default  Created
 Container sentry-self-hosted-seaweedfs-1  Creating
 Container sentry-self-hosted-postgres-1  Creating
 Container sentry-self-hosted-postgres-1  Created
 Container sentry-self-hosted-seaweedfs-1  Created
 Container sentry-self-hosted-postgres-1  Starting
 Container sentry-self-hosted-seaweedfs-1  Starting
 Container sentry-self-hosted-postgres-1  Started
 Container sentry-self-hosted-seaweedfs-1  Started
 Container sentry-self-hosted-seaweedfs-1  Waiting
 Container sentry-self-hosted-postgres-1  Waiting
 Container sentry-self-hosted-postgres-1  Healthy
 Container sentry-self-hosted-seaweedfs-1  Healthy
fetch https://dl-cdn.alpinelinux.org/alpine/v3.22/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.22/community/x86_64/APKINDEX.tar.gz
(1/26) Installing libbz2 (1.0.8-r6)
(2/26) Installing libexpat (2.7.3-r0)
(3/26) Installing libffi (3.4.8-r0)
(4/26) Installing gdbm (1.24-r0)
(5/26) Installing xz-libs (5.8.1-r0)
(6/26) Installing libgcc (14.2.0-r6)
(7/26) Installing libstdc++ (14.2.0-r6)
(8/26) Installing mpdecimal (4.0.1-r0)
(9/26) Installing ncurses-terminfo-base (6.5_p20250503-r0)
(10/26) Installing libncursesw (6.5_p20250503-r0)
(11/26) Installing libpanelw (6.5_p20250503-r0)
(12/26) Installing readline (8.2.13-r1)
(13/26) Installing sqlite-libs (3.49.2-r1)
(14/26) Installing python3 (3.12.12-r0)
(15/26) Installing python3-pycache-pyc0 (3.12.12-r0)
(16/26) Installing pyc (3.12.12-r0)
(17/26) Installing py3-six (1.17.0-r0)
(18/26) Installing py3-six-pyc (1.17.0-r0)
(19/26) Installing py3-dateutil (2.9.0-r1)
(20/26) Installing py3-dateutil-pyc (2.9.0-r1)
(21/26) Installing libmagic (5.46-r2)
(22/26) Installing py3-magic (0.4.27-r3)
(23/26) Installing py3-magic-pyc (0.4.27-r3)
(24/26) Installing s3cmd-pyc (2.4.0-r1)
(25/26) Installing python3-pyc (3.12.12-r0)
(26/26) Installing s3cmd (2.4.0-r1)
Executing busybox-1.37.0-r18.trigger
OK: 58 MiB in 44 packages
ERROR: Bucket 'nodestore' already exists
ERROR: S3 error: 409 (BucketAlreadyExists): The requested bucket name is not available. The bucket name can not be an existing collection, and the bucket namespace is shared by all users of the system. Please select a different name and try again.
Error in install/bootstrap-s3-nodestore.sh:64.
'$s3cmd --access_key=sentry --secret_key=sentry --no-ssl --region=us-east-1 --host=localhost:8333 --host-bucket='localhost:8333/%(bucket)' mb s3://nodestore' exited with status 13
-> ./install.sh:main:39
--> install/bootstrap-s3-nodestore.sh:source:64

install/error-handling.sh: line 81: /usr/bin/docker: Argument list too long

Event ID

No response

[klemen-df]

my comment here

[aldy505]

Hey I put up a fix, usually takes 20 mins for the CI to finish. When that's merged into master, please do:

git switch master
git pull
# change nightly tag into 25.11.1
# on .env file, or .env.custom
# (if any)
sed -i 's/nightly/25.11.1/g' .env
./install.sh

[gokulakrishnagk333]

We’re still hitting the Bucket 'nodestore' already exists error on 25.11.1 even after following the suggested workaround.

Environment

Self-hosted version: 25.11.1

Nodestore: SeaweedFS, nodestore bucket already exists

COMPOSE_PROFILES: feature-complete

We’ve also enabled profiling-related feature flags in sentry.conf.py, including:

We’ve also enabled profiling-related feature flags in sentry.conf.py, including:

SENTRY_USE_PROFILING = True
SENTRY_FEATURES["projects:continuous-profiling-vroomrs-processing"] = True
SENTRY_FEATURES["projects:transaction-profiling-vroomrs-processing"] = True

What I did

First upgrade from 25.11.0 → 25.11.1:

git checkout 25.11.1
./install.sh

This completed successfully.
The SeaweedFS profiles migration ran and created s3://profiles/.
Sentry started and worked (traces, errors, etc.).
Later, after editing sentry/sentry.conf.py to add the profiling vroomrs flags, I followed the recommended workaround from this issue:

git switch master
git pull
sed -i 's/nightly/25.11.1/g' .env
./install.sh

Resulting error

On this second ./install.sh run, I get:

(22/26) Installing py3-magic (0.4.27-r3)
(23/26) Installing py3-magic-pyc (0.4.27-r3)
(24/26) Installing s3cmd-pyc (2.4.0-r1)
(25/26) Installing python3-pyc (3.12.12-r0)
(26/26) Installing s3cmd (2.4.0-r1)
Executing busybox-1.37.0-r18.trigger
OK: 58 MiB in 44 packages
ERROR: Bucket 'nodestore' already exists
ERROR: S3 error: 409 (BucketAlreadyExists): The requested bucket name is not available. The bucket name can not be an existing collection, and the bucket namespace is shared by all users of the system. Please select a different name and try again.
Error in install/bootstrap-s3-nodestore.sh:64.
'$s3cmd --access_key=sentry --secret_key=sentry --no-ssl --region=us-east-1 --host=localhost:8333 --host-bucket='localhost:8333/%(bucket)' mb s3://nodestore' exited with status 13
-> ./install.sh:main:39
--> install/bootstrap-s3-nodestore.sh:source:64

At this point s3://nodestore is already present in SeaweedFS, so it looks like bootstrap-s3-nodestore.sh still treats the 409 from mb s3://nodestore as fatal when rerunning ./install.sh on 25.11.1.

Question

Is there an updated version of install/bootstrap-s3-nodestore.sh or additional steps we should take so that:

Rerunning ./install.sh on 25.11.1 does not fail when the nodestore bucket already exists, and
We can safely apply config/feature flag changes without hitting this error?
Happy to provide more logs or config snippets if needed.

[aldy505]

@gokulakrishnagk333 the second install.sh failed because #4076 is not merged yet