`taskworker` emmiting "sentry.digests.backends.base.InvalidState: Timeline is not in the ready state." errors

[aldy505]

Disclaimer: I know taskbroker and taskworker are different, feel free to transfer this issue to sentry, but I think it's owned by the same team.

Environment

self-hosted 25.9.0

Steps to Reproduce

Just spin up an instance, and let it work like usual.

Expected Result

No to minimal errors.

Actual Result

I've seeing significant amount of lags on Kafka for taskworker topic, as seen here:

Therefore I see the logs for taskworker container, and found redis ResponseError stating: timeline is not in the ready state, cannot be digested script. Here is the full logs:

taskworker-1  | 01:58:52 [INFO] sentry.tasks.post_process: post_process.skipped (cache_key='e:99212d765fda478991d113575214a83b:38' reason='missing_cache')
taskworker-1  | 01:58:52 [INFO] sentry.tasks.post_process: post_process.skipped (cache_key='e:b6398a0071ac4527ab5681b7bc0ed7c6:13' reason='missing_cache')
taskworker-1  | 01:58:52 [INFO] sentry.tasks.post_process: post_process.skipped (cache_key='e:221d154300c0416eac83e855ed8fbdb1:38' reason='missing_cache')
taskworker-1  | 02:01:36 [INFO] sentry.delayed_processing: delayed_workflow.disabled (option='delayed_workflow.rollout')
taskworker-1  | Traceback (most recent call last):
taskworker-1  |   File "/usr/src/sentry/src/sentry/digests/backends/redis.py", line 204, in digest
taskworker-1  |     response = script(
taskworker-1  |         [key],
taskworker-1  |     ...<8 lines>...
taskworker-1  |         connection,
taskworker-1  |     )
taskworker-1  |   File "/.venv/lib/python3.13/site-packages/redis/client.py", line 4021, in __call__
taskworker-1  |     return client.evalsha(self.sha, len(keys), *args)
taskworker-1  |            ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
taskworker-1  |   File "/.venv/lib/python3.13/site-packages/redis/client.py", line 3091, in evalsha
taskworker-1  |     return self.execute_command('EVALSHA', sha, numkeys, *keys_and_args)
taskworker-1  |            ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
taskworker-1  |   File "/.venv/lib/python3.13/site-packages/sentry_sdk/integrations/redis/_sync_common.py", line 109, in sentry_patched_execute_command
taskworker-1  |     value = old_execute_command(self, name, *args, **kwargs)
taskworker-1  |   File "/.venv/lib/python3.13/site-packages/redis/client.py", line 878, in execute_command
taskworker-1  |     return self.parse_response(conn, command_name, **options)
taskworker-1  |            ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
taskworker-1  |   File "/.venv/lib/python3.13/site-packages/redis/client.py", line 892, in parse_response
taskworker-1  |     response = connection.read_response()
taskworker-1  |   File "/.venv/lib/python3.13/site-packages/redis/connection.py", line 752, in read_response
taskworker-1  |     raise response
taskworker-1  | redis.exceptions.ResponseError: user_script:272: err(invalid_state): timeline is not in the ready state, cannot be digested script: b725ca33e5b36f318ab1150b8ac955a3d997c872, on @user_script:272.
taskworker-1  |
taskworker-1  | The above exception was the direct cause of the following exception:
taskworker-1  |
taskworker-1  | Traceback (most recent call last):
taskworker-1  |   File "/usr/src/sentry/src/sentry/tasks/digests.py", line 75, in deliver_digest
taskworker-1  |     with digests.backend.digest(key, minimum_delay=minimum_delay) as records:
taskworker-1  |          ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
taskworker-1  |   File "/usr/local/lib/python3.13/contextlib.py", line 141, in __enter__
taskworker-1  |     return next(self.gen)
taskworker-1  |   File "/usr/src/sentry/src/sentry/digests/backends/redis.py", line 218, in digest
taskworker-1  |     raise InvalidState("Timeline is not in the ready state.") from e
taskworker-1  | sentry.digests.backends.base.InvalidState: Timeline is not in the ready state.
taskworker-1  | 02:01:36 [INFO] sentry.tasks.digests: Skipped digest delivery: Timeline is not in the ready state.
taskworker-1  | Traceback (most recent call last):
taskworker-1  |   File "/usr/src/sentry/src/sentry/digests/backends/redis.py", line 204, in digest
taskworker-1  |     response = script(
taskworker-1  |         [key],
taskworker-1  |     ...<8 lines>...
taskworker-1  |         connection,
taskworker-1  |     )
taskworker-1  |   File "/.venv/lib/python3.13/site-packages/redis/client.py", line 4021, in __call__
taskworker-1  |     return client.evalsha(self.sha, len(keys), *args)
taskworker-1  |            ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
taskworker-1  |   File "/.venv/lib/python3.13/site-packages/redis/client.py", line 3091, in evalsha
taskworker-1  |     return self.execute_command('EVALSHA', sha, numkeys, *keys_and_args)
taskworker-1  |            ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
taskworker-1  |   File "/.venv/lib/python3.13/site-packages/sentry_sdk/integrations/redis/_sync_common.py", line 109, in sentry_patched_execute_command
taskworker-1  |     value = old_execute_command(self, name, *args, **kwargs)
taskworker-1  |   File "/.venv/lib/python3.13/site-packages/redis/client.py", line 878, in execute_command
taskworker-1  |     return self.parse_response(conn, command_name, **options)
taskworker-1  |            ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
taskworker-1  |   File "/.venv/lib/python3.13/site-packages/redis/client.py", line 892, in parse_response
taskworker-1  |     response = connection.read_response()
taskworker-1  |   File "/.venv/lib/python3.13/site-packages/redis/connection.py", line 752, in read_response
taskworker-1  |     raise response
taskworker-1  | redis.exceptions.ResponseError: user_script:272: err(invalid_state): timeline is not in the ready state, cannot be digested script: b725ca33e5b36f318ab1150b8ac955a3d997c872, on @user_script:272.
taskworker-1  |
taskworker-1  | The above exception was the direct cause of the following exception:
taskworker-1  |
taskworker-1  | Traceback (most recent call last):
taskworker-1  |   File "/usr/src/sentry/src/sentry/tasks/digests.py", line 75, in deliver_digest
taskworker-1  |     with digests.backend.digest(key, minimum_delay=minimum_delay) as records:
taskworker-1  |          ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
taskworker-1  |   File "/usr/local/lib/python3.13/contextlib.py", line 141, in __enter__
taskworker-1  |     return next(self.gen)
taskworker-1  |   File "/usr/src/sentry/src/sentry/digests/backends/redis.py", line 218, in digest
taskworker-1  |     raise InvalidState("Timeline is not in the ready state.") from e
taskworker-1  | sentry.digests.backends.base.InvalidState: Timeline is not in the ready state.
taskworker-1  | 02:01:36 [INFO] sentry.tasks.digests: Skipped digest delivery: Timeline is not in the ready state.
taskworker-1  | Traceback (most recent call last):
taskworker-1  |   File "/usr/src/sentry/src/sentry/digests/backends/redis.py", line 204, in digest
taskworker-1  |     response = script(
taskworker-1  |         [key],
taskworker-1  |     ...<8 lines>...
taskworker-1  |         connection,
taskworker-1  |     )
taskworker-1  |   File "/.venv/lib/python3.13/site-packages/redis/client.py", line 4021, in __call__
taskworker-1  |     return client.evalsha(self.sha, len(keys), *args)
taskworker-1  |            ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
taskworker-1  |   File "/.venv/lib/python3.13/site-packages/redis/client.py", line 3091, in evalsha
taskworker-1  |     return self.execute_command('EVALSHA', sha, numkeys, *keys_and_args)
taskworker-1  |            ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
taskworker-1  |   File "/.venv/lib/python3.13/site-packages/sentry_sdk/integrations/redis/_sync_common.py", line 109, in sentry_patched_execute_command
taskworker-1  |     value = old_execute_command(self, name, *args, **kwargs)
taskworker-1  |   File "/.venv/lib/python3.13/site-packages/redis/client.py", line 878, in execute_command
taskworker-1  |     return self.parse_response(conn, command_name, **options)
taskworker-1  |            ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
taskworker-1  |   File "/.venv/lib/python3.13/site-packages/redis/client.py", line 892, in parse_response
taskworker-1  |     response = connection.read_response()
taskworker-1  |   File "/.venv/lib/python3.13/site-packages/redis/connection.py", line 752, in read_response
taskworker-1  |     raise response
taskworker-1  | redis.exceptions.ResponseError: user_script:272: err(invalid_state): timeline is not in the ready state, cannot be digested script: b725ca33e5b36f318ab1150b8ac955a3d997c872, on @user_script:272.
taskworker-1  |
taskworker-1  | The above exception was the direct cause of the following exception:
taskworker-1  |
taskworker-1  | Traceback (most recent call last):
taskworker-1  |   File "/usr/src/sentry/src/sentry/tasks/digests.py", line 75, in deliver_digest
taskworker-1  |     with digests.backend.digest(key, minimum_delay=minimum_delay) as records:
taskworker-1  |          ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
taskworker-1  |   File "/usr/local/lib/python3.13/contextlib.py", line 141, in __enter__
taskworker-1  |     return next(self.gen)
taskworker-1  |   File "/usr/src/sentry/src/sentry/digests/backends/redis.py", line 218, in digest
taskworker-1  |     raise InvalidState("Timeline is not in the ready state.") from e
taskworker-1  | sentry.digests.backends.base.InvalidState: Timeline is not in the ready state.
taskworker-1  | 02:01:36 [INFO] sentry.tasks.digests: Skipped digest delivery: Timeline is not in the ready state.
taskworker-1  | 02:01:36 [INFO] sentry.tasks.post_process: post_process.skipped (cache_key='e:9d908c90fa754983b5b8d3dfef43a279:2' reason='missing_cache')
taskworker-1  | 02:01:36 [INFO] sentry.tasks.post_process: post_process.skipped (cache_key='e:f56040d4a7064db6a8069360c54c1180:38' reason='missing_cache')
taskworker-1  | 02:01:36 [INFO] sentry.tasks.post_process: post_process.skipped (cache_key='e:891bcb15b65f4ce782f6636377765609:45' reason='missing_cache')
taskworker-1  | 02:01:36 [INFO] sentry.tasks.post_process: post_process.skipped (cache_key='e:78edb24338c1498fae6bc85f23516ca9:48' reason='missing_cache')
taskworker-1  | 02:01:36 [INFO] sentry.tasks.post_process: post_process.skipped (cache_key='e:cbf845182d3a4c94af9814270f215241:38' reason='missing_cache')

Honestlly I have no idea on what's causing this and what might be the solution.

[markstory]

I've not seen the 'Timeline is not in the ready state error' before. We're not currently getting that in production. Based on the context of the code surrounding that error, it looks like the digests lua script that is run inside of redis has data missing. You could try clearing the digest related keys from redis. Digest keys begin with d:t: and d:s.