DAG runs perfectly when I trigger it manually, but when it runs on its scheduled interval, some tasks fail with AirflowTaskTimeout errors

[lakabosch]

The failure seems random - different tasks fail each time, and there's no clear pattern. I've checked the task logs and there's no application-level error. The DAG has about 15 tasks with mixed operators (Python, Bash, Snowflake), and the issues started when we added more DAGs to the same Airflow instance

[aayostem]

could be caused by

• worker resource exhaustion If using Celery/KubernetesExecutor, workers might be overloaded.
• scheduler overload, it can't keep up with parsing and scheduling all DAGs.

# Increase worker resources or adjust concurrency
[celery]
worker_concurrency = 8  # Reduce if tasks are memory-intensive

# Or use pools for resource management
# In Airflow UI: Admin -> Pools
# Create pools for different resource requirements

if it scheduler overload

# Increase scheduler resources
[scheduler]
max_threads = 4                    # Increase from default 2
parsing_processes = 2              # If you have multiple CPU cores
scheduler_heartbeat_sec = 3        # Faster heartbeat detection

# Reduce parsing frequency
[core]
dagbag_import_timeout = 30         # Increase timeout
min_serialized_dag_fetch_sec = 15  # Reduce frequency for serialized DAGs