Remove WAL-E and make WAL-G the default backup tool (#1143)

Author: idanovinda

ENVIRONMENT.rst

@@ -11,11 +11,11 @@ Environment Configuration Settings
 - **ETCD_KEY**: Etcd client certificate key. Can be empty if the key is part of certificate.
 - **PGHOME**: filesystem path where to put PostgreSQL home directory (/home/postgres by default)
 - **APIPORT**: TCP port to Patroni API connections (8008 by default)
-- **BACKUP_SCHEDULE**: cron schedule for doing backups via WAL-E (if WAL-E is enabled, '00 01 * * *' by default)
+- **BACKUP_SCHEDULE**: cron schedule for doing backups via WAL-G ('00 01 * * *' by default)
 - **CLONE_TARGET_TIMELINE**: timeline id of the backup for restore, 'latest' by default.
 - **CRONTAB**: anything that you want to run periodically as a cron job (empty by default)
 - **PGROOT**: a directory where we put the pgdata (by default /home/postgres/pgroot). One may adjust it to point to the mount point of the persistent volume, such as EBS.
-- **WALE_TMPDIR**: directory to store WAL-E temporary files. PGROOT/../tmp by default, make sure it has a few GBs of free space.
+- **WALE_TMPDIR** or **WALG_TMPDIR**: directory to store WAL-G temporary files. PGROOT/../tmp by default, make sure it has a few GBs of free space.
 - **PGDATA**: location of PostgreSQL data directory, by default PGROOT/pgdata.
 - **PGUSER_STANDBY**: username for the replication user, 'standby' by default.
 - **PGPASSWORD_STANDBY**: a password for the replication user, 'standby' by default.
@@ -47,22 +47,22 @@ Environment Configuration Settings
 - **SSL_RESTAPI_PRIVATE_KEY**: content of the REST Api SSL private key in the SSL_PRIVATE_KEY_FILE file (by default /run/certs/server.key).
 - **SSL_TEST_RELOAD**: whenever to test for certificate rotation and reloading (by default True if SSL_PRIVATE_KEY_FILE has been set).
 - **RESTAPI_CONNECT_ADDRESS**: when you configure Patroni RESTAPI in SSL mode some safe API (i.e. switchover) perform hostname validation. In this case could be convenient configure ````restapi.connect_address````as a hostname instead of IP. For example, you can configure it as "$(POD_NAME).<service name>".
-- **WALE_BACKUP_THRESHOLD_MEGABYTES**: maximum size of the WAL segments accumulated after the base backup to consider WAL-E restore instead of pg_basebackup.
-- **WALE_BACKUP_THRESHOLD_PERCENTAGE**: maximum ratio (in percents) of the accumulated WAL files to the base backup to consider WAL-E restore instead of pg_basebackup.
-- **WALE_ENV_DIR**: directory where to store WAL-E environment variables
+- **WALG_BACKUP_THRESHOLD_MEGABYTES** or **WALE_BACKUP_THRESHOLD_MEGABYTES**: maximum size of the WAL segments accumulated after the base backup to consider WAL-G restore instead of pg_basebackup.
+- **WALG_BACKUP_THRESHOLD_PERCENTAGE** or **WALE_BACKUP_THRESHOLD_PERCENTAGE**: maximum ratio (in percents) of the accumulated WAL files to the base backup to consider WAL-G restore instead of pg_basebackup.
+- **WALG_ENV_DIR** or **WALE_ENV_DIR**: directory where to store WAL-G environment variables
 - **WAL_RESTORE_TIMEOUT**: timeout (in seconds) for restoring a single WAL file (at most 16 MB) from the backup location, 0 by default. A duration of 0 disables the timeout.
-- **WAL_S3_BUCKET**: (optional) name of the S3 bucket used for WAL-E base backups.
+- **WAL_S3_BUCKET**: (optional) name of the S3 bucket used for WAL-G base backups.
 - **AWS_ACCESS_KEY_ID**: (optional) aws access key
 - **AWS_SECRET_ACCESS_KEY**: (optional) aws secret key
 - **AWS_REGION**: (optional) region of S3 bucket
 - **AWS_ENDPOINT**: (optional) in format 'https://s3.AWS_REGION.amazonaws.com:443', if not specified will be calculated from AWS_REGION
-- **WALE_S3_ENDPOINT**: (optional) in format 'https+path://s3.AWS_REGION.amazonaws.com:443', if not specified will be calculated from AWS_ENDPOINT or AWS_REGION
-- **WALE_S3_PREFIX**: (optional) the full path to the backup location on S3 in the format s3://bucket-name/very/long/path. If not specified Spilo will generate it from WAL_S3_BUCKET.
-- **WAL_GS_BUCKET**: ditto for the Google Cloud Storage (WAL-E supports both S3 and GCS).
-- **WALE_GS_PREFIX**: (optional) the full path to the backup location on the Google Cloud Storage in the format gs://bucket-name/very/long/path. If not specified Spilo will generate it from WAL_GS_BUCKET.
-- **GOOGLE_APPLICATION_CREDENTIALS**: credentials for WAL-E when running in Google Cloud.
+- **WALG_S3_ENDPOINT** or **WALE_S3_ENDPOINT**: (optional) in format 'https+path://s3.AWS_REGION.amazonaws.com:443', if not specified will be calculated from AWS_ENDPOINT or AWS_REGION
+- **WALG_S3_PREFIX** or **WALE_S3_PREFIX**: (optional) the full path to the backup location on S3 in the format s3://bucket-name/very/long/path. If not specified Spilo will generate it from WAL_S3_BUCKET.
+- **WAL_GS_BUCKET**: ditto for the Google Cloud Storage (WAL-G supports both S3 and GCS).
+- **WALG_GS_PREFIX** or **WALE_GS_PREFIX**: (optional) the full path to the backup location on the Google Cloud Storage in the format gs://bucket-name/very/long/path. If not specified Spilo will generate it from WAL_GS_BUCKET.
+- **GOOGLE_APPLICATION_CREDENTIALS**: credentials for WAL-G when running in Google Cloud.
 - **WAL_SWIFT_BUCKET**: ditto for the OpenStack Object Storage (Swift)
-- **SWIFT_AUTHURL**: see wal-e documentation https://github.com/wal-e/wal-e#swift
+- **SWIFT_AUTHURL**: see wal-g documentation https://wal-g.readthedocs.io/STORAGES/#swift
 - **SWIFT_TENANT**:
 - **SWIFT_TENANT_ID**:
 - **SWIFT_USER**:
@@ -79,7 +79,7 @@ Environment Configuration Settings
 - **SWIFT_PROJECT_ID**:
 - **SWIFT_PROJECT_DOMAIN_NAME**:
 - **SWIFT_PROJECT_DOMAIN_ID**:
-- **WALE_SWIFT_PREFIX**: (optional) the full path to the backup location on the Swift Storage in the format swift://bucket-name/very/long/path. If not specified Spilo will generate it from WAL_SWIFT_BUCKET.
+- **WALG_SWIFT_PREFIX** or **WALE_SWIFT_PREFIX**: (optional) the full path to the backup location on the Swift Storage in the format swift://bucket-name/very/long/path. If not specified Spilo will generate it from WAL_SWIFT_BUCKET.
 - **SSH_USERNAME**: (optional) the username for WAL backups.
 - **SSH_PORT**: (optional) the ssh port for WAL backups.
 - **SSH_PRIVATE_KEY_PATH**: (optional) the path to the private key used for WAL backups.
@@ -109,18 +109,17 @@ Environment Configuration Settings
 - **KUBERNETES_BOOTSTRAP_LABELS**: a JSON describing names and values of labels used by Patroni as ``kubernetes.bootstrap_labels``. Default is empty.
 - **INITDB_LOCALE**: database cluster's default UTF-8 locale (en_US by default)
 - **ENABLE_WAL_PATH_COMPAT**: old Spilo images were generating wal path in the backup store using the following template ``/spilo/{WAL_BUCKET_SCOPE_PREFIX}{SCOPE}{WAL_BUCKET_SCOPE_SUFFIX}/wal/``, while new images adding one additional directory (``{PGVERSION}``) to the end. In order to avoid (unlikely) issues with restoring WALs (from S3/GC/and so on) when switching to ``spilo-13`` please set the ``ENABLE_WAL_PATH_COMPAT=true`` when deploying old cluster with ``spilo-13`` for the first time. After that the environment variable could be removed. Change of the WAL path also mean that backups stored in the old location will not be cleaned up automatically.
-- **WALE_DISABLE_S3_SSE**, **WALG_DISABLE_S3_SSE**: by default wal-e/wal-g are configured to encrypt files uploaded to S3. In order to disable it you can set this environment variable to ``true``.
+- **WALG_DISABLE_S3_SSE** or **WALE_DISABLE_S3_SSE**: by default wal-g is configured to encrypt files uploaded to S3. In order to disable it you can set this environment variable to ``true``.
 - **USE_OLD_LOCALES**: whether to use old locales from Ubuntu 18.04 in the Ubuntu 22.04-based image. Default is false.
 
 wal-g
 -----
 
-`wal-g` is used by default for Azure and SSH backups and restore.
-In case of S3, `wal-e` is used for backups and `wal-g` for restore.
-
-- **USE_WALG_BACKUP**: (optional) Enforce using `wal-g` instead of `wal-e` for backups (Boolean)
-- **USE_WALG_RESTORE**: (optional) Enforce using `wal-g` instead of `wal-e` for restores (Boolean)
-
+wal-g is used everywhere in Spilo to perform backups and restore from them. **Support for wal-e has been removed**.
+Backward compatibility is ensured for environment variables containing **WALE**, the env-dir layout, and bootstrap method names.
+This allows existing configurations and clusters to continue working without requiring immediate changes.
+Regardless of which variable is set, all backups and restores will be performed using wal-g.
+However, if both **WALE** and **WALG** variables are present, the latter will take precedence.
 - **WALG_DELTA_MAX_STEPS**, **WALG_DELTA_ORIGIN**, **WALG_DOWNLOAD_CONCURRENCY**, **WALG_UPLOAD_CONCURRENCY**, **WALG_UPLOAD_DISK_CONCURRENCY**, **WALG_DISK_RATE_LIMIT**, **WALG_NETWORK_RATE_LIMIT**, **WALG_COMPRESSION_METHOD**, **WALG_BACKUP_COMPRESSION_METHOD**, **WALG_BACKUP_FROM_REPLICA**, **WALG_SENTINEL_USER_DATA**, **WALG_PREVENT_WAL_OVERWRITE**: (optional) configuration options for wal-g.
 - **WALG_S3_CA_CERT_FILE**: (optional) TLS CA certificate for wal-g (see [wal-g configuration](https://github.com/wal-g/wal-g#configuration))
 - **WALG_SSH_PREFIX**: (optional) the ssh prefix to store WAL backups at in the format ssh://host.example.com/path/to/backups/ See `Wal-g <https://github.com/wal-g/wal-g#configuration>`__ documentation for details.

postgres-appliance/Dockerfile

@@ -68,15 +68,14 @@ RUN bash base.sh
 # Install wal-g
 COPY --from=dependencies-builder /builddeps/wal-g /usr/local/bin/
 
-COPY build_scripts/patroni_wale.sh build_scripts/compress_build.sh /builddeps/
+COPY build_scripts/patroni.sh build_scripts/compress_build.sh /builddeps/
 
-# Install patroni and wal-e
+# Install patroni
 ENV PATRONIVERSION=4.0.6
-ENV WALE_VERSION=1.1.1
 
 WORKDIR /
 
-RUN bash /builddeps/patroni_wale.sh
+RUN bash /builddeps/patroni.sh
 
 RUN if [ "$COMPRESS" = "true" ]; then bash /builddeps/compress_build.sh; fi
 
@@ -101,7 +100,7 @@ ENV LC_ALL=en_US.utf-8 \
     RW_DIR=/run \
     DEMO=$DEMO
 
-ENV WALE_ENV_DIR=$RW_DIR/etc/wal-e.d/env \
+ENV WALG_ENV_DIR=$RW_DIR/etc/wal-e.d/env \
     LOG_ENV_DIR=$RW_DIR/etc/log.d/env \
     PGROOT=$PGHOME/pgdata/pgroot
 

postgres-appliance/bootstrap/clone_with_wale.py renamed to postgres-appliance/bootstrap/clone_with_walg.py

@@ -25,7 +25,7 @@ def read_configuration():
     parser.add_argument('--recovery-target-time',
                         help='the timestamp up to which recovery will proceed (including time zone)',
                         dest='recovery_target_time_string')
-    parser.add_argument('--dry-run', action='store_true', help='find a matching backup and build the wal-e '
+    parser.add_argument('--dry-run', action='store_true', help='find a matching backup and build the wal-g.'
                         'command to fetch that backup without running it')
     args = parser.parse_args()
 
@@ -40,8 +40,8 @@ def read_configuration():
     return options(args.scope, args.datadir, recovery_target_time, args.dry_run)
 
 
-def build_wale_command(command, datadir=None, backup=None):
-    cmd = ['wal-g' if os.getenv('USE_WALG_RESTORE') == 'true' else 'wal-e'] + [command]
+def build_walg_command(command, datadir=None, backup=None):
+    cmd = ['wal-g', command]
     if command == 'backup-fetch':
         if datadir is None or backup is None:
             raise Exception("backup-fetch requires datadir and backup arguments")
@@ -79,7 +79,7 @@ def choose_backup(backup_list, recovery_target_time):
 
 
 def list_backups(env):
-    backup_list_cmd = build_wale_command('backup-list')
+    backup_list_cmd = build_walg_command('backup-list')
     output = subprocess.check_output(backup_list_cmd, env=env)
     reader = csv.DictReader(fix_output(output), dialect='excel-tab')
     return list(reader)
@@ -89,7 +89,7 @@ def get_clone_envdir():
     from spilo_commons import get_patroni_config
 
     config = get_patroni_config()
-    restore_command = shlex.split(config['bootstrap']['clone_with_wale']['recovery_conf']['restore_command'])
+    restore_command = shlex.split(config['bootstrap']['clone_with_walg']['recovery_conf']['restore_command'])
     if len(restore_command) > 4 and restore_command[0] == 'envdir':
         return restore_command[1]
     raise Exception('Failed to find clone envdir')
@@ -117,10 +117,9 @@ def get_possible_versions():
     return [ver for _, ver in sorted(versions.items(), reverse=True)]
 
 
-def get_wale_environments(env):
-    use_walg = env.get('USE_WALG_RESTORE') == 'true'
-    prefix = 'WALG_' if use_walg else 'WALE_'
-    # len('WALE__PREFIX') = 12
+def get_walg_environments(env):
+    prefix = 'WALG_'
+    # len('WALG_PREFIX') = 12
     names = [name for name in env.keys() if name.endswith('_PREFIX') and name.startswith(prefix) and len(name) > 12]
     if len(names) != 1:
         raise Exception('Found find {0} {1}*_PREFIX environment variables, expected 1'
@@ -141,7 +140,7 @@ def get_wale_environments(env):
 
 def find_backup(recovery_target_time, env):
     old_value = None
-    for name, value in get_wale_environments(env):
+    for name, value in get_walg_environments(env):
         logger.info('Trying %s for clone', value)
         if not old_value:
             old_value = env[name]
@@ -164,12 +163,12 @@ def run_clone_from_s3(options):
 
     backup_name, update_envdir = find_backup(options.recovery_target_time, env)
 
-    backup_fetch_cmd = build_wale_command('backup-fetch', options.datadir, backup_name)
+    backup_fetch_cmd = build_walg_command('backup-fetch', options.datadir, backup_name)
     logger.info("cloning cluster %s using %s", options.name, ' '.join(backup_fetch_cmd))
     if not options.dry_run:
         ret = subprocess.call(backup_fetch_cmd, env=env)
         if ret != 0:
-            raise Exception("wal-e backup-fetch exited with exit code {0}".format(ret))
+            raise Exception("wal-g backup-fetch exited with exit code {0}".format(ret))
 
         if update_envdir:  # We need to update file in the clone envdir or restore_command will fail!
             envdir = get_clone_envdir()

postgres-appliance/build_scripts/patroni_wale.sh renamed to postgres-appliance/build_scripts/patroni.sh

@@ -1,8 +1,8 @@
 #!/bin/bash
 
-## -------------------------
-## Install patroni and wal-e
-## -------------------------
+## ----------------
+## Install patroni
+## ----------------
 
 export DEBIAN_FRONTEND=noninteractive
 
@@ -26,25 +26,17 @@ if [ "$DEMO" != "true" ]; then
         python3-etcd \
         python3-consul \
         python3-kazoo \
-        python3-boto \
         python3-boto3 \
         python3-botocore \
         python3-cachetools \
-        python3-cffi \
-        python3-gevent \
         python3-pyasn1-modules \
         python3-rsa \
-        python3-s3transfer \
-        python3-swiftclient
+        python3-s3transfer
 
     find /usr/share/python-babel-localedata/locale-data -type f ! -name 'en_US*.dat' -delete
 
-    pip3 install filechunkio protobuf \
-            'git+https://github.com/zalando-pg/wal-e.git@ipv6-imds#egg=wal-e[aws,google,swift]' \
+    pip3 install protobuf \
             'git+https://github.com/zalando/pg_view.git@master#egg=pg-view'
-
-    # https://github.com/wal-e/wal-e/issues/318
-    sed -i 's/^\(    for i in range(0,\) num_retries):.*/\1 100):/g' /usr/lib/python3/dist-packages/boto/utils.py
 else
     EXTRAS=""
 fi

postgres-appliance/build_scripts/prepare.sh

@@ -19,8 +19,6 @@ rm -fr /etc/cron.??*
 truncate --size 0 /etc/crontab
 
 if [ "$DEMO" != "true" ]; then
-    # Required for wal-e
-    apt-get install -y pv lzop
     # install etcdctl
     ETCDVERSION=3.3.27
     curl -L https://github.com/coreos/etcd/releases/download/v${ETCDVERSION}/etcd-v${ETCDVERSION}-linux-"$(dpkg --print-architecture)".tar.gz \

postgres-appliance/launch.sh

@@ -51,10 +51,11 @@ chmod -R go-w "$PGROOT"
 chmod 01777 "$RW_DIR/tmp"
 chmod 0700 "$PGDATA"
 
+WALG_ENV_DIR="${WALG_ENV_DIR:-$WALE_ENV_DIR}"
 if [ "$DEMO" = "true" ]; then
     python3 /scripts/configure_spilo.py patroni pgqd certificate pam-oauth2
 elif python3 /scripts/configure_spilo.py all; then
-    CMD="/scripts/patroni_wait.sh -t 3600 -- envdir $WALE_ENV_DIR /scripts/postgres_backup.sh $PGDATA"
+    CMD="/scripts/patroni_wait.sh -t 3600 -- envdir $WALG_ENV_DIR /scripts/postgres_backup.sh $PGDATA"
     if [ "$(id -u)" = "0" ]; then
         su postgres -c "PATH=$PATH $CMD" &
     else

postgres-appliance/major_upgrade/inplace_upgrade.py

@@ -21,7 +21,7 @@
 RSYNC_PORT = 5432
 
 
-def patch_wale_prefix(value, new_version):
+def patch_walg_prefix(value, new_version):
     from spilo_commons import is_valid_pg_version
 
     if '/spilo/' in value and '/wal/' in value:  # path crafted in the configure_spilo.py?
@@ -51,20 +51,20 @@ def update_configs(new_version):
 
     write_patroni_config(config, True)
 
-    # update wal-e/wal-g envdir files
+    # update wal-g envdir files
     restore_command = shlex.split(config['postgresql'].get('recovery_conf', {}).get('restore_command', ''))
     if len(restore_command) > 6 and restore_command[0] == 'envdir':
         envdir = restore_command[1]
 
         try:
             for name in os.listdir(envdir):
-                # len('WALE__PREFIX') = 12
-                if len(name) > 12 and name.endswith('_PREFIX') and name[:5] in ('WALE_', 'WALG_'):
+                # len('WALG__PREFIX') = 12
+                if len(name) > 12 and name.endswith('_PREFIX') and name.startswith('WALG_'):
                     name = os.path.join(envdir, name)
                     try:
                         with open(name) as f:
                             value = f.read().strip()
-                        new_value = patch_wale_prefix(value, new_version)
+                        new_value = patch_walg_prefix(value, new_version)
                         if new_value != value:
                             write_file(new_value, name, True)
                     except Exception as e: