aman
← all writeups

dnf installed a different Postgres

The PGDG repo won't install on Amazon Linux 2023, dnf quietly installed AL2023's own postgresql17 instead, and every path in the bootstrap script was wrong. Then cloud-init refused to rerun the fixed script.

Date
2026-06-04
Type
note
Impact
The self-managed Postgres EC2 instance for Keycloak came up with no database, which blocked moving Keycloak off in-cluster Postgres. Beta environment only.

The instance booted and user-data ran. There was no database. postgresql17-server showed as installed. /usr/pgsql-17/bin/postgresql-17-setup didn’t exist, and neither did postgresql-17.service. The package was there and the files weren’t.

What actually happens

The bootstrap script was written for PGDG on RHEL: add the pgdg-redhat-repo RPM, install postgresql17-server, init with /usr/pgsql-17/bin/postgresql-17-setup.

The host was Amazon Linux 2023 on Graviton. The PGDG repo RPM needs /etc/redhat-release, and AL2023 doesn’t have it, so the repo install failed. Then dnf install postgresql17-server still succeeded, because AL2023 ships its own postgresql17 package, 17.x-amzn2023. Same package name, different layout:

PGDG AL2023 native
binaries /usr/pgsql-17/bin /usr/bin
unit postgresql-17.service postgresql.service
init postgresql-17-setup initdb postgresql-setup --initdb

The fix was to drop PGDG. AL2023 already ships PG17. Set PGDATA in a systemd drop-in (Environment=PGDATA=...), which both postgresql-setup and the unit read, then run /usr/bin/postgresql-setup --initdb behind a PG_VERSION check.

Why the fix didn’t apply

I pushed the corrected user_data with Terraform. The instance stopped, started, and did nothing. cloud-init runs user-data once per instance ID. With user_data_replace_on_change = false, the default, Terraform swaps the stored script to /var/lib/cloud/instance/scripts/part-001 and never runs it.

What to do instead

  • On AL2023, use the native postgresql17 packages. Check dnf info first to see which repo a package really comes from.
  • To apply a user-data fix, use terraform apply -replace=<instance>, or run the stored script over SSM: sudo bash /var/lib/cloud/instance/scripts/part-001. That’s only safe if the script is idempotent, so guard mkfs and initdb.
  • The instance had no SSH. /var/log/user-data.log, pulled through ssm send-command, was the only view into what happened.

A successful package install tells you a package was installed. It doesn’t tell you it was the one you meant.

← all writeups