> ## Documentation Index
> Fetch the complete documentation index at: https://formbricks.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshoot an upgrade

> Diagnose a blocked migration and choose between repair and restore.

Keep maintenance active when an upgrade fails. Record the failed step, source and target images,
and sanitized error code. Retry only after checking for partial changes and concurrent migration runners.

* **Preparation blocked:** use [preparation results](https://formbricks.com/docs/self-hosting/upgrade/prepare.md#results-and-recovery).
* **Database migration failed:** follow the diagnostic steps below.
* **Restore required:** use [backup and restore](https://formbricks.com/docs/self-hosting/operate/backup-restore.md).
* **`StatefulSet.apps "formbricks-redis-master" is invalid: spec: Forbidden`:** see
  [Upgrade fails on the Valkey StatefulSet](https://formbricks.com/docs/self-hosting/upgrade/kubernetes.md#valkey-statefulset-forbidden).

A completed migration is not the final acceptance check. Return to your deployment recipe before
restarting writers or reopening traffic.

## If a migration fails

From Formbricks 5.3 on, one runner applies every database migration in order: the `formbricks-migrate` service on
Docker Compose, the `formbricks-migration` Job on Kubernetes (the application pods themselves when the chart sets
`migration.enabled: false`), or `formbricks-authzed-prepare` during a v5 to v6 upgrade. When a migration fails,
the runner stops and no later migration runs. A failed schema migration then blocks every later run until you
resolve it; a failed data migration is retried on the next run.

> [!WARNING]
>
> Do not delete volumes, restore a backup or roll the images back before you know which migration failed and
> why. Most failures are fixed in place. The output below can contain SQL and data from your instance, so keep
> it out of public issues.

### Open a SQL session

The steps below run SQL as the user that owns the Formbricks tables:

> [tab] One-Click / Docker Compose
>
> ```bash theme={null}
> docker compose exec postgres psql -U postgres -d formbricks
> ```

> [tab] Kubernetes
>
> ```bash theme={null}
> kubectl exec -it -n formbricks formbricks-postgresql-0 -- psql -U formbricks -d formbricks
> ```
>
> > [!NOTE]
> >
> > If your PostgreSQL pod has a different name, run `kubectl get pods -n formbricks` to find it.

With an external or managed PostgreSQL, connect with `psql` as the user in `MIGRATE_DATABASE_URL`, or in
`DATABASE_URL` if you do not set it. The migrations themselves need a direct or session-pooled connection: a
transaction-mode pooler such as PgBouncer can split their session settings across server connections.

### 1. Find the failed migration

The runner's output names it. `docker compose run … formbricks-migrate` and the update script print it to your
terminal; `docker compose logs --tail=200 formbricks-migrate` has it only when the service ran under
`docker compose up`. On Kubernetes, read `kubectl logs -n formbricks job/formbricks-migration` (by default the
chart deletes the Job 300 seconds after it fails). A first failure prints `Schema migration batch failed:` with
every migration in that batch; the error names the failed one as `Migration name:`, or as `migration_name="…"`
when the real error was hidden. Every run after that stops with `P3009` and names it again. If you have no log,
including with `formbricks-authzed-prepare`, which prints only `upgrade_database_migration_failed` (plus
`"reason":"timeout"` when the step ran out its time limit rather than failing), the database always knows:

```sql theme={null}
SELECT migration_name, started_at FROM "_prisma_migrations"
WHERE finished_at IS NULL AND rolled_back_at IS NULL;
```

If that returns no row but the log says `Data migration … failed`, see [Data migrations](#data-migrations)
below. If the log just stops without an error, the runner was stopped (Compose stops it after 300 seconds) and
its last statement may still be running in PostgreSQL: check that first, as step 3 describes.

### 2. Find the real error

The runner's error usually shows it after `Database error:`. If it shows only `current transaction is aborted,
commands ignored until end of transaction block`, the migration runs in a transaction of its own and the
[real error was hidden](https://github.com/prisma/prisma/issues/15295). Find it in the PostgreSQL log, on the
`ERROR` line just before that one:

```bash theme={null}
docker compose logs postgres | grep -E "ERROR|STATEMENT"
kubectl logs -n formbricks formbricks-postgresql-0 | grep -E "ERROR|STATEMENT"
```

On a managed service, use your provider's database log. `canceling statement due to lock timeout` is the most
common cause: another session held a lock on the table the migration changes, for example a long-running query,
a backup, or an old Formbricks pod still running. Stop it, or retry at a quieter time. If the error itself lists
recovery steps, as some migrations' guards do, follow those.

### 3. Check what the migration left behind

First make sure nothing is still running it. On Compose, the runner has exited. On Kubernetes, the Job has
finished retrying (`kubectl wait -n formbricks --for=condition=Failed job/formbricks-migration --timeout=10m`
returns, or the Job is gone). With `migration.enabled: false` the application pods are the runner and retry on
every restart, so scale them to 0 (`kubectl scale deploy -n formbricks formbricks --replicas=0`) until step 4. A
statement keeps running in PostgreSQL after its runner is stopped, so also list what the database is still
running:

```sql theme={null}
SELECT pid, now() - query_start AS running_for, left(query, 80) AS query,
       pid IN (SELECT pid FROM pg_locks WHERE locktype = 'advisory' AND objid = 72707369) AS migration_lock
FROM pg_stat_activity
WHERE datname = current_database() AND state IS DISTINCT FROM 'idle' AND pid <> pg_backend_pid();
```

`migration_lock` marks a schema migration: Prisma holds that lock while it applies one. Data migrations take no
such lock, so also look for statements on the tables the migration changes. Wait for every migration statement
to end.

Then read the failed migration's file. A released migration never changes, so the copy in your image is the one
that ran:

```bash theme={null}
migration_name='<the name from step 1>'
docker compose run --rm --no-deps --entrypoint cat formbricks-migrate \
  "packages/database/dist/migration/${migration_name}/migration.sql"
```

It is also on GitHub under any release that ships it, for example
`https://github.com/formbricks/formbricks/blob/6.0.2/packages/database/migration/<migration_name>/migration.sql`.

* **The runner printed an error, and the whole file is one `BEGIN; … COMMIT;` block, with no statement before
  `BEGIN;` or after `COMMIT;`:** PostgreSQL rolled the whole file back, so there is nothing to undo. A file that
  commits a block and then runs more statements, such as `20250911192630_remove_deprecated_fields_and_tables`,
  belongs under the next case.
* **Otherwise, including any migration whose runner was stopped instead of printing an error:** the statements
  before the failing one may have taken effect, even inside a `BEGIN; … COMMIT;` block. For a stopped runner,
  possibly every statement in the file did: a file Prisma sends as one string runs to its end without a client.
  Check which did (`\d "Table"` for schema changes, a `SELECT` for data) and undo those before you continue. If
  the new version is already serving, stop it first (`docker compose stop formbricks`, or scale the Deployment to
  0\) and keep it stopped until you run the migrations again. Undoing a `DROP` means recreating the object so the
  retry can drop it again; if a change cannot be undone, such as an `UPDATE` whose old values are gone, stop and
  restore the pre-upgrade backup instead. You need not undo:
  * `IF NOT EXISTS`, `IF EXISTS` and `CREATE OR REPLACE` statements are safe to repeat, apart from a failed
    `CREATE INDEX CONCURRENTLY`. That leaves an invalid index which `IF NOT EXISTS` then skips: list invalid
    indexes with `SELECT indexrelid::regclass FROM pg_index WHERE NOT indisvalid;` and drop only those the
    migration's file creates, with `SET lock_timeout = 0;` and then `DROP INDEX CONCURRENTLY`.
  * An `UPDATE`, `INSERT` or `DELETE` is safe to repeat only if running it twice changes nothing.

### 4. Mark it rolled back and run the migrations again

Once the cause is fixed, mark the failed migration rolled back. This is what
`prisma migrate resolve --rolled-back` does. It should report `UPDATE 1`; `UPDATE 0` means the name is wrong or
the migration is not recorded as failed:

```sql theme={null}
UPDATE "_prisma_migrations" SET rolled_back_at = now()
WHERE migration_name = '<migration_name>' AND finished_at IS NULL AND rolled_back_at IS NULL;
```

Then run the migrations again:

> [tab] One-Click / Docker Compose
>
> ```bash theme={null}
> docker compose run --rm --no-deps formbricks-migrate
> ```
>
> The service stops the runner after 300 seconds. If a migration needs longer, run it without that limit:
>
> ```bash theme={null}
> docker compose run --rm --no-deps formbricks-migrate "node packages/database/dist/scripts/apply-migrations.js"
> ```
>
> During a v5 to v6 upgrade, re-run `formbricks-authzed-prepare` as the
> [v6 maintenance upgrade guide](https://formbricks.com/docs/self-hosting/upgrade/versions/6.0.md) describes instead.

> [tab] Kubernetes
>
> Delete the failed Job, then run the same `helm upgrade` (or Argo CD sync) again. Helm does not re-run a Job
> that still exists, and refuses to change one (`field is immutable`):
>
> ```bash theme={null}
> kubectl delete job -n formbricks formbricks-migration --ignore-not-found
> ```
>
> With `migration.enabled: false`, scale the Deployment back up instead; its pods run the migrations on start.
>
> Otherwise, if you scaled the Deployment to 0 in step 3, scale it back up once the Job succeeds
> (`kubectl scale deploy -n formbricks formbricks --replicas=<your count>`). `helm upgrade` restores the count
> only when `autoscaling.enabled` is false and `deployment.replicas` is set, and an autoscaler does not scale up
> from 0.
>
> During a v5 to v6 upgrade, delete and re-create the `formbricks-v6-prepare` Job as the
> [v6 maintenance upgrade guide](https://formbricks.com/docs/self-hosting/upgrade/versions/6.0.md) describes instead.

### Data migrations

Most data migrations run in a transaction of their own and leave nothing behind when they fail; the large ones
commit in batches and are written to resume where they stopped. Either way the runner records the failure and
retries the migration on the next run: fix the cause and run the migrations again. The exception is
`Migration is pending. Please resolve the issue manually.`, left by an interrupted run of a version before 5.3.
Make sure nothing is still running it (step 3), then mark it failed so the next run retries it:

```sql theme={null}
SELECT name, status, started_at FROM "DataMigration" WHERE status <> 'applied';
UPDATE "DataMigration" SET status = 'failed' WHERE name = '<name>' AND status = 'pending';
```


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.