Upgrade Qdrant

Overview

LILT stores the vectors behind translation-memory matching and nearest-neighbour suggestions in Qdrant. The installer upgrades Qdrant in place, on its existing persistent volume, when you run the normal install command. Qdrant has two properties that make its upgrade different from every other component in LILT:

Qdrant migrates its storage one minor version at a time and cannot skip one. If your installation is more than one minor version behind the release you are installing, the installer stops rather than crossing several versions in a single restart, which can leave the storage partly migrated.
Qdrant cannot be rebuilt from the LILT database. A snapshot is the only way to recover its contents.

A snapshot is your only recovery path for Qdrant. No tool rebuilds the vectors from the LILT database, so an upgrade that damages the storage with no snapshot in hand is not recoverable. Losing Qdrant degrades translation-memory adaptation and nearest-neighbour matches.

Every command on this page assumes the LILT namespace is lilt. If you installed into a different namespace, substitute it for NAMESPACE.

Connect to Qdrant

Every curl command on this page reaches Qdrant through a port forward:

NAMESPACE=lilt
kubectl -n $NAMESPACE port-forward svc/qdrant 6333:6333 &

Leave this running while you work, and stop it when you finish.

Collections

Qdrant stores vectors in collections. LILT creates and names them itself, one per organization and language direction, so you never choose a name — you look up the ones your installation has. Names look like memory_segments_source_organization_BM25_9058, where the trailing number is the LILT organization ID. Wherever a command on this page shows COLLECTION, substitute one of these names:

curl -s localhost:6333/collections | jq -r '.result.collections[].name'

The backup procedure loops over that list, so it needs no names from you. The restore procedures act on one collection at a time, so they do. If the list comes back empty on a running Qdrant, nothing has been indexed yet and there is nothing to back up.

Version

To see the version that is running, read the image from the pods rather than from the StatefulSet. A rollout that failed leaves the StatefulSet naming the new image while the storage on disk is still the old version:

kubectl -n $NAMESPACE get pods -l app.kubernetes.io/name=qdrant \
  -o jsonpath='{range .items[*]}{.metadata.name}{"  "}{.status.containerStatuses[0].image}{"\n"}{end}'

Where Qdrant keeps snapshots

By default, Qdrant writes snapshots to /qdrant/snapshots, which is a directory on the same volume as its data. A volume you lose takes its snapshots with it. Because of that, always copy a snapshot off the pod, as the following procedure does. To keep snapshots on their own volume instead, contact LILT support: the change adds a second volume to the Qdrant StatefulSet, which Kubernetes cannot apply to a running StatefulSet without recreating it.

Back up before you upgrade

Take a snapshot before every upgrade. Each version step changes the data on disk and cannot be reversed.

Snapshot every collection and record its point count:

for c in $(curl -s localhost:6333/collections | jq -r '.result.collections[].name'); do
  snap=$(curl -s -X POST "localhost:6333/collections/$c/snapshots" | jq -r '.result.name')
  pts=$(curl -s "localhost:6333/collections/$c" | jq -r '.result.points_count')
  printf '%s\t%s\t%s\n' "$c" "$pts" "$snap"
done | tee qdrant-backup.tsv

Keep qdrant-backup.tsv. The point counts it records are how you tell a complete restore from a partial one.

Confirm that each snapshot completed. A snapshot with a size of 0, or one missing from this list, did not finish:
curl -s localhost:6333/collections/COLLECTION/snapshots \ | jq -r '.result[] | "\(.name)\t\(.size)"'
Substitute a collection name for COLLECTION.

Copy the snapshots off the pod:

POD=$(kubectl -n $NAMESPACE get pods -l app.kubernetes.io/name=qdrant \
  -o jsonpath='{.items[0].metadata.name}')
mkdir -p qdrant-backup
kubectl -n $NAMESPACE cp "$POD:/qdrant/snapshots" qdrant-backup

kubectl cp is slow on a large collection. To download one snapshot over the API instead:

curl -s "localhost:6333/collections/COLLECTION/snapshots/SNAPSHOT" \
  --output "qdrant-backup/SNAPSHOT"

Record the Qdrant version you noted earlier alongside the files. A snapshot restores only into the version that produced it or the next minor version, so the version is part of the backup.

Upgrade

Qdrant is upgraded by one component script, which the full installer also runs. For a Qdrant version change you do not need to re-run the whole installer:

sh install_scripts/install-qdrant.sh

A component script does not read install.env. Only install-lilt.sh and install-lilt-eks.sh load that file, so when you run a component on its own you have to load it yourself:

set -a; . ./install.env; set +a
sh install_scripts/install-qdrant.sh

On an AWS EKS installation, also set VALS_FILE, because the script defaults to qdrant/on-prem-values.yaml and the EKS installer package ships only qdrant/eks-values.yaml:

set -a; . ./install.env; set +a
VALS_FILE=eks-values.yaml sh install_scripts/install-qdrant.sh

Skipping install.env on AWS goes wrong in two ways. With DNS_DOMAIN unset the script stops with ERROR: DNS_DOMAIN is required. Worse, if DNS_DOMAIN happens to be exported but REGISTRY_BASE is not, the values resolve to a LILT-internal registry host your cluster cannot pull from, and the script completes — leaving the pod unable to start its new image. The check that catches a missing REGISTRY_BASE lives in the entrypoint, not in the component script.

The script runs helm upgrade, restarts the StatefulSet, and then waits up to five minutes for the pod to become ready. Qdrant migrates its storage while it starts, so on a large collection that wait can expire before the migration finishes. If it does, the script reports a failed rollout; check the pod before you re-run anything:

kubectl -n $NAMESPACE logs -f statefulset/qdrant

A pod still logging migration progress is working, not stuck. Wait for it to become ready, then confirm your collections as described in Verify the restore.

If the installer stops with a version error

The installer stops with a message like this:

install-qdrant: ERROR refusing to change Qdrant 1.16 -> 1.19 in place.

This means your Qdrant is more than one minor version behind the release you are installing. Qdrant cannot skip a minor version when it migrates its storage, and it cannot be downgraded, so you walk the existing instance up one minor version at a time until it is one version below the release.

Do not set QDRANT_ALLOW_VERSION_SKIP=true to get past this message. The check exists because crossing several minor versions in one restart can leave the storage silently half-migrated, and there is no way back.

Read these points before you start:

Run the climb inside the maintenance window, before you upgrade LILT itself. The climb moves the Qdrant server ahead of the LILT services still talking to it, and those services are only tested against the Qdrant version their release shipped with. Open the maintenance window, climb, then run the installer, which upgrades LILT to matching versions. Never climb while LILT is serving traffic.
Expect downtime at each step. A single-replica Qdrant stops and starts on the same volume, and it migrates the storage as it starts.
Every step is irreversible. Take the snapshot first.

Find the repository the running pod uses. The intermediate versions ship in your release under the same name:

REPO=$(kubectl -n $NAMESPACE get statefulset qdrant \
  -o jsonpath='{.spec.template.spec.containers[0].image}' | sed 's/:[^:]*$//')
echo "$REPO"

Read the intermediate tags your release ships. Do not type version numbers of your own: only the delivered tags exist in your registry, and their exact form varies between releases:
RUNGS=$(sed -n 's#.*/qdrant:$[^[:space:]]*$#\1#p' additional-shipped-images.yaml \ | while IFS= read -r t; do printf '%s\t%s\n' "${t#v}" "$t"; done \ | sort -V | cut -f2) echo "$RUNGS"
The tags print lowest version first. Confirm that each one you need is present in your registry before you start, because an air-gapped installation cannot fetch a missing one part-way through the climb and there is no way back:
for t in $RUNGS; do echo "$REPO:$t"; done
Start from the tag just above the lowest version any pod reports, and skip any rung at or below it.

Read the chart pin from the install script, so these commands cannot go stale against your release:

. install_scripts/utils.sh
eval "$(grep -E '^CHART_(NAME|VERSION)=' install_scripts/install-qdrant.sh)"
CHART="$(chart_ref "$CHART_NAME")"

Apply each remaining tag from $RUNGS, in order, one at a time. Wait for the pod to become ready before you move to the next — do not paste the whole sequence at once, because each step migrates the storage on disk:
helm upgrade qdrant "$CHART" -n $NAMESPACE --version "$CHART_VERSION" \ --reuse-values --set qdrant.image.tag=RUNG_TAG kubectl -n $NAMESPACE rollout status statefulset/qdrant --timeout=600s
Substitute one tag from $RUNGS for RUNG_TAG, then repeat for the next. Confirm the running image before continuing, since a failed rollout leaves the StatefulSet naming the new tag while the storage is still on the old version:
kubectl -n $NAMESPACE get pods -l app.kubernetes.io/name=qdrant \ -o jsonpath='{range .items[*]}{.status.containerStatuses[0].image}{"\n"}{end}'
Run the component script again. The remaining step is a single minor version, so the check passes and the script completes the upgrade with the values your release ships:
sh install_scripts/install-qdrant.sh
Load install.env first, and on AWS prefix the command with VALS_FILE=eks-values.yaml, as described in Upgrade.

Restore from a snapshot

A restore replaces the contents of the collection it targets. Run it in a maintenance window with LILT quiesced, otherwise LILT writes new vectors while the restore is in progress. Both procedures use priority=snapshot, which makes the snapshot’s data win over anything already in the collection. That is what a restore means. The default, replica, keeps the live data instead.

When the snapshot is still on the Qdrant volume

curl -s -X PUT "localhost:6333/collections/COLLECTION/snapshots/recover" \
  -H 'Content-Type: application/json' \
  -d '{"location": "file:///qdrant/snapshots/COLLECTION/SNAPSHOT", "priority": "snapshot"}'

Qdrant creates the collection if it does not exist, so this also works against an empty store after you lose a volume, as long as the snapshot survived. That is what copying snapshots off the pod is for.

When you are restoring a copy from outside the cluster

curl -s -X POST "localhost:6333/collections/COLLECTION/snapshots/upload?priority=snapshot" \
  -H 'Content-Type: multipart/form-data' \
  -F "snapshot=@qdrant-backup/SNAPSHOT"

The upload is a single request and cannot resume. For a snapshot of several gigabytes, copy the file into the pod first and then use the preceding procedure:

kubectl -n $NAMESPACE cp "qdrant-backup/SNAPSHOT" \
  "$POD:/qdrant/snapshots/COLLECTION/SNAPSHOT"

Verify the restore

Compare the point counts against the baseline you recorded:

while read -r c pts snap; do
  now=$(curl -s "localhost:6333/collections/$c" | jq -r '.result.points_count')
  [ "$now" = "$pts" ] && status=OK || status="MISMATCH (expected $pts)"
  printf '%-60s %-12s %s\n' "$c" "$now" "$status"
done < qdrant-backup.tsv

Every collection must report OK, and its status must be green:

curl -s localhost:6333/collections/COLLECTION | jq -r '.result.status'

A collection that reports yellow is still building its index. One that reports red failed to restore, so recover it again.

​Overview

​Connect to Qdrant

​Collections

​Version

​Where Qdrant keeps snapshots

​Back up before you upgrade

​Upgrade

​If the installer stops with a version error

​Restore from a snapshot

​When the snapshot is still on the Qdrant volume

​When you are restoring a copy from outside the cluster

​Verify the restore

​Related pages

Overview

Connect to Qdrant

Collections

Version

Where Qdrant keeps snapshots

Back up before you upgrade

Upgrade

If the installer stops with a version error

Restore from a snapshot

When the snapshot is still on the Qdrant volume

When you are restoring a copy from outside the cluster

Verify the restore

Related pages