Skip to content

OpenSearch Backup and Restore

This guide covers snapshot management, Velero backups, and recovery procedures for the OpenSearch cluster. It specifies how you handle snapshots with OpenSearch in Kubernetes.

How It Works

The snapshot and restore process relies on the OpenSearch snapshot engine and native Kubernetes resource backups to ensure data integrity and cluster recoverability.

  1. Repository Binding (Remote Object Storage)

    OpenSearch uses native repository plugins (such as S3, Azure, or GCS) to bind the cluster to an external, offsite storage bucket or container.

    • Storage Path: All snapshots are written to a designated base path within the configured remote storage bucket.
    • Authentication: Connection credentials are managed securely via the storage client, utilizing cloud-native authentication methods (such as AWS IAM roles, Azure Managed Identities, or mounted Kubernetes secrets).

  2. Point-in-Time, Incremental Backups

    • Point-in-Time Consistency: When a snapshot command is executed, OpenSearch freezes the current state of the Lucene index segments. Any writes or updates that occur while the snapshot is running are safely isolated and do not corrupt the snapshot.
    • Incremental Storage: Only new or modified Lucene segments that do not exist in the repository are uploaded. This minimizes storage consumption, reduces network overhead, and speeds up subsequent backup runs.
    • Metadata Preservation: Along with index data, the snapshot captures cluster metadata, including index templates, mappings, aliases, and global settings.

  3. Data Restoration (System vs. User Indices)

    When restoring a snapshot to a clean or recreated cluster:

    • User Indices: All business data and custom indices are restored directly from the snapshot.
    • Security & System Indices: System-critical indices (such as .opendistro_security, .opensearch-observability, and security audit logs) are explicitly excluded from the snapshot restore (-d "{\"indices\": ...}"). Overwriting these dynamically on a running cluster can lock out administrators or break cluster authentication.

Create a Restore Snapshot in OpenSearch

Before performing upgrades or critical maintenance, create a manual OpenSearch snapshot.

Mount usage for curl commands

In the following code examples, there will be used cat /mnt/admin-credentials/username and cat /mnt/admin-credentials/password which is only possible if a secret is mounted in the pod.

Mount secret to pod

If the secret is not mounted in the pod, then we recommend manually create and mount the secret, passwords will be saved in the bash history for the pod.

First create the secret:

kubectl create secret generic opensearch-admin-secret \
  --from-literal=username=admin \
  --from-literal=password='YourAdminPassword' \
  -n opensearch-namespace

Define it in the opensearch-cluster-manager pod:

apiVersion: v1
kind: Pod
metadata:
  name: opensearch-cluster-manager-0
spec:
  containers:
  - name: opensearch
    # 1. Mount the volume to the custom path
    volumeMounts:
    - name: admin-creds-volume
      mountPath: "/mnt/admin-credentials"
      readOnly: true

  # 2. Define the volume based on the Secret
  volumes:
  - name: admin-creds-volume
    secret:
      secretName: opensearch-admin-secret

Validate Data Consistency

To ensure zero data loss, compare the document counts and a sample record before the maintenance/deletion against the newly restored cluster.

Verify the total document count matches your baseline:

kubectl exec -n opensearch-namespace opensearch-cluster-manager-0 -c opensearch -- bash -c 'curl -s -k -X GET -u "$(cat /mnt/admin-credentials/username):$(cat /mnt/admin-credentials/password)" "https://localhost:9200/top_queries-2026.07.10-44506/_count"' | jq .

You should see an output as such:

{
  "count": 397,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  }
}
After the restore you should run the command again and validate that the count is the same as before you took the snapshot.

Check Repository Config

Find the snapshots repository:

kubectl exec -n opensearch-namespace opensearch-cluster-manager-0 -c opensearch -- bash -c 'curl -s -k -u "$(cat /mnt/admin-credentials/username):$(cat /mnt/admin-credentials/password)" "https://localhost:9200/_snapshot"' | jq .

The output will show:

{
  "name": {
    "type": "<type>",
    "settings": {
      "container": "<name_of_container>",
      "client": "<type_of_client>",
      "base_path": "<the_base_path>"
    }
  }
}

⚠️ WARNING: Verify Storage Location

Before continuing, check the "type" field in the repository output above. It must indicate a remote storage location (e.g., "azure"). If the type is "fs" (File System), your snapshots are being saved locally to the pod's storage volume. Do not proceed with this guide if the type is "fs", as deleting the cluster and PVCs will permanently destroy your backups.

Create the Snapshot

Create a snapshot with the current date:

kubectl exec -n opensearch-namespace opensearch-cluster-manager-0 -c opensearch -- bash -c 'curl -s -k -X PUT -u "$(cat /mnt/admin-credentials/username):$(cat /mnt/admin-credentials/password)" "https://localhost:9200/_snapshot/kamstrup-os-snapshots/<snapshot_name>"' | jq .

Verify Snapshot Creation

Test to see if the snapshot was created successfully:

kubectl exec -n opensearch-namespace opensearch-cluster-manager-0 -c opensearch -- bash -c 'curl -s -k -X GET -u "$(cat /mnt/admin-credentials/username):$(cat /mnt/admin-credentials/password)" "https://localhost:9200/_snapshot/kamstrup-os-snapshots/<snapshot_name>"' | jq .

Restore Backup

If an upgrade fails and the cluster is unrecoverable, follow these steps to restore the OpenSearch data and dashboard.

The best way to restore the OpenSearch snapshot is to create a new OpenSearch cluster and restore the backup there.

The cleanest way is to:

  • Delete the PVCs for datanodes and managerpods in the OpenSearch namespace.
  • Delete the OpenSearch cluster.
  • Flux will recreate the cluster

Delete The Opensearch Cluster

Locate the Opensearch cluster: 

kubectl get opensearchcluster -n opensearch-namespace

Delete the Opensearch cluster which will ensure that all the pods and dependancies gets deleted, however it wont delte the pvc's, which you will have to delte manually

Delete the Opensearch Cluster: 

kubectl delete opensearchcluster <your-cluster-name> -n opensearch-namespace

Delete the pvc's: 

kubectl delete pvc -l opensearch.cluster=<name> -n opensearch-namespace

After deleteing the Opensearch cluster and a new one has been created with either flux or manually applyed by help, aswell as the cluster is green, you can proceed.

View Available Snapshots

List all snapshots inside the repository:

kubectl exec -n opensearch-namespace opensearch-cluster-manager-0 -c opensearch -- bash -c 'curl -s -k -X GET -u "$(cat /mnt/admin-credentials/username):$(cat /mnt/admin-credentials/password)" "https://localhost:9200/_snapshot/kamstrup-os-snapshots/_all"' | jq '.snapshots[] | {snapshot: .snapshot, state: .state}'

Restore OpenSearch Data

If you need to recreate the OpenSearch cluster, run the following command to restore data from your snapshot.

Note

Specify the snapshot name in place of <SnapshotName>.

kubectl exec -n opensearch-namespace opensearch-cluster-manager-0 -c opensearch -- bash -c 'curl -s -k -X POST -u "$(cat /mnt/admin-credentials/username):$(cat /mnt/admin-credentials/password)" -H "Content-Type: application/json" "https://localhost:9200/_snapshot/kamstrup-os-snapshots/<SnapshotName>/_restore?wait_for_completion=true" -d "{\"indices\": \"*,-.opendistro_security,-security-auditlog-*,-.opensearch-observability,-.plugins-ml-config,-.opensearch-sap-log-types-config\"}"' | jq .

If some indices cannot be restored because cluster recreation generated new indices with conflicting names, delete the local conflicting indices and retry the restore:

kubectl exec -n opensearch-namespace opensearch-cluster-manager-0 -c opensearch -- bash -c 'curl -s -k -X DELETE -u "$(cat /mnt/admin-credentials/username):$(cat /mnt/admin-credentials/password)" "https://localhost:9200/<index_name>"'

Validate Recovery

Check that your recovered indices are present and active:

kubectl exec -n opensearch-namespace opensearch-cluster-manager-0 -c opensearch -- bash -c 'curl -s -k -X GET -u "$(cat /mnt/admin-credentials/username):$(cat /mnt/admin-credentials/password)" "https://localhost:9200/_cat/indices?format=json"' | jq .

Verify index creation dates:

kubectl exec -n opensearch-namespace opensearch-cluster-manager-0 -c opensearch -- bash -c 'curl -s -k -X GET -u "$(cat /mnt/admin-credentials/username):$(cat /mnt/admin-credentials/password)" "https://localhost:9200/_cat/indices?format=json&h=index,docs.count,creation.date.string"' | jq .

Verify the total document count matches your baseline:

kubectl exec -n opensearch-namespace opensearch-cluster-manager-0 -c opensearch -- bash -c 'curl -s -k -X GET -u "$(cat /mnt/admin-credentials/username):$(cat /mnt/admin-credentials/password)" "https://localhost:9200/top_queries-2026.07.10-44506/_count"' | jq .

Verify that the "count" is the same as before the backup was taken:

{
  "count": 397,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  }
}

Restore Dashboards & Permissions

The Fix: Force-Syncing Your Security State

To resolve this issue and get your dashboard up and running, force Kubernetes and Flux to re-apply the credentials to OpenSearch.

  1. Clear current Kubernetes security states: Delete all existing OpenSearch security objects in the namespace. The opensearch operator will re-create them:
kubectl delete opensearchusers,opensearchroles,opensearchuserrolebindings,opensearchtenants -n opensearch-namespace --all
  1. Re-run the security update job Delete the old, completed update job. This forces Kubernetes to start a new pod that pushes your user configurations, role mappings, and dashboard credentials back into OpenSearch:
kubectl delete job opensearch-securityconfig-update -n opensearch-namespace
  1. Restart the Dashboard Pod Once the security job finishes, restart the dashboard pod so it can establish a fresh session using the newly injected credentials:
kubectl delete pod -n opensearch-namespace -l app=opensearch-dashboards

Validate Cluster Health

Verify the final health status of the cluster:

Command to curl to the application:

kubectl exec -n opensearch-namespace opensearch-cluster-manager-0 -c opensearch -- bash -c 'curl -s -k -X GET -u "$(cat /mnt/admin-credentials/username):$(cat /mnt/admin-credentials/password)" "https://localhost:9200/_cluster/health"' | jq .


{
  "cluster_name": "opensearch",
  "status": "green",
  "timed_out": false,
  "number_of_nodes": 7,
  "number_of_data_nodes": 4,
  "discovered_master": true,
  "discovered_cluster_manager": true,
  "active_primary_shards": 13,
  "active_shards": 39,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 0,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 100.0
}
Command to see the clusterhealth in the kubernetes cluster:

kubectl get opensearch -n opensearch-namespace

NAME         HEALTH   NODES   VERSION   PHASE     AGE
opensearch   green    7       2.19.5    RUNNING   45h