Zero-Downtime Agent Migration & Backup Recovery Runbooks

Zero-Downtime Agent Migration & Backup Recovery Runbooks

Migrating a stateless microservice is one of the most solved problems in cloud engineering: spin up a container replica on the new cluster, pass a readiness probe, shift DNS or load balancer traffic, and terminate the old pod.

Migrating an autonomous AI agent like OpenClaw, Hermes Agent, or n8n breaks that playbook completely.

Autonomous AI agents are inherently stateful, long-running cognitive daemons. At any given second, a production agent runtime may be holding:

1. Multi-turn conversation sessions and pending task queues in a relational database (PostgreSQL or SQLite).
2. High-dimensional vector memory indices (e.g., pgvector, Qdrant, Chroma, or sqlite-vec) representing months of user interactions, indexed documentation, and episodic learnings.
3. In-flight reasoning loops executing multi-step tool calls, such as browser automation sessions, shell commands, or external API transactions.
4. Inbound event streams and webhooks originating from GitHub, Stripe, Telegram, Slack, or internal monitoring systems.

If you treat an AI agent like a stateless container and perform a naive "stop old host, copy files, start new host" migration, you expose your operations to three severe failures:

• Cognitive split-brain and duplicate execution: If both old and new instances run concurrently without coordination, both agents may process the same inbound webhooks, firing duplicate API requests—such as placing duplicate orders, sending duplicate customer emails, or executing duplicate database mutations.
• Vector index corruption: Copying raw vector database files while background indexing or HNSW graph updates are occurring creates corrupted index segments, resulting in silent retrieval failures or sudden container crashes post-migration.
• Inbound trigger black holes: Webhooks arriving during DNS propagation or container restarts receive HTTP 502/504 errors, causing lost events unless an ingress buffering layer intercepts and retries them.

This runbook provides an engineering blueprint for migrating autonomous AI agents between hosts with zero downtime, zero lost memory state, and zero duplicate tool executions, backed by a production-tested disaster recovery and point-in-time recovery (PITR) strategy.

1. Anatomy of Agent State: What Must Move and What Must Drain

Before touching a single network route or filesystem volume, you must categorize your agent's state into four operational tiers. Each tier requires a different synchronization and cutover strategy.

| Tier | State Component | Typical Storage Engine | Consistency Requirement | Migration Strategy |

| :--- | :--- | :--- | :--- | :--- |

| Tier 1: Relational & Transactional | Message history, task queues, workflow execution status, tool audit logs | PostgreSQL, SQLite (WAL mode) | Strict serializability; zero uncommitted writes | Hot snapshot + continuous replication or WAL delta replay |

| Tier 2: Vector Memory & Embeddings | Episodic recall, semantic document chunks, HNSW / IVF index graphs | pgvector, Qdrant, Chroma, sqlite-vec | Read consistency; atomic index flush | Collection snapshot export + segment file verification |

| Tier 3: File System Blobs & Config | Agent custom skills (SKILL.md), cloned git repos, browser session cookies, local auth tokens | Local persistent volume / Docker volume | File integrity; read-only lock during cutover | Multi-pass rsync with hard links and checksum validation |

| Tier 4: Transient Cognitive State | In-flight LLM reasoning loops, open browser instances, streaming API connections | Container memory (RAM) | Ephemeral; cannot be live-migrated across processes | Graceful execution drain (SIGTERM with 30s timeout) |

The Danger of In-Flight Reasoning Loops

Unlike a web server where an HTTP request completes in 50 milliseconds, an autonomous agent's ReAct (Reason + Act) loop can take 30 to 180 seconds to complete. During this window, the agent queries an LLM, inspects the response, calls an external bash tool, waits for output, and writes observations back to memory.

If you sever the process midway through a tool execution:

• The external side-effect has already occurred (e.g., a cloud server was provisioned or an email was sent).
• The agent's local memory store is never updated with the tool's return value.
• On reboot, the agent inspects its state, assumes the step failed or never ran, and re-executes the tool call—producing a dangerous duplicate side-effect.

Therefore, zero-downtime agent migration requires an explicit drain phase: the active agent stops accepting new triggers, completes its active cognitive cycles, commits its final observations to disk, and transitions to passive read-only mode before traffic flips.

2. Architectural Blueprint: The Blue/Green Ingress Buffer

To achieve zero downtime without duplicate executions, we deploy a Blue/Green migration topology with an Ingress Buffer.

```mermaid

flowchart TD

subgraph External["External Triggers"]

WH["Webhooks (GitHub, Stripe, Slack)"]

Users["User Chat Messages (API / WebUI)"]

end

subgraph Ingress["Ingress & Buffering Layer"]

Proxy["Reverse Proxy (Nginx / Caddy)"]

Buffer["Ingress Buffer / Retry Queue"]

Proxy --> Buffer

end

subgraph BlueCluster["Blue Environment (Old Host)"]

BlueAgent["Blue Agent Runtime"]

BlueDB[("Blue State & Vector Store")]

BlueAgent <--> BlueDB

end

subgraph GreenCluster["Green Environment (New Host)"]

GreenAgent["Green Agent Runtime"]

GreenDB[("Green State & Vector Store")]

GreenAgent <--> GreenDB

end

WH --> Proxy

Users --> Proxy

Buffer -.->|"1. Active Route (Phase 1-2)"| BlueAgent

Buffer ==>|"4. Cutover Route (Phase 3)"| GreenAgent

BlueDB ==>|"2. Continuous State Replication & WAL Sync"| GreenDB

```

The Ingress Buffering Mechanism

During the brief cutover window (typically 5 to 15 seconds) when Blue is draining and Green is initializing, external webhook senders will continue firing events.

Rather than relying on external services to implement graceful exponential backoff (many third-party webhooks give up after a single failure or retry hours later), your ingress reverse proxy must buffer incoming requests.

Here is a production Nginx reverse proxy configuration that buffers inbound webhook triggers, absorbs transient upstream disconnects, and automatically retries requests against the backup upstream during promotion:

```nginx

/etc/nginx/conf.d/agent_ingress.conf

upstream agent_cluster {

# Blue is primary initially; Green is backup

server 10.0.1.10:8000 max_fails=2 fail_timeout=10s;

server 10.0.2.10:8000 backup;

keepalive 32;

}

server {

listen 443 ssl http2;

server_name agent.example.com;

ssl_certificate /etc/letsencrypt/live/agent.example.com/fullchain.pem;

ssl_certificate_key /etc/letsencrypt/live/agent.example.com/privkey.pem;

# Buffer client request bodies to disk/memory before proxying

client_body_buffer_size 128k;

client_max_body_size 50M;

location / {

proxy_pass http://agent_cluster;

proxy_http_version 1.1;

proxy_set_header Connection "";

proxy_set_header Host $host;

proxy_set_header X-Real-IP $remote_addr;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

proxy_set_header X-Forwarded-Proto $scheme;

# Transparently retry on error or during upstream reload

proxy_next_upstream error timeout invalid_header http_502 http_503 http_504;

proxy_next_upstream_tries 3;

proxy_next_upstream_timeout 30s;

# Timeouts accommodated for long LLM reasoning streams

proxy_connect_timeout 5s;

proxy_send_timeout 300s;

proxy_read_timeout 300s;

}

}

```

3. Step-by-Step Migration Runbook

Follow these four structured phases to execute the migration without losing state or dropping triggers.

Phase 1: Environment Pre-Flight & Baseline File Mirroring

On the target host (Green), provision the container runtime environment matching the resource envelope defined in your HostAgentics Cloud plan (e.g., matching CPU, RAM, and persistent disk mount points).

1. Baseline Filesystem Mirroring (Pass 1)

While the Blue agent is running normally, perform an initial baseline transfer of static workspaces, custom skills, and configurations using rsync. This moves 99% of bulk data over the wire without impacting production:

```bash

#!/usr/bin/env bash

Step 1: Initial baseline data synchronization (Blue -> Green)

set -euo pipefail

TARGET_HOST="green.internal.hostagentics.com"

DATA_DIR="/var/lib/hostagentics/data"

echo "==> Executing Pass 1 baseline rsync..."

rsync -avzP \

--hard-links \

--sparse \

--numeric-ids \

--exclude="*.sock" \

--exclude="*.tmp" \

--exclude="*.pid" \

"${DATA_DIR}/" \

"root@${TARGET_HOST}:${DATA_DIR}/"

echo "==> Pass 1 baseline rsync completed successfully."

```

2. Vector Database Snapshotting

If you run Qdrant, Chroma, or pgvector, you must generate an immutable, point-in-time snapshot to avoid capturing half-written index segments.

For Qdrant:

```bash

Trigger an internal snapshot of the agent memory collection

curl -X POST "http://localhost:6333/collections/agent_memories/snapshots" \

-H "Content-Type: application/json"

Download the created snapshot directly to the Green host

SNAPSHOT_NAME=$(curl -s "http://localhost:6333/collections/agent_memories/snapshots" | jq -r '.result[-1].name')

scp "/var/lib/qdrant/snapshots/agent_memories/${SNAPSHOT_NAME}" "root@${TARGET_HOST}:/tmp/"

On Green host: recover collection from snapshot

ssh "root@${TARGET_HOST}" "curl -X POST 'http://localhost:6333/collections/agent_memories/snapshots/recover' \

-H 'Content-Type: application/json' \

-d '{\"location\": \"file:///tmp/${SNAPSHOT_NAME}\"}'"

```

For SQLite-based agent memory (e.g., OpenClaw default memory store):

Never copy an active SQLite database with standard cp or rsync while in WAL (Write-Ahead Logging) mode, as the -wal and -shm shared memory files may be in an inconsistent state. Use the official SQLite Online Backup API via the CLI:

```bash

Safely snapshot live SQLite database to a consistent single-file backup

sqlite3 /var/lib/hostagentics/data/openclaw.db ".backup '/var/lib/hostagentics/data/openclaw_snapshot.db'"

Verify integrity of the generated snapshot

sqlite3 /var/lib/hostagentics/data/openclaw_snapshot.db "PRAGMA integrity_check;"

```

Phase 2: Dual-State Replication & Warm Standby

At this stage, Green has all data up to the snapshot point. Now, launch the Green container stack in Warm Standby mode:

1. Green's background consumers and crons are disabled (AGENT_CRON_ENABLED=false, AGENT_WORKER_CONCURRENCY=0).
2. Green's database engines are running and accepting streaming replication or delta WAL files.
3. Green answers local health checks (curl http://127.0.0.1:8000/healthz returns HTTP 200).

Verify that Green's runtime environment matches expectations:

• Model provider API keys and endpoints respond correctly.
• MCP (Model Context Protocol) tool servers initialize without permission errors.
• Internal network paths to databases and caching layers are reachable.

Phase 3: The 15-Second Cutover Window

Now execute the atomic transition. Run this cutover script on the Blue host:

```bash

#!/usr/bin/env bash

Step 3: Zero-downtime cutover script

set -euo pipefail

TARGET_HOST="green.internal.hostagentics.com"

PROXY_HOST="proxy.internal.hostagentics.com"

DATA_DIR="/var/lib/hostagentics/data"

echo "=================================================="

echo "Starting Zero-Downtime Agent Cutover Procedure"

echo "=================================================="

1. Put Blue Agent into Drain Mode (stops accepting new webhook triggers)

echo "==> Signaling Blue Agent to enter DRAIN mode..."

curl -s -X POST "http://127.0.0.1:8000/api/admin/drain" \

-H "Authorization: Bearer ${ADMIN_SECRET}" \

-d '{"timeout_seconds": 25}'

2. Wait for active ReAct reasoning loops to finish

echo "==> Waiting for active tool executions to settle..."

while [ $(curl -s "http://127.0.0.1:8000/api/admin/active-tasks" | jq '.count') -gt 0 ]; do

sleep 1

done

echo "==> Active tasks drained to zero."

3. Stop Blue Agent process to release all database locks

echo "==> Halting Blue Agent container..."

docker stop --time 15 hostagentics-agent-blue

4. Flush SQLite WAL to database disk file

sqlite3 "${DATA_DIR}/openclaw.db" "PRAGMA wal_checkpoint(TRUNCATE);"

5. Final Pass Delta rsync (fast: only transfers files changed during the drain)

echo "==> Performing final delta rsync..."

rsync -avz \

--hard-links \

--delete \

"${DATA_DIR}/" \

"root@${TARGET_HOST}:${DATA_DIR}/"

6. Promote Green Agent on the Target Host

echo "==> Promoting Green Agent to PRIMARY..."

ssh "root@${TARGET_HOST}" 'bash -s' << 'EOF'

set -e

# Enable worker loops and crons

sed -i 's/AGENT_WORKER_CONCURRENCY=0/AGENT_WORKER_CONCURRENCY=4/g' /etc/hostagentics/agent.env

sed -i 's/AGENT_CRON_ENABLED=false/AGENT_CRON_ENABLED=true/g' /etc/hostagentics/agent.env

# Restart container with active configuration

docker restart hostagentics-agent-green

# Await healthy probe

until curl -sf http://127.0.0.1:8000/healthz > /dev/null; do

echo "Waiting for Green Agent to report healthy..."

sleep 1

done

EOF

7. Atomically switch Reverse Proxy upstream routing

echo "==> Flipping Reverse Proxy upstream to Green..."

ssh "root@${PROXY_HOST}" 'bash -s' << 'EOF'

set -e

sed -i 's/server 10.0.1.10:8000 max_fails=2 fail_timeout=10s;/server 10.0.1.10:8000 down;/g' /etc/nginx/conf.d/agent_ingress.conf

sed -i 's/server 10.0.2.10:8000 backup;/server 10.0.2.10:8000 max_fails=2 fail_timeout=10s;/g' /etc/nginx/conf.d/agent_ingress.conf

nginx -s reload

EOF

echo "=================================================="

echo "Cutover complete! All traffic now serving from Green."

echo "=================================================="

```

4. Post-Migration Verification & Semantic Recall Audit

Once traffic switches to Green, you must not assume the agent is functioning simply because the container returns HTTP 200. You must execute a programmatic semantic integrity audit to verify that the agent's long-term memory, vector similarity scoring, and tool integrations survived the migration uncorrupted.

Save and execute this verification script against the Green agent:

```python

#!/usr/bin/env python3

"""

Post-Migration Agent Memory and Semantic Integrity Verification Script.

Audits vector recall, database record counts, and tool connectivity.

"""

import sys

import json

import urllib.request

import urllib.error

AGENT_ENDPOINT = "http://127.0.0.1:8000"

ADMIN_TOKEN = "your-admin-verification-token"

def request(path, payload=None):

url = f"{AGENT_ENDPOINT}{path}"

headers = {

"Authorization": f"Bearer {ADMIN_TOKEN}",

"Content-Type": "application/json"

}

data = json.dumps(payload).encode("utf-8") if payload else None

req = urllib.request.Request(url, data=data, headers=headers)

try:

with urllib.request.urlopen(req, timeout=10) as resp:

return json.loads(resp.read().decode("utf-8"))

except urllib.error.HTTPError as e:

print(f"[FAIL] HTTP {e.code} on {path}: {e.read().decode('utf-8')}")

sys.exit(1)

except Exception as e:

print(f"[FAIL] Connection error on {path}: {str(e)}")

sys.exit(1)

def verify_agent():

print("Starting Post-Migration Semantic Integrity Check...\n")

# 1. Health and runtime verification

print("1. Checking runtime status...")

status = request("/api/healthz")

assert status.get("status") == "healthy", "Agent health check failed"

print(" [PASS] Container is healthy and worker loops are active.")

# 2. Database Record Count Sanity Check

print("2. Verifying relational memory records...")

db_stats = request("/api/admin/db-stats")

messages = db_stats.get("total_messages", 0)

sessions = db_stats.get("total_sessions", 0)

print(f" [INFO] Found {sessions} sessions and {messages} conversation messages.")

assert sessions > 0, "No historical sessions found; state was lost!"

print(" [PASS] Relational message state verified.")

# 3. Vector Store Semantic Recall Verification

print("3. Testing vector memory similarity recall...")

# Query for a known foundational context chunk saved in previous months

test_query = {

"query": "infrastructure deployment credentials and cluster topology",

"top_k": 3

}

vector_results = request("/api/memory/vector-search", test_query)

matches = vector_results.get("results", [])

assert len(matches) > 0, "Vector memory returned 0 results! Vector index corrupted or empty."

top_score = matches[0].get("score", 0.0)

print(f" [INFO] Top semantic similarity score: {top_score:.4f}")

assert top_score > 0.70, f"Semantic score too low ({top_score:.4f}); embeddings drifted or index misaligned."

print(" [PASS] Vector memory embeddings and nearest-neighbor search operational.")

# 4. Tool Registry and Credential Verification

print("4. Validating tool execution permissions...")

tool_check = request("/api/tools/verify-credentials")

failed_tools = tool_check.get("failed_integrations", [])

if failed_tools:

print(f" [FAIL] The following integrations failed authentication: {failed_tools}")

sys.exit(1)

print(" [PASS] All MCP and API integration credentials valid.")

print("\n=======================================================")

print("ALL INTEGRITY CHECKS PASSED: Green agent is fully operational!")

print("=======================================================")

if __name__ == "__main__":

verify_agent()

```

5. Disaster Recovery Runbook: Rapid Point-in-Time Recovery (PITR)

Even with automated migration runbooks, real-world infrastructure incidents happen: accidental volume deletions, corrupted SQLite headers, or rogue agent tool loops that delete local directories.

To recover from catastrophic data loss, follow this Point-in-Time Recovery (PITR) procedure.

```mermaid

sequenceDiagram

autonumber

actor Admin as DevOps Engineer

participant Snapshot as HostAgentics Cloud Storage

participant Target as Recovery Host

participant Agent as Restored Agent Runtime

Admin->>Snapshot: Locate latest verified daily snapshot + WAL archive

Snapshot-->>Admin: Return manifest with SHA256 checksums

Admin->>Target: Download and verify artifact checksums

Admin->>Target: Mount fresh persistent volume

Admin->>Target: Extract baseline snapshot into /var/lib/hostagentics/data

Admin->>Target: Replay continuous WAL deltas up to target timestamp

Admin->>Target: Execute SQLite / Postgres PRAGMA integrity_check

Target-->>Admin: Database reported OK

Admin->>Agent: Boot container in recovery mode with crons disabled

Admin->>Agent: Run semantic recall audit script

Agent-->>Admin: Recall verified

Admin->>Agent: Promote to live production & enable triggers

```

The 3-2-1 Backup Verification Protocol

A backup file that has never been restored is not a backup; it is merely an unverified assumption. In HostAgentics Cloud, backup recovery relies on the following operational standards:

1. Volume Snapshots with Manifest Checksums: Every automated daily snapshot generates an accompanying JSON manifest recording total byte count, SHA256 hashes of database files, and vector index metadata.
2. Storage Separation: Backups are written to distinct object storage clusters isolated from the running compute instance. If the compute host suffers a hardware failure, the storage snapshot remains fully accessible.
3. Automated Restoration Sandbox: Restoration drills should be executed monthly on an isolated scratch container to measure real-world Recovery Time Objective (RTO).

Disaster Recovery Execution Steps

When recovering from a catastrophic failure:

Step 1: Provision Clean Volume and Download Artifacts

Never restore backup files directly over a corrupted or suspect disk. Provision a clean, empty volume on the target host:

```bash

mkdir -p /var/lib/hostagentics/restore

cd /var/lib/hostagentics/restore

Download verified snapshot and sha256 checksum manifest

s3cmd get s3://hostagentics-backups/instances/inst_99482/snapshot_latest.tar.gz .

s3cmd get s3://hostagentics-backups/instances/inst_99482/snapshot_latest.sha256 .

Cryptographically verify the archive before extraction

sha256sum -c snapshot_latest.sha256

```

Step 2: Unpack and Replay Point-in-Time WAL Logs

Extract the database files and replay any incremental write-ahead log (WAL) segments collected between the snapshot timestamp and the incident timestamp:

```bash

tar -xzvf snapshot_latest.tar.gz -C /var/lib/hostagentics/restore/

For SQLite: replay WAL and verify integrity

sqlite3 /var/lib/hostagentics/restore/openclaw.db "PRAGMA integrity_check;"

sqlite3 /var/lib/hostagentics/restore/openclaw.db "PRAGMA foreign_key_check;"

Swap restored directory into live data location

systemctl stop hostagentics-agent || docker stop hostagentics-agent

mv /var/lib/hostagentics/data /var/lib/hostagentics/data_corrupted_backup_$(date +%s)

mv /var/lib/hostagentics/restore /var/lib/hostagentics/data

chown -R 1000:1000 /var/lib/hostagentics/data

```

Step 3: Start in Quarantined Observation Mode

Do not expose the recovered agent directly to public webhook triggers immediately. Boot the container with external webhooks and scheduled cron triggers temporarily disabled:

```bash

docker run -d \

--name hostagentics-agent-recovery \

-e AGENT_WORKER_CONCURRENCY=0 \

-e AGENT_CRON_ENABLED=false \

-e WEBHOOK_ACCEPT_INBOUND=false \

-v /var/lib/hostagentics/data:/app/data \

hostagentics/openclaw:latest

```

Execute the semantic verification script (test_agent_memory_integrity.py). Once semantic recall and database statistics are confirmed, enable incoming traffic and re-enable cron triggers.

6. Production Operations Checklist

Keep this operational checklist on hand whenever executing host migrations or planning contingency exercises.

Pre-Migration Readiness

• [ ] DNS TTL Reduction: Lower DNS TTL on agent domain names to 60 seconds at least 24 hours prior to cutover.
• [ ] Ingress Reverse Proxy Deployed: Confirm Nginx or Caddy reverse proxy is positioned in front of the agent with request buffering and proxy_next_upstream retry rules enabled.
• [ ] Baseline Rsync Completed: Bulk workspaces, logs, and custom skills synchronized to Green host.
• [ ] Vector Store Snapshot Exported: Immutable collection snapshot exported and restored to Green vector database.
• [ ] Warm Standby Validated: Green host container running in passive mode; internal health checks passing.

Cutover Execution

• [ ] Drain Mode Initiated: Blue agent signaled to stop accepting new requests; active ReAct loops allowed up to 30 seconds to complete.
• [ ] Blue Process Halted: Blue container stopped to release database write locks and flush SQLite WAL files.
• [ ] Final Delta Rsync: Incremental changes copied over within 2–5 seconds.
• [ ] Green Promotion: Green worker concurrency and cron timers enabled; container restarted.
• [ ] Proxy Upstream Switched: Reverse proxy reloaded to route all inbound traffic to Green.

Post-Migration Verification

• [ ] Relational Record Count Verified: Check that conversation session counts and message IDs match pre-migration counts.
• [ ] Vector Recall Test Passed: Execute test vector similarity search; verify cosine similarity score is within expected bounds (>0.70).
• [ ] Tool Authentication Validated: Confirm API keys, OAuth tokens, and MCP connections authenticate successfully.
• [ ] Webhook Deliveries Audited: Inspect ingress reverse proxy logs for incoming webhooks; confirm HTTP 200 responses and zero HTTP 502/504 errors.
• [ ] 24-Hour Observation Window: Maintain Blue host in stopped, read-only state for 24 hours before permanently decommissioning storage.

Summary

Migrating an autonomous AI agent requires an operational discipline that recognizes the uniqueness of agentic systems. Agents are not purely stateless web servers, nor are they static databases—they are hybrid cognitive runtimes that continuously accumulate memory, reason across multi-step tool graphs, and react to real-time events.

By combining an Ingress Buffering Proxy, SQLite/Postgres Online Snapshots, Controlled Execution Draining, and Automated Semantic Recall Auditing, you eliminate the risks of split-brain executions, vector corruption, and lost triggers.

On HostAgentics Cloud, runtimes benefit from isolated container environments, persistent NVMe storage volumes, and automated daily provider snapshots designed to make these operational runbooks predictable, robust, and repeatable.