Operations: recovery and hardening

This chapter is the runbook for the public Thor Tigress Cub: what is public, where every setting lives, what starts at boot, what to do when something leaks or breaks, and the gaps that are known and not yet closed. It changes no code; each fix it describes is a set of commands or clicks.

What is public, and what protects the Thor

Check the certificate record yourself:

curl -s "https://api.certspotter.com/v1/issuances?domain=arpanpathak.taildb9a39.ts.net&expand=dns_names"

It lists the certificate issued on 2026-10-05 at 07:15 UTC. Every Funnel address is public this way the moment Funnel is turned on, so keeping the name out of documents would not hide it. The key is the protection, plus the fact that only port 8080 is reachable and only a fixed set of paths answers (chapter “Security”).

What the public name does allow: anyone can load the page, see the invite screen, call /health, and try keys. A new name helps against a flood of unwanted traffic; it does nothing for a leaked key, which works on any name.

Everything that makes it work

On the Thor

On each machine you work from

Optional; what the guides in this book set up. Paths are the same on Linux and macOS.

Domain and GitHub

Where the Tailscale name is written

jetson-thor/web/index.html, chat.css and chat.js (the chat itself) use relative paths and never contain the name, so the chat needs no change.

Boot: what starts, in what order

1. tailscaled (system service) starts, joins the tailnet and restores Funnel for arpanpathak.taildb9a39.ts.net → 127.0.0.1:8080.
2. Because of linger, the user’s systemd starts without anyone logging in: thor-chat (llama-server, loads the Nano), searxng, and thor-tigress-agent (ordered after thor-chat).
3. Each service restarts itself 5 seconds after a crash (Restart=on-failure).

Measured: the last start of thor-chat (2026-10-05 05:17:55) loaded the model and listened 6 seconds later, with the model file already in memory cache. Not measured yet: the time from power-on to the first answer, when the two model files (33 GB and 35 GB) have to come from disk. With both files in the memory cache, a restart on 2026-10-06 had both models loaded in 32 seconds.

After a reboot, check from the Thor:

systemctl is-active tailscaled
systemctl --user is-active thor-chat thor-tigress-agent searxng
tailscale funnel status                              # Funnel on, / → 127.0.0.1:8080
curl -s 127.0.0.1:8080/health                         # {"status":"ok"}
curl -s https://arpanpathak.taildb9a39.ts.net/health  # the same, from outside

Runbook

The key leaked

Signs: someone uninvited is chatting, or a personal key was pasted somewhere public. Whose key it is decides the command.

ssh thor
thor-tigress-serve keyring keys          # who is active; the keys are masked
thor-tigress-serve keyring revoke KEY    # one leaked key: end that person
thor-tigress-serve keyring revoke-all    # every active key, at once

The agent re-reads the file when the keyring changes, so the next request is refused: no restart, and a reply in progress is not cut off. The records stay, so the people who should keep access are approved again with approve EMAIL, and each is sent a new key.

If what leaked was the service key, the one the agent sends to llama-server, rotate it and restart both services:

thor-tigress-serve key
systemctl --user restart thor-chat thor-tigress-agent
cat ~/.config/thor-chat/api-key

Then copy the new service key to each machine you work from: (umask 077; ssh thor cat .config/thor-chat/api-key > ~/.config/thor-chat/api-key). Personal keys are unaffected by that rotation.

Abuse: stop everything now

ssh thor 'sudo tailscale funnel --bg 8080 off'

The public address stops answering within seconds; the Thor, the model and the tailnet keep working, so your own devices still reach it over Tailscale or the SSH tunnel. To turn it back on: sudo tailscale funnel --bg 8080. Rotate the key before turning it back on if the abuse came from a key holder.

Move to a new Tailscale address

Use this when the name itself attracts unwanted traffic. It does not replace rotating the key.

1. Choose the new name. Either:

• New machine name (the part before the first dot): Tailscale admin console → Machines → the Thor → ⋯ → Edit machine name → untick “Auto-generate from OS hostname” → type the new name → Update name. The address becomes <new-name>.taildb9a39.ts.net.
• New tailnet name (the taildb9a39 part, for every device): admin console → DNS → Tailnet DNS name → Rename tailnet… → pick one of the offered names.

2. Bring Funnel up on the new name, on the Thor:

tailscale status --self --json | python3 -c "import sys,json;print(json.load(sys.stdin)['Self']['DNSName'])"
sudo tailscale funnel --bg 8080      # re-applies Funnel for the current name
tailscale funnel status
curl -s https://<new-address>/health

The first HTTPS request may take a few seconds while Tailscale gets the new certificate. The new name enters the public certificate logs at that moment.

3. Update every place that names it. On any machine with the two repositories cloned (paths below assume ~/Projects):

OLD=arpanpathak.taildb9a39.ts.net
NEW=<new-address>

gh repo clone arpanpathak/voltforge.tech ~/Projects/voltforge.tech 2>/dev/null || git -C ~/Projects/voltforge.tech pull
cd ~/Projects/voltforge.tech
grep -rl "$OLD" --include='*.html' --include='*.md' . | xargs sed -i "s/$OLD/$NEW/g"
git commit -am "Forward to $NEW" && git push

cd ~/Projects/thor-thunder-tigress-platform
grep -rl "$OLD" book/src jetson-thor | xargs sed -i "s/$OLD/$NEW/g"
git commit -am "New Tailscale address" && git push

(On macOS, sed -i '' instead of sed -i.) GitHub Pages republishes the forwarding page within a minute or two; the DNS at Namecheap does not change, because voltforge.tech points at GitHub, not at the Thor. This is the main benefit of the forwarding page.

4. Tell people. Their saved key and history belong to the old address; on the new one they paste the key again. Agents need the new --openai-url or ANTHROPIC_BASE_URL.

5. Check.

curl -s https://voltforge.tech/thor-tigress-cub | grep -o 'url=[^"]*'   # the new address
curl -s https://arpanpathak.taildb9a39.ts.net/health                      # old name: no answer

Change a DNS record

All records live at Namecheap → Domain List → Manage next to voltforge.tech → Advanced DNS → Host Records.

• Edit: click the value, change it, click the green tick.
• Add: Add New Record, choose the type, fill Host and Value, green tick.
• Delete: the trash icon at the end of the row.

Records have a time to live of about 30 minutes, so old answers can linger that long. Check what the world sees, not your own machine’s cache:

curl -s "https://dns.google/resolve?name=voltforge.tech&type=A" | python3 -m json.tool | grep data
curl -s "https://dns.google/resolve?name=www.voltforge.tech&type=CNAME" | python3 -m json.tool | grep data

If the A records stop pointing at GitHub, GitHub Pages loses the certificate for voltforge.tech and will request it again once they are back.

A service is down

Keep it updated

Known gaps

Not fixed yet; each needs a decision or work.