# Operations Guide

## Normal Daily Operation

Timer:

```bash
systemctl status digest-orchestrator.timer --no-pager
systemctl list-timers --all | grep digest
```

Manual real run:

```bash
sudo -iu agent
cd /workspace/digest
./run-digest-orchestrator.sh --ignore-ack
```

Manual run for a specific date:

```bash
./run-digest-orchestrator.sh --ignore-ack --date-iso 2026-06-11
```

Zero-token simulation:

```bash
./run-digest-orchestrator.sh --simulate --ignore-ack --date-iso 2030-01-01 --attempt-timeout-seconds 15 --poll-seconds 1
```

## Orchestrator Behavior

Defaults:

- Workdir: `/workspace/digest`
- Max attempts: 4
- Attempt timeout: 1200 seconds (20 minutes)
- Poll interval: 5 seconds
- Completion freshness: 7200 seconds (2 hours)
- Kill grace: 10 seconds
- Lock file: `/tmp/claude-digest-orchestrator.lock`

Completion is strict:

1. `COMPLETED-DAYS.md` contains today's `YYYY-MM-DD` as a full line.
2. `COMPLETED-DAYS.md` was modified within the freshness window.

On success:

- Kills Claude process group.
- Writes `.orchestrator-ack-YYYY-MM-DD`.
- Exits `0`.

On failure:

- Writes raw logs under `logs/raw/`.
- Writes report under `logs/past-logs/YYYY-MM-DD.md`.
- Exits nonzero after max attempts.

## VPN Health Operation

Timer:

```bash
systemctl status digest-vpn-health.timer --no-pager
systemctl status digest-vpn-health.service --no-pager
```

Manual check:

```bash
cd /workspace/digest
./vpn-health.sh check
```

Manual repair:

```bash
./vpn-health.sh ensure
```

Manual repair with reboot guard:

```bash
sudo /workspace/digest/vpn-health.sh --allow-reboot --log-file /workspace/digest/logs/vpn-health.log
```

What it checks:

- NordVPN status includes `Status: Connected`.
- DNS resolves `api.openai.com`, `www.reddit.com`, `api.telegram.org`.
- HTTP probes succeed for `https://api.openai.com/`, `https://www.reddit.com/`, `https://api.telegram.org/`.

Repair order:

1. If disconnected: `nordvpn connect us`.
2. If connected but unhealthy: disconnect/reconnect.
3. Restart `nordvpnd`.
4. After repeated failures, optionally reboot if `--allow-reboot` is set.

## Logs

Important files:

- `logs/vpn-health.log`: 5-minute VPN repair timer log.
- `logs/raw/run_YYYY-MM-DD_*.log`: orchestrator run log.
- `logs/raw/claude_attemptN_YYYY-MM-DD_*.log`: Claude output per attempt.
- `logs/raw/vpn_health_YYYY-MM-DD_*.log`: per-run preflight health logs.
- `logs/past-logs/YYYY-MM-DD.md`: readable summary.

Useful commands:

```bash
journalctl -u digest-orchestrator.service -n 200 --no-pager
journalctl -u digest-vpn-health.service -n 200 --no-pager
tail -n 120 /workspace/digest/logs/vpn-health.log
tail -n 120 /workspace/digest/logs/past-logs/$(date +%Y-%m-%d).md
```

## Link Validation

```bash
cd /workspace/digest
./validate-links.sh digest-archive/YYYY-MM-DD/index.php
```

Exit code `1` can mean warnings, not only broken links. The script intentionally fails on 403 warnings because those must be manually verified.

Latest observed validation for 2026-06-11:

- 27 total links
- 24 OK
- 0 broken
- 3 warnings: OpenAI, Product Hunt, x.ai returned 403 likely bot protection

## Web Feedback

Generated PHP writes feedback to:

`/workspace/digest/SUGGESTIONS-FROM-USER.MD`

Make this writable by PHP-FPM user `www-data`:

```bash
sudo chown agent:www-data /workspace/digest/SUGGESTIONS-FROM-USER.MD
sudo chmod 664 /workspace/digest/SUGGESTIONS-FROM-USER.MD
```

Then test:

```bash
sudo -u www-data test -w /workspace/digest/SUGGESTIONS-FROM-USER.MD && echo ok
```

## Common Failure Modes

### Claude not found from systemd

Check PATH inside `run-digest-orchestrator.sh`.

Current exported PATH includes `/home/agent/.local/bin`.

Verify as `agent`:

```bash
sudo -iu agent command -v claude
sudo -iu agent claude --version
```

### Reddit MCP unavailable

Verify:

```bash
sudo -iu agent claude mcp list
npx -y reddit-mcp-buddy --help
```

If missing:

```bash
sudo -iu agent claude mcp add reddit-mcp-buddy -- npx -y reddit-mcp-buddy
```

### VPN group permissions

Symptoms:

- `sg nordvpn -c "nordvpn status"` fails.
- Orchestrator preflight fails.

Fix:

```bash
sudo usermod -aG nordvpn agent
sudo systemctl restart digest-orchestrator.service
```

The user must start a new login session for group changes, but systemd services usually pick up group membership after restart.

### SSH/Web broken after NordVPN

Do not enable NordVPN kill switch.

Check allowlist:

```bash
sg nordvpn -c "nordvpn settings"
```

Required allowlisted ports:

- 22
- 80
- 443

Add:

```bash
sg nordvpn -c "nordvpn allowlist add port 22"
sg nordvpn -c "nordvpn allowlist add port 80"
sg nordvpn -c "nordvpn allowlist add port 443"
```

### Digest generated but orchestrator keeps retrying

Check:

```bash
grep -Fx "$(date +%Y-%m-%d)" /workspace/digest/COMPLETED-DAYS.md
stat /workspace/digest/COMPLETED-DAYS.md
```

The date must be present and the file mtime must be fresh.

### PHP page shows 404 or downloads instead of renders

Check:

```bash
sudo nginx -t
systemctl status php8.3-fpm --no-pager
curl -I https://www.tonki.dev/digest/digest-archive/2026-06-11/
```

Nginx root must be `/workspace`, and PHP location must pass to `/run/php/php8.3-fpm.sock`.

