# Cron Plan (Phase 1 Initial Rollout)

Status: `planned` (not installed yet).

This document defines the first cron bundle requested:
- `3` ingest jobs (`scrape_slot.sh` for `morning`, `midday`, `evening`)
- `2` process jobs (`process_day.sh`)

All commands are copy/paste ready.

## 1) Scope

This rollout only covers Phase 1:
- `PHASES/PHASE-01-INGEST/scrape_slot.sh`
- `PHASES/PHASE-01-INGEST/process_day.sh`

Later phases (captions, orchestrator, notifications) are intentionally excluded from this initial cron batch.

## 2) Runtime Assumptions

- Repo root: `/workspace/twitter-digest`
- Python venv exists at `/workspace/twitter-digest/venv`
- Cookies file exists at `PHASES/PHASE-01-INGEST/cookies.json`
- Scripts are executable:
  - `PHASES/PHASE-01-INGEST/scrape_slot.sh`
  - `PHASES/PHASE-01-INGEST/process_day.sh`
- Server timezone is the one returned by `date -Is` at runtime (currently `UTC-06:00` on this server).

## 3) Why 3 Scrapes + 2 Process Jobs

- Scrape runs: collect changing For You feed at three points in the day.
- Process runs: build/refresh `with_images.csv`, `no_images.csv`, `manifest.json`, and downloaded images from the merged `raw_feed.csv`.

For this initial bundle, process is scheduled twice:
- Midday (`13:05`) after the midday scrape.
- Evening (`18:05`) after the evening scrape.

## 4) Job Matrix

1. `08:00` - scrape morning
2. `13:00` - scrape midday
3. `18:00` - scrape evening
4. `13:05` - process day (midday refresh)
5. `18:05` - process day (final refresh)

## 5) Exact Crontab Block (Ready to Add)

```cron
# ===== Twitter Digest - Phase 1 Initial Rollout =====
# Ingest scrapes
0 8 * * * cd /workspace/twitter-digest && bash PHASES/PHASE-01-INGEST/scrape_slot.sh morning $(date +\%F) >> PHASES/PHASE-01-INGEST/logs/scrape/cron_morning.log 2>&1
0 13 * * * cd /workspace/twitter-digest && bash PHASES/PHASE-01-INGEST/scrape_slot.sh midday $(date +\%F) >> PHASES/PHASE-01-INGEST/logs/scrape/cron_midday.log 2>&1
0 18 * * * cd /workspace/twitter-digest && bash PHASES/PHASE-01-INGEST/scrape_slot.sh evening $(date +\%F) >> PHASES/PHASE-01-INGEST/logs/scrape/cron_evening.log 2>&1

# Day processing (2 runs)
5 13 * * * cd /workspace/twitter-digest && bash PHASES/PHASE-01-INGEST/process_day.sh $(date +\%F) >> PHASES/PHASE-01-INGEST/logs/process/cron_midday.log 2>&1
5 18 * * * cd /workspace/twitter-digest && bash PHASES/PHASE-01-INGEST/process_day.sh $(date +\%F) >> PHASES/PHASE-01-INGEST/logs/process/cron_evening.log 2>&1
# ===== End =====
```

## 6) Safe Install Command

```bash
(crontab -l 2>/dev/null; cat <<'EOF'
# ===== Twitter Digest - Phase 1 Initial Rollout =====
0 8 * * * cd /workspace/twitter-digest && bash PHASES/PHASE-01-INGEST/scrape_slot.sh morning $(date +\%F) >> PHASES/PHASE-01-INGEST/logs/scrape/cron_morning.log 2>&1
0 13 * * * cd /workspace/twitter-digest && bash PHASES/PHASE-01-INGEST/scrape_slot.sh midday $(date +\%F) >> PHASES/PHASE-01-INGEST/logs/scrape/cron_midday.log 2>&1
0 18 * * * cd /workspace/twitter-digest && bash PHASES/PHASE-01-INGEST/scrape_slot.sh evening $(date +\%F) >> PHASES/PHASE-01-INGEST/logs/scrape/cron_evening.log 2>&1
5 13 * * * cd /workspace/twitter-digest && bash PHASES/PHASE-01-INGEST/process_day.sh $(date +\%F) >> PHASES/PHASE-01-INGEST/logs/process/cron_midday.log 2>&1
5 18 * * * cd /workspace/twitter-digest && bash PHASES/PHASE-01-INGEST/process_day.sh $(date +\%F) >> PHASES/PHASE-01-INGEST/logs/process/cron_evening.log 2>&1
# ===== End =====
EOF
) | crontab -
```

## 7) Verification Checklist After Install

1. Confirm cron entries exist:

```bash
crontab -l
```

2. Confirm logs are written after each scheduled run:

```bash
tail -n 50 PHASES/PHASE-01-INGEST/logs/scrape/cron_morning.log
tail -n 50 PHASES/PHASE-01-INGEST/logs/scrape/cron_midday.log
tail -n 50 PHASES/PHASE-01-INGEST/logs/scrape/cron_evening.log
tail -n 50 PHASES/PHASE-01-INGEST/logs/process/cron_midday.log
tail -n 50 PHASES/PHASE-01-INGEST/logs/process/cron_evening.log
```

3. Confirm artifacts for the target date:

```bash
DATE=YYYY-MM-DD
ls -la PHASES/PHASE-01-INGEST/tweets/archive/$DATE/raw_feed_runs
ls -la PHASES/PHASE-01-INGEST/tweets/archive/$DATE/raw_feed.csv
ls -la PHASES/PHASE-01-INGEST/tweets/archive/$DATE/with_images.csv
ls -la PHASES/PHASE-01-INGEST/tweets/archive/$DATE/no_images.csv
ls -la PHASES/PHASE-01-INGEST/tweets/archive/$DATE/manifest.json
```

4. Confirm row-level sanity:

```bash
DATE=YYYY-MM-DD
echo "raw_feed lines: $(wc -l < PHASES/PHASE-01-INGEST/tweets/archive/$DATE/raw_feed.csv)"
echo "with_images lines: $(wc -l < PHASES/PHASE-01-INGEST/tweets/archive/$DATE/with_images.csv)"
echo "no_images lines: $(wc -l < PHASES/PHASE-01-INGEST/tweets/archive/$DATE/no_images.csv)"
```

## 8) Known Failure Signals

- Missing or invalid cookies:
  - scraper fails when loading Twitter session.
- Rate limit:
  - scraper logs `Rate limited` and may stop early.
- Missing `raw_feed.csv`:
  - `process_day.sh` exits with `[process] ERROR: missing .../raw_feed.csv`.
- Partial network/download failures:
  - `manifest.json` shows non-zero `download_failures`.

## 9) Planned Next Cron Bundle (Later)

After this Phase 1 bundle is stable, add:
1. Phase 2 captions cron(s)
2. Phase 5 orchestrator cron
3. Phase 6 notification cron
