# Twitter Digest Orchestration — Full Build Plan From Scratch (for your server AI agent)

**Root path (new):** `/agent/workspace/twitter-digest/`
**Goal:** 3× daily scrapes → incremental image downloads → **z-ai captions MUST finish** → 22:00 digest agent generates digest page using your `twitter-app` template, with **tweet text/author/date shown above embeds**, continuity across last 7 days, Telegram notify, correct permissions, and feedback appends to `INTEREST.md`.

---

## 1) What you have right now (baseline)

In `/agent/workspace/twitter-digest/` you already have:

* `bin/twitter-scraper` (scrapes “For You” feed → CSV)
* `bin/twitter-process` (splits feed, downloads images, writes `with_images.csv`, `manifest.json`)
* `bin/twitter-analyze` (deep dive on a tweet + top comments)
* `bin/tweet-csv-merge` (merge CSVs, dedupe by tweet_id)
* `bin/tweet-filter` (already upgraded)
* `twitter-app/` (your HTML/CSS/JS template)

**Critical limitation currently:**
If you run `twitter-process` multiple times per day, it will **wipe** `image_description` and re-download. That **must be fixed** because you will caption images after each scrape.

---

## 2) What “done” looks like (end state)

### Daily data folder

`/agent/workspace/twitter-digest/tweets/archive/YYYY-MM-DD/` contains:

* `raw_feed_runs/HHMM.csv` (each scrape creates a new file)
* `raw_feed.csv` (merged canonical feed for the day)
* `no_images.csv`
* `with_images.csv` (incremental; never loses existing captions)
* `manifest.json` (incremental; includes all images so far)
* `images/` (downloaded images)
* `image_descriptions.jsonl` (append-only per-image captions from z-ai)
* `images_checkpoint.md` (progress log for humans/agent)
* `images_done.ok` (created when validator passes)

### Daily digest folder

`/agent/workspace/twitter-digest/digests/archive/YYYY-MM-DD/` contains:

* `digest.json` (structured plan: sections + narratives + selected tweets)
* `index.php` (renders your `twitter-app` layout using `digest.json`)
* `digest_checkpoint.md` (resumable state)
* `published.ok` (created after permissions + telegram)

### Global files

* `/agent/workspace/twitter-digest/INTEREST.md` (plain language, append-only user additions)
* `/agent/workspace/twitter-digest/COMPLETED-DAYS.md` (append-only)
* `/agent/workspace/twitter-digest/logs/...` (attempt logs)

---

## 3) Timeline (the actual automation schedule)

Example times (server local time):

* **08:00** scrape → process
* **08:10** caption (z-ai) until complete for everything downloaded so far
* **13:00** scrape → process
* **13:10** caption until complete
* **18:00** scrape → process
* **18:10** caption until complete
* **22:00** **finalize + caption until complete (MUST PASS) → digest agent → publish → telegram**

**Non-negotiable rule:** digest does **not** run unless captions are complete (validator passes).

---

## 4) Build steps (what your server AI agent must do)

## Phase A — Create folders + baseline files

### A1) Create directories

Run:

```bash
mkdir -p /agent/workspace/twitter-digest/{scripts,agents/images,agents/digest,logs/{scrape,process,caption,digest},digests/archive}
```

### A2) Create plain-language `INTEREST.md` (root)

Create: `/agent/workspace/twitter-digest/INTEREST.md`

```md
# What I want in my Twitter Digest

## Topics I care about most
- AI tools and workflows that help builders ship faster (agents, coding, automation)
- Startup / tech strategy that has real leverage (distribution, positioning, business models)
- Developer tools and infrastructure (practical, not hype)
- Important industry shifts (OpenAI / Anthropic / Google / Meta) only if meaningful

## Usually skip
- Politics / outrage / culture-war content
- Celebrity drama
- Generic motivational fluff
- Low-signal memes (unless it teaches something)

## What “high value” means
- Something new I can apply
- Real numbers, strong evidence, or first-hand knowledge
- Clear explanation (not vague takes)

## Output preferences
- Fewer items, higher quality
- 1–2 sentences: why this matters
- Connect stories that evolve across the last 7 days

## Notes from me (append-only)
```

---

## Phase B — Patch the pipeline so multi-run per day is safe

### B1) **Modify `twitter-scraper` to include tweet timestamp**

You MUST add `created_at` (ISO) to the CSV, otherwise you can’t print real tweet dates above embeds.

In `bin/twitter-scraper`, inside the loop where you append tweet dict, add something like:

* `created_at_iso`: best effort from `tweet.created_at_datetime` or `tweet.created_at`
* keep it UTC ISO string

**Change:**
Add a new CSV column in fieldnames and row dict.

Example patch logic (agent should implement cleanly):

* Fieldnames add: `created_at`
* Row add:

```python
created_at = ""
dt = getattr(tweet, "created_at_datetime", None)
if dt:
    created_at = dt.astimezone(timezone.utc).isoformat()
else:
    created_at = safe_str(getattr(tweet, "created_at", ""))
```

(Import `timezone` from datetime.)

This is required so the digest shows correct dates.

---

### B2) **Replace `twitter-process` with an incremental-safe version**

This is the biggest required fix.

**What must be true after the patch:**

* If `with_images.csv` already exists, running `twitter-process` again:

  * does **not wipe** `image_description`
  * preserves existing `downloaded_images`
  * downloads only missing images
  * updates manifest without duplicating entries

#### Replace `bin/twitter-process` with this full file

(Yes, full replacement; it’s safer than incremental diffs.)

```python
#!/usr/bin/env python3
"""
Twitter Feed Processor (INCREMENTAL-SAFE)

Splits raw feed into tweets with/without images and downloads images incrementally.

Key guarantees:
- Never wipes existing image_description if already present in with_images.csv
- Never re-downloads images if a matching file already exists in images/
- Rebuilds manifest.json deterministically from the current raw_feed.csv + images folder state
- Safe to run multiple times per day

Usage:
  twitter-process [DATE]
"""

import sys
import os
import csv
import json
import argparse
import re
from datetime import date
from pathlib import Path
from typing import Dict, List, Optional, Tuple

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

# Add venv to path
venv_path = Path(__file__).parent.parent / "venv" / "lib" / "python3.12" / "site-packages"
sys.path.insert(0, str(venv_path))

import filetype

BASE_DIR = Path(__file__).parent.parent


def create_session_with_retries():
    session = requests.Session()
    retry = Retry(
        total=3,
        backoff_factor=1,
        status_forcelist=(429, 500, 502, 503, 504),
        allowed_methods=["GET"],
    )
    adapter = HTTPAdapter(max_retries=retry)
    session.mount("http://", adapter)
    session.mount("https://", adapter)
    session.headers.update({"User-Agent": "twitter-pipeline/1.0"})
    return session


def sanitize_filename(s: str) -> str:
    s = (s or "").replace("@", "").strip()
    s = re.sub(r"[^\w\-]", "_", s)
    return s[:30] or "unknown"


def detect_extension(content: bytes) -> str:
    kind = filetype.guess(content)
    if kind:
        return f".{kind.extension}"
    return ".jpg"


def download_image(session: requests.Session, url: str, filepath_base: Path) -> Tuple[bool, Optional[Path], Optional[str]]:
    try:
        response = session.get(url, timeout=30)
        response.raise_for_status()
        content = response.content

        ext = detect_extension(content)
        filepath = filepath_base.with_suffix(ext)

        with open(filepath, "wb") as f:
            f.write(content)

        return True, filepath, None
    except Exception as e:
        return False, None, f"{type(e).__name__}: {e}"


def write_csv(path: Path, fieldnames: List[str], rows: List[Dict[str, str]]) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    with open(path, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=fieldnames, quoting=csv.QUOTE_MINIMAL)
        writer.writeheader()
        writer.writerows(rows)


def read_csv_rows(path: Path) -> List[Dict[str, str]]:
    if not path.exists():
        return []
    with open(path, "r", newline="", encoding="utf-8-sig") as f:
        r = csv.DictReader(f)
        return list(r)


def build_existing_with_images_map(with_images_path: Path) -> Dict[str, Dict[str, str]]:
    """
    tweet_id -> row
    """
    rows = read_csv_rows(with_images_path)
    out: Dict[str, Dict[str, str]] = {}
    for row in rows:
        tid = (row.get("tweet_id") or "").strip()
        if tid.isdigit():
            out[tid] = row
    return out


def find_existing_image_file(images_dir: Path, filename_base: str) -> Optional[str]:
    """
    Given base like 'user-tweetid-123-image1', find any existing file with any extension.
    Returns filename if found.
    """
    matches = sorted(images_dir.glob(f"{filename_base}.*"))
    if matches:
        return matches[0].name
    return None


def process_feed(archive_dir: Path) -> None:
    raw_file = archive_dir / "raw_feed.csv"
    if not raw_file.exists():
        print(f"Error: {raw_file} not found")
        sys.exit(1)

    images_dir = archive_dir / "images"
    images_dir.mkdir(parents=True, exist_ok=True)

    session = create_session_with_retries()

    # Existing with_images.csv map (to preserve image_description)
    with_images_file = archive_dir / "with_images.csv"
    existing_map = build_existing_with_images_map(with_images_file)

    no_images: List[Dict[str, str]] = []
    with_images: List[Dict[str, str]] = []
    manifest_images: List[Dict[str, str]] = []
    download_failures: List[Dict[str, str]] = []

    rows = read_csv_rows(raw_file)
    print(f"Processing {len(rows)} tweets...")

    base_fields = [
        "tweet_id", "tweet_url", "name", "username", "content",
        "likes", "retweets", "replies", "image_count", "image_urls", "has_video",
        "created_at"
    ]
    no_images_fields = base_fields + ["media_note"]
    with_images_fields = base_fields + ["downloaded_images", "image_description", "media_note"]

    for row in rows:
        tweet_id = (row.get("tweet_id") or "").strip()
        if not tweet_id.isdigit():
            print(f"  Skipping invalid tweet_id: {tweet_id}")
            continue

        # Parse image_count safely
        try:
            image_count = int(row.get("image_count") or 0)
        except Exception:
            image_count = 0

        has_images = image_count > 0

        if not has_images:
            row["media_note"] = "[Video - not downloaded]" if row.get("has_video") == "True" else ""
            no_images.append(row)
            continue

        # Images pathing
        image_urls = row.get("image_urls", "")
        urls = [u for u in image_urls.split("|") if u.strip()] if image_urls else []
        username = sanitize_filename(row.get("username", "unknown"))

        downloaded_files: List[str] = []
        for idx, url in enumerate(urls, 1):
            if not url:
                continue

            filename_base = f"{username}-tweetid-{tweet_id}-image{idx}"

            # If file already exists, do not download again
            existing_file = find_existing_image_file(images_dir, filename_base)
            if existing_file:
                downloaded_files.append(existing_file)
                manifest_images.append({
                    "file": existing_file,
                    "tweet_id": tweet_id,
                    "username": row.get("username", ""),
                    "image_num": idx,
                    "url": url
                })
                continue

            # Download if missing
            print(f"  Downloading {filename_base}...")
            success, final_path, error = download_image(session, url, images_dir / filename_base)
            if success and final_path:
                downloaded_files.append(final_path.name)
                manifest_images.append({
                    "file": final_path.name,
                    "tweet_id": tweet_id,
                    "username": row.get("username", ""),
                    "image_num": idx,
                    "url": url
                })
            else:
                print(f"    Failed: {error}")
                download_failures.append({"tweet_id": tweet_id, "url": url, "error": error or "unknown"})

        row["downloaded_images"] = "|".join(downloaded_files)

        # Preserve existing image_description if present
        prev = existing_map.get(tweet_id, {})
        prev_desc = (prev.get("image_description") or "").strip()
        row["image_description"] = prev_desc  # may be empty; later filled by apply script

        row["media_note"] = "[Also has video - not downloaded]" if row.get("has_video") == "True" else ""
        with_images.append(row)

    # Write CSVs always
    write_csv(archive_dir / "no_images.csv", no_images_fields, no_images)
    write_csv(archive_dir / "with_images.csv", with_images_fields, with_images)

    manifest = {
        "date": archive_dir.name,
        "total_tweets": len(rows),
        "tweets_without_images": len(no_images),
        "tweets_with_images": len(with_images),
        "total_images": len(manifest_images),
        "download_failures": len(download_failures),
        "images": manifest_images,
        "failures": download_failures
    }

    with open(archive_dir / "manifest.json", "w", encoding="utf-8") as f:
        json.dump(manifest, f, indent=2)

    print(f"Done! Output in {archive_dir}")
    print(f"  - no_images.csv: {len(no_images)}")
    print(f"  - with_images.csv: {len(with_images)}")
    print(f"  - images listed in manifest: {len(manifest_images)}")
    if download_failures:
        print(f"  - failures: {len(download_failures)}")


def main():
    parser = argparse.ArgumentParser(description="Process Twitter feed (incremental-safe)")
    parser.add_argument("date", nargs="?", default=None, help="Date (YYYY-MM-DD) or path to archive dir")
    args = parser.parse_args()

    if args.date:
        if os.path.isdir(args.date):
            archive_dir = Path(args.date)
        else:
            archive_dir = BASE_DIR / "tweets" / "archive" / args.date
    else:
        archive_dir = BASE_DIR / "tweets" / "archive" / date.today().isoformat()

    print(f"Twitter Feed Processor (incremental)")
    print(f"  Archive: {archive_dir}")
    process_feed(archive_dir)


if __name__ == "__main__":
    main()
```

---

## Phase C — Add image caption pipeline (z-ai) + validator + “run until complete”

### C1) Create `scripts/apply_image_descriptions.py`

Create file: `/agent/workspace/twitter-digest/scripts/apply_image_descriptions.py`

```python
#!/usr/bin/env python3
"""
Apply image_descriptions.jsonl into with_images.csv (tweet-level combined string).
- Append-only jsonl; last description for a file wins.
- Produces image_description like:
  img1(file): ... || img2(file): ...
"""

from __future__ import annotations
import csv, json, os
from pathlib import Path
from typing import Dict, List

ROOT = Path("/agent/workspace/twitter-digest")

def single_line(s: str) -> str:
    return " ".join((s or "").replace("\r", " ").replace("\n", " ").split()).strip()

def load_jsonl_map(jsonl_path: Path) -> Dict[str, str]:
    m: Dict[str, str] = {}
    if not jsonl_path.exists():
        return m
    for line in jsonl_path.read_text(encoding="utf-8").splitlines():
        line = line.strip()
        if not line:
            continue
        obj = json.loads(line)
        fn = str(obj.get("file","")).strip()
        desc = single_line(str(obj.get("description","")).strip())
        if fn and desc:
            m[fn] = desc
    return m

def build_desc(downloaded_images: str, file_map: Dict[str, str]) -> str:
    files = [p.strip() for p in (downloaded_images or "").split("|") if p.strip()]
    parts: List[str] = []
    for i, fn in enumerate(files, start=1):
        desc = file_map.get(fn, "").strip()
        if not desc:
            parts.append(f"img{i}({fn}): [MISSING DESCRIPTION]")
        else:
            parts.append(f"img{i}({fn}): {desc}")
    return " || ".join(parts)

def main(date_iso: str) -> int:
    day = ROOT / "tweets" / "archive" / date_iso
    with_images = day / "with_images.csv"
    jsonl = day / "image_descriptions.jsonl"

    if not with_images.exists():
        print(f"ERROR: missing {with_images}")
        return 2

    file_map = load_jsonl_map(jsonl)

    backup = day / "with_images.before_apply.csv"
    if not backup.exists():
        backup.write_bytes(with_images.read_bytes())

    tmp = day / ".with_images.csv.tmp"

    with with_images.open("r", encoding="utf-8-sig", newline="") as fin:
        reader = csv.DictReader(fin)
        if not reader.fieldnames:
            print("ERROR: no header")
            return 2
        fns = list(reader.fieldnames)
        if "image_description" not in fns:
            fns.append("image_description")
        rows = list(reader)

    updated = 0
    for r in rows:
        downloaded = (r.get("downloaded_images") or "").strip()
        if not downloaded:
            continue
        new_desc = build_desc(downloaded, file_map)
        if new_desc and (r.get("image_description") or "").strip() != new_desc:
            r["image_description"] = new_desc
            updated += 1

    with tmp.open("w", encoding="utf-8", newline="") as fout:
        w = csv.DictWriter(fout, fieldnames=fns, quoting=csv.QUOTE_MINIMAL)
        w.writeheader()
        w.writerows(rows)

    os.replace(tmp, with_images)
    print(f"OK: applied captions. updated_rows={updated}")
    return 0

if __name__ == "__main__":
    import sys
    if len(sys.argv) < 2:
        print("Usage: apply_image_descriptions.py YYYY-MM-DD")
        raise SystemExit(2)
    raise SystemExit(main(sys.argv[1]))
```

Make executable:

```bash
chmod +x /agent/workspace/twitter-digest/scripts/apply_image_descriptions.py
```

---

### C2) Create validator `scripts/validate_captions.py`

Create file: `/agent/workspace/twitter-digest/scripts/validate_captions.py`

```python
#!/usr/bin/env python3
"""
Validator: captions complete iff every manifest.images[].file has a non-empty caption in image_descriptions.jsonl.
"""

from __future__ import annotations
import json
from pathlib import Path
from typing import Dict

ROOT = Path("/agent/workspace/twitter-digest")

def load_jsonl_map(jsonl: Path) -> Dict[str, str]:
    out = {}
    if not jsonl.exists():
        return out
    for line in jsonl.read_text(encoding="utf-8").splitlines():
        line = line.strip()
        if not line:
            continue
        obj = json.loads(line)
        fn = str(obj.get("file","")).strip()
        desc = str(obj.get("description","")).strip()
        if fn and desc and not desc.startswith("CAPTION_FAILED"):
            out[fn] = desc
    return out

def main(date_iso: str) -> int:
    day = ROOT / "tweets" / "archive" / date_iso
    manifest = day / "manifest.json"
    jsonl = day / "image_descriptions.jsonl"

    if not manifest.exists():
        print("ERROR: manifest.json missing")
        return 2

    data = json.loads(manifest.read_text(encoding="utf-8"))
    imgs = data.get("images", [])
    expected = [i.get("file","") for i in imgs if i.get("file")]

    m = load_jsonl_map(jsonl)
    missing = [fn for fn in expected if fn not in m or not m[fn].strip()]

    if missing:
        print(f"NOT_READY: missing_captions={len(missing)}")
        for fn in missing[:25]:
            print(f" - {fn}")
        return 1

    print("OK: all captions present")
    return 0

if __name__ == "__main__":
    import sys
    if len(sys.argv) < 2:
        print("Usage: validate_captions.py YYYY-MM-DD")
        raise SystemExit(2)
    raise SystemExit(main(sys.argv[1]))
```

Make executable:

```bash
chmod +x /agent/workspace/twitter-digest/scripts/validate_captions.py
```

---

### C3) Create z-ai image agent instructions: `agents/images/CLAUDE.md`

Create: `/agent/workspace/twitter-digest/agents/images/CLAUDE.md`

```md
# IMAGE CAPTION AGENT (z-ai)

You run under z-ai (claude --dangerously-skip-permissions).

Your job:
- Caption ONLY missing images for a specific date.
- Append results to: tweets/archive/{DATE}/image_descriptions.jsonl
- Do NOT edit CSV files directly.

Inputs:
- tweets/archive/{DATE}/manifest.json
- tweets/archive/{DATE}/images/
- tweets/archive/{DATE}/image_descriptions.jsonl (may exist)

Rules:
1) Read manifest.json and list all images[].file.
2) Read existing image_descriptions.jsonl; build set of already-captioned files.
3) For each missing file:
   - open the local image file under images/
   - write ONE jsonl line:
     {"file":"...","tweet_id":"...","username":"...","image_num":1,"description":"..."}
4) Description requirements:
   - 2–5 sentences, literal, no fluff.
   - If image has text: include important text verbatim, especially numbers and claims.
   - If chart/table: summarize the takeaway + key numbers.
5) Output must be single-line JSON (no newlines inside description).

If image is unreadable:
- still append a line with:
  "description": "CAPTION_FAILED: <reason>"
```

---

### C4) Create the caption runner that **loops until complete**

Create: `/agent/workspace/twitter-digest/scripts/caption_until_complete.sh`

```bash
#!/usr/bin/env bash
set -euo pipefail

ROOT="/agent/workspace/twitter-digest"
DATE_ISO="${1:-$(date +%Y-%m-%d)}"
DAY_DIR="${ROOT}/tweets/archive/${DATE_ISO}"
LOCK="/tmp/twitter-caption-${DATE_ISO}.lock"

mkdir -p "${ROOT}/logs/caption" "${DAY_DIR}"

exec 9>"${LOCK}"
flock -n 9 || { echo "[caption] already running"; exit 0; }

echo "[caption] start date=${DATE_ISO} at $(date -Is)"

# If already done, exit fast
if [[ -f "${DAY_DIR}/images_done.ok" ]]; then
  echo "[caption] already complete (images_done.ok exists)"
  exit 0
fi

ATTEMPT=0
while true; do
  ATTEMPT=$((ATTEMPT+1))
  echo "[caption] attempt=${ATTEMPT} at $(date -Is)"

  # Run z-ai with a per-attempt timeout so we can restart cleanly if it hangs.
  # IMPORTANT: This script loops forever until validate passes.
  timeout 20m z-ai -p "Read ${ROOT}/agents/images/CLAUDE.md and caption missing images for date ${DATE_ISO}. Only append to ${DAY_DIR}/image_descriptions.jsonl."

  # Apply captions into with_images.csv (deterministic)
  python3 "${ROOT}/scripts/apply_image_descriptions.py" "${DATE_ISO}"

  # Validate
  if python3 "${ROOT}/scripts/validate_captions.py" "${DATE_ISO}"; then
    echo "[caption] COMPLETE at $(date -Is)"
    echo "ok $(date -Is)" > "${DAY_DIR}/images_done.ok"
    exit 0
  fi

  echo "[caption] not complete yet; sleeping 30s"
  sleep 30
done
```

Make executable:

```bash
chmod +x /agent/workspace/twitter-digest/scripts/caption_until_complete.sh
```

✅ This satisfies your rule: **it does not proceed until captions are complete**, ever.
It still restarts every 20 minutes per attempt to avoid deadlock, but **overall it runs until finished**.

---

## Phase D — Scrape + merge + process wrappers (multi-run per day)

### D1) Scrape wrapper (writes unique file, then merges canonical)

Create: `/agent/workspace/twitter-digest/scripts/scrape_slot.sh`

```bash
#!/usr/bin/env bash
set -euo pipefail

ROOT="/agent/workspace/twitter-digest"
SLOT="${1:?usage: scrape_slot.sh morning|midday|evening}"
DATE_ISO="${2:-$(date +%Y-%m-%d)}"
HHMM="$(date +%H%M)"
DAY_DIR="${ROOT}/tweets/archive/${DATE_ISO}"
RUN_DIR="${DAY_DIR}/raw_feed_runs"
LOCK="/tmp/twitter-scrape-${DATE_ISO}-${SLOT}.lock"

mkdir -p "${RUN_DIR}" "${ROOT}/logs/scrape"
exec 9>"${LOCK}"
flock -n 9 || { echo "[scrape] already running"; exit 0; }

cd "${ROOT}"
source "${ROOT}/venv/bin/activate"

OUT="${RUN_DIR}/${HHMM}_${SLOT}.csv"

echo "[scrape] slot=${SLOT} date=${DATE_ISO} out=${OUT} start=$(date -Is)"
./bin/twitter-scraper feed 6 --delay 30-60 --output "${OUT}"

echo "[scrape] merge to raw_feed.csv"
./bin/tweet-csv-merge "${RUN_DIR}"/*.csv -o "${DAY_DIR}/raw_feed.csv" --keep first

echo "[scrape] done end=$(date -Is)"
```

Make executable:

```bash
chmod +x /agent/workspace/twitter-digest/scripts/scrape_slot.sh
```

---

### D2) Process wrapper (incremental-safe twitter-process)

Create: `/agent/workspace/twitter-digest/scripts/process_day.sh`

```bash
#!/usr/bin/env bash
set -euo pipefail

ROOT="/agent/workspace/twitter-digest"
DATE_ISO="${1:-$(date +%Y-%m-%d)}"
LOCK="/tmp/twitter-process-${DATE_ISO}.lock"

mkdir -p "${ROOT}/logs/process"
exec 9>"${LOCK}"
flock -n 9 || { echo "[process] already running"; exit 0; }

cd "${ROOT}"
source "${ROOT}/venv/bin/activate"

echo "[process] date=${DATE_ISO} start=$(date -Is)"
./bin/twitter-process "${DATE_ISO}"
echo "[process] end=$(date -Is)"
```

Make executable:

```bash
chmod +x /agent/workspace/twitter-digest/scripts/process_day.sh
```

---

## Phase E — Digest output (normal Claude) + your `twitter-app` integration

### E1) Digest agent instructions: `agents/digest/CLAUDE.md`

Create: `/agent/workspace/twitter-digest/agents/digest/CLAUDE.md`

```md
# TWITTER DIGEST AGENT (normal Claude)

Goal:
Generate today's digest.json and ensure index.php renders it using the twitter-app layout.

Inputs:
- /agent/workspace/twitter-digest/INTEREST.md
- tweets/archive/{DATE}/raw_feed.csv
- tweets/archive/{DATE}/with_images.csv  (must have image_description filled; validator already ensures)
- tweets/archive/{DATE}/no_images.csv
- tweets/archive/{DATE}/manifest.json
- Previous 7 days:
  - digests/archive/*/digest.json (prefer)
  - and/or digests/archive/*/index.php for human narrative reference

Hard rules:
1) You MUST read the previous 7 days before writing today's narrative.
2) You MUST maintain evolving storylines: "Yesterday we saw X, today it evolved into Y".
3) You MUST pick tweets that match INTEREST.md (plain language).
4) If a tweet is unclear or a big claim, you SHOULD:
   - run ./bin/twitter-analyze <tweet_url>
   - and/or do a quick web lookup for context/verification

Output files:
- digests/archive/{DATE}/digest.json
- digests/archive/{DATE}/digest_checkpoint.md (resumable state)

digest.json schema (must follow):
{
  "date": "YYYY-MM-DD",
  "title": "Twitter Digest — <human date>",
  "generated_at": "ISO timestamp",
  "sections": [
    {
      "id": "topic-ai",
      "icon": "🤖",
      "title": "AI & Dev Tools",
      "tags": ["Claude Code", "Agents", "Open Source"],
      "blocks": [
        {"type":"narrative","html":"<p>...</p>"},
        {"type":"tweet","tweet": {
          "tweet_id":"...",
          "tweet_url":"https://x.com/.../status/...",
          "name":"...",
          "username":"@...",
          "created_at":"ISO timestamp",
          "content":"tweet text",
          "likes":0,"retweets":0,"replies":0,
          "image_description":"...",
          "why":"1–2 sentences why it matters"
        }},
        {"type":"narrative","html":"<p>...</p>"}
      ]
    }
  ],
  "closing_html":"<p>...</p>"
}

Notes:
- narrative blocks are HTML strings to allow <strong> and <em>.
- tweet blocks MUST include tweet text + author + date so index.php can render context blocks above embeds.
```

---

### E2) Create the day renderer `digests/archive/YYYY-MM-DD/index.php` template

You will use **one identical index.php** for all days. It reads `digest.json` and renders your layout.

Create: `/agent/workspace/twitter-digest/scripts/install_digest_index_template.sh`

```bash
#!/usr/bin/env bash
set -euo pipefail

ROOT="/agent/workspace/twitter-digest"

cat > "${ROOT}/digests/index.php.template" <<'PHP'
<?php
// ==========================
// Feedback: append to INTEREST.md
// ==========================
$submitted = false;
$error = null;

if ($_SERVER['REQUEST_METHOD'] === 'POST') {
    $suggestion = trim($_POST['suggestion'] ?? '');
    if ($suggestion !== '') {
        $digestDate = basename(__DIR__);
        $stamp = date('Y-m-d H:i:s');

        // Path: digests/archive/YYYY-MM-DD/ -> ../../../INTEREST.md
        $interestPath = __DIR__ . '/../../../INTEREST.md';

        $block = "\n\n## Addition from user on {$digestDate}\n";
        $block .= "**Submitted:** {$stamp}\n\n";
        $block .= $suggestion . "\n";

        if (@file_put_contents($interestPath, $block, FILE_APPEND | LOCK_EX) === false) {
            $error = "Failed to write to INTEREST.md";
        } else {
            $submitted = true;
        }
    }
}

// ==========================
// Load digest.json
// ==========================
$digestPath = __DIR__ . '/digest.json';
if (!file_exists($digestPath)) {
    http_response_code(500);
    echo "<h1>digest.json missing</h1>";
    exit;
}
$digest = json_decode(file_get_contents($digestPath), true);
if (!$digest) {
    http_response_code(500);
    echo "<h1>digest.json invalid</h1>";
    exit;
}

$dateIso = $digest['date'] ?? basename(__DIR__);
$title = $digest['title'] ?? ("Twitter Digest — " . $dateIso);
$generatedAt = $digest['generated_at'] ?? '';

function esc($s) { return htmlspecialchars((string)$s, ENT_QUOTES, 'UTF-8'); }
?>
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <meta name="description" content="Your daily Twitter digest - curated tweets and insights">
    <meta name="theme-color" content="#1d9bf0">

    <title><?php echo esc($title); ?></title>

    <link rel="preconnect" href="https://platform.twitter.com">
    <meta name="twitter:widgets:theme" content="light">
    <meta name="twitter:widgets:border-color" content="transparent">

    <!-- Assets from twitter-app -->
    <link rel="icon" type="image/svg+xml" href="../../../twitter-app/twitter-logo.svg">
    <link rel="stylesheet" href="../../../twitter-app/styles.css">

    <style>
      /* Minimal extra styling for tweet context blocks */
      .tweet-context { margin: 10px 0 8px; padding: 10px 12px; border: 1px solid rgba(0,0,0,0.08); border-radius: 10px; background: rgba(255,255,255,0.6); }
      [data-theme="dark"] .tweet-context { background: rgba(0,0,0,0.15); border-color: rgba(255,255,255,0.12); }
      .tweet-context__meta { display:flex; gap:10px; align-items:center; flex-wrap:wrap; font-size: 13px; opacity: 0.8; }
      .tweet-context__meta a { text-decoration:none; }
      .tweet-context__text { margin-top: 8px; font-size: 15px; line-height: 1.6; }
      .tweet-context__why { margin-top: 8px; font-size: 14px; opacity: 0.9; }
    </style>
</head>
<body>
    <div class="container">
        <div class="main-card">

            <!-- HEADER (same structure as your template) -->
            <header class="digest-header content-item">
                <img src="../../../twitter-app/twitter-logo.svg" alt="Twitter" class="digest-header__logo">
                <div class="digest-header__text">
                    <h1 class="digest-header__title">Twitter Digest</h1>
                    <time class="digest-header__date" datetime="<?php echo esc($dateIso); ?>">
                      <?php echo esc($dateIso); ?>
                    </time>
                </div>
            </header>

            <?php foreach (($digest['sections'] ?? []) as $section): ?>
                <section aria-labelledby="<?php echo esc($section['id']); ?>">

                    <div class="topic-header content-item">
                        <span class="topic-header__icon" aria-hidden="true"><?php echo esc($section['icon'] ?? ''); ?></span>
                        <h2 id="<?php echo esc($section['id']); ?>" class="topic-header__label"><?php echo esc($section['title'] ?? ''); ?></h2>
                    </div>

                    <?php if (!empty($section['tags'])): ?>
                    <div class="topic-tags content-item">
                        <?php foreach ($section['tags'] as $tag): ?>
                            <span class="topic-tag"><?php echo esc($tag); ?></span>
                        <?php endforeach; ?>
                    </div>
                    <?php endif; ?>

                    <?php foreach (($section['blocks'] ?? []) as $block): ?>
                        <?php if (($block['type'] ?? '') === 'narrative'): ?>
                            <div class="narrative content-item">
                                <?php echo $block['html'] ?? ''; ?>
                            </div>
                        <?php elseif (($block['type'] ?? '') === 'tweet'): ?>
                            <?php $t = $block['tweet'] ?? []; ?>
                            <div class="tweet-context content-item content-item--fast">
                                <div class="tweet-context__meta">
                                    <span><strong><?php echo esc($t['username'] ?? ''); ?></strong></span>
                                    <?php if (!empty($t['created_at'])): ?>
                                      <time datetime="<?php echo esc($t['created_at']); ?>"><?php echo esc(substr($t['created_at'], 0, 10)); ?></time>
                                    <?php endif; ?>
                                    <?php if (!empty($t['tweet_url'])): ?>
                                      <a href="<?php echo esc($t['tweet_url']); ?>" target="_blank" rel="noopener noreferrer">Open</a>
                                    <?php endif; ?>
                                    <span>❤ <?php echo esc($t['likes'] ?? 0); ?></span>
                                    <span>↻ <?php echo esc($t['retweets'] ?? 0); ?></span>
                                    <span>💬 <?php echo esc($t['replies'] ?? 0); ?></span>
                                </div>
                                <div class="tweet-context__text"><?php echo nl2br(esc($t['content'] ?? '')); ?></div>
                                <?php if (!empty($t['why'])): ?>
                                  <div class="tweet-context__why"><strong>Why it matters:</strong> <?php echo esc($t['why']); ?></div>
                                <?php endif; ?>
                                <?php if (!empty($t['image_description'])): ?>
                                  <div class="tweet-context__why"><strong>Images:</strong> <?php echo esc($t['image_description']); ?></div>
                                <?php endif; ?>
                            </div>

                            <div class="tweet-container content-item">
                                <blockquote class="twitter-tweet"><a href="<?php echo esc($t['tweet_url'] ?? ''); ?>"></a></blockquote>
                            </div>
                        <?php endif; ?>
                    <?php endforeach; ?>

                </section>
            <?php endforeach; ?>

            <!-- CLOSING -->
            <div class="narrative narrative--closing content-item">
                <?php echo $digest['closing_html'] ?? '<p><strong>That’s a wrap.</strong> See you tomorrow.</p>'; ?>
            </div>

            <!-- FEEDBACK -->
            <?php if ($submitted): ?>
              <div class="narrative content-item"><p><strong>Thanks.</strong> Your suggestion was appended to INTEREST.md.</p></div>
            <?php elseif ($error): ?>
              <div class="narrative content-item"><p><strong>Error:</strong> <?php echo esc($error); ?></p></div>
            <?php endif; ?>

            <form method="POST" class="feedback content-item" data-feedback>
                <button type="button" class="feedback__trigger" data-feedback-trigger>
                    ✏️ Suggest changes for tomorrow
                </button>
                <div class="feedback__content" data-feedback-content>
                    <textarea
                        name="suggestion"
                        class="feedback__textarea"
                        placeholder="What would you like me to change? More of something, less of another, different topics..."
                        aria-label="Your feedback"
                    ></textarea>
                    <div class="feedback__actions">
                        <button type="button" class="btn btn--secondary" data-feedback-cancel>Cancel</button>
                        <button type="submit" class="btn btn--primary">Send</button>
                    </div>
                </div>
            </form>

        </div>
    </div>

    <script async src="https://platform.twitter.com/widgets.js"></script>
    <script src="../../../twitter-app/app.js"></script>
</body>
</html>
PHP
```

Run:

```bash
chmod 644 /agent/workspace/twitter-digest/digests/index.php.template
```

---

### E3) Patch `twitter-app/app.js` so it doesn’t “fake-submit”

Right now it shows an alert and never submits.

**Change needed:** remove the “alert-only submit” behavior. Let the form submit normally (type=submit already in template above).

Minimal patch: in `FeedbackManager.bindEvents`, REMOVE submitBtn click handler, and remove `submit()` function call logic.

**Simplest acceptable behavior:**

* button is type submit → browser posts → PHP appends to INTEREST.md → page reload with success message.
* JS still handles open/close.

---

## Phase F — Digest runner + “do not run until captions complete”

### F1) Create digest runner script

Create: `/agent/workspace/twitter-digest/scripts/run_digest.sh`

```bash
#!/usr/bin/env bash
set -euo pipefail

ROOT="/agent/workspace/twitter-digest"
DATE_ISO="${1:-$(date +%Y-%m-%d)}"
DAY_TWEETS="${ROOT}/tweets/archive/${DATE_ISO}"
DAY_DIGEST="${ROOT}/digests/archive/${DATE_ISO}"
LOCK="/tmp/twitter-digest-${DATE_ISO}.lock"

mkdir -p "${ROOT}/logs/digest" "${DAY_DIGEST}"

exec 9>"${LOCK}"
flock -n 9 || { echo "[digest] already running"; exit 0; }

cd "${ROOT}"
source "${ROOT}/venv/bin/activate"

echo "[digest] start date=${DATE_ISO} at $(date -Is)"

# 1) Ensure merged raw_feed.csv exists
if [[ ! -f "${DAY_TWEETS}/raw_feed.csv" ]]; then
  echo "[digest] raw_feed.csv missing; merging from raw_feed_runs"
  ./bin/tweet-csv-merge "${DAY_TWEETS}/raw_feed_runs/"*.csv -o "${DAY_TWEETS}/raw_feed.csv" --keep first
fi

# 2) Ensure process done (incremental-safe)
./bin/twitter-process "${DATE_ISO}"

# 3) Ensure captions complete (MUST FINISH)
"${ROOT}/scripts/caption_until_complete.sh" "${DATE_ISO}"

# 4) Run digest agent (normal Claude) to generate digest.json
# Agent must read last 7 days too.
claude -p "Read ${ROOT}/agents/digest/CLAUDE.md and generate digests/archive/${DATE_ISO}/digest.json for date ${DATE_ISO}. Use tweets/archive/${DATE_ISO}/*.csv and previous 7 days digests."

# 5) Install index.php (copy template)
cp -f "${ROOT}/digests/index.php.template" "${DAY_DIGEST}/index.php"

# 6) Permissions for web server
chmod -R 755 "${ROOT}/digests/archive"
find "${ROOT}/digests/archive" -type f -name "*.php" -exec chmod 644 {} \;
find "${ROOT}/digests/archive" -type f -name "*.json" -exec chmod 644 {} \;

# 7) Completion ledger
touch "${DAY_DIGEST}/published.ok"
echo "${DATE_ISO}" >> "${ROOT}/COMPLETED-DAYS.md"

echo "[digest] complete at $(date -Is)"
```

Make executable:

```bash
chmod +x /agent/workspace/twitter-digest/scripts/run_digest.sh
```

✅ This enforces your rule: **digest will not proceed unless z-ai captions are complete**.

---

## Phase G — Telegram notification + checklist

### G1) Add a notify script (keep token out of code)

Create: `/agent/workspace/twitter-digest/scripts/notify_telegram.sh`

```bash
#!/usr/bin/env bash
set -euo pipefail

DATE_ISO="${1:-$(date +%Y-%m-%d)}"
ROOT="/agent/workspace/twitter-digest"

: "${TELEGRAM_BOT_TOKEN:?missing env TELEGRAM_BOT_TOKEN}"
: "${TELEGRAM_CHAT_ID:?missing env TELEGRAM_CHAT_ID}"
: "${PUBLIC_BASE_URL:?missing env PUBLIC_BASE_URL (e.g. https://yourdomain.com/twitter-digest)}"

URL="${PUBLIC_BASE_URL}/digests/archive/${DATE_ISO}/"

TEXT="📝 <b>Twitter digest ready</b>%0A%0A📅 ${DATE_ISO}%0A🔗 ${URL}"

curl -s -X POST "https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/sendMessage" \
  -d "chat_id=${TELEGRAM_CHAT_ID}" \
  -d "parse_mode=HTML" \
  -d "text=${TEXT}" >/dev/null

echo "[telegram] sent ${DATE_ISO}"
```

Then in `run_digest.sh` after publishing, call it:

```bash
# export TELEGRAM_BOT_TOKEN=...
# export TELEGRAM_CHAT_ID=...
# export PUBLIC_BASE_URL=...
"${ROOT}/scripts/notify_telegram.sh" "${DATE_ISO}"
```

**Checklist before sending:**

* `digests/archive/YYYY-MM-DD/digest.json` exists and valid JSON
* `digests/archive/YYYY-MM-DD/index.php` exists
* `tweets/archive/YYYY-MM-DD/images_done.ok` exists (captions complete)
* permissions applied

---

## 5) Cron schedule (final)

Edit crontab for the user running jobs (example):

```cron
# Scrape + process
0 8 * * *  /agent/workspace/twitter-digest/scripts/scrape_slot.sh morning >> /agent/workspace/twitter-digest/logs/scrape/morning.log 2>&1 && /agent/workspace/twitter-digest/scripts/process_day.sh >> /agent/workspace/twitter-digest/logs/process/morning.log 2>&1
0 13 * * * /agent/workspace/twitter-digest/scripts/scrape_slot.sh midday  >> /agent/workspace/twitter-digest/logs/scrape/midday.log 2>&1 && /agent/workspace/twitter-digest/scripts/process_day.sh >> /agent/workspace/twitter-digest/logs/process/midday.log 2>&1
0 18 * * * /agent/workspace/twitter-digest/scripts/scrape_slot.sh evening >> /agent/workspace/twitter-digest/logs/scrape/evening.log 2>&1 && /agent/workspace/twitter-digest/scripts/process_day.sh >> /agent/workspace/twitter-digest/logs/process/evening.log 2>&1

# Caption 10 minutes later (runs until complete for currently-known images)
10 8 * * *  /agent/workspace/twitter-digest/scripts/caption_until_complete.sh >> /agent/workspace/twitter-digest/logs/caption/morning.log 2>&1
10 13 * * * /agent/workspace/twitter-digest/scripts/caption_until_complete.sh >> /agent/workspace/twitter-digest/logs/caption/midday.log 2>&1
10 18 * * * /agent/workspace/twitter-digest/scripts/caption_until_complete.sh >> /agent/workspace/twitter-digest/logs/caption/evening.log 2>&1

# Final digest (will internally force caption completion before proceeding)
0 22 * * *  /agent/workspace/twitter-digest/scripts/run_digest.sh >> /agent/workspace/twitter-digest/logs/digest/night.log 2>&1
```

---

## 6) Testing (no waiting)

### Test full pipeline for today

```bash
cd /agent/workspace/twitter-digest
./scripts/scrape_slot.sh test
./scripts/process_day.sh
./scripts/caption_until_complete.sh
./scripts/run_digest.sh
```

### Test caption validator

```bash
python3 scripts/validate_captions.py $(date +%Y-%m-%d)
```

### Test digest rendering locally

```bash
cd /agent/workspace/twitter-digest/digests/archive/$(date +%Y-%m-%d)
php -S 0.0.0.0:8000
```

---

## 7) Hard requirements recap (so your agent doesn’t “kinda do it”)

* ✅ Image captioning runs after each scrape (T+10m)
* ✅ Image captions **must finish** (loop forever, retry every 20m attempt)
* ✅ Digest agent runs only after captions complete (enforced in `run_digest.sh`)
* ✅ Digest agent reads previous 7 days and maintains evolving narratives
* ✅ `index.php` shows tweet author/date/text above embeds (rendered from `digest.json`)
* ✅ Web feedback appends to `INTEREST.md` (no DB, no extra services)
* ✅ Telegram notify + permissions are included in publish step

---

If you want the *next* upgrade after this: add a tiny `scripts/validate_digest_json.py` (schema check) so the digest agent can’t output malformed JSON and silently publish garbage.
