From b80f2fb29d149f8c38e7606160c81baa65ca54fa Mon Sep 17 00:00:00 2001 From: Bryan Johnson Date: Wed, 27 May 2026 22:07:36 -0700 Subject: [PATCH] =?UTF-8?q?v0.8.9:=20manifest-sync=20live=20progress=20ind?= =?UTF-8?q?icator=20=E2=80=94=20silent=20~3-min=20relaunch=20no=20longer?= =?UTF-8?q?=20looks=20frozen?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Root cause: sync_from_manifest fully downloads all 48 manifest entries sequentially (authenticated HTTPS via proxy + Cloudflare), then cmp-compares locally to find the few that changed — 48 silent round-trips, ~3 min, no output. Add _sync_progress/_sync_progress_done: live in-place "checking N/48 " (switching to "downloading N/48 " on real changes) via \r\033[K only — MobaXterm-safe (no scroll-region/cursor-save/abs-pos). Gates on [ -t 2 ]; non-TTY emits a plain heartbeat every 10 files (no \r). Current filename shown so a hang is visible by name; per-file curl --max-time bounds each stall. Hash-skip speedup deferred: MANIFEST is paths-only (no hashes), so local skip-unchanged needs a manifest-format + release-tooling change — filed for v0.9.x. Sync correctness unchanged. Co-Authored-By: Clover (Claude Opus 4.7) --- CHANGELOG.md | 50 +++++++++++++++++++++++++++++++ VERSION | 2 +- larry.sh | 84 +++++++++++++++++++++++++++++++++++++++++++++++++++- 3 files changed, 134 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a77db49..9626441 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,56 @@ All notable changes to `cloverleaf-larry` / `larry-anywhere` are recorded here. Versioning is loose-semver; bumps trigger the in-process self-update on every running client via `LARRY_BASE_URL` + `MANIFEST`. +## v0.8.9 — 2026-05-27 + +Manifest-sync live progress indicator (Clover). Symptom: Bryan's auto-update +relaunch is very slow and **looks frozen** — a multi-minute silent gap between +`update found: X -> Y … relaunching` and `manifest sync: N updated, 0 failed, M +total`, with NO output in between. Observed: `[21:32:29]` → `[21:35:37]` = ~3 min +of silence to update just 3 of 48 files; earlier `[20:19:42]` → `[20:20:32]` = +~50s for 22 files. Bryan cannot tell working-but-slow from hung. + +**Root cause (confirmed by reading `sync_from_manifest`).** Phase-A sync does NOT +do cheap HEAD/hash-checks — it **fully downloads EVERY manifest entry** over an +authenticated HTTPS round-trip (Gitea via the corporate proxy + Cloudflare), +then uses `cmp -s` locally to decide whether the bytes actually changed +(discarding unchanged ones). With 48 entries that is **48 sequential full +downloads**, every relaunch, regardless of how few files changed. The loop emits +nothing until the trailing summary `log` line → the entire window is silent → +looks hung. The "3 files changed" in the summary is just how many survived the +`cmp`; all 48 were still fetched. + +- **Live in-place progress over the WHOLE sync.** New `_sync_progress` / + `_sync_progress_done` helpers render `checking N/48 lib/foo.sh` per entry, + rewriting the line via `\r\033[K` (carriage-return + clear-line). When a fetch + actually lands a changed file, the frame switches to `downloading N/48 + lib/foo.sh` so real writes are distinguishable from the common unchanged case. + The current filename is always shown, so a genuine stall is VISIBLE — you see + exactly which file it is stuck on instead of a blank freeze. +- **MobaXterm-safe escapes only.** Uses solely `\r` + `ESC[K` (the same + primitive already at the readline prompt, audited safe in the v0.8.7 escape + inventory). Deliberately NO DECSTBM scroll-region, cursor save/restore, or + absolute-row addressing — the exact escapes MobaXterm mis-honors. Verified + under a pty: only `ESC[0m`, `ESC[2m`, `ESC[K` ever emitted; zero forbidden + escapes. Independent of `LARRY_NO_STREAM` / mouse mode (gates only on a TTY). +- **Non-TTY safe.** Gates on `[ -t 2 ]`. When stderr is a pipe/log it emits a + plain newline-terminated heartbeat every 10 files (no `\r`), so captured logs + stay clean — verified zero `\r` bytes in piped output. +- **Heartbeat for hangs.** The per-file fetch already carries curl `--max-time` + (5s for VERSION, 15s otherwise); a stuck file now shows its name, times out, + counts as a fail, and the loop advances — never an infinite silent stall. +- **Speedup deferred (by design).** The clean fix — fetch the manifest once, + compare per-file hashes locally, and SKIP unchanged files with NO per-file + network round-trip — requires the MANIFEST to carry hashes. It currently + carries **paths only** (no hashes/sizes), so the optimization would require a + manifest-format + release-tooling change (higher risk, separate change). Filed + as a follow-up: `MANIFEST` should emit `pathsha256` so v0.9.x can turn 48 + network round-trips into 1 fetch + local compare. This release ships the + indicator (Bryan's explicit ask); correctness of the sync is unchanged. + +Note: the v0.8.9 relaunch itself is still the slow path (the indicator helps +from the NEXT update onward), but you now see live forward progress during it. + ## v0.8.8 — 2026-05-27 Force unconditional 429 header capture (Clover). Symptom: Bryan's MobaXterm diff --git a/VERSION b/VERSION index 6201b5f..55485e1 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.8.8 +0.8.9 diff --git a/larry.sh b/larry.sh index e71a2b1..16b714a 100755 --- a/larry.sh +++ b/larry.sh @@ -65,7 +65,7 @@ set -o pipefail # ───────────────────────────────────────────────────────────────────────────── # Config # ───────────────────────────────────────────────────────────────────────────── -LARRY_VERSION="0.8.8" +LARRY_VERSION="0.8.9" LARRY_HOME="${LARRY_HOME:-$HOME/.larry}" # ───────────────────────────────────────────────────────────────────────────── @@ -474,6 +474,65 @@ _record_origin() { _LARRY_LAST_ORIGIN_URL="$2" } +# ───────────────────────────────────────────────────────────────────────────── +# v0.8.9: manifest-sync progress indicator. +# +# WHY: phase-A sync (sync_from_manifest) fetches EVERY manifest entry over an +# authenticated HTTPS round-trip (Gitea via corporate proxy + Cloudflare), then +# uses `cmp` to decide if the byte stream actually changed. With 48 entries that +# is 48 sequential round-trips — ~3 min on Bryan's work-box — and it was +# COMPLETELY SILENT between "update found … relaunching" and "manifest sync: …". +# Looked frozen. This indicator shows live per-file forward progress so a real +# hang is visible (you see WHICH file it stalls on) and a slow-but-working sync +# is obviously alive. +# +# MobaXterm-safety: uses ONLY carriage-return + clear-line (\r\033[K), the same +# primitive already used at the readline prompt (see read_user_input) and +# audited MobaXterm-safe in the v0.8.7 escape inventory. Deliberately NO +# DECSTBM scroll-region, cursor-save/restore, or absolute-row addressing — those +# are exactly the escapes MobaXterm mis-honors (see render_status_line note). +# +# These run BEFORE lib/cygwin-safe.sh is sourced (self_update is early), so we +# strip CRs inline via parameter expansion rather than calling strip_cr. +# +# Output goes to stderr (fd 2) to sit alongside log()/warn()/err(), and the TTY +# gate is `[ -t 2 ]`. Non-TTY (pipe/log redirect): no \r — we emit a plain +# newline-terminated heartbeat line every $_SYNC_PROGRESS_PLAIN_EVERY files so a +# captured log shows progress without \r garbage. +# ───────────────────────────────────────────────────────────────────────────── +_SYNC_PROGRESS_PLAIN_EVERY=10 # non-TTY: emit a plain heartbeat every N files + +# _sync_progress PHASE CUR TOTAL LABEL — render one in-place progress frame. +# PHASE : short verb, e.g. "syncing" / "checking" / "downloading" +# CUR : current 1-based index +# TOTAL : denominator +# LABEL : current filename (or any short context string) +# TTY: rewrites the current line via \r\033[K (no scroll, no newline). +# non-TTY: prints a plain line only on the first, every-Nth, and (caller uses +# _sync_progress_done for) final frame, so logs stay readable. +_sync_progress() { + local phase="$1" cur="$2" total="$3" label="${4:-}" + # Strip any CR that rode in on the label (defensive; pre-cygwin-safe). + label="${label//$'\r'/}" + if [ -t 2 ]; then + printf '\r\033[K%s[%s]%s %s %s/%s %s' \ + "$C_DIM" "$(date +%H:%M:%S)" "$C_RESET" "$phase" "$cur" "$total" "$label" >&2 + else + # Plain mode: heartbeat on first frame and every Nth frame. No \r. + if [ "$cur" -eq 1 ] || [ $(( cur % _SYNC_PROGRESS_PLAIN_EVERY )) -eq 0 ]; then + printf '%s[%s]%s %s %s/%s\n' \ + "$C_DIM" "$(date +%H:%M:%S)" "$C_RESET" "$phase" "$cur" "$total" >&2 + fi + fi +} + +# _sync_progress_done — clear the in-place progress line on a TTY so the +# following summary log line lands clean. No-op on a non-TTY (nothing to clear). +_sync_progress_done() { + [ -t 2 ] && printf '\r\033[K' >&2 + return 0 +} + sync_from_manifest() { local base="$1" local manifest="$LARRY_HOME/.manifest.new" @@ -491,6 +550,16 @@ sync_from_manifest() { local self="$0" case "$self" in /*) ;; *) self="$PWD/$self" ;; esac + # v0.8.9: pre-count manifest entries so the progress indicator has a + # denominator. Cheap local pass (no network) over the just-fetched manifest. + local total=0 _l + while IFS= read -r _l; do + case "$_l" in ''|'#'*) continue ;; esac + _l="${_l%%[[:space:]]*}" + [ -z "$_l" ] && continue + total=$((total + 1)) + done < "$manifest" + local count=0 updated=0 failed=0 path tmp dest while IFS= read -r path; do case "$path" in ''|'#'*) continue ;; esac @@ -502,6 +571,13 @@ sync_from_manifest() { # the running script mid-execution. [ "$path" = "larry.sh" ] && continue + # v0.8.9: live progress BEFORE the network round-trip, so a fetch that + # hangs (slow file / proxy stall) shows exactly which file it is stuck on + # rather than freezing silently. fetch_validate's --max-time bounds each + # hang to ${_kind} fetch timeout (15s); on timeout it fails loud, counts as + # a fail, and the loop advances to the next file — never an infinite stall. + _sync_progress checking "$count" "$total" "$path" + dest="$LARRY_HOME/$path" tmp="$dest.new" mkdir -p "$(dirname "$dest")" 2>/dev/null @@ -517,6 +593,9 @@ sync_from_manifest() { esac if fetch_validate "$base/$path" "$tmp" "$_kind" 15 && [ -s "$tmp" ]; then if [ ! -f "$dest" ] || ! cmp -s "$dest" "$tmp"; then + # Distinguish a real change (a download that lands) from the common + # case of an unchanged file, so the operator sees actual writes. + _sync_progress downloading "$count" "$total" "$path" mv "$tmp" "$dest" case "$path" in *.sh) chmod +x "$dest" 2>/dev/null || true ;; esac updated=$((updated + 1)) @@ -530,6 +609,9 @@ sync_from_manifest() { done < "$manifest" rm -f "$manifest" + # v0.8.9: clear the in-place progress line so the summary lands clean. + _sync_progress_done + if [ "$updated" -gt 0 ] || [ "$failed" -gt 0 ]; then log "manifest sync: $updated updated, $failed failed, $count total (from $base)" fi