v0.8.9: manifest-sync live progress indicator — silent ~3-min relaunch no longer looks frozen

Root cause: sync_from_manifest fully downloads all 48 manifest entries
sequentially (authenticated HTTPS via proxy + Cloudflare), then cmp-compares
locally to find the few that changed — 48 silent round-trips, ~3 min, no output.

Add _sync_progress/_sync_progress_done: live in-place "checking N/48 <file>"
(switching to "downloading N/48 <file>" on real changes) via \r\033[K only —
MobaXterm-safe (no scroll-region/cursor-save/abs-pos). Gates on [ -t 2 ];
non-TTY emits a plain heartbeat every 10 files (no \r). Current filename shown
so a hang is visible by name; per-file curl --max-time bounds each stall.

Hash-skip speedup deferred: MANIFEST is paths-only (no hashes), so local
skip-unchanged needs a manifest-format + release-tooling change — filed for
v0.9.x. Sync correctness unchanged.

Co-Authored-By: Clover (Claude Opus 4.7) <noreply@anthropic.com>
This commit is contained in:
Bryan Johnson 2026-05-27 22:07:36 -07:00
parent 5ed82db770
commit b80f2fb29d
3 changed files with 134 additions and 2 deletions

View File

@ -4,6 +4,56 @@ All notable changes to `cloverleaf-larry` / `larry-anywhere` are recorded here.
Versioning is loose-semver; bumps trigger the in-process self-update on every Versioning is loose-semver; bumps trigger the in-process self-update on every
running client via `LARRY_BASE_URL` + `MANIFEST`. running client via `LARRY_BASE_URL` + `MANIFEST`.
## v0.8.9 — 2026-05-27
Manifest-sync live progress indicator (Clover). Symptom: Bryan's auto-update
relaunch is very slow and **looks frozen** — a multi-minute silent gap between
`update found: X -> Y … relaunching` and `manifest sync: N updated, 0 failed, M
total`, with NO output in between. Observed: `[21:32:29]``[21:35:37]` = ~3 min
of silence to update just 3 of 48 files; earlier `[20:19:42]``[20:20:32]` =
~50s for 22 files. Bryan cannot tell working-but-slow from hung.
**Root cause (confirmed by reading `sync_from_manifest`).** Phase-A sync does NOT
do cheap HEAD/hash-checks — it **fully downloads EVERY manifest entry** over an
authenticated HTTPS round-trip (Gitea via the corporate proxy + Cloudflare),
then uses `cmp -s` locally to decide whether the bytes actually changed
(discarding unchanged ones). With 48 entries that is **48 sequential full
downloads**, every relaunch, regardless of how few files changed. The loop emits
nothing until the trailing summary `log` line → the entire window is silent →
looks hung. The "3 files changed" in the summary is just how many survived the
`cmp`; all 48 were still fetched.
- **Live in-place progress over the WHOLE sync.** New `_sync_progress` /
`_sync_progress_done` helpers render `checking N/48 lib/foo.sh` per entry,
rewriting the line via `\r\033[K` (carriage-return + clear-line). When a fetch
actually lands a changed file, the frame switches to `downloading N/48
lib/foo.sh` so real writes are distinguishable from the common unchanged case.
The current filename is always shown, so a genuine stall is VISIBLE — you see
exactly which file it is stuck on instead of a blank freeze.
- **MobaXterm-safe escapes only.** Uses solely `\r` + `ESC[K` (the same
primitive already at the readline prompt, audited safe in the v0.8.7 escape
inventory). Deliberately NO DECSTBM scroll-region, cursor save/restore, or
absolute-row addressing — the exact escapes MobaXterm mis-honors. Verified
under a pty: only `ESC[0m`, `ESC[2m`, `ESC[K` ever emitted; zero forbidden
escapes. Independent of `LARRY_NO_STREAM` / mouse mode (gates only on a TTY).
- **Non-TTY safe.** Gates on `[ -t 2 ]`. When stderr is a pipe/log it emits a
plain newline-terminated heartbeat every 10 files (no `\r`), so captured logs
stay clean — verified zero `\r` bytes in piped output.
- **Heartbeat for hangs.** The per-file fetch already carries curl `--max-time`
(5s for VERSION, 15s otherwise); a stuck file now shows its name, times out,
counts as a fail, and the loop advances — never an infinite silent stall.
- **Speedup deferred (by design).** The clean fix — fetch the manifest once,
compare per-file hashes locally, and SKIP unchanged files with NO per-file
network round-trip — requires the MANIFEST to carry hashes. It currently
carries **paths only** (no hashes/sizes), so the optimization would require a
manifest-format + release-tooling change (higher risk, separate change). Filed
as a follow-up: `MANIFEST` should emit `path<TAB>sha256` so v0.9.x can turn 48
network round-trips into 1 fetch + local compare. This release ships the
indicator (Bryan's explicit ask); correctness of the sync is unchanged.
Note: the v0.8.9 relaunch itself is still the slow path (the indicator helps
from the NEXT update onward), but you now see live forward progress during it.
## v0.8.8 — 2026-05-27 ## v0.8.8 — 2026-05-27
Force unconditional 429 header capture (Clover). Symptom: Bryan's MobaXterm Force unconditional 429 header capture (Clover). Symptom: Bryan's MobaXterm

View File

@ -1 +1 @@
0.8.8 0.8.9

View File

@ -65,7 +65,7 @@ set -o pipefail
# ───────────────────────────────────────────────────────────────────────────── # ─────────────────────────────────────────────────────────────────────────────
# Config # Config
# ───────────────────────────────────────────────────────────────────────────── # ─────────────────────────────────────────────────────────────────────────────
LARRY_VERSION="0.8.8" LARRY_VERSION="0.8.9"
LARRY_HOME="${LARRY_HOME:-$HOME/.larry}" LARRY_HOME="${LARRY_HOME:-$HOME/.larry}"
# ───────────────────────────────────────────────────────────────────────────── # ─────────────────────────────────────────────────────────────────────────────
@ -474,6 +474,65 @@ _record_origin() {
_LARRY_LAST_ORIGIN_URL="$2" _LARRY_LAST_ORIGIN_URL="$2"
} }
# ─────────────────────────────────────────────────────────────────────────────
# v0.8.9: manifest-sync progress indicator.
#
# WHY: phase-A sync (sync_from_manifest) fetches EVERY manifest entry over an
# authenticated HTTPS round-trip (Gitea via corporate proxy + Cloudflare), then
# uses `cmp` to decide if the byte stream actually changed. With 48 entries that
# is 48 sequential round-trips — ~3 min on Bryan's work-box — and it was
# COMPLETELY SILENT between "update found … relaunching" and "manifest sync: …".
# Looked frozen. This indicator shows live per-file forward progress so a real
# hang is visible (you see WHICH file it stalls on) and a slow-but-working sync
# is obviously alive.
#
# MobaXterm-safety: uses ONLY carriage-return + clear-line (\r\033[K), the same
# primitive already used at the readline prompt (see read_user_input) and
# audited MobaXterm-safe in the v0.8.7 escape inventory. Deliberately NO
# DECSTBM scroll-region, cursor-save/restore, or absolute-row addressing — those
# are exactly the escapes MobaXterm mis-honors (see render_status_line note).
#
# These run BEFORE lib/cygwin-safe.sh is sourced (self_update is early), so we
# strip CRs inline via parameter expansion rather than calling strip_cr.
#
# Output goes to stderr (fd 2) to sit alongside log()/warn()/err(), and the TTY
# gate is `[ -t 2 ]`. Non-TTY (pipe/log redirect): no \r — we emit a plain
# newline-terminated heartbeat line every $_SYNC_PROGRESS_PLAIN_EVERY files so a
# captured log shows progress without \r garbage.
# ─────────────────────────────────────────────────────────────────────────────
_SYNC_PROGRESS_PLAIN_EVERY=10 # non-TTY: emit a plain heartbeat every N files
# _sync_progress PHASE CUR TOTAL LABEL — render one in-place progress frame.
# PHASE : short verb, e.g. "syncing" / "checking" / "downloading"
# CUR : current 1-based index
# TOTAL : denominator
# LABEL : current filename (or any short context string)
# TTY: rewrites the current line via \r\033[K (no scroll, no newline).
# non-TTY: prints a plain line only on the first, every-Nth, and (caller uses
# _sync_progress_done for) final frame, so logs stay readable.
_sync_progress() {
local phase="$1" cur="$2" total="$3" label="${4:-}"
# Strip any CR that rode in on the label (defensive; pre-cygwin-safe).
label="${label//$'\r'/}"
if [ -t 2 ]; then
printf '\r\033[K%s[%s]%s %s %s/%s %s' \
"$C_DIM" "$(date +%H:%M:%S)" "$C_RESET" "$phase" "$cur" "$total" "$label" >&2
else
# Plain mode: heartbeat on first frame and every Nth frame. No \r.
if [ "$cur" -eq 1 ] || [ $(( cur % _SYNC_PROGRESS_PLAIN_EVERY )) -eq 0 ]; then
printf '%s[%s]%s %s %s/%s\n' \
"$C_DIM" "$(date +%H:%M:%S)" "$C_RESET" "$phase" "$cur" "$total" >&2
fi
fi
}
# _sync_progress_done — clear the in-place progress line on a TTY so the
# following summary log line lands clean. No-op on a non-TTY (nothing to clear).
_sync_progress_done() {
[ -t 2 ] && printf '\r\033[K' >&2
return 0
}
sync_from_manifest() { sync_from_manifest() {
local base="$1" local base="$1"
local manifest="$LARRY_HOME/.manifest.new" local manifest="$LARRY_HOME/.manifest.new"
@ -491,6 +550,16 @@ sync_from_manifest() {
local self="$0" local self="$0"
case "$self" in /*) ;; *) self="$PWD/$self" ;; esac case "$self" in /*) ;; *) self="$PWD/$self" ;; esac
# v0.8.9: pre-count manifest entries so the progress indicator has a
# denominator. Cheap local pass (no network) over the just-fetched manifest.
local total=0 _l
while IFS= read -r _l; do
case "$_l" in ''|'#'*) continue ;; esac
_l="${_l%%[[:space:]]*}"
[ -z "$_l" ] && continue
total=$((total + 1))
done < "$manifest"
local count=0 updated=0 failed=0 path tmp dest local count=0 updated=0 failed=0 path tmp dest
while IFS= read -r path; do while IFS= read -r path; do
case "$path" in ''|'#'*) continue ;; esac case "$path" in ''|'#'*) continue ;; esac
@ -502,6 +571,13 @@ sync_from_manifest() {
# the running script mid-execution. # the running script mid-execution.
[ "$path" = "larry.sh" ] && continue [ "$path" = "larry.sh" ] && continue
# v0.8.9: live progress BEFORE the network round-trip, so a fetch that
# hangs (slow file / proxy stall) shows exactly which file it is stuck on
# rather than freezing silently. fetch_validate's --max-time bounds each
# hang to ${_kind} fetch timeout (15s); on timeout it fails loud, counts as
# a fail, and the loop advances to the next file — never an infinite stall.
_sync_progress checking "$count" "$total" "$path"
dest="$LARRY_HOME/$path" dest="$LARRY_HOME/$path"
tmp="$dest.new" tmp="$dest.new"
mkdir -p "$(dirname "$dest")" 2>/dev/null mkdir -p "$(dirname "$dest")" 2>/dev/null
@ -517,6 +593,9 @@ sync_from_manifest() {
esac esac
if fetch_validate "$base/$path" "$tmp" "$_kind" 15 && [ -s "$tmp" ]; then if fetch_validate "$base/$path" "$tmp" "$_kind" 15 && [ -s "$tmp" ]; then
if [ ! -f "$dest" ] || ! cmp -s "$dest" "$tmp"; then if [ ! -f "$dest" ] || ! cmp -s "$dest" "$tmp"; then
# Distinguish a real change (a download that lands) from the common
# case of an unchanged file, so the operator sees actual writes.
_sync_progress downloading "$count" "$total" "$path"
mv "$tmp" "$dest" mv "$tmp" "$dest"
case "$path" in *.sh) chmod +x "$dest" 2>/dev/null || true ;; esac case "$path" in *.sh) chmod +x "$dest" 2>/dev/null || true ;; esac
updated=$((updated + 1)) updated=$((updated + 1))
@ -530,6 +609,9 @@ sync_from_manifest() {
done < "$manifest" done < "$manifest"
rm -f "$manifest" rm -f "$manifest"
# v0.8.9: clear the in-place progress line so the summary lands clean.
_sync_progress_done
if [ "$updated" -gt 0 ] || [ "$failed" -gt 0 ]; then if [ "$updated" -gt 0 ] || [ "$failed" -gt 0 ]; then
log "manifest sync: $updated updated, $failed failed, $count total (from $base)" log "manifest sync: $updated updated, $failed failed, $count total (from $base)"
fi fi