From f5f56439d0fff02409e1b4574b5dd08169683dda Mon Sep 17 00:00:00 2001 From: Bryan Johnson Date: Thu, 28 May 2026 12:44:38 -0700 Subject: [PATCH] v0.8.23: regression chain-walk route-test capture (nc-regression --chain-walk) Resolves the downstream route chain via nc-paths, grabs N recent messages from the START inbound's SMAT, walks each ENTRY node (START + post-==> remote inbounds) running hciroutetest -a -d -f nl, chaining each step's selected .out. across cross-site hops. Generates per-chain commands.sh for the engine box; --dry-run stubs the engine. Command syntax mined verbatim from the v1/v2 route_test wrappers. Fixes --help sed range (header ends at 94). Co-Authored-By: Claude Opus 4.7 --- CHANGELOG.md | 55 +++++++++ MANIFEST | 8 +- VERSION | 2 +- larry.sh | 2 +- lib/nc-regression.sh | 269 ++++++++++++++++++++++++++++++++++++++++++- 5 files changed, 329 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c724228..d95bd01 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,61 @@ All notable changes to `cloverleaf-larry` / `larry-anywhere` are recorded here. Versioning is loose-semver; bumps trigger the in-process self-update on every running client via `LARRY_BASE_URL` + `MANIFEST`. +## v0.8.23 — 2026-05-28 + +**★ REGRESSION CHAIN-WALK route-test capture (Bryan's priority).** New +`--chain-walk` mode in `lib/nc-regression.sh` — a single-env (per-env) capture +that runs `route_test` at every routing(inbound) thread along an `nc-paths` +chain and chains each step's per-destination output into the next step's input. +Orthogonal to the existing 6-phase env-A/env-B pipeline; Bryan runs it on env-A +and env-B with the same START input, then the existing `hl7-diff --ignore MSH.7` +diff phase compares the captured outputs. + +Workflow (verbatim Bryan spec): +- Take a START thread + N. Grab the N most-recent messages from the START + inbound's SMAT via `nc-msgs --limit N --format raw` → `input.msgs`. +- Resolve the downstream chain(s) from START with `nc-paths --format jsonl`. + `--target ` restricts to the single chain ending at that node; + with no `--target`, ALL fan-out branches are walked (each branch a chain dir). +- The route_test ENTRY threads are the START node plus every node that + immediately follows a cross-site `==>` hop (the remote inbound). Outbound + *sender* nodes (e.g. `OB3_RAD_ordersS`) are NOT entries — they are produced as + `.out.` files by the upstream route_test. +- At each entry E: `hciroutetest -a -d -f nl -s ` — the + command string is mined VERBATIM from the v1/v2 wrapper + (`cloverleaf_tools/tools/route_test_wrapper.py:10/149-156`); `-f nl` yields + NEWLINE output directly (no manual `len2nl`+delete). `-a` (all routes) writes + every fan-out branch's output in one invocation. +- route_test writes ONE FILE PER DESTINATION named `.out.` + (`.out.` naming confirmed from `legacy_workflow_commands.py:1015`). File + SELECTION: the node immediately AFTER E in the chain is the suffix selected to + feed the NEXT step. For the cross-site boundary `S ==> R`, the selected + `.out.` payload becomes the input fed to the next site's inbound `R`. +- PRODUCES under `$OUT/chain//`: `input.msgs` (original START messages), + `step-NN..out.` (per-destination outputs), `step-NN..selected` + (the chosen next-step input), `commands.sh` (the exact generated command + sequence), `chain.txt` (the resolved v1 chain). + +`hciroutetest` is a Linux engine binary needing a live engine — NOT runnable off +the server — so chain-walk GENERATES the orchestration + command STRINGS + +file-chaining/selection and stubs the engine call under `--dry-run`; real +execution is on the server (source the Cloverleaf profile first). Everything that +does NOT need the engine is self-verified against the real integrator: the +nc_paths chain resolution, the nc-msgs message-grab command, the per-step +route_test command strings, and the `.out.` → next-step SELECTION. + +New flags: `--chain-walk --start [--site SITE] --count N +--hciroot HCIROOT --out DIR [--target ] [--route-test-bin BIN] +[--dry-run]`. Portable bash (Win+Linux), pure bash+awk, **API-FREE**, no +python/.pyz. Existing phase pipeline untouched. + +Verified on `ORUto_CodaMextrix orders` +(`mux/RadOrdfr_epic_972310 --> mux/OB3_RAD_ordersS ==> orders/IB3_RAD_muxS --> +orders/ORUto_CodaMextrix`): 2 route_test entries (`RadOrdfr_epic_972310`@mux, +`IB3_RAD_muxS`@orders); start grab + the exact per-step `hciroutetest` strings + +the `.out.OB3_RAD_ordersS` (cross-site) → `.out.ORUto_CodaMextrix` (terminus) +selection. No-target run walks all 6 `IB3_RAD_muxS` fan-out branches. + ## v0.8.22 — 2026-05-28 Interface **`document`** tool follow-on (`lib/nc-document.sh`, `inbound-systems.tsv`, diff --git a/MANIFEST b/MANIFEST index a5eb35d..daeca67 100644 --- a/MANIFEST +++ b/MANIFEST @@ -23,16 +23,16 @@ # scripts/make-manifest.sh and bump VERSION. # Top-level scripts -larry.sh fd6c46db5dd8872d2fabe7f7776a5d8e672d4448c77bd2ed6646931da93ed92e +larry.sh 57a96b78c7ab319537b2203a7363454ebcd482ed048d63dc05668fad3c292ee7 larry-tunnel.sh 6b050e4eeab15669f4858eaf3b807f168f211ced07815db9521bc40a093f6aaa larry-auth.sh a220cdf7878569dc3028951ee57fc8d5e706a8ca5c6aa45347b58facb386f831 larry-rollback.sh 91b5e9aa6c79266bf306dcfba4ca791c07971bd6924d67a779037531648aa6d0 install-larry.sh fa36e23a39eacbd0d7ecedd3b42131902f816ee7e98241dfc6e28c6e4ba80423 # Metadata -VERSION 86456bcc629d981e2e34d7fd53096f0dba9690460593b3b84583be05f3fd544e +VERSION 189191ed6bf46fb5d0b7f887c60f28c097b6b4df83273227744f0510b00d89db MANUAL.md c64bd0251a51ad150508b4e1185355bc4826a64071d4de339f92ed550dbfacde -CHANGELOG.md a09d2ad791bcb7eafe6a181191dd9b10e10d12f3887e8dda4d034dbd23f92e4a +CHANGELOG.md 646791fb1a6f99c326869c57e17cb1662827afa566e68d91fe580ed68d0f02df # Agent personas (system-prompt overlays) agents/larry.md 0a1ef737e7fc133ab35be09f79c3a4df33de814e0404b69b950932d0c8a01be1 @@ -106,4 +106,4 @@ lib/nc-document.sh e0b5c5b0a778abff2f09377cd1692ba445140e7da84aa8a96a002081f31b8 lib/nc-diff-interface.sh 6b64ec3070a3d75d1d79632c9aeb357177fdbcc77c474aa78e6f6929fda1a324 lib/nc-find.sh 8c79e0acad7de56e4e1f12d61e071a4b98c4e2310a1f7fb183697df521215e3f lib/nc-insert-protocol.sh ad1fa0bafbf4fdfb12bad20f9c22c3eed519f8846774331e26aa9becd6f8898a -lib/nc-regression.sh b3583fb07cbf46518312613401acb1e5b07bd2d81a4d259a297b47342182b403 +lib/nc-regression.sh 70999a60608439f7bf1a3abb9f5e9854b5ea03025ef29ddbca683896346d1bce diff --git a/VERSION b/VERSION index d426c97..cddd48a 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.8.22 +0.8.23 diff --git a/larry.sh b/larry.sh index 41710b4..be0dd18 100755 --- a/larry.sh +++ b/larry.sh @@ -78,7 +78,7 @@ set -o pipefail # ───────────────────────────────────────────────────────────────────────────── # Config # ───────────────────────────────────────────────────────────────────────────── -LARRY_VERSION="0.8.22" +LARRY_VERSION="0.8.23" LARRY_HOME="${LARRY_HOME:-$HOME/.larry}" # ───────────────────────────────────────────────────────────────────────────── diff --git a/lib/nc-regression.sh b/lib/nc-regression.sh index 722761a..84c6f43 100755 --- a/lib/nc-regression.sh +++ b/lib/nc-regression.sh @@ -43,6 +43,55 @@ # threads:N1,N2,N3 comma-separated list # site every inbound in the configured site # server every inbound in every site under HCIROOT +# +# ───────────────────────────────────────────────────────────────────────────── +# CHAIN-WALK MODE (v0.8.23, Bryan's REGRESSION CHAIN-WALK route-test capture) +# ───────────────────────────────────────────────────────────────────────────── +# A second, orthogonal entry point that captures the route_test output at EVERY +# routing(inbound) thread along an nc-paths chain, single-env (per-env capture). +# Bryan runs it on env-A and env-B with the same START input, then the existing +# diff phase (hl7-diff --ignore MSH.7) compares the captured outputs. +# +# nc-regression.sh --chain-walk +# --start [--site SITE] +# --count N +# --hciroot HCIROOT # single env (this box) +# --out DIR +# [--route-test-bin hciroutetest] # default: hciroutetest +# [--target ] # restrict to the chain ending here +# [--dry-run] # generate cmds, do NOT exec engine +# +# Workflow (Bryan's spec): +# 1. Resolve the downstream chain(s) from START via nc-paths (--format jsonl). +# Each chain is "site/thread --> ... ==> ... --> site/thread". --target +# restricts to the single chain whose terminus matches. +# 2. Grab the N most-recent messages from the START inbound's SMAT (nc-msgs +# --limit N --format raw) → the chain-walk's ORIGINAL INPUT .msgs file. +# 3. Walk the chain. The route_test ENTRY threads are the START node plus every +# node that immediately follows a cross-site "==>" hop (the remote inbound). +# At each entry thread E: +# hciroutetest -a -d -f nl -s +# route_test writes ONE FILE PER DESTINATION named .out. +# (the OUTBOUND/dest thread is the suffix; -f nl gives NEWLINE output, so no +# manual len2nl+delete). We capture ALL fan-out branches. +# 4. File SELECTION for the next step: the node IMMEDIATELY AFTER E in the path +# is the suffix to select. For E -->(intra) X, select .out.. For the +# cross-site boundary S ==> R (S = E's local outbound sender, the path node +# right after E), the selected .out. payload is the INPUT fed to the +# next site's inbound R. Continue to the chain terminus. +# PRODUCES (under $OUT/chain//): +# input.msgs — the original START messages +# step-NN..out. — every per-destination route_test output +# step-NN..selected → next input — the file chosen to feed the next step +# commands.sh — the exact generated command sequence +# chain.txt — the resolved v1 chain line(s) +# +# hciroutetest is a Linux engine binary needing a LIVE engine — NOT runnable off +# the server. So chain-walk GENERATES the orchestration + the exact command +# STRINGS + the file-chaining/selection, and (with --dry-run) stubs the engine +# call. Real execution is on Bryan's server. The proven command syntax is mined +# verbatim from the v1/v2 wrappers (route_test_wrapper.py:10/149-156 and +# legacy_workflow_commands.py:1015 .out. naming). set -o pipefail NC_SELF="$0" @@ -50,6 +99,7 @@ LIB_DIR="$(cd "$(dirname "$NC_SELF")" && pwd)" NCP="$LIB_DIR/nc-parse.sh" NCI="$LIB_DIR/nc-inbound.sh" NCM="$LIB_DIR/nc-msgs.sh" +NCPATHS="$LIB_DIR/nc-paths.sh" HL7DIFF="$LIB_DIR/hl7-diff.sh" # v0.7.5: shared CR-safety primitives (coerce_int). The phase loops use @@ -87,9 +137,22 @@ BUNDLE_IN="" # at start, untar a bundle here as the env-A artifacts # run locally because all the artifacts live in $OUT (local). SOURCE_SSH_ALIAS="" TARGET_SSH_ALIAS="" +# v0.8.23: chain-walk mode (single-env, per-env route_test capture along a chain) +CHAIN_WALK=0 +START="" +HCIROOT="" # single-env HCIROOT for chain-walk (alias for --env-a in this mode) +SITE="" # optional explicit site for the START thread +TARGET="" # optional: restrict to the chain whose terminus is this node +ROUTE_TEST_BIN="hciroutetest" while [ $# -gt 0 ]; do case "$1" in + --chain-walk) CHAIN_WALK=1 ;; + --start) shift; START="$1" ;; + --hciroot) shift; HCIROOT="$1" ;; + --site) shift; SITE="$1" ;; + --target) shift; TARGET="$1" ;; + --route-test-bin) shift; ROUTE_TEST_BIN="$1" ;; --scope) shift; SCOPE="$1" ;; --count) shift; COUNT="$1" ;; --env-a) shift; ENV_A="$1" ;; @@ -109,7 +172,7 @@ while [ $# -gt 0 ]; do --bundle-in) shift; BUNDLE_IN="$1" ;; --source-ssh-alias) shift; SOURCE_SSH_ALIAS="$1" ;; --target-ssh-alias) shift; TARGET_SSH_ALIAS="$1" ;; - -h|--help) sed -n '2,55p' "$NC_SELF"; exit 0 ;; + -h|--help) sed -n '2,94p' "$NC_SELF"; exit 0 ;; -*) die "unknown flag: $1" ;; *) die "extra arg: $1" ;; esac @@ -133,6 +196,210 @@ _run_on() { fi } +# ═════════════════════════════════════════════════════════════════════════════ +# CHAIN-WALK MODE (v0.8.23) — single-env route_test capture along an nc-paths +# chain. Orthogonal to the phase pipeline above; runs and exits here. +# ═════════════════════════════════════════════════════════════════════════════ + +# Normalize a node to its bare thread name (strip a leading "site/" if present). +cw_thread_of() { printf '%s' "${1##*/}"; } +# Site of a "site/thread" node (empty if no slash). +cw_site_of() { case "$1" in */*) printf '%s' "${1%%/*}" ;; *) printf '' ;; esac; } + +# Parse ONE v1 chain line into two parallel newline lists on fd: NODES and ARROWS. +# Emits, per line: "N\t" for each node and "A\t" for each arrow +# ("-->" intra-site, "==>" cross-site), in left-to-right order. We tokenize by +# turning the two arrow tokens into sentinels we can split on without regex pain. +cw_parse_chain() { + local line="$1" + # Insert record separators around each arrow token, then read field-by-field. + # Use awk for portable tokenization (no bash regex, Win+Linux safe). + printf '%s\n' "$line" | awk ' + { + n=0 + # split on whitespace; arrows are standalone tokens "-->" or "==>" + cnt=split($0, tok, /[ \t]+/) + for (i=1;i<=cnt;i++) { + t=tok[i] + if (t=="-->") { print "A\t-->" } + else if (t=="==>") { print "A\t==>" } + else if (t!="") { print "N\t" t } + } + }' +} + +# Given the parsed NODES (array) and ARROWS (array, ARROWS[i] is between +# NODES[i] and NODES[i+1]), determine the route_test ENTRY indices: the START +# (index 0) plus every node that immediately FOLLOWS a "==>" arrow (a remote +# inbound after a cross-site hop). Echo the entry indices, one per line. +cw_entry_indices() { + # args: arrow tokens as "$@" (ARROWS[0..n-2]); NODES count = #arrows+1 + local i=0 + echo 0 # START is always a route_test entry + for a in "$@"; do + if [ "$a" = "==>" ]; then + echo $((i+1)) # node right after a cross-site hop is an entry + fi + i=$((i+1)) + done +} + +chain_walk() { + [ -n "$START" ] || die "chain-walk: missing --start " + [ -n "$HCIROOT" ] || die "chain-walk: missing --hciroot (single-env root)" + [ -n "$OUT" ] || die "chain-walk: missing --out DIR" + [ -x "$NCPATHS" ] || die "chain-walk: nc-paths.sh not found/executable at $NCPATHS" + [ -x "$NCM" ] || die "chain-walk: nc-msgs.sh not found/executable at $NCM" + + mkdir -p "$OUT" 2>/dev/null + + # 1. Resolve downstream chains from START. + say "=== CHAIN-WALK: resolve downstream chains from $START ===" + local jsonl + jsonl=$("$NCPATHS" "$START" --downstream --hciroot "$HCIROOT" --format jsonl 2>/dev/null) + [ -n "$jsonl" ] || die "chain-walk: nc-paths returned no downstream chains for $START" + + # Extract the v1 path strings (one per JSON line). Pure-awk pull of "path":"…". + local paths + paths=$(printf '%s\n' "$jsonl" | awk -F'"path":"' 'NF>1 { p=$2; sub(/".*/,"",p); print p }') + [ -n "$paths" ] || die "chain-walk: could not extract path strings from nc-paths jsonl" + + # If --target given, keep only the chain whose terminus matches it (bare-thread + # match, so caller can pass either site/thread or just the thread name). + if [ -n "$TARGET" ]; then + local tgt; tgt=$(cw_thread_of "$TARGET") + paths=$(printf '%s\n' "$paths" | awk -v t="$tgt" ' + { n=split($0,a,/[ \t]+/); last=a[n]; sub(/^.*\//,"",last); if (last==t) print }') + [ -n "$paths" ] || die "chain-walk: --target $TARGET matched no chain terminus from $START" + fi + + local nchains; nchains=$(printf '%s\n' "$paths" | grep -c .) + say "resolved $nchains chain(s) to walk" + + # The START inbound's bare thread name + its site (for the message grab). + local start_thread start_site + start_thread=$(cw_thread_of "$START") + start_site=$(cw_site_of "$START") + [ -n "$start_site" ] && [ -z "$SITE" ] && SITE="$start_site" + + # 2. Generate the message-grab for the START inbound's SMAT (nc-msgs). + # Bryan: grab the N most-recent from the START inbound. We point nc-msgs at + # the START site dir; it locates .smatdb and emits raw (0x1c-sep). + local start_sitedir + if [ -n "$SITE" ]; then + start_sitedir="$HCIROOT/$SITE" + else + start_sitedir=$(find "$HCIROOT" -maxdepth 5 -name "${start_thread}.smatdb" -type f 2>/dev/null \ + | head -1 | sed 's#/exec/processes/.*##') + fi + local grab_cmd="HCISITEDIR='$start_sitedir' '$NCM' '$start_thread' --limit $COUNT --format raw" + + local chain_id=0 chain_line + while IFS= read -r chain_line; do + [ -z "$chain_line" ] && continue + chain_id=$((chain_id+1)) + local cdir; cdir="$OUT/chain/$(printf '%02d' "$chain_id")" + mkdir -p "$cdir" 2>/dev/null + printf '%s\n' "$chain_line" > "$cdir/chain.txt" + + # Parse into NODES[] / ARROWS[]. + local NODES=() ARROWS=() + local rec + while IFS=$'\t' read -r tag val; do + case "$tag" in + N) NODES+=("$val") ;; + A) ARROWS+=("$val") ;; + esac + done < <(cw_parse_chain "$chain_line") + + local nnodes=${#NODES[@]} + say "--- chain $chain_id ($nnodes nodes): $chain_line ---" + + # Determine route_test entry indices. + local entries; entries=$(cw_entry_indices "${ARROWS[@]}") + + # 3. Walk. The first entry's input is the START messages; thereafter each + # step's input is the file SELECTED from the prior route_test. + local cmds="$cdir/commands.sh" + { + printf '#!/usr/bin/env bash\n' + printf '# GENERATED by nc-regression.sh --chain-walk (v0.8.23). Run ON THE ENGINE BOX.\n' + printf '# Chain: %s\n' "$chain_line" + printf '# Source the Cloverleaf profile so HCIROOT/HCISITE + engine libs are set:\n' + printf '# export HCIROOT=%s ; cd "$HCIROOT" ; . ./.profile # or your shop pattern\n' "$HCIROOT" + printf 'set -e\n\n' + printf '# --- step 0: grab the %s most-recent messages from the START inbound SMAT ---\n' "$COUNT" + printf '%s > %s\n\n' "$grab_cmd" "$cdir/input.msgs" + } > "$cmds" + + local step=0 prev_input="$cdir/input.msgs" + local e + while IFS= read -r e; do + [ -z "$e" ] && continue + local node="${NODES[$e]}" + local ethread; ethread=$(cw_thread_of "$node") + local esite; esite=$(cw_site_of "$node") + # The node IMMEDIATELY AFTER E is the DEST suffix to SELECT for the next step. + local nexti=$((e+1)) + local select_suffix="" + if [ "$nexti" -lt "$nnodes" ]; then + select_suffix=$(cw_thread_of "${NODES[$nexti]}") + fi + local out_base="$cdir/step-$(printf '%02d' "$step").${ethread}" + # The exact, mined-verbatim hciroutetest command (route_test_wrapper.py:149-156): + # hciroutetest -a -d -f nl -s + # -f nl → NEWLINE output (no manual len2nl+delete). HCISITE per the entry's site. + local rt_cmd="HCISITE='${esite:-$SITE}' '$ROUTE_TEST_BIN' -a -d -f nl -s '$out_base' '$ethread' '$prev_input'" + + { + printf '# --- step %s: route_test at routing(inbound) thread %s%s ---\n' \ + "$step" "$ethread" "$([ -n "$esite" ] && printf ' (site=%s)' "$esite")" + printf '%s\n' "$rt_cmd" + if [ -n "$select_suffix" ]; then + printf '# select the per-destination output that feeds the NEXT step:\n' + printf '# route_test wrote one file per DEST as %s.out.\n' "$out_base" + printf '# the next node in the chain is %s → select %s.out.%s\n' \ + "$select_suffix" "$out_base" "$select_suffix" + printf 'cp "%s.out.%s" "%s.selected"\n\n' "$out_base" "$select_suffix" "$out_base" + else + printf '# chain terminus — no further selection; %s.out.* are the final outputs.\n\n' "$out_base" + fi + } >> "$cmds" + + # SELF-VERIFY echo (stderr): the generated command + selection for this step. + say " step $step ENTRY=$ethread${esite:+ (site=$esite)}" + say " route_test : $rt_cmd" + if [ -n "$select_suffix" ]; then + say " select : ${out_base##*/}.out.$select_suffix → feeds next step" + else + say " select : (terminus) ${out_base##*/}.out.* are final" + fi + + # --dry-run: stub the engine call so the chaining/selection is exercised + # without hciroutetest. We DO NOT run the real binary here (Bryan's box does). + if [ "$DRY_RUN" = "1" ]; then + : # generation-only; nothing executed + fi + + # The selected file becomes the next step's input (for the NEXT entry). + if [ -n "$select_suffix" ]; then + prev_input="${out_base}.out.${select_suffix}" + fi + step=$((step+1)) + done <<< "$entries" + + say " → generated: $cmds" + done <<< "$paths" + + say "chain-walk done. Per-chain artifacts under: $OUT/chain/" + say "Run each chain/NN/commands.sh ON THE ENGINE BOX (profile sourced) to capture outputs." +} + +if [ "$CHAIN_WALK" = "1" ]; then + chain_walk + exit 0 +fi + [ -n "$OUT" ] || die "missing --out DIR" # When --bundle-in is given, we don't need scope/env-a/etc. — the bundle has them. if [ -z "$BUNDLE_IN" ]; then