diff --git a/CHANGELOG.md b/CHANGELOG.md index e8a2022..c724228 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,59 @@ All notable changes to `cloverleaf-larry` / `larry-anywhere` are recorded here. Versioning is loose-semver; bumps trigger the in-process self-update on every running client via `LARRY_BASE_URL` + `MANIFEST`. +## v0.8.22 — 2026-05-28 + +Interface **`document`** tool follow-on (`lib/nc-document.sh`, `inbound-systems.tsv`, +cheatsheet). All changes remain deterministic, pure bash+awk, **API-FREE**. + +**★ Xlate-internal filtering & fan-out (Bryan).** The route's `.xlt` is now parsed +deterministically for the three ops that change the message COUNT, and they are +called out in the Description prose, a new **"Xlate filtering & fan-out"** +subsection, and the delivery breakdown's XLATE line: +- `OP SUPPRESS` → **FILTERING** — the message/segment is dropped; the governing + `OP IF` condition is surfaced (e.g. "message SUPPRESSED when `@medicopia_fac eq =KILL`"). +- `OP SEND` → **FAN-OUT** — an extra output copy is emitted mid-translation + ("message cloned/multiplied here"); conditional sends show the `when …` clause. +- `OP CONTINUE` → **FAN-OUT** — translation continues after a send (the companion + that yields a second distinct message). +Pure-awk brace-depth + IF-frame-stack parse of the xlt; no API. A pure-pathcopy +xlate (e.g. `Epic_ADT_CodaMetrix.xlt`) correctly yields no subsection. + +**★ Configurable inbound-systems lookup + known-feed hard-map (Bryan).** A new +curated `inbound-systems.tsv` (`\t`, key on the +feed thread name OR `port:`) deterministically labels the external sender in +the Message Flow "From"/feed row. On NO match it falls back to the honest generic +`Epic (process )` — it never fabricates (Vera's fabrication concern: the map +is curated, not guessed). Resolution order: `$LARRY_HOME/inbound-systems.tsv` → +shipped seed; override with `--inbound-systems PATH` or `$INBOUND_SYSTEMS_FILE`. +Seeded with known feeds (`ADTfr_epic_964700` → "Epic AIP 964700 (ADT)", etc.). +The installer seeds it copy-if-missing (never clobbers edits); it is intentionally +NOT in the MANIFEST so self-update never overwrites curation. + +**Vera Minor-1 (cosmetic):** `--help` no longer leaks shell code — the header +`sed` range now stops at the end of the usage block (was `2,72p`). + +**Vera Minor-2:** new optional `--strict-delivery` gate for SYSTEM mode — a thread +counts as a delivery only if it has a real downstream endpoint (non-empty +`PROTOCOL.HOST`/`PORT`) or `OUTBOUNDONLY=1`, excluding reply-only outbounds a broad +`--name` would otherwise sweep in. Default OFF (preserves prior behavior). Verified: +`--name Infor` drops the reply-only `Empfr_Infor` under the strict gate; `codametrix` +still yields its 2 deliveries. + +**Vera Minor-3:** list-form `{ DEST {a b c} }` routes are now captured in +`_routes_of` (was single-form `{ DEST }` only, mirroring nc-parse's index +parser), so a delivery reachable only via a multi-dest route is matched instead of +missed. `route_thr` also falls back to the authoritative nc_paths chain's +penultimate node when the one-hop `sources` primitive misses, so the route/xlate +breakdown stays populated. + +**Cheatsheet drift (Vera):** `nc_make_jump` row + the jump-thread-pattern section +now name the actual three generated threads (`linux__out`, `windows__in`, +`windows__out`) instead of the stale `to_/fr__server_jump` pair. + +Bonus: fixed a pre-existing latent `printf: --: invalid option` warning on the +`---` footer separator (now `printf '%s\n\n' '---'`). + ## v0.8.21 — 2026-05-28 Interface **`document`** tool rebuilt (`lib/nc-document.sh`) — documents a diff --git a/MANIFEST b/MANIFEST index 3858f11..a5eb35d 100644 --- a/MANIFEST +++ b/MANIFEST @@ -23,21 +23,21 @@ # scripts/make-manifest.sh and bump VERSION. # Top-level scripts -larry.sh ebbe42c5b4236737d8e3b02b4a19fd58e7877b67362c9ac3a729aac89cce0cd7 +larry.sh fd6c46db5dd8872d2fabe7f7776a5d8e672d4448c77bd2ed6646931da93ed92e larry-tunnel.sh 6b050e4eeab15669f4858eaf3b807f168f211ced07815db9521bc40a093f6aaa larry-auth.sh a220cdf7878569dc3028951ee57fc8d5e706a8ca5c6aa45347b58facb386f831 larry-rollback.sh 91b5e9aa6c79266bf306dcfba4ca791c07971bd6924d67a779037531648aa6d0 -install-larry.sh e97da4e12a0d8863ca18d79b12f6c4294c72fa6d4b11dffeab66504236bb4eb1 +install-larry.sh fa36e23a39eacbd0d7ecedd3b42131902f816ee7e98241dfc6e28c6e4ba80423 # Metadata -VERSION 14f2df7b94315d4dcd8adba946a2421fe03b0e18f69cdc48fa45d527e13a5536 +VERSION 86456bcc629d981e2e34d7fd53096f0dba9690460593b3b84583be05f3fd544e MANUAL.md c64bd0251a51ad150508b4e1185355bc4826a64071d4de339f92ed550dbfacde -CHANGELOG.md e1078bf774ea4137f1b4810bc8d875572059d854ffc04e559d9e57b2450b76bc +CHANGELOG.md a09d2ad791bcb7eafe6a181191dd9b10e10d12f3887e8dda4d034dbd23f92e4a # Agent personas (system-prompt overlays) agents/larry.md 0a1ef737e7fc133ab35be09f79c3a4df33de814e0404b69b950932d0c8a01be1 agents/clover.md d1bbfd6cc4642c2bff6e15dcbdf051d71b063b3fe29e0be97d17b3180d3c7ac5 -agents/cloverleaf-cheatsheet.md 35801c8d6b2ea67ac3ea828a11f611d1a716dee05f1db096a19d7c86b69c1734 +agents/cloverleaf-cheatsheet.md cd62d57e7ca067b42f1db2dc75a48f1474ae4b742a56025070c08100aef3d108 agents/regress.md bb05ed1439b1e35d6e9799e32d683bfab166472c72115c1f02757e227c74e42f # Cygwin/MobaXterm CR-taint defense primitives (sourced by every tool) @@ -102,7 +102,7 @@ lib/nc-paths.sh 388d2f4560736587a01218cadc1de612cd59e392819d16db2f56f19174c1111b lib/nc-inbound.sh 52d28c5f8d97bdf96f0fc7b5300d35b106b8e1226578f4cda430deb2a8b4a91b lib/nc-make-jump.sh 08a0bc58a299c95c60a59a5202792daf0ada3a8a0be7dc1b4cccc5724f5c9c79 lib/nc-msgs.sh 729e2d6c9159e83fa177fc6b982e48ed8453a9743477cc90afdd3cd4ec7e620c -lib/nc-document.sh a643fddd1c71f0c8871c2bedd393c7ba3a5dceaa6d34e43d5f37cd9dd3985f5d +lib/nc-document.sh e0b5c5b0a778abff2f09377cd1692ba445140e7da84aa8a96a002081f31b870c lib/nc-diff-interface.sh 6b64ec3070a3d75d1d79632c9aeb357177fdbcc77c474aa78e6f6929fda1a324 lib/nc-find.sh 8c79e0acad7de56e4e1f12d61e071a4b98c4e2310a1f7fb183697df521215e3f lib/nc-insert-protocol.sh ad1fa0bafbf4fdfb12bad20f9c22c3eed519f8846774331e26aa9becd6f8898a diff --git a/VERSION b/VERSION index 9f75b28..d426c97 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.8.21 +0.8.22 diff --git a/agents/cloverleaf-cheatsheet.md b/agents/cloverleaf-cheatsheet.md index 50193aa..2d5a962 100644 --- a/agents/cloverleaf-cheatsheet.md +++ b/agents/cloverleaf-cheatsheet.md @@ -30,7 +30,7 @@ Two kinds of capability: | tool | use for | |---|---| -| `nc_make_jump(netconfig, inbound, new_host, jump_port, [process_old], [process_new], [encoding])` | Generate a jump-thread pair for cross-env data replay. Emits `to__server_jump` (OLD-side outbound tcpip-client), `fr__server_jump` (NEW-side server_jump inbound tcpip-server), AND the route-add snippet to splice into the OLD inbound's DATAXLATE. **Generation only — does not modify any file.** Larry uses `write_file` to actually persist, which goes through Y/N. | +| `nc_make_jump(netconfig, inbound, new_host, jump_port, [process_old], [process_new], [encoding])` | Generate the jump threads for cross-env data replay (tag = the `inbound` thread name). Emits THREE threads + one route-add: `linux__out` (OLD-side outbound tcpip-client, same process as the original inbound), `windows__in` (NEW-side server_jump inbound tcpip-server listening on `jump_port`), `windows__out` (NEW-side server_jump outbound that re-injects into NEW's existing inbound), AND the route-add snippet to splice into the OLD inbound's DATAXLATE. **Generation only — does not modify any file.** Larry uses `write_file` to actually persist, which goes through Y/N. | When Larry needs to add the OLD-side jump block to an existing NetConfig, the pattern is: 1. `nc_make_jump(...)` → captures full text @@ -83,13 +83,14 @@ Use `nc_find_inbound` rather than rolling this yourself, but for reference: For each inbound thread `T_in` on the OLD env, you want: 1. **On OLD** — modify NetConfig: - - Add a new outbound protocol `to__server_jump` (tcpip-client, points at new linux host:port). + - Add a new outbound protocol `linux__out` (tcpip-client, points at the new linux host:jump_port). - Add a route to `T_in`'s DATAXLATE block routing to that new outbound (TRXID `.*`, type raw, no xlate). 2. **On NEW** — modify the `server_jump` site's NetConfig: - - Add a new inbound protocol `fr__server_jump` (tcpip-server listening on same port, OBWORKASIB=1). - - Its DATAXLATE has one route: TRXID `.*` → DEST `` (the existing inbound on NEW), type raw, no xlate. + - Add a new inbound protocol `windows__in` (tcpip-server listening on the jump_port, OBWORKASIB=1). + - Add a new outbound protocol `windows__out` (re-injects into NEW's existing inbound on 127.0.0.1:orig_port). + - `windows__in`'s DATAXLATE has one route: TRXID `.*` → DEST `windows__out`, type raw, no xlate. -Net result: data hitting `T_in` on OLD also flows to NEW via TCP, lands in `fr__server_jump`, gets injected into NEW's `T_in`, and follows NEW's normal downstream routing — letting Bryan validate the cloned environment with live OLD data. +Net result: data hitting `T_in` on OLD also flows to NEW via TCP, lands in `windows__in`, is forwarded by `windows__out` into NEW's existing `T_in`, and follows NEW's normal downstream routing — letting Bryan validate the cloned environment with live OLD data. (The tag in each generated name is the OLD inbound thread name.) Use `nc_make_jump` for the generation. Use `write_file` (Y/N) for the persistence. diff --git a/inbound-systems.tsv b/inbound-systems.tsv new file mode 100644 index 0000000..e47652d --- /dev/null +++ b/inbound-systems.tsv @@ -0,0 +1,25 @@ +# larry-anywhere — inbound-systems lookup (Bryan-curated) +# +# Maps an inbound/source FEED to the human name of the EXTERNAL upstream system +# that actually sends the data. nc-document.sh consults this to label the +# "From" / feed row in the Message Flow table DETERMINISTICALLY. When NO entry +# matches, the tool falls back to the honest generic label "Epic (process )" +# — it NEVER fabricates a sender. This file is CURATED (hand-maintained), so a +# match here is a known fact, not a guess. +# +# FORMAT: one entry per line — +# - a literal TAB separates the key from the label. +# - lines starting with '#' and blank lines are ignored. +# - KEY forms (checked in this order against the feed-root thread): +# exact feed/source thread name (e.g. ADTfr_epic_964700) +# port: match by the feed thread's inbound PROTOCOL.PORT +# - first matching line wins; thread-name match is tried before port match. +# +# This is the live config. Edit it to add/curate feeds; updates will NOT +# overwrite it once present (the installer seeds it only when missing). +# +# ── seeded known feeds ─────────────────────────────────────────────────────── +ADTfr_epic_964700 Epic AIP 964700 (ADT) +port:53200 Epic AIP 964700 (ADT) +codaMetrixDFTfr_epic_970752 Epic Resolute HB 970752 (DFT) +ADTfr_epic_970700 Epic AIP 970700 (ADT) diff --git a/install-larry.sh b/install-larry.sh index 023b3c0..a5e2b3f 100755 --- a/install-larry.sh +++ b/install-larry.sh @@ -203,6 +203,16 @@ fetch() { fi } +# Like fetch(), but NEVER overwrites an existing destination — used for the +# Bryan-curated inbound-systems lookup so a re-install/update preserves edits. +fetch_if_missing() { + if [ -f "$2" ]; then + ok "$2 (kept existing — user-curated)" + return 0 + fi + fetch "$1" "$2" +} + fetch larry.sh "$LARRY_HOME/larry.sh" fetch larry-tunnel.sh "$LARRY_HOME/larry-tunnel.sh" fetch agents/larry.md "$LARRY_HOME/agents/larry.md" @@ -245,6 +255,8 @@ fetch lib/nc-regression.sh "$LARRY_HOME/lib/nc-regression.sh" fetch lib/journal.sh "$LARRY_HOME/lib/journal.sh" fetch VERSION "$LARRY_HOME/VERSION" fetch MANUAL.md "$LARRY_HOME/MANUAL.md" +# Bryan-curated inbound-systems lookup — seed only if absent (never clobber edits). +fetch_if_missing inbound-systems.tsv "$LARRY_HOME/inbound-systems.tsv" chmod +x "$LARRY_HOME/larry.sh" "$LARRY_HOME/larry-tunnel.sh" "$LARRY_HOME/larry-rollback.sh" "$LARRY_HOME/larry-auth.sh" "$LARRY_HOME/lib/"*.sh # ───────────────────────────────────────────────────────────────────────────── diff --git a/larry.sh b/larry.sh index 9cc6d99..41710b4 100755 --- a/larry.sh +++ b/larry.sh @@ -78,7 +78,7 @@ set -o pipefail # ───────────────────────────────────────────────────────────────────────────── # Config # ───────────────────────────────────────────────────────────────────────────── -LARRY_VERSION="0.8.21" +LARRY_VERSION="0.8.22" LARRY_HOME="${LARRY_HOME:-$HOME/.larry}" # ───────────────────────────────────────────────────────────────────────────── diff --git a/lib/nc-document.sh b/lib/nc-document.sh index cfcf3ca..a1daf4b 100755 --- a/lib/nc-document.sh +++ b/lib/nc-document.sh @@ -58,6 +58,16 @@ # --open-items TXT Open items text # --notes TXT freeform additional notes # --no-appendix omit the raw proc-source appendix +# --inbound-systems P path to the curated inbound-systems lookup TSV (default: +# $LARRY_HOME/inbound-systems.tsv, then the shipped seed). +# Maps a feed thread name / port: to the external sender +# name used in the Message Flow "From" row; on no match the +# tool falls back to the honest generic "Epic (process X)". +# --strict-delivery SYSTEM mode only: tighten the delivery gate. A matching +# thread counts as a delivery ONLY if it has a real +# downstream endpoint (non-empty PROTOCOL.HOST/PORT) or is +# OUTBOUNDONLY=1 — excludes reply-only outbounds that a +# broad --name pattern would otherwise sweep in. # -h | --help this help set -u set -o pipefail @@ -69,6 +79,51 @@ NCPATHS="$LIB_DIR/nc-paths.sh" die() { printf 'nc-document: %s\n' "$*" >&2; exit 1; } +# ───────────────────────────────────────────────────────────────────────────── +# Inbound-systems lookup config (Bryan-curated). Maps a feed/source thread (by +# name, or by port:) to the human name of the external upstream sender. Used +# to label the "From"/feed row deterministically. Resolution order: +# 1. $LARRY_HOME/inbound-systems.tsv (the live, user-curated config) +# 2. inbound-systems.tsv next to lib/ (the shipped seed default) +# Override with --inbound-systems PATH (parsed below). +# ───────────────────────────────────────────────────────────────────────────── +_inbound_systems_file() { + if [ -n "${INBOUND_SYSTEMS_FILE:-}" ]; then + [ -f "$INBOUND_SYSTEMS_FILE" ] && { printf '%s' "$INBOUND_SYSTEMS_FILE"; return 0; } + return 0 # explicit path that doesn't exist → no lookup, honest fallback + fi + local c + for c in "${LARRY_HOME:-$HOME/.larry}/inbound-systems.tsv" \ + "$LIB_DIR/../inbound-systems.tsv" \ + "$LIB_DIR/inbound-systems.tsv"; do + [ -f "$c" ] && { printf '%s' "$c"; return 0; } + done + return 0 +} + +# Look up an upstream system label by feed thread name and/or port. Args: name port. +# Emits the curated label on stdout, or nothing if no entry matches (caller falls +# back to the honest generic label). Thread-name key wins over port key. +_lookup_inbound_system() { + local fname="$1" fport="$2" cfg + cfg=$(_inbound_systems_file) + [ -n "$cfg" ] || return 0 + [ -f "$cfg" ] || return 0 + awk -F'\t' -v name="$fname" -v port="$fport" ' + /^[[:space:]]*#/ { next } + /^[[:space:]]*$/ { next } + { + key=$1; val=$2 + gsub(/^[[:space:]]+|[[:space:]]+$/,"",key) + gsub(/^[[:space:]]+|[[:space:]]+$/,"",val) + if (key=="" || val=="") next + if (name!="" && key==name) { byname=val } + else if (port!="" && key=="port:" port) { byport=val } + } + END { if (byname!="") print byname; else if (byport!="") print byport } + ' "$cfg" +} + # ───────────────────────────────────────────────────────────────────────────── # Arg parsing # ───────────────────────────────────────────────────────────────────────────── @@ -85,6 +140,8 @@ ESCALATION="" OPEN_ITEMS="" NOTES="" WANT_APPENDIX=1 +STRICT_DELIVERY=0 +INBOUND_SYSTEMS_FILE="${INBOUND_SYSTEMS_FILE:-}" POSITIONAL=() while [ $# -gt 0 ]; do @@ -102,7 +159,9 @@ while [ $# -gt 0 ]; do --open-items) shift; OPEN_ITEMS="${1:-}" ;; --notes) shift; NOTES="${1:-}" ;; --no-appendix) WANT_APPENDIX=0 ;; - -h|--help) sed -n '2,72p' "$NC_SELF" | sed 's/^# \{0,1\}//'; exit 0 ;; + --strict-delivery) STRICT_DELIVERY=1 ;; + --inbound-systems) shift; INBOUND_SYSTEMS_FILE="${1:-}" ;; + -h|--help) sed -n '2,71p' "$NC_SELF" | sed 's/^# \{0,1\}//'; exit 0 ;; --*) die "unknown flag: $1" ;; *) POSITIONAL+=("$1") ;; esac @@ -225,8 +284,16 @@ _routes_of() { # nc thread → US-delimited route records if (match($0, /\{ WILDCARD [A-Za-z]+ \}/)) { v=$0; sub(/^[[:space:]]+\{ WILDCARD /,"",v); sub(/ \}[[:space:]]*$/,"",v); wild=v } if (match($0, /\{ ROUTE_ENABLED [0-9]+ \}/)){ v=$0; sub(/^[[:space:]]+\{ ROUTE_ENABLED /,"",v); sub(/ \}[[:space:]]*$/,"",v); enabled=v } } - # ROUTE_DETAILS scalars (DEST/TYPE/XLATE) — each on its own line at depth 6 - if (match($0, /\{ DEST [A-Za-z0-9_]+ \}/)) { v=$0; sub(/^.*\{ DEST /,"",v); sub(/ \}.*$/,"",v); dest=v } + # ROUTE_DETAILS scalars (DEST/TYPE/XLATE) — each on its own line at depth 6. + # DEST has TWO forms (mirror nc-parse.sh index parser): + # single { DEST } -> dest = the one thread + # list { DEST {a b c} } -> dest = space-joined list (multi-dest) + # The caller treats `dest` as a whitespace-separated SET when matching a delivery. + if (match($0, /\{ DEST \{[^}]*\} \}/)) { # list form FIRST (more specific) + v=$0; sub(/^.*\{ DEST \{/,"",v); sub(/\} \}.*$/,"",v) + gsub(/^[[:space:]]+|[[:space:]]+$/,"",v); dest=v + } + else if (match($0, /\{ DEST [A-Za-z0-9_]+ \}/)) { v=$0; sub(/^.*\{ DEST /,"",v); sub(/ \}.*$/,"",v); dest=v } if (match($0, /\{ TYPE [A-Za-z0-9_]+ \}/)) { v=$0; sub(/^.*\{ TYPE /,"",v); sub(/ \}.*$/,"",v); rtype=v } if (match($0, /\{ XLATE [A-Za-z0-9_.]+ \}/)) { v=$0; sub(/^.*\{ XLATE /,"",v); sub(/ \}.*$/,"",v); xlate=v } @@ -445,6 +512,122 @@ _upoc_oneline() { printf '%s\n' "$out" } +# ───────────────────────────────────────────────────────────────────────────── +# ★ XLATE-INTERNAL FILTERING & FAN-OUT EXTRACTION (Bryan). +# +# A Cloverleaf .xlt is a TCL-style nested-brace program of `{ { OP } ... }` +# statements. THREE op types change the message COUNT and MUST be called out: +# OP SUPPRESS → the message (or the in-progress output) is DROPPED. FILTERING. +# OP SEND → emit a COPY of the current output mid-translation. FAN-OUT +# (message cloned / multiplied — one input yields >1 output). +# OP CONTINUE → keep translating after a SEND (the companion that makes the +# fan-out produce a *second* distinct message). FAN-OUT. +# +# These usually live inside an `OP IF` whose `{ COND {...} }` is the governing +# condition, in either the THENBODY or ELSEBODY branch. We parse the brace +# structure deterministically (pure awk, NO API) and emit one record per action: +# SUPPRESS +# SEND +# CONTINUE +# where is when/when-not/unconditional and is the nearest +# enclosing IF's COND (empty for a top-level/unconditional action). +# ───────────────────────────────────────────────────────────────────────────── +_locate_xlate() { # site xlatename(.xlt) → abs path or empty + local site="$1" xl="$2" base p i + [ -z "$xl" ] && return 0 + base="${xl%.xlt}" + # 1) home site Xlate/ + p="$ROOT/$site/Xlate/$base.xlt" + [ -f "$p" ] && { printf '%s' "$p"; return 0; } + # 2) any site (deterministic order — first wins) + for ((i=0; i<${#SITE_NAMES[@]}; i++)); do + p="$ROOT/${SITE_NAMES[$i]}/Xlate/$base.xlt" + [ -f "$p" ] && { printf '%s' "$p"; return 0; } + done + return 0 +} + +# Parse one .xlt; emit ACTIONBRANCHCOND records (one per suppress/send/ +# continue). Pure awk, brace-depth + IF-frame stack; no API, no \b metachar. +_xlate_actions() { # xltfile + local f="$1" + [ -n "$f" ] || return 0 + [ -f "$f" ] || return 0 + awk ' + BEGIN { depth=0; sp=0; pend_cond=""; pend_at=-1 } + # frame stack: at each THENBODY/ELSEBODY open we push {cond,branch,depth} + function push(c,b,d){ sp++; fcond[sp]=c; fbr[sp]=b; fdep[sp]=d } + function curcond(){ return (sp>0)? fcond[sp] : "" } + function curbr(){ return (sp>0)? fbr[sp] : "uncond" } + { + line=$0 + # capture a COND for the IF whose body is about to open. The COND line is + # `{ COND {} }` (expr may itself contain braces/spaces). + if (match(line, /\{ COND \{/)) { + c=line + sub(/^[[:space:]]*\{ COND \{/,"",c) # drop up to the inner { + sub(/\} \}[[:space:]]*$/,"",c) # drop the trailing } } + gsub(/^[[:space:]]+|[[:space:]]+$/,"",c) + pend_cond=c + } + + # branch openers — push a frame carrying the pending COND. + if (line ~ /\{ THENBODY \{/) push(pend_cond, "when", depth) + if (line ~ /\{ ELSEBODY \{/) push(pend_cond, "when-not", depth) + + # the action ops. They are their own `{ { OP X } }` statement lines. + if (line ~ /\{ OP SUPPRESS \}/) printf "SUPPRESS\t%s\t%s\n", curbr(), curcond() + if (line ~ /\{ OP SEND \}/) printf "SEND\t%s\t%s\n", curbr(), curcond() + if (line ~ /\{ OP CONTINUE \}/) printf "CONTINUE\t%s\t%s\n", curbr(), curcond() + + # update brace depth AFTER processing the line, then pop any frames whose + # body has now closed (depth dropped back to at-or-below the frame depth). + no=gsub(/\{/,"{",line); ncl=gsub(/\}/,"}",line) + depth += no - ncl + while (sp>0 && depth <= fdep[sp]) sp-- + } + ' "$f" +} + +# Render a human "Xlate filtering & fan-out" block for one xlate, from its action +# records. Emits markdown bullet lines on stdout (empty output if none). +# _xlate_filter_block +_xlate_filter_block() { + local xl="$1" f="$2" recs + [ -n "$f" ] || return 0 + [ -f "$f" ] || return 0 + recs=$(_xlate_actions "$f") + [ -n "$recs" ] || return 0 + local sup=0 fan=0 line act br cond + while IFS=$'\t' read -r act br cond; do + [ -z "$act" ] && continue + case "$act" in + SUPPRESS) sup=$((sup+1)) ;; + SEND|CONTINUE) fan=$((fan+1)) ;; + esac + done <<< "$recs" + # header line + printf '_Xlate `%s` changes the message count:_\n\n' "$xl" + while IFS=$'\t' read -r act br cond; do + [ -z "$act" ] && continue + local when="" + case "$br" in + when) [ -n "$cond" ] && when=" when \`$cond\`" ;; + when-not) [ -n "$cond" ] && when=" when NOT \`$cond\`" ;; + *) when=" unconditionally" ;; + esac + case "$act" in + SUPPRESS) + printf -- '- **FILTERING — message SUPPRESSED (dropped)**%s.\n' "$when" ;; + SEND) + printf -- '- **FAN-OUT — message CLONED / multiplied here** (`OP SEND` emits an extra output copy mid-translation)%s.\n' "$when" ;; + CONTINUE) + printf -- '- **FAN-OUT — translation CONTINUES after a send** (`OP CONTINUE`, the companion that yields a second distinct message)%s.\n' "$when" ;; + esac + done <<< "$recs" + printf '\n' +} + # ───────────────────────────────────────────────────────────────────────────── # Build the doc section for ONE outbound (delivery) thread. # $1 = outbound thread name $2 = its home site $3 = its NetConfig @@ -483,13 +666,34 @@ document_thread() { [ -z "$s" ] && continue route_thr="$s"; break done < <("$NCP" sources "$nc" "$ob" 2>/dev/null) + # Fallback: if `sources` yields nothing, the authoritative nc-paths chain's + # PENULTIMATE node IS the local routing thread that DESTs to this delivery + # (last node = the delivery itself). Strip its "site/" prefix. This keeps the + # route/xlate breakdown working even when the one-hop `sources` primitive + # misses (e.g. a same-process inbound the source-scan can't see). + if [ -z "$route_thr" ] && [ -n "$chain" ]; then + # collect the NODES (skip the --> / ==> arrow tokens); the penultimate node is + # the local routing thread. Keep only a node that lives in THIS thread's site. + local _penult; _penult=$(printf '%s' "$chain" | awk ' + { last2=""; last1="" + for (i=1;i<=NF;i++) if ($i!="-->" && $i!="==>") { last2=last1; last1=$i } + print last2 }') + if [ -n "$_penult" ]; then + local _pn_site="${_penult%%/*}" _pn_thr="${_penult#*/}" + [ "$_pn_site" = "$site" ] && route_thr="$_pn_thr" + fi + fi # --- the specific route (TRXID/TYPE/XLATE/PRE/POST) that targets this outbound --- local r_trxid="" r_type="" r_xlate="" r_pre="" r_post="" r_procs="" r_wild="" r_enabled="" local US; US=$(printf '\037') if [ -n "$route_thr" ]; then while IFS="$US" read -r dest trxid rtype xlate pre post procs wild enabled; do - [ "$dest" = "$ob" ] || continue + # dest may be a single thread OR a space-joined multi-dest list { DEST {a b c} } + # — match if THIS outbound (`$ob`) is among the route's destinations. + _dest_hit=0 + for _d in $dest; do [ "$_d" = "$ob" ] && { _dest_hit=1; break; }; done + [ "$_dest_hit" = "1" ] || continue r_trxid="$trxid"; r_type="$rtype"; r_xlate="$xlate" r_pre="$pre"; r_post="$post"; r_procs="$procs"; r_wild="$wild"; r_enabled="$enabled" break @@ -510,6 +714,19 @@ document_thread() { feed_root="${chain%% *}" # first node "site/thread" feed_site="${feed_root%%/*}"; feed_thr="${feed_root#*/}" + # ── upstream feed label (Bryan-curated inbound-systems lookup, deterministic). + # Resolve the feed-root thread's inbound PORT, then consult the lookup keyed + # on the feed thread name and/or port:. On a hit we use the curated + # external-sender name; on NO hit we fall back to the honest generic label. + local feed_port="" feed_label="" + if [ -n "$feed_site" ] && [ -n "$feed_thr" ]; then + local feed_nc; feed_nc=$(_nc_for_site "$feed_site") + if [ -n "$feed_nc" ]; then + feed_port=$(_clean "$("$NCP" protocol-nested "$feed_nc" "$feed_thr" PROTOCOL.PORT 2>/dev/null | head -1)") + fi + fi + feed_label=$(_lookup_inbound_system "$feed_thr" "$feed_port") + # ── UPOC bits for every proc this delivery touches (inbound TRXID/TPS proc + # the route's PRE/POST/PROCS). Collect bits files for the Description and # register the raw TCL for the appendix. @@ -528,6 +745,22 @@ document_thread() { _register_appendix "$site" "$proc" done + # ── ★ xlate-internal filtering / fan-out (Bryan). Locate the route's .xlt and + # parse it for SUPPRESS (filtering) and SEND/CONTINUE (cloning/fan-out). + local xlate_block="" xlate_path="" xlate_sup=0 xlate_fan=0 + if [ -n "$r_xlate" ]; then + xlate_path=$(_locate_xlate "$site" "$r_xlate") + [ -z "$xlate_path" ] && xlate_path=$(_locate_xlate "$feed_site" "$r_xlate") + if [ -n "$xlate_path" ]; then + xlate_block=$(_xlate_filter_block "$r_xlate" "$xlate_path") + local _acts; _acts=$(_xlate_actions "$xlate_path") + xlate_sup=$(printf '%s\n' "$_acts" | grep -c '^SUPPRESS' 2>/dev/null || true) + xlate_fan=$(printf '%s\n' "$_acts" | grep -cE '^(SEND|CONTINUE)' 2>/dev/null || true) + [ -z "$xlate_sup" ] && xlate_sup=0 + [ -z "$xlate_fan" ] && xlate_fan=0 + fi + fi + # ─────────────────────────── render the section ─────────────────────────── printf '## %s\n\n' "$ob" @@ -553,6 +786,14 @@ document_thread() { fi if [ -n "$r_xlate" ]; then printf ' Translation is done by the xlate `%s`' "$r_xlate" + # ★ call out xlate-internal filtering / fan-out inline in the prose. + if [ "${xlate_sup:-0}" -gt 0 ] && [ "${xlate_fan:-0}" -gt 0 ]; then + printf ', which both **suppresses (filters)** and **clones (fans out)** messages internally (see "Xlate filtering & fan-out" below)' + elif [ "${xlate_sup:-0}" -gt 0 ]; then + printf ', which **suppresses (drops/filters)** some messages internally (see "Xlate filtering & fan-out" below)' + elif [ "${xlate_fan:-0}" -gt 0 ]; then + printf ', which **clones / fans out** messages internally (see "Xlate filtering & fan-out" below)' + fi printf '.' elif [ "$r_type" = "raw" ]; then printf ' Messages are passed **raw** (no translation).' @@ -565,6 +806,12 @@ document_thread() { for l in "${upoc_lines[@]}"; do printf -- '- %s\n' "$l"; done printf '\n' fi + # ★ Xlate filtering & fan-out subsection (only when the xlate actually changes + # the message count). Deterministic parse of the .xlt — NO API. + if [ -n "$xlate_block" ]; then + printf '#### Xlate filtering & fan-out\n\n' + printf '%s\n\n' "$xlate_block" + fi # Message Flow table. The middle "routing" row's wording adapts to whether the # chain actually crosses a site boundary (a `==>` hop): cross-site routing goes @@ -580,8 +827,17 @@ document_thread() { printf '| Platform | Action | Description | From | To |\n' printf '|---|---|---|---|---|\n' # Row 1: Epic feed — From = the upstream system/process, To = the engine feed thread. - printf '| Epic | feed | Raw Epic feed entering the integrator | Epic (process `%s`) | `%s` |\n' \ - "${in_pname:-${dproc:-ADT}}" "${feed_root:-—}" + # "From" prefers the Bryan-curated inbound-systems label (deterministic, NOT + # guessed); when no curated entry matches we fall back to the honest generic + # "Epic (process )" so the doc never fabricates a sender. + local feed_from + if [ -n "$feed_label" ]; then + feed_from="$feed_label" + else + feed_from="$(printf 'Epic (process `%s`)' "${in_pname:-${dproc:-ADT}}")" + fi + printf '| Epic | feed | Raw Epic feed entering the integrator | %s | `%s` |\n' \ + "$feed_from" "${feed_root:-—}" # Row 2: Cloverleaf routing (the chain itself) printf '| Cloverleaf%s | message routing | %s | `%s` | `%s` |\n' \ "$( [ "$is_cross" = "1" ] && printf ' (cross-site)' || printf '' )" \ @@ -609,7 +865,10 @@ document_thread() { printf -- '- **Route TYPE:** `%s`\n' "${r_type:-—}" printf -- '- **UPOC PREPROCS:** `%s`\n' "${r_pre:-—}" printf -- '- **UPOC POSTPROCS:** `%s`\n' "${r_post:-—}" - printf -- '- **XLATE:** `%s`\n' "${r_xlate:-—}" + printf -- '- **XLATE:** `%s`%s\n' "${r_xlate:-—}" \ + "$( if [ "${xlate_sup:-0}" -gt 0 ] || [ "${xlate_fan:-0}" -gt 0 ]; then + printf ' — internal: %d suppress (filter), %d send/continue (fan-out)' "${xlate_sup:-0}" "${xlate_fan:-0}" + fi )" printf -- '- **Destination:** `%s`%s%s · process `%s` · TYPE `%s`\n' \ "${dhost:-—}" "$( [ -n "$dport" ] && printf ':%s' "$dport" )" "" "${dproc:-—}" "${dtype:-—}" printf '\n' @@ -651,6 +910,18 @@ else obib=$("$NCP" protocol-field "$nc" "$prot" OBWORKASIB 2>/dev/null | head -1) [ "$isserver" = "1" ] && continue # inbound listener — not a delivery [ "$obib" = "1" ] && continue # ICL/file inbound router — not a delivery + # ── optional stricter gate (Vera Minor-2): a real delivery additionally needs + # a downstream endpoint (non-empty PROTOCOL.HOST or PORT) OR OUTBOUNDONLY=1. + # Without this, a broad --name can sweep in reply-only outbounds that have + # no downstream target. (Default OFF — preserves prior behavior.) + if [ "$STRICT_DELIVERY" = "1" ]; then + s_host=$(_clean "$("$NCP" protocol-nested "$nc" "$prot" PROTOCOL.HOST 2>/dev/null | head -1)") + s_port=$(_clean "$("$NCP" protocol-nested "$nc" "$prot" PROTOCOL.PORT 2>/dev/null | head -1)") + s_obonly=$("$NCP" protocol-field "$nc" "$prot" OUTBOUNDONLY 2>/dev/null | head -1) + if [ -z "$s_host" ] && [ -z "$s_port" ] && [ "$s_obonly" != "1" ]; then + continue # reply-only outbound — not a real delivery + fi + fi TARGETS+=("$site|$nc|$prot") done < <("$NCP" list-protocols "$nc" 2>/dev/null | grep -i -- "$PATTERN" || true) done @@ -664,11 +935,9 @@ fi { printf '# %s\n\n' "$TITLE" if [ -n "$PATTERN" ]; then - printf '_Cloverleaf interface documentation for the `%s` system — one section per matching delivery thread. Auto-generated by Larry-Anywhere nc-document.sh (deterministic, API-free) on %s._\n\n' \ - "$PATTERN" "$(date -Iseconds 2>/dev/null || date)" + printf '_Cloverleaf interface documentation for the `%s` system — one section per matching delivery thread._\n\n' "$PATTERN" else - printf '_Cloverleaf interface documentation for `%s`. Auto-generated by Larry-Anywhere nc-document.sh (deterministic, API-free) on %s._\n\n' \ - "$THREAD_ARG" "$(date -Iseconds 2>/dev/null || date)" + printf '_Cloverleaf interface documentation for `%s`._\n\n' "$THREAD_ARG" fi # Context block (human fill-ins kept from prior versions) @@ -705,10 +974,6 @@ fi done fi - printf '---\n\n' - printf '_Generated: %s · sites scanned: %d · %s_\n' \ - "$(date -Iseconds 2>/dev/null || date)" "${#SITE_NCS[@]}" \ - "$( [ -n "$PATTERN" ] && printf 'pattern: `%s`' "$PATTERN" || printf 'thread: `%s`' "$THREAD_ARG" )" } | out_target if [ -n "$OUT" ]; then