Bryan Johnson
9233d23cda
New `larry tools cl-format-migrate <dir>`: fixes the "' before prologue start" errors on Windows-authored tables/xlates whose CRLF line endings choke the Linux Cloverleaf parser (and Cloverleaf's own hcitblconvert/hcixltconvert). Root cause field-confirmed by Bryan: `file` shows broken=CRLF/working=LF, both version:6.0 (no format bump) — the stray CR is read as a bogus char before the prologue keyword. - Default: strip ONLY trailing CR (sed 's/\r$//'), backup-first via journal.sh, idempotent (SKIP already-LF), per-file verify (no trailing CRLF remains + prologue block intact), summary report. Bryan proved a CR-stripped table loads clean. - Opt-in --run-native-convert: chains hcitblconvert/hcixltconvert after a successful strip (Cloverleaf's real format bump), resolving the bare name against <site>/Tables|Xlate then $HCIROOT/Tables|Xlate, one name per call; separate journal checkpoint so a native-step failure auto-rolls-back to the CR-stripped-safe state. Orphan file -> NATIVE_SKIPPED (never guesses a site). Default stays strip-only pending Bryan's hcitblconvert-post-strip confirm. - Unified tbl+xlt (both choke on CRLF identically). - QoL REPL: cdr->$HCIROOT, cds->$HCIROOT/$HCISITE, cdd->$HCIROOT/$HCISITE/data; bare site name -> setsite+cd, first-match-wins ordering so it never shadows an existing command. Vera CODE-PASS (re-gate after fixing a Major verify defect: _count_cr matched any CR byte -> false FAILED_VERIFY loop on files with a legitimate mid-line CR; now _count_trailing_cr on \r$, matching the strip). Own-fixture verified: convert/preserve-mid-line-CR/converge-to-SKIP, full regression + --run-native-convert mocked suite, bash -n 3.2.57, manifest --check clean. Live-acceptance: Bryan (Gundersen box; dry-run -> single-file --confirm pilot -> batch).
494 lines
23 KiB
Bash
Executable File
494 lines
23 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# cl-format-migrate.sh — CRLF->LF format-migration for Cloverleaf .tbl AND
|
|
# .xlt files, plus an OPT-IN chain to Cloverleaf's own native converters
|
|
# (hcitblconvert / hcixltconvert). Per the 2026-07-23 design doc
|
|
# (Deliverables/2026-07-23-cloverleaf-table-xlate-format-migration-design.md)
|
|
# and TWO rounds of Bryan-direct live evidence from the same session:
|
|
#
|
|
# ROUND 1 (the original trigger): `file` on a broken vs. a working .tbl
|
|
# showed BOTH are prologue `version: 6.0` (identical) — the ONLY
|
|
# difference is line endings:
|
|
# getwell_skipfacility_pv1_034.tbl -> ASCII text, WITH CRLF (broken)
|
|
# getwellntw_sndingevent_msh04.tbl -> ASCII text (working)
|
|
#
|
|
# ROUND 2 (Gundersen engine box, same session): Bryan found BOTH native
|
|
# Cloverleaf converters exist in $HCIROOT/bin — `hcitblconvert` (tables)
|
|
# and `hcixltconvert` (xlates) — and BOTH FAIL on the CRLF files with the
|
|
# IDENTICAL error:
|
|
# $ hcitblconvert getwell_skipfacility_pv1_034.tbl
|
|
# getwell_skipfacility_pv1_034.tbl
|
|
# ' before prologue start
|
|
# $ hcixltconvert
|
|
# ' before prologue start
|
|
# Please correct the error(s) and re-run.
|
|
# That leading `'` is the stray CR being read as a bogus char before the
|
|
# `prologue` keyword — Cloverleaf's OWN native converters choke on CRLF
|
|
# for the SAME root-cause reason our own parser does. Also learned:
|
|
# - No -h/--help flag on either binary; args are TABLE/XLATE NAMES
|
|
# (e.g. `getwell_skipfacility_pv1_034.tbl`, WITH extension), not full
|
|
# paths. Resolution is against the site's Tables/Xlate dir:
|
|
# $HCIROOT/$HCISITE/Tables/<name> then $HCIROOT/Tables/<name> (tables;
|
|
# nc-table.sh's existing lookup order) — Xlate mirrors this with
|
|
# Xlate/xlate (nc-xlate.sh's existing lookup order).
|
|
# - A bare `*` glob pulls in non-table entries and errors — so this
|
|
# tool NEVER globs at the native binary; it iterates its OWN
|
|
# discovered file list and invokes the binary once per name.
|
|
#
|
|
# DESIGN (unified, per Larry's 2026-07-23 follow-up dispatch):
|
|
# Step 1 (both .tbl and .xlt, ALWAYS the core/default step): strip ONLY
|
|
# trailing carriage returns (`sed 's/\r$//'` / dos2unix semantics —
|
|
# never `tr -d '\r'`). Journaled (backup+diff+atomic write), verified,
|
|
# idempotent (already-pure-LF files are SKIPPED). This alone is
|
|
# PROVEN (Round 1) to make a table load.
|
|
# Step 2 (OPT-IN via --run-native-convert, default OFF): chain the
|
|
# matching native binary (hcitblconvert for .tbl, hcixltconvert for
|
|
# .xlt) on top of the CR-stripped file, so Cloverleaf's OWN converter
|
|
# does whatever real format-level work it does (metadata/version
|
|
# re-serialization) — "no format-guessing by us." Only ever run on a
|
|
# file THIS invocation just CR-stripped (never on an already-clean
|
|
# SKIPPED file — untouched files stay untouched). Kept opt-in until
|
|
# Bryan confirms hcitblconvert succeeds cleanly on a CR-stripped table
|
|
# (per Larry's explicit instruction — do NOT make this the default
|
|
# yet). A pre-native checkpoint is journaled separately from the CR-
|
|
# strip entry, so a native-convert FAILURE auto-rolls-back to the
|
|
# already-verified CR-stripped-only state (never leaves a file in an
|
|
# uncertain post-native state) while the CR-strip's own success is
|
|
# preserved and reported independently.
|
|
#
|
|
# Usage:
|
|
# cl-format-migrate.sh [<dir>] [--type tbl|xlt|all] [--dry-run]
|
|
# [--confirm yes] [--out <report-dir>] [--deep-verify]
|
|
# [--run-native-convert]
|
|
#
|
|
# <dir> directory to scan RECURSIVELY for *.tbl / *.xlt.
|
|
# DEFAULTS TO THE CURRENT DIRECTORY (.) if omitted —
|
|
# same cwd-default convention as v0.9.11 hl7-diff-set.
|
|
# --type T tbl (default) | xlt | all
|
|
# --dry-run classify + report what WOULD happen; write NOTHING
|
|
# --confirm yes required to actually write on a REAL (non-dry-run)
|
|
# conversion. Without it: plan-only refusal per file,
|
|
# consistent with nc-set-field.sh / nc-provision-jumps.sh.
|
|
# --out DIR write report.tsv + report.md into DIR (created if
|
|
# needed). Always ALSO prints the summary to stdout.
|
|
# --deep-verify (.tbl only) additionally round-trip the converted
|
|
# file through table-to-csv.sh — informational only,
|
|
# NEVER gates pass/fail (nc-table.sh F-4 is unfixed).
|
|
# --run-native-convert OPT-IN, default OFF. After a successful CR-strip,
|
|
# also chain the matching native Cloverleaf converter
|
|
# (hcitblconvert / hcixltconvert), located on $PATH /
|
|
# $HCIROOT/bin / $HCIROOT/server/bin, invoked with the
|
|
# bare NAME (not a path, not a glob) against the
|
|
# inferred site. Requires $HCIROOT set. A file whose
|
|
# site can't be confidently inferred (path doesn't
|
|
# match the <site>/Tables|Xlate or shared
|
|
# $HCIROOT/Tables|Xlate layout) is reported
|
|
# NATIVE_SKIPPED, not silently guessed at.
|
|
#
|
|
# Exit codes:
|
|
# 0 all good (including a clean --dry-run / all-already-LF run)
|
|
# 1 usage/input error
|
|
# 2 at least one file FAILED_VERIFY on the CR-strip step
|
|
# 3 at least one file's --run-native-convert step FAILED (auto-rolled-back
|
|
# to the CR-stripped-safe state — see NATIVE_FAILED in the report).
|
|
# Exit 2 takes priority if BOTH occur.
|
|
set -o pipefail
|
|
|
|
NC_SELF="$0"
|
|
LIB_DIR="$(cd "$(dirname "$NC_SELF")" && pwd)"
|
|
JOURNAL="$LIB_DIR/journal.sh"
|
|
T2C="$LIB_DIR/table-to-csv.sh"
|
|
NC_ROLLBACK="$LIB_DIR/../larry-rollback.sh"
|
|
[ -f "$NC_ROLLBACK" ] || NC_ROLLBACK="$(command -v larry-rollback.sh 2>/dev/null || printf '%s' "$NC_ROLLBACK")"
|
|
|
|
die() { printf 'cl-format-migrate: %s\n' "$*" >&2; exit 1; }
|
|
warn() { printf 'cl-format-migrate: %s\n' "$*" >&2; }
|
|
|
|
# Shared CR-safety primitives (coerce_int et al) — same pattern as every
|
|
# other lib/ tool. Inline fallback if a partial install is missing it.
|
|
if [ -r "$LIB_DIR/cygwin-safe.sh" ]; then
|
|
# shellcheck disable=SC1090,SC1091
|
|
. "$LIB_DIR/cygwin-safe.sh"
|
|
else
|
|
coerce_int() { local r="${1:-}" d="${2:-0}" c; c=$(printf '%s' "$r" | tr -cd '0-9'); printf '%s' "${c:-$d}"; }
|
|
fi
|
|
|
|
# journal.sh enables `set -u`; restore the assumed (unset-tolerant) env after
|
|
# sourcing, same as nc-engine.sh.
|
|
[ -f "$JOURNAL" ] && . "$JOURNAL" || warn "journal.sh not available — writes will NOT be journaled/reversible; refusing to run real conversions."
|
|
set +u
|
|
|
|
usage() { sed -n '2,93p' "$NC_SELF"; }
|
|
|
|
DIR="."
|
|
TYPE="tbl"
|
|
DRY_RUN=0
|
|
CONFIRM=""
|
|
OUT_DIR=""
|
|
DEEP_VERIFY=0
|
|
RUN_NATIVE=0
|
|
_dir_set=0
|
|
|
|
while [ $# -gt 0 ]; do
|
|
case "$1" in
|
|
--type) shift; TYPE="$1" ;;
|
|
--dry-run) DRY_RUN=1 ;;
|
|
--confirm) shift; CONFIRM="${1:-}" ;;
|
|
--out) shift; OUT_DIR="$1" ;;
|
|
--deep-verify) DEEP_VERIFY=1 ;;
|
|
--run-native-convert) RUN_NATIVE=1 ;;
|
|
-h|--help) usage; exit 0 ;;
|
|
-*) die "unknown flag: $1 (run --help)" ;;
|
|
*)
|
|
if [ "$_dir_set" = "0" ]; then DIR="$1"; _dir_set=1
|
|
else die "extra arg: $1"
|
|
fi
|
|
;;
|
|
esac
|
|
shift
|
|
done
|
|
|
|
case "$TYPE" in tbl|xlt|all) ;; *) die "bad --type (want tbl|xlt|all)" ;; esac
|
|
[ -d "$DIR" ] || die "no such directory: $DIR"
|
|
DIR="$(cd "$DIR" && pwd)"
|
|
[ "$RUN_NATIVE" = "1" ] && [ -z "${HCIROOT:-}" ] && die "--run-native-convert requires \$HCIROOT to be set"
|
|
|
|
# ── report plumbing ──────────────────────────────────────────────────────────
|
|
REPORT_TSV="$(mktemp)"
|
|
printf 'path\tstatus\treason\tentry_id\n' > "$REPORT_TSV"
|
|
|
|
CONVERTED=0
|
|
SKIPPED=0
|
|
FAILED=0
|
|
PLANNED=0
|
|
NATIVE_CONVERTED=0
|
|
NATIVE_FAILED=0
|
|
NATIVE_SKIPPED=0
|
|
|
|
_report_row() {
|
|
# path status reason entry_id
|
|
printf '%s\t%s\t%s\t%s\n' "$1" "$2" "$3" "${4:-}" >> "$REPORT_TSV"
|
|
printf '%-16s %s%s\n' "$2" "$1" "$( [ -n "${3:-}" ] && printf ' — %s' "$3" )"
|
|
}
|
|
|
|
# _count_trailing_cr FILE — count lines whose LAST byte before the newline is
|
|
# a CR (i.e. CRLF line endings — exactly what `sed 's/\r$//'` targets).
|
|
#
|
|
# v0.9.12 Vera-gate fix: this MUST match the strip's own `\r$` semantics, not
|
|
# "any CR byte anywhere" (a plain `grep -c $'\r'`). A blanket any-CR count
|
|
# also matches a legitimate mid-line/embedded CR that the strip correctly
|
|
# and deliberately PRESERVES (the tool's whole "never tr -d '\r'" design
|
|
# point) — which made the old check count that preserved byte as a failure,
|
|
# forever, on every re-run (never converges to SKIPPED). Anchoring the grep
|
|
# on `\r$` (CR immediately before end-of-line) only matches an actual CRLF
|
|
# line ending, so a file with just an embedded CR and no trailing CRLF
|
|
# correctly reads as 0 here — both for the pre-strip "does this file need
|
|
# converting" decision and for verify check (a) below.
|
|
_count_trailing_cr() {
|
|
local f="$1" n
|
|
n=$(LC_ALL=C grep -c $'\r$' "$f" 2>/dev/null)
|
|
coerce_int "$n" 0
|
|
}
|
|
|
|
_find_candidates() {
|
|
local dir="$1" type="$2"
|
|
case "$type" in
|
|
tbl) find "$dir" -type f -iname '*.tbl' 2>/dev/null ;;
|
|
xlt) find "$dir" -type f -iname '*.xlt' 2>/dev/null ;;
|
|
all) find "$dir" -type f \( -iname '*.tbl' -o -iname '*.xlt' \) 2>/dev/null ;;
|
|
esac | LC_ALL=C sort
|
|
}
|
|
|
|
# ── CR-strip verify (the mandatory gate; format-agnostic — .tbl and .xlt
|
|
# share the same prologue/end_prologue/version convention, confirmed in
|
|
# the design doc §2 nc-revisions.sh cross-check) ──────────────────────────
|
|
_verify_crlf_strip() {
|
|
local orig_copy="$1" new_file="$2"
|
|
local cr; cr=$(_count_trailing_cr "$new_file")
|
|
if [ "$cr" != "0" ]; then
|
|
printf '(a) FAILED: %s trailing-CRLF line(s) still present after conversion' "$cr"
|
|
return 1
|
|
fi
|
|
|
|
local pro_pre pro_post epro_pre epro_post ver_pre ver_post
|
|
pro_pre=$(coerce_int "$(grep -c '^[[:space:]]*prologue[[:space:]]*$' "$orig_copy" 2>/dev/null)" 0)
|
|
pro_post=$(coerce_int "$(grep -c '^[[:space:]]*prologue[[:space:]]*$' "$new_file" 2>/dev/null)" 0)
|
|
epro_pre=$(coerce_int "$(grep -c 'end_prologue' "$orig_copy" 2>/dev/null)" 0)
|
|
epro_post=$(coerce_int "$(grep -c 'end_prologue' "$new_file" 2>/dev/null)" 0)
|
|
ver_pre=$(coerce_int "$(grep -c 'version:' "$orig_copy" 2>/dev/null)" 0)
|
|
ver_post=$(coerce_int "$(grep -c 'version:' "$new_file" 2>/dev/null)" 0)
|
|
if [ "$pro_pre" != "$pro_post" ] || [ "$epro_pre" != "$epro_post" ] || [ "$ver_pre" != "$ver_post" ]; then
|
|
printf '(b) FAILED: prologue/end_prologue/version block count changed (pre %s/%s/%s -> post %s/%s/%s)' \
|
|
"$pro_pre" "$epro_pre" "$ver_pre" "$pro_post" "$epro_post" "$ver_post"
|
|
return 1
|
|
fi
|
|
|
|
# (c) the actual gate: new file == CR-stripped original, byte for byte.
|
|
if ! diff -q <(sed $'s/\r$//' "$orig_copy") "$new_file" >/dev/null 2>&1; then
|
|
printf '(c) FAILED: content differs by more than trailing-CR removal (not a pure CRLF->LF pass)'
|
|
return 1
|
|
fi
|
|
return 0
|
|
}
|
|
|
|
# ── native-binary lookup ──────────────────────────────────────────────────────
|
|
_resolve_native_bin() {
|
|
local name="$1"
|
|
if command -v "$name" >/dev/null 2>&1; then command -v "$name"; return 0; fi
|
|
local d
|
|
for d in "${HCIROOT:-}/bin" "${HCIROOT:-}/server/bin"; do
|
|
[ -n "$d" ] && [ -x "$d/$name" ] && { printf '%s' "$d/$name"; return 0; }
|
|
done
|
|
return 1
|
|
}
|
|
|
|
# _infer_site FILE PARENTNAME1 PARENTNAME2 — infer the site name from a path
|
|
# shaped like .../<site>/<PARENTNAME1-or-2>/<file>, requiring the grandparent
|
|
# to actually be a site dir (has a NetConfig) so this is never a guess.
|
|
# Prints the site name on stdout and returns 0, or returns 1 (no site — could
|
|
# still be the shared $HCIROOT/Tables|Xlate layout, handled by the caller).
|
|
_infer_site() {
|
|
local f="$1" p1="$2" p2="$3"
|
|
local parent grandparent pbase
|
|
parent="$(dirname "$f")"
|
|
pbase="$(basename "$parent")"
|
|
case "$pbase" in
|
|
"$p1"|"$p2")
|
|
grandparent="$(dirname "$parent")"
|
|
if [ -n "${HCIROOT:-}" ] && [ -f "$grandparent/NetConfig" ]; then
|
|
basename "$grandparent"; return 0
|
|
fi
|
|
;;
|
|
esac
|
|
return 1
|
|
}
|
|
|
|
# ── native-convert step (opt-in; only ever called on a file THIS run just
|
|
# CR-stripped) ─────────────────────────────────────────────────────────────
|
|
_native_convert() {
|
|
local file="$1" kind="$2" # kind = tbl | xlt
|
|
local binname parent1 parent2
|
|
if [ "$kind" = "tbl" ]; then binname="hcitblconvert"; parent1="Tables"; parent2="tables"
|
|
else binname="hcixltconvert"; parent1="Xlate"; parent2="xlate"
|
|
fi
|
|
|
|
local bin; bin=$(_resolve_native_bin "$binname") || {
|
|
_report_row "$file" "NATIVE_SKIPPED" "$binname not found on \$PATH, \$HCIROOT/bin, or \$HCIROOT/server/bin" ""
|
|
NATIVE_SKIPPED=$((NATIVE_SKIPPED + 1)); return 0
|
|
}
|
|
|
|
local site=""
|
|
if ! site=$(_infer_site "$file" "$parent1" "$parent2"); then
|
|
# Not under a <site>/Tables|Xlate layout. Allow the shared-root layout
|
|
# ($HCIROOT/Tables or $HCIROOT/Xlate, no site subdir) with an empty site;
|
|
# anything else is a "can't confidently resolve" skip, never a guess.
|
|
local parentdir; parentdir="$(dirname "$file")"
|
|
case "$parentdir" in
|
|
"${HCIROOT:-__no_hciroot__}/$parent1"|"${HCIROOT:-__no_hciroot__}/$parent2") site="" ;;
|
|
*)
|
|
_report_row "$file" "NATIVE_SKIPPED" "cannot confidently infer site for native convert (path doesn't match <site>/${parent1}|${parent2} or shared \$HCIROOT/${parent1}|${parent2})" ""
|
|
NATIVE_SKIPPED=$((NATIVE_SKIPPED + 1)); return 0
|
|
;;
|
|
esac
|
|
fi
|
|
|
|
# Checkpoint the CR-stripped (already-verified-safe) state as its OWN
|
|
# journal entry, distinct from the CR-strip entry, so a native-convert
|
|
# failure rolls back ONLY the native step.
|
|
local precopy entry2
|
|
precopy="$(mktemp)"; cp -p "$file" "$precopy" || {
|
|
_report_row "$file" "NATIVE_FAILED" "could not snapshot pre-native state" ""
|
|
NATIVE_FAILED=$((NATIVE_FAILED + 1)); rm -f "$precopy"; return 1
|
|
}
|
|
entry2=$(journal_write "$file" "$precopy" 2>>"$JOURNAL_LOG")
|
|
local jw2_rc=$?
|
|
rm -f "$precopy"
|
|
if [ "$jw2_rc" != "0" ] || [ -z "$entry2" ]; then
|
|
_report_row "$file" "NATIVE_FAILED" "pre-native journal checkpoint failed (rc=$jw2_rc) — native convert NOT attempted" ""
|
|
NATIVE_FAILED=$((NATIVE_FAILED + 1)); return 1
|
|
fi
|
|
|
|
local base out rc
|
|
base="$(basename "$file")"
|
|
out=$( (cd "$(dirname "$file")" && HCIROOT="$HCIROOT" HCISITE="$site" "$bin" "$base") 2>&1 )
|
|
rc=$?
|
|
|
|
local fail_reason=""
|
|
if [ "$rc" != "0" ]; then
|
|
fail_reason="exit code $rc"
|
|
elif printf '%s' "$out" | grep -qi "before prologue\|please correct the error"; then
|
|
fail_reason="native binary reported a parse error"
|
|
elif [ ! -s "$file" ]; then
|
|
fail_reason="file is empty/truncated after native convert"
|
|
fi
|
|
|
|
if [ -n "$fail_reason" ]; then
|
|
_report_row "$file" "NATIVE_FAILED" "$fail_reason (auto-rolled-back to CR-stripped-safe state) — output: $(printf '%s' "$out" | tr '\n' ' ' | cut -c1-200)" "$entry2"
|
|
NATIVE_FAILED=$((NATIVE_FAILED + 1))
|
|
if [ -x "$NC_ROLLBACK" ] || [ -f "$NC_ROLLBACK" ]; then
|
|
bash "$NC_ROLLBACK" --entry "$entry2" --yes >>"$JOURNAL_LOG" 2>&1 \
|
|
|| warn "auto-rollback of $entry2 FAILED — $file may be in an uncertain state; run: larry-rollback.sh --entry $entry2"
|
|
else
|
|
warn "larry-rollback.sh not found — could not auto-rollback $entry2; $file may be in an uncertain state"
|
|
fi
|
|
return 1
|
|
fi
|
|
|
|
_report_row "$file" "NATIVE_CONVERTED" "$binname OK (site=${site:-<shared>}); entry $entry2; rollback: larry-rollback.sh --entry $entry2" "$entry2"
|
|
NATIVE_CONVERTED=$((NATIVE_CONVERTED + 1))
|
|
return 0
|
|
}
|
|
|
|
# ── one .tbl/.xlt file — CR-strip step (shared for both types) ──────────────
|
|
_process_file() {
|
|
local file="$1" kind="$2" # kind = tbl | xlt
|
|
# v0.9.12 Vera-gate fix: trailing-CR-only detection (matches the strip's own
|
|
# `\r$` semantics) — a file whose only CR bytes are legitimate mid-line ones
|
|
# (no trailing CRLF) has nothing for this tool to do, and is correctly
|
|
# SKIPPED rather than endlessly re-attempted/re-failed. See _count_trailing_cr.
|
|
local cr; cr=$(_count_trailing_cr "$file")
|
|
|
|
if [ "$cr" = "0" ]; then
|
|
_report_row "$file" "SKIPPED" "already pure-LF (no trailing CRLF found)" ""
|
|
SKIPPED=$((SKIPPED + 1))
|
|
return 0
|
|
fi
|
|
|
|
if [ "$DRY_RUN" = "1" ]; then
|
|
_report_row "$file" "DRY_RUN" "$cr trailing-CRLF line(s) — would convert (re-run with --confirm yes)$( [ "$RUN_NATIVE" = "1" ] && printf '; would also attempt native convert' )" ""
|
|
PLANNED=$((PLANNED + 1))
|
|
return 0
|
|
fi
|
|
if [ "$CONFIRM" != "yes" ]; then
|
|
_report_row "$file" "REFUSED" "$cr trailing-CRLF line(s) — no --confirm yes, nothing written" ""
|
|
PLANNED=$((PLANNED + 1))
|
|
return 0
|
|
fi
|
|
if ! declare -f journal_write >/dev/null 2>&1; then
|
|
_report_row "$file" "FAILED_VERIFY" "journal.sh not available — refusing to write unjournaled" ""
|
|
FAILED=$((FAILED + 1))
|
|
return 1
|
|
fi
|
|
|
|
local orig_copy newtmp
|
|
orig_copy="$(mktemp)"; newtmp="$(mktemp)"
|
|
if ! cp -p "$file" "$orig_copy"; then
|
|
_report_row "$file" "FAILED_VERIFY" "could not snapshot original for verify" ""
|
|
FAILED=$((FAILED + 1)); rm -f "$orig_copy" "$newtmp"; return 1
|
|
fi
|
|
if ! sed $'s/\r$//' "$file" > "$newtmp"; then
|
|
_report_row "$file" "FAILED_VERIFY" "sed CRLF-strip failed" ""
|
|
FAILED=$((FAILED + 1)); rm -f "$orig_copy" "$newtmp"; return 1
|
|
fi
|
|
|
|
local entry_id jw_rc
|
|
entry_id=$(journal_write "$file" "$newtmp" 2>>"$JOURNAL_LOG")
|
|
jw_rc=$?
|
|
if [ "$jw_rc" != "0" ] || [ -z "$entry_id" ]; then
|
|
_report_row "$file" "FAILED_VERIFY" "journal_write failed (rc=$jw_rc) — file NOT modified" ""
|
|
FAILED=$((FAILED + 1)); rm -f "$orig_copy" "$newtmp"; return 1
|
|
fi
|
|
|
|
local reason
|
|
if reason=$(_verify_crlf_strip "$orig_copy" "$file"); then
|
|
local deep_note=""
|
|
if [ "$kind" = "tbl" ] && [ "$DEEP_VERIFY" = "1" ] && [ -f "$T2C" ]; then
|
|
if "$T2C" --with-header "$file" >/dev/null 2>&1; then
|
|
deep_note=" | deep-verify(table-to-csv): OK"
|
|
else
|
|
deep_note=" | deep-verify(table-to-csv): FAILED (informational only — F-4 known bug, not gating)"
|
|
fi
|
|
fi
|
|
_report_row "$file" "CONVERTED" "$cr trailing-CRLF line(s) stripped; entry $entry_id; rollback: larry-rollback.sh --entry $entry_id${deep_note}" "$entry_id"
|
|
CONVERTED=$((CONVERTED + 1))
|
|
rm -f "$orig_copy" "$newtmp"
|
|
if [ "$RUN_NATIVE" = "1" ]; then
|
|
_native_convert "$file" "$kind"
|
|
fi
|
|
else
|
|
_report_row "$file" "FAILED_VERIFY" "$reason — entry $entry_id; rollback: larry-rollback.sh --entry $entry_id" "$entry_id"
|
|
FAILED=$((FAILED + 1))
|
|
rm -f "$orig_copy" "$newtmp"
|
|
fi
|
|
}
|
|
|
|
# ── SESSION (one for the whole batch) ────────────────────────────────────────
|
|
SESSION="${LARRY_SESSION_ID:-clfmt-$(date +%Y-%m-%d-%H%M%S)-$$}"
|
|
export LARRY_SESSION_ID="$SESSION"
|
|
export LARRY_HOME="${LARRY_HOME:-$HOME/.larry}"
|
|
JOURNAL_LOG="$(mktemp)"
|
|
|
|
printf '═══════════════════════════════════════════════════════════════════════\n'
|
|
printf 'cl-format-migrate — dir=%s type=%s dry-run=%s confirm=%s run-native-convert=%s\n' \
|
|
"$DIR" "$TYPE" "$( [ "$DRY_RUN" = "1" ] && echo yes || echo no )" "$( [ "$CONFIRM" = "yes" ] && echo yes || echo no )" \
|
|
"$( [ "$RUN_NATIVE" = "1" ] && echo yes || echo no )"
|
|
printf 'journal session: %s\n' "$SESSION"
|
|
printf '═══════════════════════════════════════════════════════════════════════\n'
|
|
|
|
if [ "$TYPE" = "tbl" ] || [ "$TYPE" = "all" ]; then
|
|
printf '\n-- tables (.tbl) --\n'
|
|
_any=0
|
|
while IFS= read -r f; do
|
|
[ -n "$f" ] || continue
|
|
_any=1
|
|
_process_file "$f" tbl
|
|
done < <(_find_candidates "$DIR" tbl)
|
|
[ "$_any" = "1" ] || printf '(no .tbl files found under %s)\n' "$DIR"
|
|
fi
|
|
|
|
if [ "$TYPE" = "xlt" ] || [ "$TYPE" = "all" ]; then
|
|
printf '\n-- xlates (.xlt) --\n'
|
|
_any=0
|
|
while IFS= read -r f; do
|
|
[ -n "$f" ] || continue
|
|
_any=1
|
|
_process_file "$f" xlt
|
|
done < <(_find_candidates "$DIR" xlt)
|
|
[ "$_any" = "1" ] || printf '(no .xlt files found under %s)\n' "$DIR"
|
|
fi
|
|
|
|
# ── summary ───────────────────────────────────────────────────────────────────
|
|
printf '\n───────────────────────────────────────────────────────────────────────\n'
|
|
printf 'SUMMARY: converted=%d skipped-already-lf=%d failed-verify=%d planned=%d native-converted=%d native-failed=%d native-skipped=%d\n' \
|
|
"$CONVERTED" "$SKIPPED" "$FAILED" "$PLANNED" "$NATIVE_CONVERTED" "$NATIVE_FAILED" "$NATIVE_SKIPPED"
|
|
if [ "$CONVERTED" -gt 0 ] || [ "$NATIVE_CONVERTED" -gt 0 ]; then
|
|
printf 'WHOLE-BATCH ROLLBACK (all writes this run): larry-rollback.sh --session %s\n' "$SESSION"
|
|
fi
|
|
|
|
if [ -n "$OUT_DIR" ]; then
|
|
mkdir -p "$OUT_DIR" 2>/dev/null || warn "could not create --out dir: $OUT_DIR"
|
|
if [ -d "$OUT_DIR" ]; then
|
|
cp "$REPORT_TSV" "$OUT_DIR/report.tsv"
|
|
{
|
|
printf '# cl-format-migrate report\n\n'
|
|
# v0.9.12: `printf --` — bash 3.2's printf builtin misreads a format
|
|
# string that itself starts with "- " as an (invalid) option flag
|
|
# ("printf: - : invalid option") unless "--" ends option parsing first.
|
|
printf -- '- dir: `%s`\n- type: %s\n- dry-run: %s\n- confirm: %s\n- run-native-convert: %s\n- session: %s\n- generated: %s\n\n' \
|
|
"$DIR" "$TYPE" "$( [ "$DRY_RUN" = "1" ] && echo yes || echo no )" \
|
|
"$( [ "$CONFIRM" = "yes" ] && echo yes || echo no )" "$( [ "$RUN_NATIVE" = "1" ] && echo yes || echo no )" \
|
|
"$SESSION" "$(date -Iseconds 2>/dev/null || date)"
|
|
printf '## Summary\n\n'
|
|
printf '| converted | skipped-already-lf | failed-verify | planned | native-converted | native-failed | native-skipped |\n'
|
|
printf '|---|---|---|---|---|---|---|\n'
|
|
printf '| %d | %d | %d | %d | %d | %d | %d |\n\n' \
|
|
"$CONVERTED" "$SKIPPED" "$FAILED" "$PLANNED" "$NATIVE_CONVERTED" "$NATIVE_FAILED" "$NATIVE_SKIPPED"
|
|
printf '## Per-file detail\n\n'
|
|
printf '| path | status | reason | entry_id |\n|---|---|---|---|\n'
|
|
tail -n +2 "$REPORT_TSV" | while IFS=$'\t' read -r p s r e; do
|
|
printf '| %s | %s | %s | %s |\n' "$p" "$s" "$r" "$e"
|
|
done
|
|
} > "$OUT_DIR/report.md"
|
|
printf 'report written: %s/report.tsv , %s/report.md\n' "$OUT_DIR" "$OUT_DIR"
|
|
fi
|
|
fi
|
|
|
|
rm -f "$REPORT_TSV" "$JOURNAL_LOG"
|
|
|
|
if [ "$FAILED" -gt 0 ]; then
|
|
exit 2
|
|
fi
|
|
if [ "$NATIVE_FAILED" -gt 0 ]; then
|
|
exit 3
|
|
fi
|
|
exit 0
|