Scripts

Workflow preflights

Run uv run python scripts/check_workflow_quality.py after a coherent Python edit batch and before committing or starting the full gate. It reuses the validator's Ruff, formatting, typing, Semgrep and documentation checks, refreshes the index, and collects all failures instead of stopping after the first one. Exit 1 means at least one check failed or the refreshed audit was unavailable. Use --dry-run to inspect its commands. The full tmux gate remains required.

Run uv run python scripts/prepare_benchmark_history.py before benchmark tests in a fresh checkout or after squash integration/branch retirement. Exit 1 lists missing commit IDs. Run it with --fetch to retrieve the exact manifest-pinned Codira fixture and reviewer revisions from origin, then verify them. That mode needs network access and leaves manifests, branches and working files intact. Exit 2 means invalid input or fetch failure. CI calls this shared helper with --fetch; local object availability alone does not establish availability in a fresh CI checkout.

For container/MCP/oracle readiness, use the existing scripts/launch_agent_efficiency_pilot.py --prepare flow and the factory's generated launch plan. Its qualify_host_context check detects rootless runtime and tmux access failures before runtime work, and its readiness receipt binds the full gate, image, fixtures, tool dispatch and task calibration. Follow the campaign factory procedure for its complete required arguments. Use the authorized host execution context for rootless Podman; a successful direct MCP server probe does not qualify native-client tool dispatch. Model canaries and paid launch remain separately authorized operations.

scripts/bootstrap_dev_environment.py

Synchronize the uv-managed .venv, install the extracted first-party analyzer and backend packages, download and smoke-test the default public ONNX embedding model, install repo-local Git configuration, and optionally run the validation surface. An HF_TOKEN is used when configured but is not required for the default public model.

scripts/install_first_party_packages.py

Install the repository-local first-party analyzer/backend package set from one authoritative package list shared by bootstrap and CI.

scripts/install_repo_git_config.py

Install the repo-local Git configuration expected by this repository, including hooks, commit template, and sanctioned aliases.

The installer mirrors repository-local aliases only. It does not configure user.name, user.email, remote URLs, tokens, or credential helpers. Aliases that contact GitHub use the operator's own Git or gh authentication.

scripts/run_with_repo_python.py

Resolve the repository Python interpreter deterministically and execute Python arguments through it.

scripts/run_repo_tool.py

Run Python-facing repository tools with cache and temporary state redirected outside the checkout. Use this helper instead of hand-setting tool cache or temporary directories under the repository.

scripts/validate_repo.py

Run the standard local validation sequence through scripts/run_repo_tool.py. This is the preferred one-command validation entry point for local changes. Both validation entry points perform a full structural index refresh with --full --defer-embeddings before auditing. This recomputes audit facts without requiring local embedding models. They reject nonempty CODIRA_CONFIG_FILE, CODIRA_TARGET_DIR, or CODIRA_OUTPUT_DIR overrides: a replacement config can silently omit the repository documentation routes, and a redirected index can audit another checkout. From the repository root, recover with:

unset CODIRA_CONFIG_FILE CODIRA_TARGET_DIR CODIRA_OUTPUT_DIR
uv run python scripts/check_workflow_quality.py
uv run python scripts/validate_repo.py

An empty audit result is meaningful only after the configured audit route has run against a successfully refreshed index. Ordinary Codira CLI commands retain their routing override support; this restriction applies to repository gates. Failed captured audits print their command and stdout/stderr diagnostics. Use --dry-run to print the delegated commands without executing them. Use --semgrep-complete to append a broad Semgrep registry scan and save its JSON report under .artifacts/analysis/semgrep/.

scripts/check_commit_messages.py

Validate commit headers for semantic-release compatibility.

This script is used by the GitHub commit-message workflow and enforces the repository's conventional-commit contract.

scripts/release_preview.py

Run the non-publishing semantic-release preview used by the guarded git rel path. It exposes github.env only to the preview child via sops exec-env.

scripts/clean_repo.py

Clean ignored repository artifacts using Git as the source of truth rather than custom filesystem heuristics.

scripts/new_decision.py

Create a new ADR file under docs/adr/ and append it to the ADR index.

scripts/provision_embedding_model.py

Prefetch or verify the local model artifact required by the active embedding engine.

Normal CLI indexing now provisions the model automatically on first use. This script remains available when operators want to pre-warm the cache explicitly.

scripts/embedding_model_manifest.py

Validate and inspect the committed embedding model candidate manifest:

uv run python scripts/embedding_model_manifest.py --list
uv run python scripts/embedding_model_manifest.py \
  --id bge-small-en-v1.5-onnx \
  --print-config

The script does not download model weights. It renders repository configuration snippets for the selected embedding engine/model entry.

scripts/download_embedding_model.py

Download and smoke-test model artifacts named in benchmarks/embedding/model-candidates.json:

uv run python scripts/download_embedding_model.py \
  --manifest benchmarks/embedding/model-candidates.json \
  --anonymous --allow-remote-code

The script obtains authenticated downloads through the SOPS-scoped Hugging Face environment. ONNX artifact paths in the manifest must resolve beneath --install-root; manifests cannot write elsewhere. A manifest entry that enables remote model code requires both --allow-remote-code and --anonymous, so that code never receives the Hugging Face credential. Use that acknowledgement only with a reviewed manifest.

For ONNX entries, the script downloads onnx/model.onnx and tokenizer.json from Hugging Face, installs them under the manifest's .codira/models/... paths, and smoke-tests the artifacts with the first-party ONNX engine. For SentenceTransformers entries, it downloads the model snapshot into the Hugging Face cache and runs a local smoke encode.

Select one candidate with --model-id:

uv run python scripts/download_embedding_model.py \
  --model-id bge-small-en-v1.5-onnx \
  --anonymous

scripts/build_retrieval_quality_dataset.py

Build a labeled retrieval-quality dataset from GitHub pull requests when a repository manifest provides github_owner/github_repo, and from local Git commit history as the deterministic fallback:

uv run python -m scripts.build_retrieval_quality_dataset \
  --repo-manifest benchmarks/retrieval-quality/repos.local.json \
  --output .artifacts/benchmarks/retrieval-quality/dataset.jsonl \
  --source git

The generated JSONL rows contain a natural-language query and expected repo-relative paths. GitHub PR examples use the PR title/body as the query and the changed files as labels. Git commit examples use the commit subject/body as the query and changed files as labels; the Git parser uses an explicit message/path separator so multiline commit bodies cannot become labels. The script writes only the requested dataset file and does not index repositories.

Use --source github to require GitHub-backed rows only, or --source git to avoid network access entirely. GitHub collection uses the operator's existing gh authentication.

scripts/grade_retrieval_quality.py

Prepare blinded snippet pools from a completed known-target comparison, check an authenticated OpenRouter judge contract, grade with checkpoints, or rescore saved responses offline. This optional assessment uses one judge and one request per nonempty query pool. It reports LLM relevance separately from known-target metrics. See LLM retrieval grading for exact preparation, SOPS preflight, execution and recovery commands.

scripts/run_retrieval_quality_benchmark.py

The --known-target mode implements the automatic source-verified benchmark for #59, with pinned sources, resume, rescoring, explicit cutoffs and cost measurements. Generate its cases with scripts/build_known_target_dataset.py. See Automatic known-target retrieval quality for complete commands and interpretation limits. The historical mode below continues to accept commit-derived changed-file labels.

Run model quality measurements against a dataset produced by scripts/build_retrieval_quality_dataset.py:

uv run python -m scripts.run_retrieval_quality_benchmark \
  --dataset .artifacts/benchmarks/retrieval-quality/dataset.jsonl \
  --repo-manifest benchmarks/retrieval-quality/repos.local.json \
  --model-manifest benchmarks/embedding/model-candidates.json \
  --backend sqlite \
  --top-k 10

The runner writes generated configs, isolated .codira output directories, logs, results.jsonl, summary.json, and report.md under .artifacts/benchmarks/retrieval-quality/runs/<timestamp>/. It uses codira emb by default and records Recall@K, MRR@K, nDCG@K, hit rate, index time, and query time. Add --include-ctx only when the mixed ctx retrieval behavior is also part of the quality question.

To recompute summary.json and report.md for an existing results.jsonl after scorer changes, run:

uv run python -m scripts.run_retrieval_quality_benchmark \
  --rescore-results .artifacts/benchmarks/retrieval-quality/runs/<timestamp>/results.jsonl

This benchmark performs full indexing unless --no-full is supplied. Do not run it during another large campaign unless CPU, RAM, and disk contention are acceptable.

scripts/run_final_embedding_model_campaign.py

Run the final engine/model measuring campaign:

uv run python -m scripts.run_final_embedding_model_campaign \
  --manifest benchmarks/embedding/uv-backed-repos.local.json \
  --model-manifest benchmarks/embedding/model-candidates.json \
  --backend duckdb \
  --runs 5 \
  --warmup 1

The wrapper writes artifacts under .artifacts/benchmarks/embedding/runs/<timestamp>/, first runs scripts/download_embedding_model.py against the model manifest, records the optional baseline path and manifests, writes one generated config per model/backend under the artifact directory, and passes that config to benchmark commands with --config-file. It does not rewrite repository .codira/config.toml files. Use --baseline PATH only to record which previous matrix should be used later during analysis. Use --preflight-only to stop after download and smoke tests. Use --backend both only when the campaign must run separate SQLite and DuckDB backend phases for each model/repository pair as well as PyTorch and ONNX Runtime embedding engines.

The wrapper applies conservative benchmark defaults for large embedding models: 768-dimensional candidates use batch_size = 1, [embeddings.indexing].max_text_chars = 2000, [embeddings.indexing].work_batch_multiplier = 256, Torch threads 4/1, and ONNX thread limits intra_op_num_threads = 4 plus inter_op_num_threads = 1. 384-dimensional ONNX candidates use batch_size = 4; 384-dimensional SentenceTransformers candidates use batch_size = 32.

scripts/benchmark_index.py

Run one instrumented index pass and emit structured JSON with phase timings, embedding batch sizes, and index summary counters.

Use this script when evaluating indexing regressions or tuning embedding batch and Torch runtime settings.

When --output is supplied, the JSON artifact is written to that path and includes run metadata: UTC timestamp, Codira version, Git commit, active plugin inventory, and profiler/tool availability.

scripts/benchmark_campaign.py

Run a manifest-driven performance measurement campaign across repository size categories.

The campaign runner builds phase-timing, Hyperfine, cProfile, and optional Pyinstrument command plans for each configured repository. Full-index profiling comes from the phase-timing helper; cProfile and Pyinstrument are reserved for context retrieval. Use --dry-run to write and inspect .artifacts/benchmarks/campaigns/<run-id>/campaign-plan.json without executing benchmark commands. --dry-run still validates the manifest before printing the plan. The dry run still performs the adaptive discovery pass used to resolve repo-specific benchmark commands.

Use --continue-on-error for torture campaigns. In that mode every planned command is attempted, command stdout and stderr are retained under <run-id>/logs/, and failures are summarized in <run-id>/failure-summary.json.

The manifest supports optional repository-local commands entries that extend the Hyperfine command set beyond the default warm index and ctx --json measurements. Each command is written as a JSON argv array excluding the codira executable itself, for example:

["sym", "build_parser", "--json"]

Supported manifest-benchmark subcommands are:

• help
• index
• cov
• sym
• symlist
• emb
• calls
• refs
• audit
• ctx
• plugins
• caps

Manifest command tokens may use {path}, {output_dir}, and {query} placeholders. For path-aware subcommands, the campaign runner appends --path and --output-dir automatically when they are omitted.

Before building the final Hyperfine command matrix, the campaign runs an adaptive discovery pass for each repository:

• scripts/benchmark_index.py --full builds the artifact index and records phase timings
• symlist --json is used to discover candidate symbols with meaningful graph connectivity
• semantic query candidates are ranked from the manifest query plus discovered symbol and module names
• adaptive commands such as sym, calls, refs, emb, ctx, and symlist are resolved to repo-specific commands with more significant output
• unresolved adaptive commands are skipped instead of aborting the whole repo campaign

Discovery index state is the same artifact index used by the later warm-index and query measurements. Selector provenance is persisted under .artifacts/benchmarks/campaigns/<run-id>/selection/*.json, discovery command output is persisted under .artifacts/benchmarks/campaigns/<run-id>/logs/discovery/, and the resolved or skipped commands are also recorded in campaign-plan.json.

Full-index timing is therefore recorded once and written to <category>-<label>-index-phases.json. Hyperfine no longer duplicates codira index --full; the utility summary combines the phase full-index timing with Hyperfine warm-index and query timings. Pass --warmup 0 to disable Hyperfine warmups entirely.

Example:

python scripts/benchmark_campaign.py benchmarks.json --dry-run
python scripts/benchmark_campaign.py benchmarks.json --runs 10
python scripts/benchmark_campaign.py benchmarks.json --runs 5 --warmup 0

scripts/run_manifest_baseline.py

Run the paired SQLite and DuckDB benchmarks/performance/bk-cpp.local.json torture baseline with fixed runtime environment defaults. Results default to .artifacts/benchmarks/backend/; set ARTIFACT_ROOT to override that root. The runner continues after individual benchmark failures.

uv run python -m scripts.run_manifest_baseline --runs 5 --warmup 1
uv run python -m scripts.run_manifest_baseline --runs 5 --warmup 0

See docs/process/performance-benchmarking.md for the manifest format, artifact layout, and plugin metadata requirements.

scripts/benchmark_release.py

Run the release-oriented Hyperfine benchmark plan for codira index --full, codira ctx --json, and codira audit --json.

The default result file is .artifacts/benchmarks/release/release-hyperfine.json. Use --dry-run to inspect the exact Hyperfine invocation before measuring.

scripts/release_audit.py

Run conservative release-readiness checks for the current branch and repository state.

scripts/release_rel.py

Run the guarded release push path used by git rel.

scripts/tag_guard.py

Validate that a proposed release tag matches the expected vX.Y.Z pattern.

scripts/changelog_guard.py

Validate that CHANGELOG.md is structurally consistent with the latest reachable release tag.

scripts/release_system_selfcheck.py

Run a read-only consistency check of the installed local release tooling.

scripts/evaluate_agent_efficiency_retrieval.py

Agent-efficiency utilities share one disposable scratch root. Set CODIRA_BENCHMARK_TEMP_ROOT to override it; the directory is created when the utilities are imported. Otherwise they use the existing workstation scratch directory, or the platform temporary directory on other machines, including CI. Choose a short path for Unix-socket-based probes. Durable evidence destinations are configured separately and do not use this scratch root.

Measure source-reference retrieval against the frozen Sentinel panel without model calls. Run from the repository root with the exact admitted Click fixture revision specified by benchmarks/agent-efficiency/fixtures/click-public.json:

uv run python scripts/evaluate_agent_efficiency_retrieval.py \
  --source /path/to/frozen/click \
  --panel benchmarks/agent-efficiency/panels/sentinel-retrieval-v1.json \
  --output '.artifacts/analysis/sentinel-retrieval-<fresh-identity>.json'

The script exports and indexes a disposable fixture under the designated scratch root. Its durable receipt contains reference recall/precision, first-reference rank, negative controls, payload bytes and provenance. The reference set is narrow: these metrics do not establish general relevance or paid task efficacy. Use a fresh output path; retain the receipt with its investigation.

scripts/python_line_inventory.py

List Python files above a physical code-line threshold, separating code, comments/docstrings and blank lines. This is an optional maintenance utility:

uv run python scripts/python_line_inventory.py --threshold 1500
uv run python scripts/python_line_inventory.py src scripts --threshold 500 --format json

It reads the selected roots and writes the inventory to stdout. Counts guide maintenance review; they are not a code-quality gate.

Retired conformance probes

The one-off Phase 0 and Phase 4 paid probe executables were retired after their conformance work completed. Offline manifest, command, fixture and executable checks remain in tests/test_agent_efficiency_phase0.py. New paid work follows the campaign factory and current preparation procedure; historical commands and evidence can be recovered through the cleanup register.

scripts/check_docs.py

Run uv run python scripts/check_docs.py from the checkout root. This builds the site using mkdocs.yml and strict warning handling, with generated HTML placed in a temporary directory and removed automatically. The quality preflight, full validation gate and git rel all invoke this check. Local artifact paths belong in code spans; source-file links should use immutable GitHub URLs.