Skip to content

Follow-up: complete portable pstack conformance hardening #2

Closed

Description

@Go7hic

Context

PR #1 adds a portability audit and fixes the highest-impact runtime-neutrality problems. This issue tracks the remaining work that should be handled in smaller, reviewable changes.

P0: make the new audit the source of truth

  • Review every repository-wide warning produced by python3 scripts/audit_portability.py.
  • Classify each warning as real portable debt, intentional adapter-specific content, or a false-positive pattern.
  • Extend SCAN_EXCLUDES only for content whose purpose is explicitly runtime-specific; do not suppress portable skill debt.
  • Add regression fixtures for every portability pattern before expanding the regex set.

P1: remove remaining vendor-shaped semantics

  • Remove concrete model identifiers from all remaining portable skills, playbooks, and shared prompt references. Use model_role and the active adapter.
  • Replace ambiguous mechanically generated phrases such as “explore / implement helper” with the one capability the step actually needs.
  • Remove runtime fields such as helper type names, background flags, and read-only flags from portable bodies. Keep them inside adapters.
  • Audit prompt templates under skills/*/references/, not only top-level SKILL.md files.
  • Audit every page under docs/guide/ for Cursor-only installation, model-path, transcript, automation, control-surface, and mode-lifetime assumptions.

P1: harden the import pipeline

  • Refactor scripts/port_to_portable.py away from broad word replacement for Task and other overloaded terms.
  • Use targeted phrase transforms plus an explicit list of hand-maintained files.
  • Make the import pipeline idempotent and fail when a second run changes generated output.
  • Emit a machine-readable port report listing every transformation by file and rule.
  • Run the strict audit automatically after both port passes.
  • Add semantic review fixtures for how, why, architect, arena, swarm, interrogate, reflect, and poteto-mode.

P1: host conformance matrix

Create smoke tests or eval fixtures for each supported host:

  • Claude Code
  • Codex
  • OpenCode
  • Droid / Factory
  • Generic Agent Skills host with a spawn tool
  • Generic single-agent host without a spawn tool

Each host should cover:

  • /how simple path: no unnecessary fan-out
  • /how complex path: 2–4 independent explorers
  • /arena: isolated candidates, cross-judge, synthesis, verification
  • /interrogate: independent reviewers and lead judgment
  • feature implementation: disjoint worker scope and lead diff review
  • bug fix: reproduce, root cause, fix, same-surface verification
  • fallback behavior when model selection or helper spawning is denied
  • no invented tool parameters from another host

P1: mode and session lifecycle

  • Document the exact lifecycle guarantee for every host: current turn, current conversation, post-compaction, and new session.
  • Add optional host-native bootstrap/reminder integrations where supported.
  • Ensure Session pickup can recover from repository evidence and decision trails without relying on hidden conversation memory.
  • Never claim sticky mode on a host that cannot enforce it.

P2: upstream synchronization and provenance

  • Add an upstream manifest containing repository, path, commit SHA, sync date, and portable patch version.
  • Add a script that fetches or accepts an upstream checkout, imports it, runs both passes, refreshes mirrors, and produces a semantic-review checklist.
  • Generate an upstream-to-portable coverage report instead of maintaining counts manually.
  • Consider a scheduled workflow that opens a draft sync PR without auto-merging it.
  • Keep NOTICE and attribution updates part of the sync check.

P2: reduce duplicated sources

  • Decide whether mirrored playbooks and adapters should remain committed artifacts or be generated during packaging.
  • When retained, provide one sync-mirrors command and keep the byte-equality audit.
  • When generated, ensure skills.sh still discovers both entry layouts correctly.

P2: formalize adapter contracts

  • Version the capability contract.
  • Define required, optional, and degradable capabilities per adapter.
  • Add adapter self-check guidance: available helper types, concurrency limit, model-selection support, external connectors, runtime control, and persistence.
  • Validate model override files against a documented schema and reject unknown roles or unavailable model identifiers.
  • Separate “tool access needed” from “write permission needed”; read-only intent must not accidentally remove connected evidence tools.

P2: workflow quality improvements

  • Add a clear Spec-compliance axis beside engineering-quality review where an originating spec exists.
  • Add explicit external-side-effect policy to PR, ticket, chat, deploy, and merge playbooks.
  • Add cost/concurrency budgets for multi-model panels and autonomous loops.
  • Add a lightweight mode for small tasks so rigor does not imply unnecessary fan-out.

Definition of done

  • Repository-wide non-strict audit has no unexplained warnings.
  • Strict changed-file audit passes for every pull request.
  • All supported-host conformance fixtures pass or document a deliberate degraded behavior.
  • Upstream import is repeatable, reviewable, pinned to a source revision, and cannot silently introduce vendor-specific semantics.

Activity

  1. self-assigned this
    on Aug 7, 2026
  2. Go7hic commented on Aug 7, 2026

    @Go7hic
    OwnerAuthor

    PR #1 has now completed the audit-baseline work that was originally listed as P0/P1 cleanup:

    • full repository portability audit: 0 error(s), 0 warning(s)
    • strict changed-file audit: 0 error(s), 0 warning(s)
    • concrete Cursor model/helper leakage removed from audited portable content
    • fixed transcript-history paths removed
    • guide and playbook references to Cursor-only built-ins and external cleanup plugins removed or converted to capability/fallback language
    • mirrored playbooks remain byte-identical

    The remaining high-value work in this issue now starts with:

    1. hardening scripts/port_to_portable.py and adding idempotency/transformation reports;
    2. building the multi-host conformance matrix;
    3. formalizing mode/session lifecycle per host;
    4. pinning upstream provenance and automating reviewable syncs;
    5. versioning and validating adapter/model-role contracts.

    Keep the completed checklist items in the issue for provenance; use this comment as the current resume point.

  3. Go7hic commented on Aug 8, 2026

    @Go7hic
    OwnerAuthor

    Status update after PR #1 merge

    PR #1 is merged. Follow-up PR for the first post-merge slice is open: audit coverage + reflect path neutrality + GitHub-first install docs.

    Done in the portable baseline (PR #1)

    • Capability-first entry/workflow skills and selected playbooks
    • Structural audit + mirror equality + CI gate

    Corrected resume point

    P1 vendor-semantic cleanup is not complete. The audit now surfaces remaining debt instead of reporting a false clean bill:

    1. Rewrite orchestrate / shipping / autopilot-full / autopilot-stack to capability language (currently 16 mirrored warnings).
    2. Harden scripts/port_to_portable.py (idempotency + transform report).
    3. Host conformance matrix.
    4. Mode/session lifecycle per host.
    5. Upstream provenance + reviewable sync.
    6. Versioned adapter / model-role contracts.
    7. Semantic rewrites for arena / interrogate (and prompt templates under references/).

    Keep checklist history for provenance; use this comment as the current resume point.

  4. Go7hic commented on Aug 8, 2026

    @Go7hic
    OwnerAuthor

    Progress

    PR #3 now also clears the previously warned long-running playbooks:

    • orchestrate
    • shipping
    • autopilot-full
    • autopilot-stack

    Repository audit is back to 0 error(s), 0 warning(s) with the expanded pattern set.

    Remaining high-value work

    1. Harden scripts/port_to_portable.py (idempotency + transform report).
    2. Host conformance matrix.
    3. Mode/session lifecycle per host.
    4. Semantic rewrites for arena / interrogate (+ remaining prompt templates).
    5. Upstream provenance + reviewable sync.
    6. Versioned adapter / model-role contracts.
  5. Go7hic commented on Aug 8, 2026

    @Go7hic
    OwnerAuthor

    PR #3 now also includes capability-first rewrites for arena and interrogate.

    Remaining high-value work:

    1. Harden scripts/port_to_portable.py (idempotency + transform report).
    2. Host conformance matrix.
    3. Mode/session lifecycle per host.
    4. Upstream provenance + reviewable sync.
    5. Versioned adapter / model-role contracts.
    6. Remaining prompt-template / leaf-skill pass as needed.
  6. Go7hic commented on Aug 8, 2026

    @Go7hic
    OwnerAuthor

    Unified in-repo close-out (PR #3)

    Re-audited issue #2 and landed the remaining in-repo work in one pass instead of skill-by-skill edits.

    Now done in-repo

    • P0 audit SoT + expanded patterns/fixtures (still 0/0)
    • Leaf thin portability blocks + canonical runtime.md
    • Import pipeline hardened (port_to_portable.py targeted, idempotent, reported)
    • Mirror sync command + upstream manifest + import helper
    • Capability contract v1.0.0 + adapter self-check / read-only-vs-tool-access notes
    • Host lifecycle matrix + workflow-quality defaults
    • Model override schema + validator
    • Conformance matrix scaffold + semantic review checklist
    • Remaining playbook rewrite artifacts / subagent-shaped wording cleaned

    Still deferred (live hosts only)

    1. Fill scripts/fixtures/conformance/HOST_MATRIX.md with real runs
    2. Certify sticky-mode behavior empirically per host
    3. Validate concrete model IDs against live host model catalogs
    4. Optional scheduled draft sync PR automation

    Use this comment as the current resume point for host-runtime work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions