diff --git a/.cspell.yml b/.cspell.yml index 631cb32d..dd58c4e3 100644 --- a/.cspell.yml +++ b/.cspell.yml @@ -20,6 +20,7 @@ ignorePaths: # ignoreRegExpList: # - CodeBlock words: + - afdocs - backstore - Chalin - CLOTributor @@ -30,6 +31,8 @@ words: - github - keda - knative + - llms + - llmstxt - kedacore - krator - krook diff --git a/docs/analysis/ai-harness/_github/copilot-instructions.md b/docs/analysis/ai-harness/_github/copilot-instructions.md new file mode 100644 index 00000000..af427936 --- /dev/null +++ b/docs/analysis/ai-harness/_github/copilot-instructions.md @@ -0,0 +1,541 @@ +# CNCF TechDocs Analysis Copilot Instructions + +The CNCF TechDocs analysis focuses on evaluating the quality and completeness of +project documentation. Human reviewers will continue performing this analysis. +The goal of this project is to provide a harness and workflow for writers to use +with Copilot. As such, predefined prompts are provided to guide Copilot in +providing analysis responses. Predefined prompts should also alleviate +inconsistencies resulting from having the writer craft the prompt. + +Given that the analysis writers are providing their own analysis, the Copilot +analysis must be written as to require minimal editing by the writers, such as +making fragments into complete sentences, or adding context to a comment. + +## Analysis Criteria + +The focus is to analyze documentation for CNCF projects and to provide comments, +recommendations, and issues. + +Review sources: + +- Analysis criteria for evaluating content: + https://github.com/cncf/techdocs/blob/main/docs/analysis/criteria.md. +- Template for the analysis report: + https://github.com/cncf/techdocs/blob/main/docs/analysis/templates/analysis.md + +## Analysis Guidelines + +Things to keep in mind while doing the analysis: + +Look for: + +- Quick wins – low-effort changes that would have a major impact. +- Large, systemic issues that you can organize a number of issues around. +- The two or three most important issues that impede documentation + effectiveness. +- Anything the project does exceptionally well. We can call these out as + examples in other evaluations! + +Don't get bogged down in detail when writing comments. Include enough detail +that you can describe how to implement a suggested solution. A sentence or two +is sufficient for most issues. + +Keep in mind the overall goal of the technical documentation: to make its users +more effective. Focus on issues that get in the way of that goal. + +It is not necessary to come up with a recommendation in every case, especially +for elements that are satisfactory or if a recommendation would result in +minimal improvement. + +Analysis criteria is further specified in the prompt files under +`.github/prompts/`. + +## Analysis responses style + +Readers of the analysis are subject-matter experts in the project's ecosystem +(named in the "Current Repository" section) who want to improve the +documentation quickly. Write for them: be concise and actionable, and skip +explanations they don't need. + +- Write in complete sentences, using active voice and the imperative mood. +- Fragments are fine inside a bulleted list that follows an introductory + sentence. +- Explain any term or acronym that a senior engineer in the project's ecosystem + might not know. +- Don't list technologies in parentheses; readers already know them. +- Use parentheses only when the content is needed to understand the + recommendation. +- When a sentence would list four or more items, end the paragraph with an + introductory sentence followed by a bulleted list of the items. +- You can refer to Markdown files instead of page titles; but unless the name is + obvious, also provide the page title. No need to include relative paths as + long as the page title is clear. +- Keep paragraphs under about 4 sentences; split longer ones. +- Treat any paragraph limit in a prompt as a maximum, not a target. +- If you're unsure whether something is technically wrong, phrase it as a + question for the SME rather than asserting it. +- Follow the project's documentation conventions as described in "Current + Repository". For Kubernetes-ecosystem projects that means UpperCamelCase for + API objects (Pod, Deployment) and code formatting for commands, fields, and + file names. + +## Analysis Criteria and Predefined Prompts + +The analysis has three sections (Project Documentation, Contributor +Documentation, and Website & Infrastructure). Each section contains multiple +areas, and each area has its own criteria and prompts for evaluation. + +The AI assisted analysis harness is organized into two types of files: criteria +and prompt Markdown files. These files are organized further by `areas` and +`sections` folders. These files live under `.github/criteria/areas`, +`.github/criteria/sections`, `.github/prompts/areas`, and +`.github/prompts/sections` respectively. Generated output lives in +`.github/output/analysis` and `.github/output/issues`. + + + +```text +.github/ + copilot-instructions.md + criteria/ + areas/.md # Per-area data: display name, file stem, questions, comment guidance + sections/
.md # Per-section data: display name, slug, member areas and comment-file stems + prompts/ + areas/ + answers.prompt.md # Shared engines + comment.prompt.md + recommendations.prompt.md + full.prompt.md + -.prompt.md # Per-area wrappers + sections/ + section-comment.prompt.md # Shared engine +
-comment.prompt.md # Per-section wrappers + issues/ + rank-recommendations.prompt.md # Ranks recommendations into input/ai-ranked-recommendations.csv + create-issues.prompt.md # Generates numbered issue files from ranked recommendations + create-umbrella-issues.prompt.md # Groups issue files into umbrella issues + templates/ + doc-issue.md # Documentation issue template (replace with the project's own) + umbrella-issue.md # Umbrella issue template + input/ + ai-ranked-recommendations.csv # Written by /rank-recommendations, read by /create-issues + output/ + analysis/--.md # Per-area outputs + analysis/--comment.md # Section-level comments + issues/-.md # Generated standalone issue files + issues/U--.md # Generated umbrella issue files + issues-coverage.md # Issue list and recommendation-to-issue mapping +``` + + + +Criteria files for an area define the per-area data: display name, output-file +stem, question set, and comment guidance. The area name is the criteria filename +without `.md` (for example, `.github/criteria/areas/information-architecture.md` +defines the `information-architecture` area). + +Criteria files for a section define the section's display name, slug, and member +areas (with the stem of each area's comment file). + +Prompt files for areas are organized into three layers so the per-area questions +(data) are separated from the task logic (verbs): + +- Per-area wrappers (`-.prompt.md`) — the entry points you invoke, + for example `/information-architecture-answers`. Each wrapper simply binds one + criteria area to one shared engine. +- Shared engines (`answers.prompt.md`, `comment.prompt.md`, + `recommendations.prompt.md`, all in `.github/prompts/areas/`) — the reusable + procedure for each output type. They are project-agnostic: the project name + and documentation label are read from the "Current Repository" section below, + so the same engines work in any repository. + +Prompt files for sections follow a similar structure, with section-level +wrappers and an engine (`section-comment.prompt.md`), all in +`.github/prompts/sections/`, to synthesize comments from the member areas. + +For areas only, there is also an optional composite engine `full.prompt.md`, +that runs the `answers`, `comment`, and `recommendations` engines for one area +in that order (recommendations depends on the answers and comment files existing +first). A per-area `full` wrapper (for example, +`/information-architecture-full`) invokes it to produce all three output files +in a single run. + +The `section-comment.prompt.md` engine writes a two-to-five paragraph overall +comment for one section by synthesizing that section's existing per-area comment +files (it does not re-analyze the documentation), and it prepends a summary +ratings table scraped from each area comment's `Rating:` line. Its data lives in +`.github/criteria/sections/
.md`, which lists the section's display +name, slug, and member areas with the stem of each area's comment file. A +per-section wrapper (for example, `/project-documentation-comment`) invokes it. +Because the per-area comment files are a precondition, run the relevant +`-comment` prompts first. By default the output is written to +`.github/output/analysis/--comment.md` (for example, +`kubevirt-project-documentation-comment.md`), overridable with a `title` input. + +Each criteria area needs three wrapper files (`answers`, `comment`, +`recommendations`), and the wrappers are mechanical: they reference the area +name and embed the display name in the description. A `full` wrapper is optional +per area. + +By default a prompt writes to `--.md` in the +`.github/output/analysis` directory (for example, +`kubevirt-information-architecture-answers.md`). Provide an optional `title` +input to override the filename. To add or change questions for an area, edit +only its `.github/criteria/areas/.md` file. + +Output types: `answers`, `comment`, `recommendations` (plus the optional `full` +composite). At the section level there is one additional output type, the +section-level `comment` produced by the `section-comment` engine and its +per-section wrappers. + +The analysis document has three sections: Project Documentation, Contributor +Documentation, and Website & Infrastructure. Each section contains multiple +criteria areas that are evaluated separately, and the results are synthesized +into section-level overall comments. + +Sections (for the section-level overall comment): + + + +| Section | Section slug | Member areas (comment-file stems) | +| ------------------------- | ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Project Documentation | project-documentation | information-architecture, new-user, content-maintainability, content-creation-process, inclusive-language | +| Contributor Documentation | contributor-documentation | communication-methods-docd, beginner-issue-backlog, new-contributor, project-governance | +| Website & Infrastructure | website-infrastructure | single-source-requirement, website-requirements, usability-accessibility-devices, branding-design, case-studies, seo-analytics, maintenance-planning, agent-readiness | + + + +Areas (the Area column is the criteria filename under `.github/criteria/areas/` +and the prefix of the per-area wrapper prompts): + + + +| Area | Display Name | Output File Stem | Section in analysis.md | +| -------------------------------- | --------------------------------------- | ------------------------------- | ------------------------- | +| agent-readiness | AI Agent Readiness | agent-readiness | Website & Infrastructure | +| beginner-issue-backlog | Beginner Friendly Issue Backlog | beginner-issue-backlog | Contributor Documentation | +| branding-design | Branding and Design | branding-design | Website & Infrastructure | +| case-studies | Case Studies | case-studies | Website & Infrastructure | +| communication-methods-documented | Communication Methods Documented | communication-methods-docd | Contributor Documentation | +| content-creation-process | Content Creation Process | content-creation-process | Project Documentation | +| content-maintainability | Content Maintainability | content-maintainability | Project Documentation | +| inclusive-language | Inclusive Language | inclusive-language | Project Documentation | +| information-architecture | Information Architecture | information-architecture | Project Documentation | +| maintenance-planning | Maintenance Planning | maintenance-planning | Website & Infrastructure | +| new-contributor-content | New Contributor Getting Started Content | new-contributor | Contributor Documentation | +| new-user-content | New User Content | new-user | Project Documentation | +| project-governance-documentation | Project Governance Documentation | project-governance | Contributor Documentation | +| seo-analytics | SEO, Analytics, and Site Search | seo-analytics | Website & Infrastructure | +| single-source-requirement | Single-Source Requirement | single-source-requirement | Website & Infrastructure | +| usability-accessibility-devices | Usability, Accessibility and Devices | usability-accessibility-devices | Website & Infrastructure | +| website-requirements | Website Requirements | website-requirements | Website & Infrastructure | + + + +### Example: invoking a predefined prompt + +To have Copilot answer the information architecture questions, invoke the +per-area wrapper as a slash command in the current directory: + +```text +run /information-architecture-answers +``` + +This runs the `information-architecture` questions through the shared `answers` +engine and writes the result to the default file +`.github/output/analysis/kubevirt-information-architecture-answers.md`. + +To override the output filename, pass a `title` input: + +```text +run /information-architecture-answers title: my-analysis +``` + +That writes the answers to `.github/output/analysis/my-analysis.md` instead of +the default filename. The same pattern applies to every area and output type — +for example, `/inclusive-language-recommendations` or +`/new-user-content-comment`. + +## Analysis.md and Output Files Mapping + +Predefined prompts create Markdown files of their AI generated output. The +content from these files, without the temporary H1 headings at the top, is +copied into specific locations in the analysis.md file. + + + +| Output file | Location in analysis.md | +| ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `-project-documentation-comment.md` | **Start**: After the ratings table that follows the first paragraph that follows the "Project Documentation" H2.
**End**: Before the sentence "The following sections contain assessments of each element of the Project Documentation rubric." | +| `-contributor-documentation-comment.md` | **Start**: After the ratings table that follows the first paragraph that follows the "Contributor Documentation" H2.
**End**: Before the sentence "The following sections contain assessments of each element of the Contributor Documentation rubric." | +| `-website-infrastructure-comment.md` | **Start**: After the ratings table that follows the first paragraph that follows the "Website & Infrastructure" H2.
**End**: Before the sentence "The following sections contain assessments of each element of the Website & Infrastructure rubric." | +| `--answers.md` | **Start**: After the first paragraph that follows the H3 heading for an area, e.g. "### Information Architecture". That first paragraph should end with the sentence: "We evaluate on the following:".
**End**: Before the H4 "Comment" heading. | +| `--comment.md` | **Start**: After the H4 "Comment" heading.
**End**: Before the next H3 or H2 heading.
**Note**: Currently, only the Strengths and Weaknesses of the area comments are pasted into analysis.md.
**Note**: The rating for the area (last line in the output) is later manually copied by a writer into the area summary ratings table for the section. | +| `--recommendations.md` | **Start**: Under the H2 headings "Project documentation recommendations", "Contributor documentation recommendations", or "Website & infrastructure recommendations" and under the H3 heading for the area.
**End**: Before the next H3 or H2 heading. | + + + +## Recommendations and Issues + +The `/rank-recommendations` prompt collects the recommendations from the +per-area recommendations files in `.github/output/analysis/` and ranks them into +`.github/input/ai-ranked-recommendations.csv`, with the following columns: + +- `Section` - Either the "Project Documentation", "Contributor documentation", + or "Website & Infrastructure" section. +- `Area` - The specific area within the section that the recommendation pertains + to. The area names in the CSV are the display names used in the analysis, not + the criteria area filenames. +- `Recommendation` - The AI-written recommendation describing the action that + should be taken. +- `Suggested Priority` - The ranking priority: `1 - Urgent`, `2 - High`, or + `3 - Backlog`. These priorities were only for the recommendation ranking and + may be adjusted when creating issues. + +The priority labels used for issues are `urgent`, `high`, and `backlog`, +matching the CSV ranking without its number prefix. The upload process maps them +to the target repository's own labels using the "Priority label mapping" table +in the "Current Repository" section. Assume all three mapped labels exist when +writing the issue files; the pre-flight checklist in the harness `index.md` +covers creating them. + +Every issue also gets the `help wanted` label. It exists in the repository and +signals that maintainers are seeking community help; CNCF uses it to surface +documentation tasks on [CLOTributor](https://clotributor.dev). + +Issues are a subset of the recommendations that have been identified as problems +or areas needing attention. Not all recommendations are issues; some may be +suggestions for improvement that do not indicate a problem. Copilot is to create +the issues based on the ranked recommendations, and then group them into +umbrella issues. Copilot may change the priority of the issues as it sees fit. +As a reference, see the umbrella issues in +[CNCF TechDocs Assistance issues tracker #221](https://github.com/cncf/techdocs/issues/221), +the KEDA +[standalone issue example](https://github.com/kedacore/keda-docs/issues/1363), +and the KEDA +[umbrella issue example](https://github.com/kedacore/keda-docs/issues/1361). The +project's own assistance request is listed in the "Current Repository" section. + +Copilot will create Markdown files of the issues with updated priorities as +applicable. These files will be stored in the `.github/output/issues/` +directory. A process will then upload them to the project's repository, so every +file in that directory must be a complete, uploadable issue. Issue creation runs +in two passes, each with its own prompt: + +1. `/create-issues` writes one file per standalone issue, numbered sequentially. +2. `/create-umbrella-issues` reads those files and writes umbrella issues that + link to them. + +In writing the issues, Copilot should use the following guidelines: + +- Source the issues only from the ranked recommendations CSV and the per-area + recommendations files in `.github/output/analysis/`. Do not add issues from + general knowledge of the project's repositories; that over-generates issues + the assessment did not identify. +- Ensure that the issues are clearly described and actionable. Follow the + headings of the documentation issue template under `.github/templates/` + (`doc-issue.md` by default; the "Current Repository" section names the file). + Replace the default with the project's own documentation issue format when it + has one. +- Each issue should take about two to four hours for a technical writer or + engineer with project context to complete. +- Small tasks can be consolidated into one issue where appropriate. Avoid + flooding the project: aim for far fewer issues than recommendations. +- Umbrella issues group three to ten related issues. Follow the + `.github/templates/umbrella-issue.md` template: an `## Umbrella Issue` + overview and a `## Sub-issues` list that links to the sub-issue files. The + umbrella does not contain the sub-issue content. +- Everything outside the front matter is the issue description and must be + GitHub-flavored Markdown. +- Ensure the following front matter (metadata) is included at the top of each + issue file. + - Create a title and append the simulated issue number as `#{issue number}`. + It becomes the GitHub issue title. + - Set `labels` to the priority label (`urgent`, `high`, or `backlog`) plus + `help wanted`. Labels become GitHub issue labels. + - Set `repo` to the issues repository URL from the "Current Repository" + section. + + ```yaml + --- + title: Add a user-facing troubleshooting page #9007 + labels: [high, help wanted] + repo: https://github.com///issues + --- + ``` + +### Issue numbers + +Issue numbers simulate GitHub issue numbers so that files can reference each +other before upload. They are five digits, start at 9001, and increment by one +for each issue in the order written. Umbrella issues continue the sequence after +the last standalone issue. The `## Issues` table in +`.github/output/issues-coverage.md` maps each number to its title, label, and +file, and is how the umbrella prompt finds the next free number. + +In an umbrella issue, each sub-issue is one list item whose link text is the +sub-issue title and number and whose target is the sub-issue file, relative to +the umbrella file: + +```markdown +- [Reorganize Concepts #9938](9938-reorganize-concepts.md) +``` + +#### Issue file names + +Save the issue files to the `.github/output/issues/` directory. Name standalone +issue files `{issue-number}-{condensed-issue-title}.md` and umbrella issue files +`U-{issue-number}-{condensed-issue-title}.md`. The condensed title is a +lowercase, hyphenated version of the issue title, no more than about five words: + +- Keep the action verb and the main subject noun; drop articles, conjunctions, + and secondary clauses. +- Name the topic (for example, `navigation`, `footer`, `virtctl`), not the + analysis area it came from. +- When the title lists several changes, use a collective noun such as + `improvements` or `updates` instead of listing them. +- Strip punctuation from version numbers (`v1.9` becomes `v19`). + +Examples of a file name built from a full title: + + + +| Issue title | File name | +| ----------------------------------------------------------------- | --------------------------------------- | +| Add a user-facing troubleshooting page #9007 | `9007-add-troubleshooting-page.md` | +| Move Release Notes out of the main navigation #9004 | `9004-move-release-notes.md` | +| Document the v1.9 VirtualMachineBackup API #9012 | `9012-document-v19-backup-api.md` | +| Create a new-user getting started path #9061 | `U-9061-create-getting-started-path.md` | +| Fix navigation gaps and reorganize information architecture #9062 | `U-9062-fix-navigation-gaps.md` | + + + +To archive a previous run, move its files into a sub-folder of +`.github/output/issues/` (for example, `prev_1/`). The prompts ignore +sub-folders and the folder's `readme.md`. + +### Rank recommendations AI prompt + +The prompt lives in `.github/prompts/issues/rank-recommendations.prompt.md`. Run +it after all seventeen area recommendations files exist: + +```text +run /rank-recommendations +``` + +It reads every `-*-recommendations.md` file in +`.github/output/analysis/`, assigns each recommendation a `1 - Urgent`, +`2 - High`, or `3 - Backlog` priority by impact and effort, and writes +`.github/input/ai-ranked-recommendations.csv`. Review the ranking before running +`/create-issues`; to adjust it, re-run with the corrections in the same message. + +### Create issues AI prompt + +The prompt lives in `.github/prompts/issues/create-issues.prompt.md` and is the +single source of truth. Run it as a slash command from the repository root: + +```text +run /create-issues +``` + +It deletes issue files from a previous run, writes the numbered standalone issue +files, and writes `.github/output/issues-coverage.md` with the issue table, the +recommendation-to-issue mapping, priority adjustments, and counts. + +To re-run after coaching Copilot on scope, run the command again and describe +the corrections in the same message, for example +`run /create-issues — split the "Getting started" issue into two`. + +### Create umbrella issues AI prompt + +The prompt lives in `.github/prompts/issues/create-umbrella-issues.prompt.md`. +Run it in a new session after `/create-issues` has finished and the writer has +reviewed the standalone issues: + +```text +run /create-umbrella-issues +``` + +It reads the standalone issue files and the `## Issues` table in +`issues-coverage.md`, groups the issues into themes of three to ten, writes the +`U-` umbrella files with the next free issue numbers, and appends the umbrellas +to the coverage report. It does not modify standalone issue files or re-read the +recommendations; if an issue needs re-scoping, re-run `/create-issues` first. + +To adjust the grouping, run the command again with the corrections in the same +message, for example +`run /create-umbrella-issues — merge the analytics and site-ownership umbrellas`. + +## Current Repository + +This section is the single place the engines and issue prompts read project +context from. Replace every `` placeholder before running any +prompt; the issue prompts stop if a placeholder is still present. See +`examples/kubevirt/current-repository.md` in the harness for a completed +example. + + + +- Project name: + +- Documentation label: + +- Ecosystem: + +- Main repository: +- Secondary repositories: + +- Published documentation URL: +- Project website URL: +- Issues repository URL: + +- Issue template file: + +- CNCF project page URL: + +- Assistance-request issue: + + + +Copilot shall output Markdown files from the AI-assisted analysis to the +`.github/output/analysis` directory and issue files to the +`.github/output/issues` directory in the current repository. + +### Priority label mapping + +The upload process maps the harness labels to the issue repository's labels. +Confirm each target label exists before uploading. + +| Harness label | Repository label | +| ------------- | ----------------------------------------------- | +| `urgent` | | +| `high` | | +| `backlog` | | +| `help wanted` | `help wanted` | + +### What This Repo Is + + + +### Content Architecture + + + +```text +docs/ + index.md +``` + +### Key Conventions + + + +#### No commits + +This work is solely for documentation analysis. Copilot makes no commits to the +documentation or application code in this repository; it writes only to +`.github/input` and `.github/output`. diff --git a/docs/analysis/ai-harness/_github/criteria/areas/agent-readiness.md b/docs/analysis/ai-harness/_github/criteria/areas/agent-readiness.md new file mode 100644 index 00000000..b06deea1 --- /dev/null +++ b/docs/analysis/ai-harness/_github/criteria/areas/agent-readiness.md @@ -0,0 +1,43 @@ +# AI agent readiness - Website and Infrastructure + +- Display name: AI Agent Readiness +- File stem: agent-readiness + +## Questions + +- Does the documentation site publish an `llms.txt` file at the site root or + docs root? +- Does `llms.txt` follow the llmstxt.org structure (H1 title, blockquote + summary, H2 sections of link lists)? +- Do all links in `llms.txt` resolve? +- Do links in `llms.txt` point to Markdown versions of pages? +- Is `llms.txt` generated at build time or otherwise kept in sync with the + published site? +- Do HTML and Markdown pages include a directive near the top pointing agents to + `llms.txt`? +- Is a Markdown version of each page served at the same URL with `.md` appended + or substituted? +- Does the Markdown version of each page contain the same content as the HTML + version? +- Are pages rendered server-side or statically generated rather than assembled + by client-side JavaScript? +- Does the main content of each page start near the top of the response rather + than behind large navigation or script payloads? +- Are pages small enough to avoid truncation in an agent's context window? +- Do tabbed components and code fences serialize cleanly to Markdown? +- Is the documentation publicly reachable without authentication gates? +- Is the documentation reachable without CDN bot challenges? +- Do pages return correct HTTP status codes? +- Do moved pages use standard HTTP redirects? +- Does the project monitor its CLOMonitor "Agent readiness" checks? +- Does the project run `afdocs check` against the site? +- Does the project run agent readiness checks in CI? + +## Comment guidance + +- Additional instructions: Reference sources for this area are the llms.txt + specification (https://llmstxt.org/), the CLOMonitor agent readiness checks + (https://clomonitor.io/docs/topics/checks/#agent-readiness), and the AFDocs + Agent-Friendly Documentation Spec (https://afdocs.dev/). Where possible, + verify by fetching the live site (`/llms.txt`, a page URL with `.md` appended) + rather than relying on the repository source alone. diff --git a/docs/analysis/ai-harness/_github/criteria/areas/beginner-issue-backlog.md b/docs/analysis/ai-harness/_github/criteria/areas/beginner-issue-backlog.md new file mode 100644 index 00000000..c9441830 --- /dev/null +++ b/docs/analysis/ai-harness/_github/criteria/areas/beginner-issue-backlog.md @@ -0,0 +1,16 @@ +# Beginner friendly issue backlog - Contributor Documentation + +- Display name: Beginner Friendly Issue Backlog +- File stem: beginner-issue-backlog + +## Questions + +- Are docs issues well-triaged? +- Is there a clearly marked way for new contributors to make code or + documentation contributions (i.e. a “good first issue” label)? +- Are issues well-documented (i.e., more than just a title)? +- Are issues maintained for staleness? + +## Comment guidance + +- Additional instructions: None. diff --git a/docs/analysis/ai-harness/_github/criteria/areas/branding-design.md b/docs/analysis/ai-harness/_github/criteria/areas/branding-design.md new file mode 100644 index 00000000..35ee03c9 --- /dev/null +++ b/docs/analysis/ai-harness/_github/criteria/areas/branding-design.md @@ -0,0 +1,15 @@ +# Branding and design - Website and Infrastructure + +- Display name: Branding and Design +- File stem: branding-design + +## Questions + +- Is there an easily recognizable brand for the project (logo + color scheme) + clearly identifiable? +- Is the brand used across the website consistently? +- Is the website’s typography clean and well-suited for reading? + +## Comment guidance + +- Additional instructions: None. diff --git a/docs/analysis/ai-harness/_github/criteria/areas/case-studies.md b/docs/analysis/ai-harness/_github/criteria/areas/case-studies.md new file mode 100644 index 00000000..6aff04fe --- /dev/null +++ b/docs/analysis/ai-harness/_github/criteria/areas/case-studies.md @@ -0,0 +1,17 @@ +# Case studies - Website and Infrastructure + +- Display name: Case Studies +- File stem: case-studies + +## Questions + +- Are there case studies available for the project and are they documented on + the website? +- Are there user testimonials available? +- Is there an active project blog? +- Are there community talks for the project and are they present on the website? +- Is there a logo wall of users/participating organizations? + +## Comment guidance + +- Additional instructions: None. diff --git a/docs/analysis/ai-harness/_github/criteria/areas/communication-methods-documented.md b/docs/analysis/ai-harness/_github/criteria/areas/communication-methods-documented.md new file mode 100644 index 00000000..5b213a7a --- /dev/null +++ b/docs/analysis/ai-harness/_github/criteria/areas/communication-methods-documented.md @@ -0,0 +1,17 @@ +# Communication methods documented - Contributor Documentation + +- Display name: Communication Methods Documented +- File stem: communication-methods-docd + +## Questions + +- Is there a Slack/Discord/Discourse/etc. community and is it prominently linked + from your website? +- Is there a direct link to your GitHub organization/repository? +- Are weekly/monthly project meetings documented? Is it clear how someone can + join those meetings? +- Are mailing lists documented? + +## Comment guidance + +- Additional instructions: None. diff --git a/docs/analysis/ai-harness/_github/criteria/areas/content-creation-process.md b/docs/analysis/ai-harness/_github/criteria/areas/content-creation-process.md new file mode 100644 index 00000000..4d4f2754 --- /dev/null +++ b/docs/analysis/ai-harness/_github/criteria/areas/content-creation-process.md @@ -0,0 +1,19 @@ +# Content creation process - Project Documentation + +- Display name: Content Creation Process +- File stem: content-creation-process + +## Questions + +- Is there a clearly documented (ongoing) contribution process for + documentation? +- Does the code release process account for documentation creation & updates? +- Who reviews and approves documentation pull requests? +- Does the website have a clear owner/maintainer? + +## Comment guidance + +- Good examples: + - https://github.com/nats-io/nats-site/blob/master/MAINTAINERS.md + - https://thanos.io/tip/contributing/how-to-contribute-to-docs.md +- Additional instructions: None. diff --git a/docs/analysis/ai-harness/_github/criteria/areas/content-maintainability.md b/docs/analysis/ai-harness/_github/criteria/areas/content-maintainability.md new file mode 100644 index 00000000..85874b8c --- /dev/null +++ b/docs/analysis/ai-harness/_github/criteria/areas/content-maintainability.md @@ -0,0 +1,17 @@ +# Content maintainability - Project Documentation + +- Display name: Content Maintainability +- File stem: content-maintainability + +## Questions + +- Is the documentation searchable? +- Are there plans for localization/internationalization with regards to site + directory structure? Is a localization framework present? +- Is there a clearly documented method for versioning of content? + +## Comment guidance + +- Good examples: + - https://kubernetes.io/docs/ +- Additional instructions: None. diff --git a/docs/analysis/ai-harness/_github/criteria/areas/inclusive-language.md b/docs/analysis/ai-harness/_github/criteria/areas/inclusive-language.md new file mode 100644 index 00000000..83523dac --- /dev/null +++ b/docs/analysis/ai-harness/_github/criteria/areas/inclusive-language.md @@ -0,0 +1,17 @@ +# Inclusive language - Project Documentation + +- Display name: Inclusive Language +- File stem: inclusive-language + +## Questions + +- Are there any customer-facing utilities, endpoints, class names, or feature + names that use non-recommended words as documented by the + [Inclusive Naming Initiative](https://inclusivenaming.org) website? +- Does the project use language like "simple", "easy", etc.? + +## Comment guidance + +- Good examples: None. +- Additional instructions: Focus the comment on any inclusive language concerns + in the documentation. diff --git a/docs/analysis/ai-harness/_github/criteria/areas/information-architecture.md b/docs/analysis/ai-harness/_github/criteria/areas/information-architecture.md new file mode 100644 index 00000000..858e9ece --- /dev/null +++ b/docs/analysis/ai-harness/_github/criteria/areas/information-architecture.md @@ -0,0 +1,26 @@ +# Information architecture - Project Documentation + +- Display name: Information Architecture +- File stem: information-architecture + +## Questions + +- Is there high level conceptual content? +- Is the documentation feature complete? +- Are there step-by-step instructions documented for features in tasks and + tutorials? +- Are there any key features that are documented but missing task documentation? +- Is the “happy path” (most common use case) documented? +- Are tasks clearly named according to user goals? +- If the documentation doesn't suffice, is there a clear escalation path for + users needing more help? (FAQ, Troubleshooting) +- If the product exposes an API, is there a complete reference that includes + documented CLIs as applicable? +- Is content up to date and accurate? +- Does the documentation need restructuring? + +## Comment guidance + +- Good examples: + - https://prometheus.io/docs +- Additional instructions: None. diff --git a/docs/analysis/ai-harness/_github/criteria/areas/maintenance-planning.md b/docs/analysis/ai-harness/_github/criteria/areas/maintenance-planning.md new file mode 100644 index 00000000..587309f7 --- /dev/null +++ b/docs/analysis/ai-harness/_github/criteria/areas/maintenance-planning.md @@ -0,0 +1,18 @@ +# Maintenance planning - Website and Infrastructure + +- Display name: Maintenance Planning +- File stem: maintenance-planning + +## Questions + +- Is the website tooling well supported by the community (i.e., Hugo with the + Docsy theme) or commonly used by CNCF projects? +- Is there active cultivating website maintainers from within the community? +- Are site build times reasonable? +- Do site maintainers have adequate permissions? +- Is the website accessible via HTTPS? +- Does HTTP access, if any, redirect to HTTPS? + +## Comment guidance + +- Additional instructions: None. diff --git a/docs/analysis/ai-harness/_github/criteria/areas/new-contributor-content.md b/docs/analysis/ai-harness/_github/criteria/areas/new-contributor-content.md new file mode 100644 index 00000000..ec32bb11 --- /dev/null +++ b/docs/analysis/ai-harness/_github/criteria/areas/new-contributor-content.md @@ -0,0 +1,14 @@ +# New contributor getting started content - Contributor Documentation + +- Display name: New Contributor Getting Started Content +- File stem: new-contributor + +## Questions + +- Do you have a community repository or section on your website? +- Is there a document specifically for new contributors/your first contribution? +- Do new users know where to get help? + +## Comment guidance + +- Additional instructions: None. diff --git a/docs/analysis/ai-harness/_github/criteria/areas/new-user-content.md b/docs/analysis/ai-harness/_github/criteria/areas/new-user-content.md new file mode 100644 index 00000000..c048353d --- /dev/null +++ b/docs/analysis/ai-harness/_github/criteria/areas/new-user-content.md @@ -0,0 +1,21 @@ +# New user content - Project Documentation + +- Display name: New User Content +- File stem: new-user + +## Questions + +- Is “Getting started” clearly labeled? (e.g “Getting started”, “Installation”, + “First steps”, etc.) +- Is installation documented step-by-step? +- If needed, is guidance provided for multiple operating systems and platforms? +- Do users know where to go after reading the getting started guide? +- Is your new user content clearly discoverable, such as on the documentation + home page? +- Is there sample code or content that can easily be copy-pasted? + +## Comment guidance + +- Good examples: + - https://falco.org/docs/getting-started/ +- Additional instructions: None. diff --git a/docs/analysis/ai-harness/_github/criteria/areas/project-governance-documentation.md b/docs/analysis/ai-harness/_github/criteria/areas/project-governance-documentation.md new file mode 100644 index 00000000..9efe6cba --- /dev/null +++ b/docs/analysis/ai-harness/_github/criteria/areas/project-governance-documentation.md @@ -0,0 +1,12 @@ +# Project Governance Documentation - Contributor Documentation + +- Display name: Project Governance Documentation +- File stem: project-governance + +## Questions + +- Is project governance clearly documented? + +## Comment guidance + +- Additional instructions: None. diff --git a/docs/analysis/ai-harness/_github/criteria/areas/seo-analytics.md b/docs/analysis/ai-harness/_github/criteria/areas/seo-analytics.md new file mode 100644 index 00000000..49935d72 --- /dev/null +++ b/docs/analysis/ai-harness/_github/criteria/areas/seo-analytics.md @@ -0,0 +1,20 @@ +# SEO, analytics, and local site search - Website and Infrastructure + +- Display name: SEO, Analytics, and Site Search +- File stem: seo-analytics + +## Questions + +- Is analytics enabled for the production server? +- Is analytics disabled for all other deploys? +- If project is using Google Analytics, has it migrated to GA4? +- Can Page-not-found (404) reports easily be generated from site analytics? +- Is site indexing supported for the production server, while disabled for + website previews and builds for non-default branches? +- Is local intra-site search available from the website? +- Are the current custodian(s) of the analytics accounts (such as Google CSE) + documented? + +## Comment guidance + +- Additional instructions: None. diff --git a/docs/analysis/ai-harness/_github/criteria/areas/single-source-requirement.md b/docs/analysis/ai-harness/_github/criteria/areas/single-source-requirement.md new file mode 100644 index 00000000..bb893a34 --- /dev/null +++ b/docs/analysis/ai-harness/_github/criteria/areas/single-source-requirement.md @@ -0,0 +1,13 @@ +# Single source requirement - Website and Infrastructure + +- Display name: Single-Source Requirement +- File stem: single-source-requirement + +## Questions + +- Does the project have a single source for its documentation? If not, is there + a reason? + +## Comment guidance + +- Additional instructions: None. diff --git a/docs/analysis/ai-harness/_github/criteria/areas/usability-accessibility-devices.md b/docs/analysis/ai-harness/_github/criteria/areas/usability-accessibility-devices.md new file mode 100644 index 00000000..22465c1e --- /dev/null +++ b/docs/analysis/ai-harness/_github/criteria/areas/usability-accessibility-devices.md @@ -0,0 +1,18 @@ +# Usability, accessibility and devices - Website and Infrastructure + +- Display name: Usability, Accessibility and Devices +- File stem: usability-accessibility-devices + +## Questions + +- Is the website usable from mobile? +- Are doc pages readable? +- Are all / most website features accessible from mobile -- such as the top-nav, + site search and in-page table of contents? +- Are color contrasts significant enough for color-impaired readers? +- Are most website features usable using a keyboard only? +- Does text-to-speech offer listeners a good experience? + +## Comment guidance + +- Additional instructions: None. diff --git a/docs/analysis/ai-harness/_github/criteria/areas/website-requirements.md b/docs/analysis/ai-harness/_github/criteria/areas/website-requirements.md new file mode 100644 index 00000000..ca6c8668 --- /dev/null +++ b/docs/analysis/ai-harness/_github/criteria/areas/website-requirements.md @@ -0,0 +1,13 @@ +# Website requirements - Website and Infrastructure + +- Display name: Website Requirements +- File stem: website-requirements + +## Questions + +- Are most of the applicable CNCF Website Guidelines satisfied? See + https://github.com/cncf/techdocs/blob/main/docs/website-guidelines-checklist.md + +## Comment guidance + +- Additional instructions: None. diff --git a/docs/analysis/ai-harness/_github/criteria/sections/contributor-documentation.md b/docs/analysis/ai-harness/_github/criteria/sections/contributor-documentation.md new file mode 100644 index 00000000..51a50ec4 --- /dev/null +++ b/docs/analysis/ai-harness/_github/criteria/sections/contributor-documentation.md @@ -0,0 +1,17 @@ +# Contributor Documentation + +- Display name: Contributor Documentation +- Section slug: contributor-documentation + +## Areas + +Each row is an area in this section: its criterion label (used for the summary +ratings table) and the stem of its existing per-area comment file +(`--comment.md`). + +| Criterion | Comment file stem | +| ----------------------------------------- | -------------------------- | +| Communication methods documented | communication-methods-docd | +| Beginner friendly issue backlog | beginner-issue-backlog | +| "New contributor" getting started content | new-contributor | +| Project governance documentation | project-governance | diff --git a/docs/analysis/ai-harness/_github/criteria/sections/project-documentation.md b/docs/analysis/ai-harness/_github/criteria/sections/project-documentation.md new file mode 100644 index 00000000..ec44bd91 --- /dev/null +++ b/docs/analysis/ai-harness/_github/criteria/sections/project-documentation.md @@ -0,0 +1,18 @@ +# Project Documentation + +- Display name: Project Documentation +- Section slug: project-documentation + +## Areas + +Each row is an area in this section: its criterion label (used for the summary +ratings table) and the stem of its existing per-area comment file +(`--comment.md`). + +| Criterion | Comment file stem | +| -------------------------- | ------------------------ | +| Information architecture | information-architecture | +| New user content | new-user | +| Content maintainability | content-maintainability | +| Content creation processes | content-creation-process | +| Inclusive language | inclusive-language | diff --git a/docs/analysis/ai-harness/_github/criteria/sections/website-infrastructure.md b/docs/analysis/ai-harness/_github/criteria/sections/website-infrastructure.md new file mode 100644 index 00000000..cd65858e --- /dev/null +++ b/docs/analysis/ai-harness/_github/criteria/sections/website-infrastructure.md @@ -0,0 +1,25 @@ +# Website & Infrastructure + +- Display name: Website & Infrastructure +- Section slug: website-infrastructure + +## Areas + +Each row is an area in this section: its criterion label (used for the summary +ratings table) and the stem of its existing per-area comment file +(`--comment.md`). + + + +| Criterion | Comment file stem | +| ------------------------------------------- | ------------------------------- | +| Single-source for all files | single-source-requirement | +| Meets min website req. (for maturity level) | website-requirements | +| Usability, accessibility, and design | usability-accessibility-devices | +| Branding and design | branding-design | +| Case studies/social proof | case-studies | +| SEO, Analytics, and site-local search | seo-analytics | +| Maintenance planning | maintenance-planning | +| AI agent readiness | agent-readiness | + + diff --git a/docs/analysis/ai-harness/_github/examples/kubevirt/current-repository.md b/docs/analysis/ai-harness/_github/examples/kubevirt/current-repository.md new file mode 100644 index 00000000..2a4be8ab --- /dev/null +++ b/docs/analysis/ai-harness/_github/examples/kubevirt/current-repository.md @@ -0,0 +1,93 @@ +# KubeVirt example: Current Repository section + +This is the "Current Repository" section of `copilot-instructions.md` as filled +in for the KubeVirt user guide analysis +([cncf/techdocs#343](https://github.com/cncf/techdocs/issues/343)). Use it as a +model when completing the fill-in template in your own copy of +`.github/copilot-instructions.md`. The accompanying `kubevirt-doc-issue.md` is +the issue template the KubeVirt run used in place of the generic +`templates/doc-issue.md`. + +## Current Repository + +- Project name: KubeVirt +- Documentation label: the KubeVirt user guide +- Ecosystem: Kubernetes +- Main repository: [KubeVirt](https://github.com/kubevirt/kubevirt) +- Secondary repositories: + [KubeVirt.github.io](https://github.com/kubevirt/kubevirt.github.io) (website + and lab docs) +- Published documentation URL: https://kubevirt.io/user-guide/ +- Project website URL: https://kubevirt.io/ +- Issues repository URL: https://github.com/kubevirt/user-guide/issues +- Issue template file: kubevirt-doc-issue.md +- CNCF project page URL: https://www.cncf.io/projects/kubevirt/ +- Assistance-request issue: cncf/techdocs#343 + +Copilot shall output Markdown files from the AI-assisted analysis to the +`.github/output/analysis` directory and issue files to the +`.github/output/issues` directory in the current repository. + +### Priority label mapping + +| Harness label | Repository label | +| ------------- | -------------------------- | +| `urgent` | `priority/critical-urgent` | +| `high` | `priority/high` | +| `backlog` | `priority/backlog` | +| `help wanted` | `help wanted` | + +### What This Repo Is + +This is the [KubeVirt user guide](https://kubevirt.io/user-guide), a +documentation-only site built with [MkDocs](https://www.mkdocs.org/) using the +`mkdocs-material` theme and `mkdocs-awesome-nav` plugin. All content lives in +`./docs` as Markdown files. There is no application code. + +### Content Architecture + +```text +docs/ + .nav.yml # Top-level navigation order + index.md + architecture.md + cluster_admin/ # Installation, feature gates, RBAC, node ops + user_workloads/ # VM lifecycle, instancetypes, virtctl, startup scripts + compute/ # CPU/memory, live migration, hugepages, NUMA + network/ # Interfaces, hotplug, binding plugins, Istio + storage/ # CDI, volumes, snapshots, clone, export + debug_virt_stack/ # Debugging guides +``` + +### Key Conventions + +#### Navigation order via `.nav.yml` + +Every subdirectory under `docs/` has a `.nav.yml` that explicitly controls page +ordering. Alphabetical order is intentionally not used. When adding a new page, +add it to the relevant `.nav.yml`. + +#### Redirects in `mkdocs.yml` + +Old page paths (e.g. `operations/`, `virtual_machines/`) are redirected to their +new locations via the `redirects` plugin in `mkdocs.yml`. When moving or +renaming a page, add an entry there. + +#### No commits + +This work is solely for documentation analysis. Copilot makes no commits to the +documentation or application code in this repository; it writes only to +`.github/input` and `.github/output`. + +#### Spelling dictionary + +The yaspeller dictionary is sourced from +`kubevirt/project-infra/images/yaspeller/.yaspeller.json`. If a technical term +causes false positives, it needs to be added there (upstream), not locally — +unless a local `yaspeller.json` override is present. + +#### MkDocs extensions in use + +Pages can use: `admonition`, `footnotes`, `toc` with `permalink`, +`pymdownx.highlight`, `pymdownx.superfences`, `attr_list`, and `pymdownx.emoji`. +These are already configured in `mkdocs.yml` — no additional setup needed. diff --git a/docs/analysis/ai-harness/_github/examples/kubevirt/kubevirt-doc-issue.md b/docs/analysis/ai-harness/_github/examples/kubevirt/kubevirt-doc-issue.md new file mode 100644 index 00000000..8b6bb877 --- /dev/null +++ b/docs/analysis/ai-harness/_github/examples/kubevirt/kubevirt-doc-issue.md @@ -0,0 +1,24 @@ +--- +title: { title } #{Issue number} +labels: [high, help wanted] +repo: https://github.com/kubevirt/user-guide/issues +--- + + + +## Description + +A clear and concise description of what the issue is. + +## What you expected + +A clear and concise description of what you expected. + +## URL + +Provide a link to the documentation. + +## Additional context + +Add any other context about the issue here. diff --git a/docs/analysis/ai-harness/_github/input/readme.md b/docs/analysis/ai-harness/_github/input/readme.md new file mode 100644 index 00000000..76403e00 --- /dev/null +++ b/docs/analysis/ai-harness/_github/input/readme.md @@ -0,0 +1,16 @@ +# About this folder + +Use this folder to place temporary prompt files, spreadsheets, or anything else +you ask Copilot to read in a session. + +One file is required by the issue pipeline: + +- `ai-ranked-recommendations.csv` — the ranked recommendations that + `/create-issues` reads. The `/rank-recommendations` prompt produces it from + the per-area recommendations files in `../output/analysis/`. Columns: + - `Section` — `Project Documentation`, `Contributor Documentation`, or + `Website & Infrastructure`. + - `Area` — the area display name used in the analysis (for example + `Information Architecture`), not the criteria filename. + - `Recommendation` — the recommendation text, one per row. + - `Suggested Priority` — `1 - Urgent`, `2 - High`, or `3 - Backlog`. diff --git a/docs/analysis/ai-harness/_github/output/analysis/readme.md b/docs/analysis/ai-harness/_github/output/analysis/readme.md new file mode 100644 index 00000000..46e9a655 --- /dev/null +++ b/docs/analysis/ai-harness/_github/output/analysis/readme.md @@ -0,0 +1,8 @@ +# Analysis output + +The area prompts (`/-answers`, `/-comment`, +`/-recommendations`, `/-full`) and the section prompts +(`/
-comment`) write their Markdown output here. The +`/rank-recommendations` and `/create-issues` prompts read from this folder. + +This readme exists so Git keeps the folder; leave it in place. diff --git a/docs/analysis/ai-harness/_github/output/issues/readme.md b/docs/analysis/ai-harness/_github/output/issues/readme.md new file mode 100644 index 00000000..442a85f7 --- /dev/null +++ b/docs/analysis/ai-harness/_github/output/issues/readme.md @@ -0,0 +1,10 @@ +# Issue output + +`/create-issues` writes numbered standalone issue files here +(`{issue-number}-{condensed-title}.md`) and `/create-umbrella-issues` writes +umbrella issue files (`U-{issue-number}-{condensed-title}.md`). The upload +process treats every `*.md` file in this folder except this readme as an issue +to create. + +To keep a previous run, move its files into a sub-folder (for example +`prev_1/`); the prompts ignore sub-folders. diff --git a/docs/analysis/ai-harness/_github/output/readme.md b/docs/analysis/ai-harness/_github/output/readme.md new file mode 100644 index 00000000..82380b8d --- /dev/null +++ b/docs/analysis/ai-harness/_github/output/readme.md @@ -0,0 +1,11 @@ +# About this folder + +This folder contains two sub-folders for Copilot to place output: + +- `analysis/` — per-area and per-section output files written by the area and + section prompts (`--.md`). +- `issues/` — numbered standalone and umbrella issue files written by + `/create-issues` and `/create-umbrella-issues`. + +The `/create-issues` prompt also writes `issues-coverage.md` directly in this +folder, outside `issues/`, so it is not uploaded as an issue. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/agent-readiness-answers.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/agent-readiness-answers.prompt.md new file mode 100644 index 00000000..5b7ac2de --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/agent-readiness-answers.prompt.md @@ -0,0 +1,20 @@ +--- +agent: agent +description: + Answer the CNCF TechDocs AI agent readiness questions for the documentation. +--- + +# AI Agent Readiness answers + +Run the shared **answers** analysis for the `agent-readiness` criteria area. + +1. Read the criteria definition in `.github/criteria/areas/agent-readiness.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/answers.prompt.md`, treating `agent-readiness` as the + selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`--answers.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/agent-readiness-comment.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/agent-readiness-comment.prompt.md new file mode 100644 index 00000000..cd19e3f3 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/agent-readiness-comment.prompt.md @@ -0,0 +1,19 @@ +--- +agent: agent +description: Write a comment on the AI agent readiness of the documentation. +--- + +# AI Agent Readiness comment + +Run the shared **comment** analysis for the `agent-readiness` criteria area. + +1. Read the criteria definition in `.github/criteria/areas/agent-readiness.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/comment.prompt.md`, treating `agent-readiness` as the + selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`--comment.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/agent-readiness-full.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/agent-readiness-full.prompt.md new file mode 100644 index 00000000..c0ec2642 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/agent-readiness-full.prompt.md @@ -0,0 +1,22 @@ +--- +agent: agent +description: + Run the full `agent-readiness` analysis (answers, comment, and + recommendations) for the documentation. +--- + +# AI Agent Readiness full analysis + +Run the shared **full** analysis for the `agent-readiness` criteria area. + +1. Read the criteria definition in `.github/criteria/areas/agent-readiness.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/full.prompt.md`, treating `agent-readiness` as the + selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the base name for all three output files (`-answers.md`, +`<title>-comment.md`, `<title>-recommendations.md`); otherwise use each engine's +default (`<project-slug>-<stem>-<type>.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/agent-readiness-recommendations.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/agent-readiness-recommendations.prompt.md new file mode 100644 index 00000000..f356bbfc --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/agent-readiness-recommendations.prompt.md @@ -0,0 +1,22 @@ +--- +agent: agent +description: + Provide recommendations for improving the AI agent readiness of the + documentation. +--- + +# AI Agent Readiness recommendations + +Run the shared **recommendations** analysis for the `agent-readiness` criteria +area. + +1. Read the criteria definition in `.github/criteria/areas/agent-readiness.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/recommendations.prompt.md`, treating `agent-readiness` + as the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-recommendations.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/answers.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/answers.prompt.md new file mode 100644 index 00000000..7f3edc95 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/answers.prompt.md @@ -0,0 +1,45 @@ +--- +agent: agent +description: + Shared engine for producing analysis answers. Normally run via a per-area + wrapper such as /information-architecture-answers. +--- + +# Shared engine: answers + +This is the reusable procedure for answering an analysis area's questions. It is +normally invoked through a per-area wrapper prompt (for example, +`/information-architecture-answers`), which selects the criteria area for you. + +## Inputs + +- Area — the criteria area to analyze. When run from a wrapper, the wrapper + names it. If you reached this engine without an area, ask the user to choose + one of the areas defined in `.github/criteria/areas/` (the area name is the + criteria filename without `.md`). +- Title (optional) — overrides the default output filename. + +## Procedure + +1. Read the criteria definition at `.github/criteria/areas/<area>.md`. It + defines the area's Display name, File stem, and Questions. +2. Determine the project name from the "Current Repository" section of the + repository Copilot instructions. Derive a project slug by lowercasing the + name and replacing spaces with hyphens (for example, "KubeVirt" becomes + `kubevirt`). +3. Answer every question in the criteria file's "Questions" section, using the + sources described in the "Current Repository" section as context. + +## Output + +- Present the answers as indented paragraphs by two spaces under each question. + Format each question as a bullet, not as a heading. Do not bold the answers. +- If an answer is "yes" or "no", provide a brief explanation. +- Keep each answer to one or two short paragraphs. +- Write the result to a Markdown file in the `.github/output/analysis` + directory: + - If a title was provided, name the file `<title>.md`. + - Otherwise name it `<project-slug>-<stem>-answers.md`. + - If a file exists by the same name, overwrite it. + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/beginner-issue-backlog-answers.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/beginner-issue-backlog-answers.prompt.md new file mode 100644 index 00000000..fad544dd --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/beginner-issue-backlog-answers.prompt.md @@ -0,0 +1,23 @@ +--- +agent: agent +description: + Answer the CNCF TechDocs beginner friendly issue backlog questions for the + documentation. +--- + +# Beginner Friendly Issue Backlog answers + +Run the shared **answers** analysis for the `beginner-issue-backlog` criteria +area. + +1. Read the criteria definition in + `.github/criteria/areas/beginner-issue-backlog.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/answers.prompt.md`, treating `beginner-issue-backlog` + as the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-answers.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/beginner-issue-backlog-comment.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/beginner-issue-backlog-comment.prompt.md new file mode 100644 index 00000000..552b3bda --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/beginner-issue-backlog-comment.prompt.md @@ -0,0 +1,22 @@ +--- +agent: agent +description: + Write a comment on the beginner friendly issue backlog of the documentation. +--- + +# Beginner Friendly Issue Backlog comment + +Run the shared **comment** analysis for the `beginner-issue-backlog` criteria +area. + +1. Read the criteria definition in + `.github/criteria/areas/beginner-issue-backlog.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/comment.prompt.md`, treating `beginner-issue-backlog` + as the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-comment.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/beginner-issue-backlog-full.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/beginner-issue-backlog-full.prompt.md new file mode 100644 index 00000000..6bb96443 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/beginner-issue-backlog-full.prompt.md @@ -0,0 +1,23 @@ +--- +agent: agent +description: + Run the full `beginner-issue-backlog` analysis (answers, comment, and + recommendations) for the documentation. +--- + +# Beginner Friendly Issue Backlog full analysis + +Run the shared **full** analysis for the `beginner-issue-backlog` criteria area. + +1. Read the criteria definition in + `.github/criteria/areas/beginner-issue-backlog.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/full.prompt.md`, treating `beginner-issue-backlog` as + the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the base name for all three output files (`<title>-answers.md`, +`<title>-comment.md`, `<title>-recommendations.md`); otherwise use each engine's +default (`<project-slug>-<stem>-<type>.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/beginner-issue-backlog-recommendations.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/beginner-issue-backlog-recommendations.prompt.md new file mode 100644 index 00000000..6120467e --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/beginner-issue-backlog-recommendations.prompt.md @@ -0,0 +1,23 @@ +--- +agent: agent +description: + Provide recommendations for improving the beginner friendly issue backlog of + the documentation. +--- + +# Beginner Friendly Issue Backlog recommendations + +Run the shared **recommendations** analysis for the `beginner-issue-backlog` +criteria area. + +1. Read the criteria definition in + `.github/criteria/areas/beginner-issue-backlog.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/recommendations.prompt.md`, treating + `beginner-issue-backlog` as the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-recommendations.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/branding-design-answers.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/branding-design-answers.prompt.md new file mode 100644 index 00000000..107f6647 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/branding-design-answers.prompt.md @@ -0,0 +1,20 @@ +--- +agent: agent +description: + Answer the CNCF TechDocs branding and design questions for the documentation. +--- + +# Branding and Design answers + +Run the shared **answers** analysis for the `branding-design` criteria area. + +1. Read the criteria definition in `.github/criteria/areas/branding-design.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/answers.prompt.md`, treating `branding-design` as the + selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-answers.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/branding-design-comment.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/branding-design-comment.prompt.md new file mode 100644 index 00000000..138cd71b --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/branding-design-comment.prompt.md @@ -0,0 +1,19 @@ +--- +agent: agent +description: Write a comment on the branding and design of the documentation. +--- + +# Branding and Design comment + +Run the shared **comment** analysis for the `branding-design` criteria area. + +1. Read the criteria definition in `.github/criteria/areas/branding-design.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/comment.prompt.md`, treating `branding-design` as the + selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-comment.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/branding-design-full.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/branding-design-full.prompt.md new file mode 100644 index 00000000..a0eca621 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/branding-design-full.prompt.md @@ -0,0 +1,22 @@ +--- +agent: agent +description: + Run the full `branding-design` analysis (answers, comment, and + recommendations) for the documentation. +--- + +# Branding and Design full analysis + +Run the shared **full** analysis for the `branding-design` criteria area. + +1. Read the criteria definition in `.github/criteria/areas/branding-design.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/full.prompt.md`, treating `branding-design` as the + selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the base name for all three output files (`<title>-answers.md`, +`<title>-comment.md`, `<title>-recommendations.md`); otherwise use each engine's +default (`<project-slug>-<stem>-<type>.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/branding-design-recommendations.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/branding-design-recommendations.prompt.md new file mode 100644 index 00000000..ffe42ad7 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/branding-design-recommendations.prompt.md @@ -0,0 +1,22 @@ +--- +agent: agent +description: + Provide recommendations for improving the branding and design of the + documentation. +--- + +# Branding and Design recommendations + +Run the shared **recommendations** analysis for the `branding-design` criteria +area. + +1. Read the criteria definition in `.github/criteria/areas/branding-design.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/recommendations.prompt.md`, treating `branding-design` + as the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-recommendations.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/case-studies-answers.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/case-studies-answers.prompt.md new file mode 100644 index 00000000..fade49fa --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/case-studies-answers.prompt.md @@ -0,0 +1,20 @@ +--- +agent: agent +description: + Answer the CNCF TechDocs case studies questions for the documentation. +--- + +# Case Studies answers + +Run the shared **answers** analysis for the `case-studies` criteria area. + +1. Read the criteria definition in `.github/criteria/areas/case-studies.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/answers.prompt.md`, treating `case-studies` as the + selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-answers.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/case-studies-comment.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/case-studies-comment.prompt.md new file mode 100644 index 00000000..9e7feeaf --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/case-studies-comment.prompt.md @@ -0,0 +1,19 @@ +--- +agent: agent +description: Write a comment on the case studies of the documentation. +--- + +# Case Studies comment + +Run the shared **comment** analysis for the `case-studies` criteria area. + +1. Read the criteria definition in `.github/criteria/areas/case-studies.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/comment.prompt.md`, treating `case-studies` as the + selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-comment.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/case-studies-full.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/case-studies-full.prompt.md new file mode 100644 index 00000000..95d0e7ba --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/case-studies-full.prompt.md @@ -0,0 +1,22 @@ +--- +agent: agent +description: + Run the full `case-studies` analysis (answers, comment, and recommendations) + for the documentation. +--- + +# Case Studies full analysis + +Run the shared **full** analysis for the `case-studies` criteria area. + +1. Read the criteria definition in `.github/criteria/areas/case-studies.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/full.prompt.md`, treating `case-studies` as the + selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the base name for all three output files (`<title>-answers.md`, +`<title>-comment.md`, `<title>-recommendations.md`); otherwise use each engine's +default (`<project-slug>-<stem>-<type>.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/case-studies-recommendations.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/case-studies-recommendations.prompt.md new file mode 100644 index 00000000..83dfe289 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/case-studies-recommendations.prompt.md @@ -0,0 +1,21 @@ +--- +agent: agent +description: + Provide recommendations for improving the case studies of the documentation. +--- + +# Case Studies recommendations + +Run the shared **recommendations** analysis for the `case-studies` criteria +area. + +1. Read the criteria definition in `.github/criteria/areas/case-studies.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/recommendations.prompt.md`, treating `case-studies` as + the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-recommendations.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/comment.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/comment.prompt.md new file mode 100644 index 00000000..acb1f2a1 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/comment.prompt.md @@ -0,0 +1,54 @@ +--- +agent: agent +description: + Shared engine for producing an analysis comment. Normally run via a per-area + wrapper such as /information-architecture-comment. +--- + +# Shared engine: comment + +Reusable procedure for writing an analysis comment for one area. Normally +invoked through a per-area wrapper prompt (for example, +`/information-architecture-comment`), which selects the criteria area for you. + +## Inputs + +- Area — the criteria area to analyze. When run from a wrapper, the wrapper + names it. +- Title (optional) — overrides the default output filename. + +## Procedure + +1. Read the criteria definition at `.github/criteria/areas/<area>.md`. Use its + Display name, File stem, and Comment guidance (good examples and any + additional instructions). +2. Determine the project name and documentation label from the "Current + Repository" section of the repository Copilot instructions (for example, "the + KubeVirt user guide"). Derive the project slug by lowercasing the name and + replacing spaces with hyphens. +3. Write one to three paragraphs for a high-level comment. No need to replicate + points from the answers output. Follow with two bulleted lists: one of + strengths and one of weaknesses. Precede the lists with labels (not bold) + "Strengths:" and "Weaknesses:". +4. You can follow with any "Additional instructions" from the criteria file. You + may reference the listed "good examples" as models of quality. +5. Add a rating for this area, using the CNCF 1–5 scale: + - 1 - Not present + - 2 - Needs improvement + - 3 - Meets standards + - 4 - Meets or exceeds standards + - 5 - Exemplary + + State the rating on its own line at the end of the comment in the form + `Rating: <number> - <label>` (for example, `Rating: 3 - Meets standards`). Do + not include any other text on the rating line. + +## Output + +- Write the result to a Markdown file in the `.github/output/analysis` + directory: + - If a title was provided, name the file `<title>.md`. + - Otherwise name it `<project-slug>-<stem>-comment.md`. + - If a file exists by the same name, overwrite it. + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/communication-methods-documented-answers.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/communication-methods-documented-answers.prompt.md new file mode 100644 index 00000000..bcfc6d65 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/communication-methods-documented-answers.prompt.md @@ -0,0 +1,23 @@ +--- +agent: agent +description: + Answer the CNCF TechDocs communication methods documented questions for the + documentation. +--- + +# Communication Methods Documented answers + +Run the shared **answers** analysis for the `communication-methods-documented` +criteria area. + +1. Read the criteria definition in + `.github/criteria/areas/communication-methods-documented.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/answers.prompt.md`, treating + `communication-methods-documented` as the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-answers.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/communication-methods-documented-comment.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/communication-methods-documented-comment.prompt.md new file mode 100644 index 00000000..42bcb0aa --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/communication-methods-documented-comment.prompt.md @@ -0,0 +1,22 @@ +--- +agent: agent +description: + Write a comment on the communication methods documented of the documentation. +--- + +# Communication Methods Documented comment + +Run the shared **comment** analysis for the `communication-methods-documented` +criteria area. + +1. Read the criteria definition in + `.github/criteria/areas/communication-methods-documented.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/comment.prompt.md`, treating + `communication-methods-documented` as the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-comment.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/communication-methods-documented-full.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/communication-methods-documented-full.prompt.md new file mode 100644 index 00000000..55b9a5ff --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/communication-methods-documented-full.prompt.md @@ -0,0 +1,24 @@ +--- +agent: agent +description: + Run the full `communication-methods-documented` analysis (answers, comment, + and recommendations) for the documentation. +--- + +# Communication Methods Documented full analysis + +Run the shared **full** analysis for the `communication-methods-documented` +criteria area. + +1. Read the criteria definition in + `.github/criteria/areas/communication-methods-documented.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/full.prompt.md`, treating + `communication-methods-documented` as the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the base name for all three output files (`<title>-answers.md`, +`<title>-comment.md`, `<title>-recommendations.md`); otherwise use each engine's +default (`<project-slug>-<stem>-<type>.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/communication-methods-documented-recommendations.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/communication-methods-documented-recommendations.prompt.md new file mode 100644 index 00000000..beab1abc --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/communication-methods-documented-recommendations.prompt.md @@ -0,0 +1,23 @@ +--- +agent: agent +description: + Provide recommendations for improving the communication methods documented of + the documentation. +--- + +# Communication Methods Documented recommendations + +Run the shared **recommendations** analysis for the +`communication-methods-documented` criteria area. + +1. Read the criteria definition in + `.github/criteria/areas/communication-methods-documented.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/recommendations.prompt.md`, treating + `communication-methods-documented` as the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-recommendations.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/content-creation-process-answers.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/content-creation-process-answers.prompt.md new file mode 100644 index 00000000..9e830c73 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/content-creation-process-answers.prompt.md @@ -0,0 +1,23 @@ +--- +agent: agent +description: + Answer the CNCF TechDocs content creation process questions for the + documentation. +--- + +# Content Creation Process answers + +Run the shared **answers** analysis for the `content-creation-process` criteria +area. + +1. Read the criteria definition in + `.github/criteria/areas/content-creation-process.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/answers.prompt.md`, treating + `content-creation-process` as the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-answers.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/content-creation-process-comment.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/content-creation-process-comment.prompt.md new file mode 100644 index 00000000..93a48629 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/content-creation-process-comment.prompt.md @@ -0,0 +1,22 @@ +--- +agent: agent +description: + Write a comment on the content creation process of the documentation. +--- + +# Content Creation Process comment + +Run the shared **comment** analysis for the `content-creation-process` criteria +area. + +1. Read the criteria definition in + `.github/criteria/areas/content-creation-process.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/comment.prompt.md`, treating + `content-creation-process` as the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-comment.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/content-creation-process-full.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/content-creation-process-full.prompt.md new file mode 100644 index 00000000..a3ea0884 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/content-creation-process-full.prompt.md @@ -0,0 +1,24 @@ +--- +agent: agent +description: + Run the full content creation process analysis (answers, comment, and + recommendations) for the documentation. +--- + +# Content Creation Process full analysis + +Run the shared **full** analysis for the `content-creation-process` criteria +area. + +1. Read the criteria definition in + `.github/criteria/areas/content-creation-process.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/full.prompt.md`, treating `content-creation-process` + as the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the base name for all three output files (`<title>-answers.md`, +`<title>-comment.md`, `<title>-recommendations.md`); otherwise use each engine's +default (`<project-slug>-<stem>-<type>.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/content-creation-process-recommendations.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/content-creation-process-recommendations.prompt.md new file mode 100644 index 00000000..85f52007 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/content-creation-process-recommendations.prompt.md @@ -0,0 +1,23 @@ +--- +agent: agent +description: + Provide recommendations for improving the content creation process of the + documentation. +--- + +# Content Creation Process recommendations + +Run the shared **recommendations** analysis for the `content-creation-process` +criteria area. + +1. Read the criteria definition in + `.github/criteria/areas/content-creation-process.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/recommendations.prompt.md`, treating + `content-creation-process` as the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-recommendations.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/content-maintainability-answers.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/content-maintainability-answers.prompt.md new file mode 100644 index 00000000..5901dc69 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/content-maintainability-answers.prompt.md @@ -0,0 +1,23 @@ +--- +agent: agent +description: + Answer the CNCF TechDocs content maintainability questions for the + documentation. +--- + +# Content Maintainability answers + +Run the shared **answers** analysis for the `content-maintainability` criteria +area. + +1. Read the criteria definition in + `.github/criteria/areas/content-maintainability.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/answers.prompt.md`, treating `content-maintainability` + as the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-answers.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/content-maintainability-comment.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/content-maintainability-comment.prompt.md new file mode 100644 index 00000000..efe06e7a --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/content-maintainability-comment.prompt.md @@ -0,0 +1,22 @@ +--- +agent: agent +description: + Write a comment on the content maintainability of the documentation. +--- + +# Content Maintainability comment + +Run the shared **comment** analysis for the `content-maintainability` criteria +area. + +1. Read the criteria definition in + `.github/criteria/areas/content-maintainability.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/comment.prompt.md`, treating `content-maintainability` + as the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-comment.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/content-maintainability-full.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/content-maintainability-full.prompt.md new file mode 100644 index 00000000..a025e891 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/content-maintainability-full.prompt.md @@ -0,0 +1,24 @@ +--- +agent: agent +description: + Run the full content maintainability analysis (answers, comment, and + recommendations) for the documentation. +--- + +# Content Maintainability full analysis + +Run the shared **full** analysis for the `content-maintainability` criteria +area. + +1. Read the criteria definition in + `.github/criteria/areas/content-maintainability.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/full.prompt.md`, treating `content-maintainability` as + the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the base name for all three output files (`<title>-answers.md`, +`<title>-comment.md`, `<title>-recommendations.md`); otherwise use each engine's +default (`<project-slug>-<stem>-<type>.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/content-maintainability-recommendations.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/content-maintainability-recommendations.prompt.md new file mode 100644 index 00000000..c9e8a657 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/content-maintainability-recommendations.prompt.md @@ -0,0 +1,23 @@ +--- +agent: agent +description: + Provide recommendations for improving the content maintainability of the + documentation. +--- + +# Content Maintainability recommendations + +Run the shared **recommendations** analysis for the `content-maintainability` +criteria area. + +1. Read the criteria definition in + `.github/criteria/areas/content-maintainability.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/recommendations.prompt.md`, treating + `content-maintainability` as the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-recommendations.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/full.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/full.prompt.md new file mode 100644 index 00000000..13103cc3 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/full.prompt.md @@ -0,0 +1,103 @@ +--- +agent: agent +description: + Shared engine for running all three analyses (answers, comment, + recommendations) for one area. Normally run via a per-area wrapper such as + /information-architecture-full. +--- + +# Shared engine: full + +Reusable procedure for running the complete analysis of one criteria area: it +gathers the material for the **answers**, **comment**, and **recommendations** +engines in a single pass, sorts it by output type, and then writes the three +files in sequence. Normally invoked through a per-area wrapper prompt (for +example, `/information-architecture-full`), which selects the criteria area for +you. + +## Inputs + +- Area — the criteria area to analyze. When run from a wrapper, the wrapper + names it. If you reached this engine without an area, ask the user to choose + one of the areas defined in `.github/criteria/areas/` (the area name is the + criteria filename without `.md`). +- Title (optional) — a base name that overrides the default output filenames. + When provided, the outputs are `<title>-answers.md`, `<title>-comment.md`, and + `<title>-recommendations.md`. + +## Procedure + +Verify output style as Claude default, and review the "Analysis responses style" +guidelines in copilot-instructions.md. + +Analyze first, write last. While answering the questions and forming the comment +you will inevitably notice things to fix; those observations belong in the +recommendations file, not in the answers or comment. Collect everything before +writing so each file contains only its own kind of content. + +### Phase 1: Read + +1. Read the criteria definition at `.github/criteria/areas/<area>.md` once, so + its Display name, File stem, Questions, and Comment guidance are available to + every step below. +2. Read all three engines — `.github/prompts/areas/answers.prompt.md`, + `.github/prompts/areas/comment.prompt.md`, and + `.github/prompts/areas/recommendations.prompt.md` — so you know the required + structure of each output before you start analyzing. + +### Phase 2: Analyze and sort + +1. Investigate the documentation as the answers engine directs. As you work, + keep three running lists: + - Findings: factual observations that answer a question (what exists, where + it is, what is missing). + - Assessments: high-level judgments about strengths and weaknesses, and the + evidence for the rating. + - Fixes: anything you would tell the project to change or add. +2. Whenever an observation suggests a fix, record the fix in the Fixes list and + keep only the underlying finding in the Findings or Assessments list. A + finding may state that something is missing or unclear; it must not say what + to do about it. +3. Draft all three documents in memory from these lists: + - Answers, from the Findings list, following the answers engine's Output + section. + - Comment, from the Assessments list, following the comment engine's + Procedure and Output sections. + - Recommendations, from the Fixes list, following the recommendations + engine's Output section. Base each recommendation on a finding or + assessment that appears in the drafted answers or comment. +4. Review the drafts against each other before writing anything to disk: + - Remove any imperative or "should" phrasing from the answers and comment; + move it to the recommendations. + - Confirm every recommendation traces back to something stated in the answers + or comment. If it does not, add the supporting finding or drop the + recommendation. + - Confirm the answers and comment do not duplicate each other; the comment + synthesizes, it does not restate. + +### Phase 3: Write + +1. Write the three files in this order, since the recommendations engine expects + the first two to exist: + 1. Answers: `.github/prompts/areas/answers.prompt.md` output. + 2. Comment: `.github/prompts/areas/comment.prompt.md` output. + 3. Recommendations: `.github/prompts/areas/recommendations.prompt.md` output. + Because the sorting in Phase 2 already kept recommendations out of the + comment, its instruction to move stray recommendations out of the comment + file should find nothing to move. + +When a Title is provided, pass it through to each engine so the three files +share the `<title>` base name described under "Inputs". + +## Output + +- Produce all three Markdown files in the `.github/output/analysis` directory: + - Answers: `<title>-answers.md`, or `<project-slug>-<stem>-answers.md` by + default. + - Comment: `<title>-comment.md`, or `<project-slug>-<stem>-comment.md` by + default. + - Recommendations: `<title>-recommendations.md`, or + `<project-slug>-<stem>-recommendations.md` by default. +- After writing them, list the three filenames you created. + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/inclusive-language-answers.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/inclusive-language-answers.prompt.md new file mode 100644 index 00000000..c6db3332 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/inclusive-language-answers.prompt.md @@ -0,0 +1,21 @@ +--- +agent: agent +description: + Answer the CNCF TechDocs inclusive language questions for the documentation. +--- + +# Inclusive Language answers + +Run the shared **answers** analysis for the `inclusive-language` criteria area. + +1. Read the criteria definition in + `.github/criteria/areas/inclusive-language.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/answers.prompt.md`, treating `inclusive-language` as + the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-answers.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/inclusive-language-comment.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/inclusive-language-comment.prompt.md new file mode 100644 index 00000000..607b6042 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/inclusive-language-comment.prompt.md @@ -0,0 +1,20 @@ +--- +agent: agent +description: Write a comment on the inclusive language of the documentation. +--- + +# Inclusive Language comment + +Run the shared **comment** analysis for the `inclusive-language` criteria area. + +1. Read the criteria definition in + `.github/criteria/areas/inclusive-language.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/comment.prompt.md`, treating `inclusive-language` as + the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-comment.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/inclusive-language-full.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/inclusive-language-full.prompt.md new file mode 100644 index 00000000..2b7f47f4 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/inclusive-language-full.prompt.md @@ -0,0 +1,23 @@ +--- +agent: agent +description: + Run the full inclusive language analysis (answers, comment, and + recommendations) for the documentation. +--- + +# Inclusive Language full analysis + +Run the shared **full** analysis for the `inclusive-language` criteria area. + +1. Read the criteria definition in + `.github/criteria/areas/inclusive-language.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/full.prompt.md`, treating `inclusive-language` as the + selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the base name for all three output files (`<title>-answers.md`, +`<title>-comment.md`, `<title>-recommendations.md`); otherwise use each engine's +default (`<project-slug>-<stem>-<type>.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/inclusive-language-recommendations.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/inclusive-language-recommendations.prompt.md new file mode 100644 index 00000000..0e107b27 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/inclusive-language-recommendations.prompt.md @@ -0,0 +1,23 @@ +--- +agent: agent +description: + Provide recommendations for improving the inclusive language of the + documentation. +--- + +# Inclusive Language recommendations + +Run the shared **recommendations** analysis for the `inclusive-language` +criteria area. + +1. Read the criteria definition in + `.github/criteria/areas/inclusive-language.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/recommendations.prompt.md`, treating + `inclusive-language` as the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-recommendations.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/information-architecture-answers.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/information-architecture-answers.prompt.md new file mode 100644 index 00000000..5e7cdbd3 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/information-architecture-answers.prompt.md @@ -0,0 +1,23 @@ +--- +agent: agent +description: + Answer the CNCF TechDocs information architecture questions for the + documentation. +--- + +# Information Architecture answers + +Run the shared **answers** analysis for the `information-architecture` criteria +area. + +1. Read the criteria definition in + `.github/criteria/areas/information-architecture.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/answers.prompt.md`, treating + `information-architecture` as the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-answers.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/information-architecture-comment.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/information-architecture-comment.prompt.md new file mode 100644 index 00000000..a8848935 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/information-architecture-comment.prompt.md @@ -0,0 +1,22 @@ +--- +agent: agent +description: + Write a comment on the information architecture of the documentation. +--- + +# Information Architecture comment + +Run the shared **comment** analysis for the `information-architecture` criteria +area. + +1. Read the criteria definition in + `.github/criteria/areas/information-architecture.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/comment.prompt.md`, treating + `information-architecture` as the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-comment.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/information-architecture-full.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/information-architecture-full.prompt.md new file mode 100644 index 00000000..c21807c6 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/information-architecture-full.prompt.md @@ -0,0 +1,24 @@ +--- +agent: agent +description: + Run the full information architecture analysis (answers, comment, and + recommendations) for the documentation. +--- + +# Information Architecture full analysis + +Run the shared **full** analysis for the `information-architecture` criteria +area. + +1. Read the criteria definition in + `.github/criteria/areas/information-architecture.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/full.prompt.md`, treating `information-architecture` + as the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the base name for all three output files (`<title>-answers.md`, +`<title>-comment.md`, `<title>-recommendations.md`); otherwise use each engine's +default (`<project-slug>-<stem>-<type>.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/information-architecture-recommendations.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/information-architecture-recommendations.prompt.md new file mode 100644 index 00000000..be53145e --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/information-architecture-recommendations.prompt.md @@ -0,0 +1,23 @@ +--- +agent: agent +description: + Provide recommendations for improving the information architecture of the + documentation. +--- + +# Information Architecture recommendations + +Run the shared **recommendations** analysis for the `information-architecture` +criteria area. + +1. Read the criteria definition in + `.github/criteria/areas/information-architecture.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/recommendations.prompt.md`, treating + `information-architecture` as the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-recommendations.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/maintenance-planning-answers.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/maintenance-planning-answers.prompt.md new file mode 100644 index 00000000..36b21417 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/maintenance-planning-answers.prompt.md @@ -0,0 +1,22 @@ +--- +agent: agent +description: + Answer the CNCF TechDocs maintenance planning questions for the documentation. +--- + +# Maintenance Planning answers + +Run the shared **answers** analysis for the `maintenance-planning` criteria +area. + +1. Read the criteria definition in + `.github/criteria/areas/maintenance-planning.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/answers.prompt.md`, treating `maintenance-planning` as + the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-answers.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/maintenance-planning-comment.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/maintenance-planning-comment.prompt.md new file mode 100644 index 00000000..32115d36 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/maintenance-planning-comment.prompt.md @@ -0,0 +1,21 @@ +--- +agent: agent +description: Write a comment on the maintenance planning of the documentation. +--- + +# Maintenance Planning comment + +Run the shared **comment** analysis for the `maintenance-planning` criteria +area. + +1. Read the criteria definition in + `.github/criteria/areas/maintenance-planning.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/comment.prompt.md`, treating `maintenance-planning` as + the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-comment.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/maintenance-planning-full.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/maintenance-planning-full.prompt.md new file mode 100644 index 00000000..ef7d7dd6 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/maintenance-planning-full.prompt.md @@ -0,0 +1,23 @@ +--- +agent: agent +description: + Run the full `maintenance-planning` analysis (answers, comment, and + recommendations) for the documentation. +--- + +# Maintenance Planning full analysis + +Run the shared **full** analysis for the `maintenance-planning` criteria area. + +1. Read the criteria definition in + `.github/criteria/areas/maintenance-planning.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/full.prompt.md`, treating `maintenance-planning` as + the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the base name for all three output files (`<title>-answers.md`, +`<title>-comment.md`, `<title>-recommendations.md`); otherwise use each engine's +default (`<project-slug>-<stem>-<type>.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/maintenance-planning-recommendations.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/maintenance-planning-recommendations.prompt.md new file mode 100644 index 00000000..9b65dca9 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/maintenance-planning-recommendations.prompt.md @@ -0,0 +1,23 @@ +--- +agent: agent +description: + Provide recommendations for improving the maintenance planning of the + documentation. +--- + +# Maintenance Planning recommendations + +Run the shared **recommendations** analysis for the `maintenance-planning` +criteria area. + +1. Read the criteria definition in + `.github/criteria/areas/maintenance-planning.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/recommendations.prompt.md`, treating + `maintenance-planning` as the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-recommendations.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/new-contributor-content-answers.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/new-contributor-content-answers.prompt.md new file mode 100644 index 00000000..31292d8e --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/new-contributor-content-answers.prompt.md @@ -0,0 +1,23 @@ +--- +agent: agent +description: + Answer the CNCF TechDocs new contributor getting started content questions for + the documentation. +--- + +# New Contributor Getting Started Content answers + +Run the shared **answers** analysis for the `new-contributor-content` criteria +area. + +1. Read the criteria definition in + `.github/criteria/areas/new-contributor-content.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/answers.prompt.md`, treating `new-contributor-content` + as the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-answers.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/new-contributor-content-comment.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/new-contributor-content-comment.prompt.md new file mode 100644 index 00000000..ad863563 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/new-contributor-content-comment.prompt.md @@ -0,0 +1,23 @@ +--- +agent: agent +description: + Write a comment on the new contributor getting started content of the + documentation. +--- + +# New Contributor Getting Started Content comment + +Run the shared **comment** analysis for the `new-contributor-content` criteria +area. + +1. Read the criteria definition in + `.github/criteria/areas/new-contributor-content.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/comment.prompt.md`, treating `new-contributor-content` + as the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-comment.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/new-contributor-content-full.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/new-contributor-content-full.prompt.md new file mode 100644 index 00000000..853dc6f0 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/new-contributor-content-full.prompt.md @@ -0,0 +1,24 @@ +--- +agent: agent +description: + Run the full `new-contributor-content` analysis (answers, comment, and + recommendations) for the documentation. +--- + +# New Contributor Getting Started Content full analysis + +Run the shared **full** analysis for the `new-contributor-content` criteria +area. + +1. Read the criteria definition in + `.github/criteria/areas/new-contributor-content.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/full.prompt.md`, treating `new-contributor-content` as + the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the base name for all three output files (`<title>-answers.md`, +`<title>-comment.md`, `<title>-recommendations.md`); otherwise use each engine's +default (`<project-slug>-<stem>-<type>.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/new-contributor-content-recommendations.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/new-contributor-content-recommendations.prompt.md new file mode 100644 index 00000000..7350a58d --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/new-contributor-content-recommendations.prompt.md @@ -0,0 +1,23 @@ +--- +agent: agent +description: + Provide recommendations for improving the new contributor getting started + content of the documentation. +--- + +# New Contributor Getting Started Content recommendations + +Run the shared **recommendations** analysis for the `new-contributor-content` +criteria area. + +1. Read the criteria definition in + `.github/criteria/areas/new-contributor-content.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/recommendations.prompt.md`, treating + `new-contributor-content` as the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-recommendations.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/new-user-content-answers.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/new-user-content-answers.prompt.md new file mode 100644 index 00000000..fc0dbb5d --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/new-user-content-answers.prompt.md @@ -0,0 +1,20 @@ +--- +agent: agent +description: + Answer the CNCF TechDocs new user content questions for the documentation. +--- + +# New User Content answers + +Run the shared **answers** analysis for the `new-user-content` criteria area. + +1. Read the criteria definition in `.github/criteria/areas/new-user-content.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/answers.prompt.md`, treating `new-user-content` as the + selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-answers.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/new-user-content-comment.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/new-user-content-comment.prompt.md new file mode 100644 index 00000000..7e6fd532 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/new-user-content-comment.prompt.md @@ -0,0 +1,19 @@ +--- +agent: agent +description: Write a comment on the new user content of the documentation. +--- + +# New User Content comment + +Run the shared **comment** analysis for the `new-user-content` criteria area. + +1. Read the criteria definition in `.github/criteria/areas/new-user-content.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/comment.prompt.md`, treating `new-user-content` as the + selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-comment.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/new-user-content-full.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/new-user-content-full.prompt.md new file mode 100644 index 00000000..bb6072a0 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/new-user-content-full.prompt.md @@ -0,0 +1,22 @@ +--- +agent: agent +description: + Run the full new user content analysis (answers, comment, and recommendations) + for the documentation. +--- + +# New User Content full analysis + +Run the shared **full** analysis for the `new-user-content` criteria area. + +1. Read the criteria definition in `.github/criteria/areas/new-user-content.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/full.prompt.md`, treating `new-user-content` as the + selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the base name for all three output files (`<title>-answers.md`, +`<title>-comment.md`, `<title>-recommendations.md`); otherwise use each engine's +default (`<project-slug>-<stem>-<type>.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/new-user-content-recommendations.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/new-user-content-recommendations.prompt.md new file mode 100644 index 00000000..5558f3f2 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/new-user-content-recommendations.prompt.md @@ -0,0 +1,22 @@ +--- +agent: agent +description: + Provide recommendations for improving the new user content of the + documentation. +--- + +# New User Content recommendations + +Run the shared **recommendations** analysis for the `new-user-content` criteria +area. + +1. Read the criteria definition in `.github/criteria/areas/new-user-content.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/recommendations.prompt.md`, treating + `new-user-content` as the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-recommendations.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/project-governance-documentation-answers.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/project-governance-documentation-answers.prompt.md new file mode 100644 index 00000000..2afa4fa9 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/project-governance-documentation-answers.prompt.md @@ -0,0 +1,22 @@ +--- +agent: agent +description: + Answer the CNCF TechDocs project governance questions for the documentation. +--- + +# Project Governance Documentation answers + +Run the shared **answers** analysis for the `project-governance-documentation` +criteria area. + +1. Read the criteria definition in + `.github/criteria/areas/project-governance-documentation.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/answers.prompt.md`, treating + `project-governance-documentation` as the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-answers.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/project-governance-documentation-comment.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/project-governance-documentation-comment.prompt.md new file mode 100644 index 00000000..c6f099a0 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/project-governance-documentation-comment.prompt.md @@ -0,0 +1,21 @@ +--- +agent: agent +description: Write a comment on the project governance of the documentation. +--- + +# Project Governance Documentation comment + +Run the shared **comment** analysis for the `project-governance-documentation` +criteria area. + +1. Read the criteria definition in + `.github/criteria/areas/project-governance-documentation.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/comment.prompt.md`, treating + `project-governance-documentation` as the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-comment.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/project-governance-documentation-full.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/project-governance-documentation-full.prompt.md new file mode 100644 index 00000000..532ec3a8 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/project-governance-documentation-full.prompt.md @@ -0,0 +1,24 @@ +--- +agent: agent +description: + Run the full `project-governance-documentation` analysis (answers, comment, + and recommendations) for the documentation. +--- + +# Project Governance Documentation full analysis + +Run the shared **full** analysis for the `project-governance-documentation` +criteria area. + +1. Read the criteria definition in + `.github/criteria/areas/project-governance-documentation.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/full.prompt.md`, treating + `project-governance-documentation` as the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the base name for all three output files (`<title>-answers.md`, +`<title>-comment.md`, `<title>-recommendations.md`); otherwise use each engine's +default (`<project-slug>-<stem>-<type>.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/project-governance-documentation-recommendations.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/project-governance-documentation-recommendations.prompt.md new file mode 100644 index 00000000..2ff152de --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/project-governance-documentation-recommendations.prompt.md @@ -0,0 +1,23 @@ +--- +agent: agent +description: + Provide recommendations for improving the project governance of the + documentation. +--- + +# Project Governance Documentation recommendations + +Run the shared **recommendations** analysis for the +`project-governance-documentation` criteria area. + +1. Read the criteria definition in + `.github/criteria/areas/project-governance-documentation.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/recommendations.prompt.md`, treating + `project-governance-documentation` as the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-recommendations.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/recommendations.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/recommendations.prompt.md new file mode 100644 index 00000000..cbf74e28 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/recommendations.prompt.md @@ -0,0 +1,51 @@ +--- +agent: agent +description: + Shared engine for producing analysis recommendations. Normally run via a + per-area wrapper such as /information-architecture-recommendations. +--- + +# Shared engine: recommendations + +Reusable procedure for producing recommendations for one area, based on the +previously generated answers and comment. Normally invoked through a per-area +wrapper prompt (for example, `/information-architecture-recommendations`), which +selects the criteria area for you. + +## Inputs + +- Area — the criteria area to analyze. When run from a wrapper, the wrapper + names it. If you reached this engine without an area, ask the user to choose + one of the areas defined in `.github/criteria/areas/` (the area name is the + criteria filename without `.md`). +- Title (optional) — overrides the default output filename. + +## Procedure + +1. Read the criteria definition at `.github/criteria/areas/<area>.md` for its + Display name and File stem. +2. Determine the project name and documentation label from the "Current + Repository" section of the repository Copilot instructions. Derive the + project slug by lowercasing the name and replacing spaces with hyphens. +3. Read the answers and comment produced for this area from the + `.github/output/analysis` directory: `<project-slug>-<stem>-answers.md` and + `<project-slug>-<stem>-comment.md`. If either file is missing, ask the user + to run the corresponding answers or comment prompt first (or to point you at + the files). If the comment file contains a recommendation, move it to the + recommendations file. +4. Base the recommendations on those two documents. + +## Output + +- Output the recommendations as a bulleted list, introduced by this exact + sentence: "The following recommendations address the <display name> of + the <project documentation label>." For example: "The following + recommendations address the information architecture of the KubeVirt user + guide." +- Write the result to a Markdown file in the `.github/output/analysis` + directory: + - If a title was provided, name the file `<title>.md`. + - Otherwise name it `<project-slug>-<stem>-recommendations.md`. + - If a file exists by the same name, overwrite it. + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/seo-analytics-answers.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/seo-analytics-answers.prompt.md new file mode 100644 index 00000000..b891d208 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/seo-analytics-answers.prompt.md @@ -0,0 +1,21 @@ +--- +agent: agent +description: + Answer the CNCF TechDocs seo, analytics & site search questions for the + documentation. +--- + +# SEO, Analytics, and Site Search answers + +Run the shared **answers** analysis for the `seo-analytics` criteria area. + +1. Read the criteria definition in `.github/criteria/areas/seo-analytics.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/answers.prompt.md`, treating `seo-analytics` as the + selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-answers.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/seo-analytics-comment.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/seo-analytics-comment.prompt.md new file mode 100644 index 00000000..827252c3 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/seo-analytics-comment.prompt.md @@ -0,0 +1,20 @@ +--- +agent: agent +description: + Write a comment on the seo, analytics & site search of the documentation. +--- + +# SEO, Analytics, and Site Search comment + +Run the shared **comment** analysis for the `seo-analytics` criteria area. + +1. Read the criteria definition in `.github/criteria/areas/seo-analytics.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/comment.prompt.md`, treating `seo-analytics` as the + selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-comment.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/seo-analytics-full.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/seo-analytics-full.prompt.md new file mode 100644 index 00000000..120e2609 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/seo-analytics-full.prompt.md @@ -0,0 +1,22 @@ +--- +agent: agent +description: + Run the full `seo-analytics` analysis (answers, comment, and recommendations) + for the documentation. +--- + +# SEO, Analytics, and Site Search full analysis + +Run the shared **full** analysis for the `seo-analytics` criteria area. + +1. Read the criteria definition in `.github/criteria/areas/seo-analytics.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/full.prompt.md`, treating `seo-analytics` as the + selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the base name for all three output files (`<title>-answers.md`, +`<title>-comment.md`, `<title>-recommendations.md`); otherwise use each engine's +default (`<project-slug>-<stem>-<type>.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/seo-analytics-recommendations.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/seo-analytics-recommendations.prompt.md new file mode 100644 index 00000000..b80724aa --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/seo-analytics-recommendations.prompt.md @@ -0,0 +1,22 @@ +--- +agent: agent +description: + Provide recommendations for improving the seo, analytics & site search of the + documentation. +--- + +# SEO, Analytics, and Site Search recommendations + +Run the shared **recommendations** analysis for the `seo-analytics` criteria +area. + +1. Read the criteria definition in `.github/criteria/areas/seo-analytics.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/recommendations.prompt.md`, treating `seo-analytics` + as the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-recommendations.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/single-source-requirement-answers.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/single-source-requirement-answers.prompt.md new file mode 100644 index 00000000..56722aa3 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/single-source-requirement-answers.prompt.md @@ -0,0 +1,23 @@ +--- +agent: agent +description: + Answer the CNCF TechDocs single-source requirement questions for the + documentation. +--- + +# Single-Source Requirement answers + +Run the shared **answers** analysis for the `single-source-requirement` criteria +area. + +1. Read the criteria definition in + `.github/criteria/areas/single-source-requirement.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/answers.prompt.md`, treating + `single-source-requirement` as the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-answers.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/single-source-requirement-comment.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/single-source-requirement-comment.prompt.md new file mode 100644 index 00000000..56dfb08e --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/single-source-requirement-comment.prompt.md @@ -0,0 +1,22 @@ +--- +agent: agent +description: + Write a comment on the single-source requirement of the documentation. +--- + +# Single-Source Requirement comment + +Run the shared **comment** analysis for the `single-source-requirement` criteria +area. + +1. Read the criteria definition in + `.github/criteria/areas/single-source-requirement.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/comment.prompt.md`, treating + `single-source-requirement` as the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-comment.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/single-source-requirement-full.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/single-source-requirement-full.prompt.md new file mode 100644 index 00000000..8f513f6a --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/single-source-requirement-full.prompt.md @@ -0,0 +1,24 @@ +--- +agent: agent +description: + Run the full `single-source-requirement` analysis (answers, comment, and + recommendations) for the documentation. +--- + +# Single-Source Requirement full analysis + +Run the shared **full** analysis for the `single-source-requirement` criteria +area. + +1. Read the criteria definition in + `.github/criteria/areas/single-source-requirement.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/full.prompt.md`, treating `single-source-requirement` + as the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the base name for all three output files (`<title>-answers.md`, +`<title>-comment.md`, `<title>-recommendations.md`); otherwise use each engine's +default (`<project-slug>-<stem>-<type>.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/single-source-requirement-recommendations.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/single-source-requirement-recommendations.prompt.md new file mode 100644 index 00000000..ced1aace --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/single-source-requirement-recommendations.prompt.md @@ -0,0 +1,23 @@ +--- +agent: agent +description: + Provide recommendations for improving the single-source requirement of the + documentation. +--- + +# Single-Source Requirement recommendations + +Run the shared **recommendations** analysis for the `single-source-requirement` +criteria area. + +1. Read the criteria definition in + `.github/criteria/areas/single-source-requirement.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/recommendations.prompt.md`, treating + `single-source-requirement` as the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-recommendations.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/usability-accessibility-devices-answers.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/usability-accessibility-devices-answers.prompt.md new file mode 100644 index 00000000..bf45b5cf --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/usability-accessibility-devices-answers.prompt.md @@ -0,0 +1,23 @@ +--- +agent: agent +description: + Answer the CNCF TechDocs usability accessibility and devices questions for the + documentation. +--- + +# Usability, Accessibility and Devices answers + +Run the shared **answers** analysis for the `usability-accessibility-devices` +criteria area. + +1. Read the criteria definition in + `.github/criteria/areas/usability-accessibility-devices.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/answers.prompt.md`, treating + `usability-accessibility-devices` as the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-answers.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/usability-accessibility-devices-comment.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/usability-accessibility-devices-comment.prompt.md new file mode 100644 index 00000000..7c5b870d --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/usability-accessibility-devices-comment.prompt.md @@ -0,0 +1,23 @@ +--- +agent: agent +description: + Write a comment on the usability accessibility and devices of the + documentation. +--- + +# Usability, Accessibility and Devices comment + +Run the shared **comment** analysis for the `usability-accessibility-devices` +criteria area. + +1. Read the criteria definition in + `.github/criteria/areas/usability-accessibility-devices.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/comment.prompt.md`, treating + `usability-accessibility-devices` as the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-comment.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/usability-accessibility-devices-full.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/usability-accessibility-devices-full.prompt.md new file mode 100644 index 00000000..623a257d --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/usability-accessibility-devices-full.prompt.md @@ -0,0 +1,24 @@ +--- +agent: agent +description: + Run the full `usability-accessibility-devices` analysis (answers, comment, and + recommendations) for the documentation. +--- + +# Usability, Accessibility and Devices full analysis + +Run the shared **full** analysis for the `usability-accessibility-devices` +criteria area. + +1. Read the criteria definition in + `.github/criteria/areas/usability-accessibility-devices.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/full.prompt.md`, treating + `usability-accessibility-devices` as the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the base name for all three output files (`<title>-answers.md`, +`<title>-comment.md`, `<title>-recommendations.md`); otherwise use each engine's +default (`<project-slug>-<stem>-<type>.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/usability-accessibility-devices-recommendations.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/usability-accessibility-devices-recommendations.prompt.md new file mode 100644 index 00000000..7f4c630a --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/usability-accessibility-devices-recommendations.prompt.md @@ -0,0 +1,23 @@ +--- +agent: agent +description: + Provide recommendations for improving the usability accessibility and devices + of the documentation. +--- + +# Usability, Accessibility and Devices recommendations + +Run the shared **recommendations** analysis for the +`usability-accessibility-devices` criteria area. + +1. Read the criteria definition in + `.github/criteria/areas/usability-accessibility-devices.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/recommendations.prompt.md`, treating + `usability-accessibility-devices` as the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-recommendations.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/website-requirements-answers.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/website-requirements-answers.prompt.md new file mode 100644 index 00000000..04b4d0db --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/website-requirements-answers.prompt.md @@ -0,0 +1,22 @@ +--- +agent: agent +description: + Answer the CNCF TechDocs website requirements questions for the documentation. +--- + +# Website Requirements answers + +Run the shared **answers** analysis for the `website-requirements` criteria +area. + +1. Read the criteria definition in + `.github/criteria/areas/website-requirements.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/answers.prompt.md`, treating `website-requirements` as + the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-answers.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/website-requirements-comment.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/website-requirements-comment.prompt.md new file mode 100644 index 00000000..3b81f29f --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/website-requirements-comment.prompt.md @@ -0,0 +1,21 @@ +--- +agent: agent +description: Write a comment on the website requirements of the documentation. +--- + +# Website Requirements comment + +Run the shared **comment** analysis for the `website-requirements` criteria +area. + +1. Read the criteria definition in + `.github/criteria/areas/website-requirements.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/comment.prompt.md`, treating `website-requirements` as + the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-comment.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/website-requirements-full.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/website-requirements-full.prompt.md new file mode 100644 index 00000000..0da3eefb --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/website-requirements-full.prompt.md @@ -0,0 +1,23 @@ +--- +agent: agent +description: + Run the full `website-requirements` analysis (answers, comment, and + recommendations) for the documentation. +--- + +# Website Requirements full analysis + +Run the shared **full** analysis for the `website-requirements` criteria area. + +1. Read the criteria definition in + `.github/criteria/areas/website-requirements.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/full.prompt.md`, treating `website-requirements` as + the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the base name for all three output files (`<title>-answers.md`, +`<title>-comment.md`, `<title>-recommendations.md`); otherwise use each engine's +default (`<project-slug>-<stem>-<type>.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/areas/website-requirements-recommendations.prompt.md b/docs/analysis/ai-harness/_github/prompts/areas/website-requirements-recommendations.prompt.md new file mode 100644 index 00000000..3fc8c187 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/areas/website-requirements-recommendations.prompt.md @@ -0,0 +1,23 @@ +--- +agent: agent +description: + Provide recommendations for improving the website requirements of the + documentation. +--- + +# Website Requirements recommendations + +Run the shared **recommendations** analysis for the `website-requirements` +criteria area. + +1. Read the criteria definition in + `.github/criteria/areas/website-requirements.md`. +2. Read and follow the shared procedure in + `.github/prompts/areas/recommendations.prompt.md`, treating + `website-requirements` as the selected area. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<stem>-recommendations.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/issues/create-issues.prompt.md b/docs/analysis/ai-harness/_github/prompts/issues/create-issues.prompt.md new file mode 100644 index 00000000..17142710 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/issues/create-issues.prompt.md @@ -0,0 +1,195 @@ +--- +agent: agent +description: + Create numbered GitHub issue files in .github/output/issues/ from the ranked + analysis recommendations, one file per issue, each scoped to 2-4 hours. Run + /create-umbrella-issues afterwards to group them. +--- + +# Create documentation issues from the analysis recommendations + +## Role and goal + +You are a senior technical writer on the CNCF TechDocs team. Turn the ranked +recommendations from the project's documentation analysis into GitHub issue +files that a technical writer or engineer with project context can pick up and +complete. Project maintainers will later upload every file in the output +directory to GitHub, so each file must be a complete, uploadable issue. + +## Project context + +Read the "Current Repository" section of `.github/copilot-instructions.md` +before you start. It supplies every project-specific value used below: + +- Project name, from which you derive `<project-slug>` (lowercase, spaces + replaced with hyphens). +- Published documentation URL (`<docs URL>`). +- Issues repository URL (`<issues repo URL>`), used for `repo:` in the front + matter. +- Issue template file name under `.github/templates/` (`<issue template>`). +- CNCF project page URL (`<CNCF project URL>`). +- Assistance-request issue (`cncf/techdocs#<NNN>`). + +If any of these fields is still a placeholder, stop and ask the writer to fill +it in. + +This prompt creates standalone issues only. Do not create umbrella issues; a +separate prompt, `/create-umbrella-issues`, groups the issues you create here +after the writer has reviewed them. + +## Sources + +Use only these sources. Do not add issues from your general knowledge of the +project's repositories or documentation; the assessment defines the scope. + +- `.github/input/ai-ranked-recommendations.csv` — the ranked recommendations. + Columns: `Section`, `Area`, `Recommendation`, `Suggested Priority` + (`1 - Urgent`, `2 - High`, `3 - Backlog`). +- `.github/output/analysis/<project-slug>-*-recommendations.md` — the per-area + recommendations files, for context on why each recommendation was made. +- `.github/output/analysis/<project-slug>-*-comment.md` — the per-area comments, + only when you need the problem statement behind a recommendation. +- `.github/templates/<issue template>` — the project's documentation issue + template (`doc-issue.md` by default). Every issue follows its headings. +- https://github.com/kedacore/keda-docs/issues/1363 — an example of a CNCF + TechDocs issue from a previous analysis. Match its scope and tone: a specific + problem, a concrete proposed change, and links to the affected pages. + +Ignore any sub-folders of `.github/output/issues/`; they are archives of +previous runs. + +## Procedure + +1. Read the CSV and the recommendations files. Group the recommendations by + section and area. +2. Decide which recommendations are issues. A recommendation is an issue when it + describes a concrete problem or gap in the documentation that a contributor + can fix. Skip recommendations that are optional improvements with minimal + impact, that depend on decisions the maintainers have not made, or that + cannot be acted on without information the assessment does not provide. + Record every skipped recommendation and the reason in the coverage report + (see Outputs). +3. Scope every issue to about two to four hours of work for a technical writer + or engineer with project context. Split a recommendation that is larger than + that into several issues; merge recommendations that are smaller into one + issue. Merged recommendations should address the same pages or the same kind + of change so the issue stays coherent. +4. Give each issue a title that is specific and imperative, for example "Add a + Getting Started page that walks through the first deployment". A reader + should understand the task from the title alone, because umbrella issues will + list titles only. +5. Number the issues. Issue numbers are five digits and start at 9001. Increment + by one for each issue, in the order you write them. Numbers simulate GitHub + issue numbers so the files can cross-reference each other before upload. +6. Assign a priority label to every issue. Start from the CSV ranking without + its number prefix (`1 - Urgent` → `urgent`, `2 - High` → `high`, + `3 - Backlog` → `backlog`). When an issue merges recommendations with + different rankings, use the highest. You may adjust a priority when merging + or splitting changes the impact; note any adjustment in the coverage report. +7. Keep the total number of issues well below the number of recommendations. A + good target is 40 to 60 issues. If you exceed that, merge further. + +## Issue file format + +Every file in `.github/output/issues/` starts with this front matter, followed +by the issue body: + +```yaml +--- +title: <GitHub issue title> #<issue number> +labels: [<urgent | high | backlog>, help wanted] +repo: <issues repo URL> +--- +``` + +- `title` becomes the GitHub issue title. Append the simulated issue number + after a space and `#`, for example + `title: Add a user-facing troubleshooting page #9007`. +- `labels` becomes the GitHub issue labels. Include exactly one of `urgent`, + `high`, or `backlog`, and always include `help wanted`. Do not add any other + labels. +- `repo` is always the issues repository URL from "Current Repository". +- Everything after the front matter is the issue description. Write it in + GitHub-flavored Markdown. + +Follow the issue template and keep its top-level headings in this order. Omit +any HTML comment from the template. The default template has these four: + +- `## Description` — state the problem or gap, where it is, and why it matters + to users. Then describe the proposed change concretely enough that a + contributor can start without further research: which pages or files to add, + edit, or move, and what the result should contain. Use a bulleted list or a + proposed outline when the change has several parts. +- `## What you expected` — describe what done looks like from the reader's point + of view, in one to three sentences. This is the acceptance criterion. +- `## URL` — link to the affected pages on `<docs URL>` and name the source + Markdown files. For issues that affect the project website or a repository + file rather than a documentation page, link to the affected page or file in + its repository. +- `## Additional context` — end with this sentence, adapted as needed: "This + issue tracks recommended changes resulting from an analysis of the + <project name> documentation commissioned by [CNCF](<CNCF project URL>). The + assistance request is tracked in + [cncf/techdocs#<NNN>](https://github.com/cncf/techdocs/issues/<NNN>)." Before + it, add anything else a contributor needs: related issues by simulated number + (for example "Depends on #9003"), decisions the maintainers must make first, + or questions for the maintainers. + +Add subheadings within a section when they help organize the content, but do not +add or remove top-level headings. + +In all issue bodies: + +- Write in complete sentences, active voice, and the imperative mood. +- Explain any term a senior engineer in the project's ecosystem might not know; + do not explain the ecosystem's basics. +- Follow the project's documentation conventions as described in "Current + Repository" (for Kubernetes-ecosystem projects, UpperCamelCase for API objects + and code formatting for commands, fields, and file names). +- If you are unsure whether something is technically wrong, phrase it as a + question for the maintainers rather than asserting it. +- Do not mention the CSV, the priority ranking, or this prompt in the issue + body. The body is for the project's contributors. + +## Outputs + +1. Issue files in `.github/output/issues/`. Name each file + `<issue number>-<slug>.md`, for example `9007-add-troubleshooting-page.md`. + Build the slug from the title with these rules: + - Use lowercase words separated by hyphens, no punctuation, and no more than + about five words. + - Keep the action verb and the main subject noun; drop articles, + conjunctions, and secondary clauses. Name the topic (`navigation`, + `footer`, `virtctl`), not the analysis area. + - When the title lists several changes, use a collective noun such as + `improvements` or `updates` instead of listing them. + - Strip punctuation from version numbers (`v1.9` becomes `v19`). + + Examples: "Add a Getting Started page that walks through the first + deployment" becomes `9001-add-getting-started-page.md`; "Move Release Notes + out of the main navigation" becomes `9004-move-release-notes.md`; "Document + the v1.9 VirtualMachineBackup API" becomes `9012-document-v19-backup-api.md`. + + Before writing, delete any `<number>-*.md` and `U-*.md` files already in + `.github/output/issues/` from a previous run so the numbering restarts + cleanly at 9001. Leave sub-folders and `readme.md` alone. + +2. A coverage report at `.github/output/issues-coverage.md` (outside the issues + directory so it is not uploaded as an issue). Overwrite any existing report. + Include, in this order: + - An `## Issues` table with the columns `Issue` (the simulated number, for + example `#9001`), `Title`, `Label`, and `File`. The + `/create-umbrella-issues` prompt reads this table to continue the numbering + and to look up titles, so list every issue file here. + - A `## Recommendation mapping` table mapping every CSV recommendation + (numbered by its row order) to the issue number that covers it, or "Not an + issue" with a one-sentence reason. A recommendation may map to several + issues; an issue may cover several recommendations. + - A `## Priority adjustments` table with the recommendation, its CSV ranking, + the label you assigned, and the reason. + - A `## Counts` list with the number of recommendations, recommendations + turned into issues, recommendations skipped, and issue files written. + +When finished, summarize the issues you created (number, title, label) and ask +the writer to review the scope of each one before running +`/create-umbrella-issues`. diff --git a/docs/analysis/ai-harness/_github/prompts/issues/create-umbrella-issues.prompt.md b/docs/analysis/ai-harness/_github/prompts/issues/create-umbrella-issues.prompt.md new file mode 100644 index 00000000..338db493 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/issues/create-umbrella-issues.prompt.md @@ -0,0 +1,136 @@ +--- +agent: agent +description: + Group the numbered issue files in .github/output/issues/ into umbrella issues + of 3-10 sub-issues each. Run after /create-issues and after the writer has + reviewed the issues. +--- + +# Create umbrella issues from the documentation issues + +## Role and goal + +You are a senior technical writer on the CNCF TechDocs team. The +`/create-issues` prompt has already turned the project's documentation analysis +into numbered, standalone issue files. Group those issues into umbrella issues +so maintainers can track related work as a unit and so CNCF can report progress +on the analysis. Each umbrella issue is itself an uploadable issue file that +links to its sub-issues; it does not repeat their content. + +## Project context + +Read the "Current Repository" section of `.github/copilot-instructions.md` for +the project name, the issues repository URL (`<issues repo URL>`), the CNCF +project page URL (`<CNCF project URL>`), and the assistance-request issue +(`cncf/techdocs#<NNN>`). If any of these is still a placeholder, stop and ask +the writer to fill it in. + +## Sources + +- `.github/output/issues/<number>-*.md` — the standalone issue files. Read every + one. Ignore `readme.md` and any sub-folders of `.github/output/issues/`; the + sub-folders are archives of previous runs. +- `.github/output/issues-coverage.md` — the coverage report. Its `## Issues` + table lists every issue number, title, label, and file; use it to look up + titles and to find the next free issue number. +- `.github/templates/umbrella-issue.md` — the umbrella issue template. +- https://github.com/kedacore/keda-docs/issues/1361 — an example umbrella issue + from a previous CNCF TechDocs analysis. Match its purpose: a short statement + of what the group achieves, followed by a checklist of sub-issues. Do not copy + its "Use-Case:" and "Specification:" heading prefixes. + +Do not read the recommendations CSV or the analysis files. The issue files are +the only input; if an issue is missing or wrongly scoped, tell the writer to fix +it with `/create-issues` instead of working around it here. + +## Procedure + +1. Read the `## Issues` table in the coverage report and every standalone issue + file. Note each issue's number, title, label, and the pages or repository it + touches. +2. Group the issues into themes. Group by the outcome a reader or maintainer + gets, not by the analysis section or area an issue came from; a theme may + span sections. Typical themes are a new-user path, navigation and information + architecture, a feature-coverage gap, site configuration, accessibility, + contributor onboarding, community channels, and site ownership. +3. Size each umbrella at three to ten sub-issues. Split a theme that has more + than ten; merge or drop a theme that has fewer than three. An issue belongs + to at most one umbrella. An issue that fits no theme stays standalone; that + is acceptable, and you should list such issues in the coverage report rather + than force them into an umbrella. +4. Order the sub-issues within each umbrella in the sequence a contributor + should work them: prerequisites first, then by priority (`urgent`, `high`, + `backlog`). +5. Assign each umbrella the highest priority label among its sub-issues. +6. Number the umbrellas. Continue the sequence from the coverage report: the + first umbrella takes the number after the highest number in the `## Issues` + table, and each further umbrella increments by one. +7. Give each umbrella a title that names the outcome, for example "Create a + new-user getting started path" or "Fix navigation gaps in the user guide". Do + not use "Umbrella" in the title; the body identifies it as an umbrella issue. + +## Umbrella file format + +Every umbrella file starts with this front matter, followed by the body: + +```yaml +--- +title: <umbrella issue title> #<issue number> +labels: [<urgent | high | backlog>, help wanted] +repo: <issues repo URL> +--- +``` + +Follow `umbrella-issue.md`, replacing its `{issues repo URL}` placeholder with +the issues repository URL, and keep its two headings: + +- `## Umbrella Issue` — in one or two paragraphs, state the problem the theme + addresses, how the sub-issues together resolve it, and the order in which to + work them if it matters. End with this sentence, adapted as needed: "This + umbrella issue tracks recommended changes resulting from an analysis of the + <project name> documentation commissioned by [CNCF](<CNCF project URL>). The + assistance request is tracked in + [cncf/techdocs#<NNN>](https://github.com/cncf/techdocs/issues/<NNN>)." +- `## Sub-issues` — a bulleted list with one line per sub-issue, in work order, + formatted as a link whose text is the sub-issue title followed by its number + and whose target is the sub-issue file name relative to the umbrella file: + + <!-- markdownlint-disable MD013 --> + + ```markdown + - [Add a Getting Started page that walks through the first deployment #9001](9001-add-getting-started-page.md) + - [Rewrite Basic Use as "Your first deployment" #9002](9002-rewrite-basic-use.md) + ``` + + <!-- markdownlint-enable MD013 --> + + The upload process replaces the `#<number>` references with real issue numbers + and links, so every title and number must match the sub-issue's front matter + exactly. Do not describe the sub-issues here; the link is enough. + +Write in complete sentences, active voice, and the imperative mood. Do not +mention the CSV, the priority ranking, or this prompt in the body. + +## Outputs + +1. Umbrella files in `.github/output/issues/`. Name each file + `U-<issue number>-<slug>.md`, for example + `U-9061-create-getting-started-path.md`. Build the slug from the title with + the same rules as standalone issues: lowercase hyphenated words, no more than + about five, keeping the action verb and main subject noun and naming the + topic rather than the analysis area. Before writing, delete any `U-*.md` + files already in `.github/output/issues/` from a previous run. Do not modify + the standalone issue files. +2. Update `.github/output/issues-coverage.md`: + - Append the umbrella issues to the `## Issues` table with the same columns + (`Issue`, `Title`, `Label`, `File`). + - Add a `## Umbrella issues` section after the `## Issues` table with one + subsection per umbrella listing its number, title, label, and sub-issue + numbers, followed by a list of standalone issues that belong to no + umbrella. + - Update the `## Counts` list with the number of umbrella issues and the + number of standalone issues not in an umbrella. + +When finished, summarize the umbrella issues you created (number, title, label, +sub-issue count) and list any issues left outside an umbrella. Ask the writer to +review the grouping. diff --git a/docs/analysis/ai-harness/_github/prompts/issues/rank-recommendations.prompt.md b/docs/analysis/ai-harness/_github/prompts/issues/rank-recommendations.prompt.md new file mode 100644 index 00000000..b981f502 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/issues/rank-recommendations.prompt.md @@ -0,0 +1,84 @@ +--- +agent: agent +description: + Collect every recommendation from the per-area recommendations files in + .github/output/analysis/ and rank them into + .github/input/ai-ranked-recommendations.csv for /create-issues. +--- + +# Rank the analysis recommendations + +## Role and goal + +You are a senior technical writer on the CNCF TechDocs team. The area prompts +have written one recommendations file per analysis area. Gather every +recommendation into a single CSV and assign each a priority so that +`/create-issues` can turn the ranked list into GitHub issues. Rank by the impact +on documentation users relative to the effort to implement; do not rewrite, +merge, or drop recommendations here. + +## Sources + +- The "Current Repository" section of `.github/copilot-instructions.md` — gives + the project name, from which you derive the project slug (lowercase, spaces + replaced with hyphens). +- `.github/output/analysis/<project-slug>-*-recommendations.md` — the per-area + recommendations files. Every bullet in these files is one recommendation. +- `.github/output/analysis/<project-slug>-*-comment.md` — the per-area comments, + for the `Rating:` line and the weaknesses that motivate each recommendation. +- `.github/criteria/sections/*.md` — map each area's comment-file stem to its + section display name and criterion label (the area display name). + +If any area listed in the section files has no recommendations file, stop and +tell the user which `/<area>-recommendations` or `/<area>-full` prompt to run +first. + +## Procedure + +1. Read every recommendations file. Record each bullet as one recommendation, + with its section display name and area display name taken from the section + criteria files. +2. Rank each recommendation on two factors: + - Impact: how much it improves users' ability to succeed with the project. + Gaps in the new-user path, missing task documentation for core features, + broken or misleading content, and accessibility blockers rank highest. The + area's rating is a signal: areas rated 1 or 2 usually hold the + highest-impact recommendations. + - Effort: a recommendation that a contributor can complete in a few hours + ranks above one of equal impact that needs maintainer decisions, new + tooling, or a large restructuring. +3. Assign a priority: + - `1 - Urgent` — high impact; blocks or badly misleads users. Aim for roughly + the top 15 to 20 percent. + - `2 - High` — clear improvement to effectiveness; should be done this cycle. + - `3 - Backlog` — worthwhile but low impact, cosmetic, or dependent on + decisions not yet made. +4. Keep the recommendation text as written in the source file, minus the leading + bullet marker. Do not paraphrase; `/create-issues` maps CSV rows back to the + source files. + +## Output + +Write `.github/input/ai-ranked-recommendations.csv`, overwriting any existing +file. This file will be used later to create issues based on the ranked +recommendations. Use these columns, in this order, with a header row: + +```csv +Section,Area,Recommendation,Suggested Priority +``` + +- `Section` — `Project Documentation`, `Contributor Documentation`, or + `Website & Infrastructure`. +- `Area` — the area display name (for example `Information Architecture`), not + the criteria filename or file stem. +- `Recommendation` — the recommendation text. Quote the field and double any + embedded quotes so the file is valid RFC 4180 CSV. +- `Suggested Priority` — `1 - Urgent`, `2 - High`, or `3 - Backlog`. + +Order rows by section (Project Documentation, Contributor Documentation, Website +& Infrastructure), then by area in the order the section file lists them, then +by priority. + +When finished, report the number of recommendations per priority and per +section, and ask the writer to review the ranking before running +`/create-issues`. diff --git a/docs/analysis/ai-harness/_github/prompts/sections/contributor-documentation-comment.prompt.md b/docs/analysis/ai-harness/_github/prompts/sections/contributor-documentation-comment.prompt.md new file mode 100644 index 00000000..96ffac43 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/sections/contributor-documentation-comment.prompt.md @@ -0,0 +1,23 @@ +--- +agent: agent +description: + Write an overall comment on the Contributor Documentation analysis section by + synthesizing its per-area comment files. +--- + +# Contributor Documentation section comment + +Run the shared **section-comment** analysis for the `contributor-documentation` +section. + +1. Read the section definition in + `.github/criteria/sections/contributor-documentation.md`. +2. Read and follow the shared procedure in + `.github/prompts/sections/section-comment.prompt.md`, treating + `contributor-documentation` as the selected section. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<section-slug>-comment.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/sections/project-documentation-comment.prompt.md b/docs/analysis/ai-harness/_github/prompts/sections/project-documentation-comment.prompt.md new file mode 100644 index 00000000..3622fa4f --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/sections/project-documentation-comment.prompt.md @@ -0,0 +1,23 @@ +--- +agent: agent +description: + Write an overall comment on the Project Documentation analysis section by + synthesizing its per-area comment files. +--- + +# Project Documentation section comment + +Run the shared **section-comment** analysis for the `project-documentation` +section. + +1. Read the section definition in + `.github/criteria/sections/project-documentation.md`. +2. Read and follow the shared procedure in + `.github/prompts/sections/section-comment.prompt.md`, treating + `project-documentation` as the selected section. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<section-slug>-comment.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/sections/section-comment.prompt.md b/docs/analysis/ai-harness/_github/prompts/sections/section-comment.prompt.md new file mode 100644 index 00000000..5971ef68 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/sections/section-comment.prompt.md @@ -0,0 +1,100 @@ +--- +agent: agent +description: + Shared engine for producing an overall comment on an analysis section by + synthesizing its per-area comment files. Normally run via a per-section + wrapper such as /project-documentation-comment. +--- + +# Shared engine: section-comment + +Reusable procedure for writing an overall comment on one analysis **section** +(for example, "Project Documentation"). Unlike the per-area `comment` engine, +this engine does not analyze the documentation directly — it synthesizes the +section's existing per-area comment files into a single overall comment. +Normally invoked through a per-section wrapper prompt (for example, +`/project-documentation-comment`), which selects the section for you. + +## Inputs + +- Section — the analysis section to comment on. When run from a wrapper, the + wrapper names it. If you reached this engine without a section, ask the user + to choose one of: `project-documentation`, `contributor-documentation`, + `website-infrastructure`. +- Title (optional) — overrides the default output filename. + +## Precondition + +This engine reads the per-area comment files that already exist in the +`.github/output/analysis` directory; it does not re-analyze the documentation. +Every area listed in the section definition must already have its per-area +comment file (produced by that area's `comment` wrapper, for example +`/information-architecture-comment`). If one or more are missing, stop and tell +the user exactly which per-area comment prompts to run first. Do not write a +partial section comment. + +## Procedure + +1. Read the section definition at `.github/criteria/sections/<section>.md`. Use + its Display name, Section slug, and the Areas table (each row gives an area's + criterion label and the stem of its per-area comment file). +2. Determine the project name and documentation label from the "Current + Repository" section of the repository Copilot instructions (for example, "the + KubeVirt user guide"). Derive the project slug by lowercasing the name and + replacing spaces with hyphens. +3. For each area row, read the existing per-area comment file + `<project-slug>-<stem>-comment.md` from the `.github/output/analysis` + directory. If any are missing, follow the Precondition above and stop. +4. From each area comment, extract its overall standing, its rating (the final + `Rating: <n> - <label>` line), and its key strengths and highest-impact gaps + or recommendations. +5. Build a summary ratings table with one row per area, in the order listed in + the section definition: + + | Criterion | Rating (1-5) | + | ----------------- | ------------- | + | <criterion label> | <n> - <label> | + + If an area comment has no `Rating:` line, use `N/A` for that row. + +6. If `website-infrastructure` is the current section, add this table and its + label for these other metrics: + + Other Metrics: + + | Criterion | [Rating (1-5)] | + | ------------------------------------------- | -------------- | + | A11y plan & implementation | [rating (1-5)] | + | Mobile-first plan & implementation | [rating (1-5)] | + | HTTPS access & HTTP redirect | [rating (1-5)] | + | Google Analytics 4 for production only | [rating (1-5)] | + | Indexing allowed for production server only | [rating (1-5)] | + | Intra-site / local search | [rating (1-5)] | + | Account custodians are documented | [rating (1-5)] | + + Fill each row from the relevant area comments (`website-requirements`, + `usability-accessibility-devices`, `seo-analytics`, `maintenance-planning`). + If the comments do not support a rating for a row, leave `[rating (1-5)]` in + place for the writer to complete. + +7. Write a two to five paragraph overall comment on the section beneath the + table(s). Synthesize rather than concatenate: lead with the section's overall + standing, describe only the most important cross-cutting themes that recur + across two or more areas. Do not describe detailed suggestions, as they + should be covered in the per-area comments and recommendations. Avoid large + paragraphs with listed details, so use bulleted lists where appropriate. Keep + it concise, actionable, and in complete sentences, following the repository's + analysis response style. + +## Output + +- Begin the file with a level-1 heading + `# <Section display name> - Overall Comment`. +- Follow the heading with the summary ratings table, then the two-to-five + paragraph comment. +- Write the result to a Markdown file in the `.github/output/analysis` + directory: + - If a title was provided, name the file `<title>.md`. + - Otherwise name it `<project-slug>-<section-slug>-comment.md`. + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/prompts/sections/website-infrastructure-comment.prompt.md b/docs/analysis/ai-harness/_github/prompts/sections/website-infrastructure-comment.prompt.md new file mode 100644 index 00000000..a5146ba1 --- /dev/null +++ b/docs/analysis/ai-harness/_github/prompts/sections/website-infrastructure-comment.prompt.md @@ -0,0 +1,23 @@ +--- +agent: agent +description: + Write an overall comment on the Website & Infrastructure analysis section by + synthesizing its per-area comment files. +--- + +# Website & Infrastructure section comment + +Run the shared **section-comment** analysis for the `website-infrastructure` +section. + +1. Read the section definition in + `.github/criteria/sections/website-infrastructure.md`. +2. Read and follow the shared procedure in + `.github/prompts/sections/section-comment.prompt.md`, treating + `website-infrastructure` as the selected section. + +Optional output filename override: `${input:title}` — if non-empty, use it as +the output filename; otherwise use the engine's default +(`<project-slug>-<section-slug>-comment.md`). + +Output the file to the `.github/output/analysis` directory. diff --git a/docs/analysis/ai-harness/_github/templates/doc-issue.md b/docs/analysis/ai-harness/_github/templates/doc-issue.md new file mode 100644 index 00000000..38288d46 --- /dev/null +++ b/docs/analysis/ai-harness/_github/templates/doc-issue.md @@ -0,0 +1,28 @@ +--- +title: { title } #{Issue number} +labels: [high, help wanted] +repo: { issues repo URL } +--- + +<!-- Generic documentation issue template. Replace this file with the target +project's own documentation issue template if it has one, keeping the four +top-level headings below (or update create-issues.prompt.md to match), and +name the file in the "Current Repository" section of copilot-instructions.md. --> + +## Description + +A clear and concise description of the problem or gap, where it is, and the +proposed change. + +## What you expected + +A clear and concise description of what done looks like for the reader. + +## URL + +Links to the affected documentation pages and source files. + +## Additional context + +Related issues, decisions the maintainers must make, and the sentence that links +the issue back to the CNCF TechDocs analysis. diff --git a/docs/analysis/ai-harness/_github/templates/umbrella-issue.md b/docs/analysis/ai-harness/_github/templates/umbrella-issue.md new file mode 100644 index 00000000..33d7df57 --- /dev/null +++ b/docs/analysis/ai-harness/_github/templates/umbrella-issue.md @@ -0,0 +1,18 @@ +--- +title: { title } #{Issue number} +labels: [high, help wanted] +repo: { issues repo URL } +--- + +## Umbrella Issue + +Describe the umbrella issue and how the sub-issues work to resolve it. + +## Sub-issues + +<!-- One line per sub-issue: the title and simulated number as link text, +the sub-issue file name (sibling of this file in .github/output/issues/) +as the target. For example: +- [Reorganize Concepts #9938](9938-reorganize-concepts.md) --> + +- [{issue title} #{issue number}]({issue-number}-{condensed-issue-title}.md) diff --git a/docs/analysis/ai-harness/agents.md b/docs/analysis/ai-harness/agents.md new file mode 100644 index 00000000..9a13ad18 --- /dev/null +++ b/docs/analysis/ai-harness/agents.md @@ -0,0 +1,1035 @@ +# AI Assisted Analysis Harness: agent instructions + +> **Prototype.** This file is a proposal to replace the `_github/` template +> folder (Copilot instructions, criteria files, prompt engines, prompt wrappers, +> and issue templates) with a single file in the [AGENTS.md](https://agents.md) +> format. It is not yet validated against a real analysis. Open decisions are +> marked `TODO` inline and collected in [Migration TODO](#migration-todo). + +This file tells an AI coding agent how to help a writer run a CNCF TechDocs +documentation analysis. Human reviewers still own the analysis. The agent +answers the criteria questions, drafts comments and ratings, proposes +recommendations, ranks them, and writes issue files. The writer reviews and +edits every output before it goes into `analysis.md` or the project's issue +tracker, so write output that needs minimal editing. + +The predefined tasks in this file exist so that every analysis uses the same +procedure. Do not improvise a different procedure when a task covers the +request. + +## How to use this file + +The writer asks for a task by name, followed by its argument. For example: + +```text +run full information-architecture +run section-comment project-documentation +run create-issues +``` + +The tasks are listed below. Each one is defined under [Tasks](#tasks). + +<!-- markdownlint-disable MD013 --> + +| Task | Argument | Result | +| ------------------------ | ---------- | ------------------------------------------------------ | +| `answers` | area ID | Answers to the area's criteria questions | +| `comment` | area ID | Comment, strengths, weaknesses, and rating for an area | +| `recommendations` | area ID | Recommendations for an area | +| `full` | area ID | All three area outputs in one run | +| `section-comment` | section ID | Ratings table and overall comment for a section | +| `rank-recommendations` | none | Ranked recommendations CSV | +| `create-issues` | none | Numbered standalone issue files and a coverage report | +| `create-umbrella-issues` | none | Umbrella issue files that link to the standalone ones | + +<!-- markdownlint-enable MD013 --> + +The area IDs and section IDs are defined under +[Sections and areas](#sections-and-areas). If the writer names a task without a +required argument, or names an area or section that is not in those tables, ask +which one they mean. Do not guess. + +Every task that writes analysis output also accepts an optional `title`, for +example `run answers information-architecture title: my-analysis`. See +[Conventions for all tasks](#conventions-for-all-tasks). + +Text that follows a task name is a correction for that run, for example +`run create-issues, and split the "Getting started" issue into two`. Apply the +correction on top of the task's procedure. + +## Working locations + +The tasks refer to four locations by the placeholder names in this table. + +<!-- markdownlint-disable MD013 --> + +| Placeholder | Meaning | Value | +| ------------ | --------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------- | +| `<techdocs>` | A clone of `cncf/techdocs`. The criteria and the analysis template are read from here. | The clone that contains this file | +| `<project>` | A clone of the project's documentation repository. The agent reads it and never writes to it. | See "Documentation repository path" in [Project context](#project-context) | +| `<work>` | The folder the agent writes to. | See "Working folder" in [Project context](#project-context) | +| `<criteria>` | The criteria file. | `<techdocs>/docs/analysis/criteria.md` | + +<!-- markdownlint-enable MD013 --> + +If `<criteria>` cannot be read from the clone, read it from +`https://github.com/cncf/techdocs/blob/main/docs/analysis/criteria.md` and tell +the writer that you did. + +The agent creates and uses this layout inside `<work>`: + +<!-- markdownlint-disable MD013 --> + +```text +<work>/ + input/ + ai-ranked-recommendations.csv # Written by rank-recommendations, read by create-issues + output/ + analysis/ + <project-slug>-<area>-<type>.md # Area outputs: answers, comment, recommendations + <project-slug>-<section>-comment.md # Section comments + issues/ + <number>-<slug>.md # Standalone issue files + U-<number>-<slug>.md # Umbrella issue files + issues-coverage.md # Issue list and recommendation-to-issue mapping +``` + +<!-- markdownlint-enable MD013 --> + +The writer can also place spreadsheets or other ad hoc inputs in `<work>/input/` +and ask the agent to read them. + +TODO: The location of `<work>` is undecided. The earlier harness wrote to +`.github/input` and `.github/output` in the project clone. Candidates are a +folder next to `analysis.md` in the `<techdocs>` clone, or a folder outside both +clones. See [Migration TODO](#migration-todo). + +## Project context + +This section is the only project-specific part of this file. Every task reads +its project values from here. Replace every `FILL IN` value before running a +task. If a task needs a field that still reads `FILL IN`, stop and ask the +writer to complete it. + +- Project name: `FILL IN` (for example, KubeVirt) +- Documentation label: `FILL IN` (how to refer to the documentation in prose, + for example, "the KubeVirt user guide") +- Ecosystem: `FILL IN` (the readers' domain, for example, Kubernetes; it sets + the terminology and style conventions) +- Main repository: `FILL IN` (name and URL) +- Secondary repositories: `FILL IN` (website or lab repositories the analysis + also covers, or "None") +- Published documentation URL: `FILL IN` +- Project website URL: `FILL IN` +- Documentation repository path: `FILL IN` (the local path of `<project>`) +- Working folder: `FILL IN` (the local path of `<work>`) +- Issues repository URL: `FILL IN` (used for `repo:` in issue front matter, for + example, `https://github.com/kubevirt/user-guide/issues`) +- CNCF project page URL: `FILL IN` (for example, + `https://www.cncf.io/projects/kubevirt/`) +- Assistance-request issue: `FILL IN` (for example, `cncf/techdocs#343`) + +The project slug, written `<project-slug>` in file names, is the project name in +lowercase with spaces replaced by hyphens. For example, "KubeVirt" becomes +`kubevirt`. + +### Priority label mapping + +The upload process maps the harness labels to the labels of the issues +repository. Confirm that each repository label exists before uploading. + +| Harness label | Repository label | +| ------------- | --------------------------------------------------- | +| `urgent` | `FILL IN` (for example, `priority/critical-urgent`) | +| `high` | `FILL IN` (for example, `priority/high`) | +| `backlog` | `FILL IN` (for example, `priority/backlog`) | +| `help wanted` | `help wanted` | + +### What the documentation repository is + +`FILL IN`: one paragraph that states what the repository publishes, the site +generator and theme, where the content lives, and whether the repository +contains application code. + +### Content architecture + +`FILL IN`: a short tree of the documentation source folders with a comment for +each folder. + +### Project conventions + +`FILL IN`: one short paragraph for each convention the agent must respect when +it reads or refers to project files. Examples are navigation configuration, +redirects, spelling dictionaries, and Markdown extensions. + +### Project issue template + +If the project has its own documentation issue format, list its top-level +headings here in order. Otherwise write "Default", and the `create-issues` task +uses the [standalone issue template](#standalone-issue-template). + +`FILL IN` + +## Rules for every task + +### No commits and no changes to the project + +This work is solely for documentation analysis. Do not commit anything. Do not +create, edit, or delete any file in `<project>`. Write only inside `<work>`. + +### What to look for + +Keep the goal of technical documentation in mind: to make its users more +effective. Focus on what gets in the way of that goal. + +Look for the following: + +- Quick wins: low-effort changes that would have a major impact. +- Large, systemic problems that a number of issues can be organized around. +- The two or three most important problems that impede documentation + effectiveness. +- Anything the project does exceptionally well, which can serve as an example in + other analyses. + +Do not get bogged down in detail. Include enough detail to describe how to +implement a suggested solution; a sentence or two is sufficient for most +problems. + +A recommendation is not required in every case. Omit one when the element is +satisfactory or when the change would bring minimal improvement. + +### Writing style + +Readers of the analysis are subject-matter experts in the project's ecosystem +who want to improve the documentation quickly. Be concise and actionable, and +skip explanations they do not need. + +- Write in complete sentences, using active voice and the imperative mood. +- Fragments are fine inside a bulleted list that follows an introductory + sentence. +- Explain any term or acronym that a senior engineer in the project's ecosystem + might not know. +- Do not list technologies in parentheses; readers already know them. +- Use parentheses only when the content is needed to understand the + recommendation. +- When a sentence would list four or more items, end the paragraph with an + introductory sentence followed by a bulleted list of the items. +- Refer to a page by its title. A Markdown file name is acceptable when the name + makes the page obvious. Relative paths are not needed when the title is clear. +- Keep paragraphs under about four sentences; split longer ones. +- Treat any paragraph limit in a task as a maximum, not a target. +- If you are unsure whether something is technically wrong, phrase it as a + question for the subject-matter expert rather than asserting it. +- Follow the documentation conventions of the project's ecosystem. For + Kubernetes-ecosystem projects, that means UpperCamelCase for API objects (Pod, + Deployment) and code formatting for commands, fields, and file names. + +## Sections and areas + +The analysis has three sections, and each section has several areas. The tables +in this part are the data that the tasks operate on. To change which areas +exist, or how they are named, edit these tables and nothing else. + +### Sections + +| Section ID | Display name | +| --------------------------- | ------------------------- | +| `project-documentation` | Project Documentation | +| `contributor-documentation` | Contributor Documentation | +| `website-infrastructure` | Website & Infrastructure | + +### Areas + +Each row is one area, listed in the order the analysis presents them. The +columns have these meanings: + +- Area ID: the name the writer uses in a task, and the `<area>` part of the + output file names. +- Display name: the name used in prose and in the `Area` column of the ranked + recommendations CSV. +- Criterion label: the label of the area's row in the section ratings table of + `analysis.md`. +- Questions heading: the heading in `<criteria>` under which the area's + questions are listed. + +<!-- markdownlint-disable MD013 --> + +| Area ID | Display name | Section ID | Criterion label | Questions heading in `<criteria>` | +| ---------------------------------- | --------------------------------------- | --------------------------- | ----------------------------------------- | ---------------------------------------------------- | +| `information-architecture` | Information Architecture | `project-documentation` | Information architecture | Information architecture questions | +| `new-user-content` | New User Content | `project-documentation` | New user content | New user content questions | +| `content-maintainability` | Content Maintainability | `project-documentation` | Content maintainability | Content maintainability and site mechanics questions | +| `content-creation-process` | Content Creation Process | `project-documentation` | Content creation processes | Content creation processes questions | +| `inclusive-language` | Inclusive Language | `project-documentation` | Inclusive language | Inclusive language questions | +| `communication-methods-documented` | Communication Methods Documented | `contributor-documentation` | Communication methods documented | Communication methods documented questions | +| `beginner-issue-backlog` | Beginner Friendly Issue Backlog | `contributor-documentation` | Beginner friendly issue backlog | Beginner friendly issue backlog questions | +| `new-contributor-content` | New Contributor Getting Started Content | `contributor-documentation` | "New contributor" getting started content | New contributor getting started content questions | +| `project-governance-documentation` | Project Governance Documentation | `contributor-documentation` | Project governance documentation | Project governance documentation questions | +| `website-requirements` | Core Website Requirements | `website-infrastructure` | Core website requirements | Core website requirements questions | +| `usability-accessibility-devices` | Usability, Accessibility, and Devices | `website-infrastructure` | Usability, accessibility, and design | Usability, accessibility, and devices questions | +| `branding-design` | Branding and Design | `website-infrastructure` | Branding and design | Branding and design questions | +| `case-studies` | Case Studies and Social Proof | `website-infrastructure` | Case studies and social proof | Case studies and social proof questions | +| `seo-analytics` | SEO, Analytics, and Site Search | `website-infrastructure` | SEO, Analytics, and site-local search | SEO, Analytics and site-local search questions | +| `maintenance-planning` | Maintenance Planning | `website-infrastructure` | Maintenance planning | Maintenance planning questions | +| `agent-readiness` | AI Agent Readiness | `website-infrastructure` | AI agent readiness | AI Agent Readiness questions | + +<!-- markdownlint-enable MD013 --> + +TODO: This table assumes the `criteria.md` structure proposed in +[cncf/techdocs#423](https://github.com/cncf/techdocs/pull/423), which is not +merged yet. That structure folds the single-source requirement into "Core +website requirements", so this table has sixteen areas where the earlier harness +had seventeen. + +### How to read an area's criteria + +The questions are not copied into this file. `<criteria>` is their only source. +To read the criteria for an area: + +1. Open `<criteria>` and find the heading whose text matches the area's + "Questions heading" exactly. +2. Take every list item under that heading, up to the next heading or the next + paragraph that is not part of the list. These are the area's questions. Use + them verbatim and in order. +3. If an `Example:` or `Examples:` list follows the questions, treat its links + as models of quality for the area. +4. Read the descriptive paragraphs between the area's own heading and its + questions heading for context on what the area evaluates. + +If the heading is missing from `<criteria>`, stop and report it. Do not answer +questions from memory or from another version of the criteria. + +### Area-specific instructions + +Apply these in addition to the task procedure. An area that is not listed has +none. + +- `inclusive-language`: Focus the comment on any inclusive-language concerns in + the documentation. +- `agent-readiness`: The reference sources for this area are the llms.txt + specification (`https://llmstxt.org/`), the CLOMonitor agent readiness checks + (`https://clomonitor.io/docs/topics/checks/#agent-readiness`), and the AFDocs + Agent-Friendly Documentation Spec (`https://afdocs.dev/`). Where possible, + verify by fetching the live site (`/llms.txt`, or a page URL with `.md` + appended) rather than relying on the repository source alone. + +## Tasks + +### Conventions for all tasks + +- Read [Project context](#project-context) first. Derive `<project-slug>` from + the project name. +- Write analysis output to `<work>/output/analysis/`. Create the folder if it + does not exist. +- Name each file as its task specifies. If the writer provides a `title`, name + the file `<title>.md` instead. For the `full` task, the title is a base name: + write `<title>-answers.md`, `<title>-comment.md`, and + `<title>-recommendations.md`. +- If a file with the same name exists, overwrite it. +- Begin each analysis output file with a level-1 heading that names its content, + for example `# Information Architecture - Answers`. The writer drops this + heading when pasting the content into `analysis.md`. +- Follow [Rules for every task](#rules-for-every-task). + +### Task: answers + +Answer the criteria questions for one area. + +Procedure: + +1. Read the area's row in the [Areas](#areas) table and its criteria, as + described in + [How to read an area's criteria](#how-to-read-an-areas-criteria). +2. Investigate the documentation in `<project>`, the published documentation, + and the other sources named in [Project context](#project-context). +3. Answer every question. + +Output: + +- Format each question as a bullet, not as a heading. Indent the answer by two + spaces under its question. Do not bold the answers. +- If an answer is "yes" or "no", add a brief explanation. +- Keep each answer to one or two short paragraphs. +- State what exists, where it is, and what is missing. Do not say what to do + about it; that belongs in the recommendations. +- Write the file `<project-slug>-<area>-answers.md`. + +### Task: comment + +Write a high-level comment and a rating for one area. + +Procedure: + +1. Read the area's row in the [Areas](#areas) table, its criteria, and any + [area-specific instructions](#area-specific-instructions). +2. Write one to three paragraphs of high-level comment. Do not repeat points + from the answers output. +3. Follow with two bulleted lists, one of strengths and one of weaknesses. + Precede the lists with the plain-text labels "Strengths:" and "Weaknesses:". + Do not bold the labels. +4. Refer to the examples listed in the criteria as models of quality where that + helps. +5. Rate the area on the CNCF scale: + - 1 - Not present + - 2 - Needs improvement + - 3 - Meets standards + - 4 - Meets or exceeds standards + - 5 - Exemplary + +Output: + +- End the comment with the rating on its own line, in the form + `Rating: <number> - <label>`, for example `Rating: 3 - Meets standards`. Put + no other text on that line. The `section-comment` and `rank-recommendations` + tasks read it. +- Do not put recommendations in the comment. +- Write the file `<project-slug>-<area>-comment.md`. + +### Task: recommendations + +Write recommendations for one area, based on its answers and comment. + +Procedure: + +1. Read the area's row in the [Areas](#areas) table. +2. Read `<project-slug>-<area>-answers.md` and + `<project-slug>-<area>-comment.md` from `<work>/output/analysis/`. If either + file is missing, ask the writer to run the `answers` or `comment` task first, + or to point you at the files. +3. If the comment file contains a recommendation, move it to the recommendations + file. +4. Base every recommendation on those two files. + +Output: + +- Introduce the recommendations with this exact sentence, filling in the two + values: "The following recommendations address the `<display name>` of + `<documentation label>`." For example: "The following recommendations address + the information architecture of the KubeVirt user guide." +- Follow the sentence with a bulleted list, one recommendation per top-level + bullet. The `rank-recommendations` task treats each top-level bullet as one + recommendation. +- Write the file `<project-slug>-<area>-recommendations.md`. + +### Task: full + +Run the `answers`, `comment`, and `recommendations` tasks for one area in a +single pass. Start a new session for each area so that context from one area +does not leak into the next. + +Analyze first and write last. While answering the questions and forming the +comment, you will notice things to fix. Those observations belong in the +recommendations file, not in the answers or the comment. Collect everything +before writing so that each file contains only its own kind of content. + +Phase 1, read: + +1. Read the area's row in the [Areas](#areas) table, its criteria, and any + [area-specific instructions](#area-specific-instructions). +2. Reread the `answers`, `comment`, and `recommendations` tasks so that you know + the required structure of each output before you start. + +Phase 2, analyze and sort: + +1. Investigate the documentation as the `answers` task directs. As you work, + keep three running lists: + - Findings: factual observations that answer a question (what exists, where + it is, and what is missing). + - Assessments: high-level judgments about strengths and weaknesses, and the + evidence for the rating. + - Fixes: anything you would tell the project to change or add. +2. Whenever an observation suggests a fix, record the fix in the Fixes list and + keep only the underlying finding in the Findings or Assessments list. A + finding may state that something is missing or unclear. It must not say what + to do about it. +3. Draft all three documents before writing any of them: the answers from the + Findings list, the comment from the Assessments list, and the recommendations + from the Fixes list. +4. Review the drafts against each other: + - Remove any imperative or "should" phrasing from the answers and the + comment, and move it to the recommendations. + - Confirm that every recommendation traces back to something stated in the + answers or the comment. If it does not, add the supporting finding or drop + the recommendation. + - Confirm that the answers and the comment do not duplicate each other. The + comment synthesizes; it does not restate. + +Phase 3, write: + +1. Write the answers file, then the comment file, then the recommendations file, + each as its own task specifies. +2. List the three file names you created. + +### Task: section-comment + +Write an overall comment for one section by synthesizing the comment files of +its areas. This task does not analyze the documentation again. + +Precondition: every area of the section in the [Areas](#areas) table must +already have its file `<project-slug>-<area>-comment.md` in +`<work>/output/analysis/`. If any are missing, stop and tell the writer exactly +which `comment` or `full` tasks to run first. Do not write a partial section +comment. + +Procedure: + +1. Read the section's row in the [Sections](#sections) table and its areas, in + table order, from the [Areas](#areas) table. +2. Read each area's comment file. From each, extract the overall standing, the + rating from the final `Rating:` line, the key strengths, and the + highest-impact gaps. +3. Build a summary ratings table with one row for each area, in table order. Use + the area's criterion label and its rating in the form `<number> - <label>`. + If a comment file has no `Rating:` line, use `N/A`. + + | Criterion | Rating (1-5) | + | ------------------------ | ------------------- | + | Information architecture | 3 - Meets standards | + +4. Write a two to five paragraph overall comment beneath the table. Synthesize + rather than concatenate. Lead with the section's overall standing, then + describe only the most important themes that recur across two or more areas. + Leave detailed suggestions to the area comments and recommendations. Use a + bulleted list instead of a long paragraph of details. + +Output: + +- Begin the file with the level-1 heading + `# <Section display name> - Overall Comment`, followed by the ratings table + and then the comment. +- Write the file `<project-slug>-<section>-comment.md`. + +TODO: The earlier engine also filled an "Other Metrics" table for the Website & +Infrastructure section. cncf/techdocs#423 removes that table from the analysis +template, so this task omits it. Restore the step if the table stays. + +### Task: rank-recommendations + +Gather every recommendation into one CSV file and assign each a priority, so +that the `create-issues` task can turn the ranked list into issues. Rank only. +Do not rewrite, merge, or drop recommendations. + +Sources: + +- `<work>/output/analysis/<project-slug>-<area>-recommendations.md` for every + area in the [Areas](#areas) table. Each top-level bullet is one + recommendation. +- `<work>/output/analysis/<project-slug>-<area>-comment.md` for the `Rating:` + line and the weaknesses that motivate each recommendation. + +If any area has no recommendations file, stop and tell the writer which +`recommendations` or `full` task to run first. + +Procedure: + +1. Read every recommendations file. Record each top-level bullet as one + recommendation, with the display names of its section and area. +2. Rank each recommendation on two factors: + - Impact: how much it improves users' ability to succeed with the project. + Gaps in the new-user path, missing task documentation for core features, + broken or misleading content, and accessibility blockers rank highest. The + area's rating is a signal: areas rated 1 or 2 usually hold the + highest-impact recommendations. + - Effort: a recommendation that a contributor can complete in a few hours + ranks above one of equal impact that needs maintainer decisions, new + tooling, or a large restructuring. +3. Assign a priority: + - `1 - Urgent`: high impact; blocks or badly misleads users. Aim for roughly + the top 15 to 20 percent. + - `2 - High`: a clear improvement to effectiveness that should be done in + this cycle. + - `3 - Backlog`: worthwhile but low impact, cosmetic, or dependent on + decisions not yet made. +4. Keep the recommendation text as written in the source file, minus the leading + bullet marker. Do not paraphrase, because the `create-issues` task maps CSV + rows back to the source files. + +Output: + +Write `<work>/input/ai-ranked-recommendations.csv`, overwriting any existing +file. Use these columns, in this order, with a header row: + +```csv +Section,Area,Recommendation,Suggested Priority +``` + +- `Section`: the section display name. +- `Area`: the area display name, not the area ID. +- `Recommendation`: the recommendation text. Quote the field and double any + embedded quotes so that the file is valid RFC 4180 CSV. +- `Suggested Priority`: `1 - Urgent`, `2 - High`, or `3 - Backlog`. + +Order the rows by section and area in the order of the [Areas](#areas) table, +then by priority. + +When finished, report the number of recommendations for each priority and for +each section. Ask the writer to review the ranking before running +`create-issues`. + +### Task: create-issues + +Act as a senior technical writer on the CNCF TechDocs team. Turn the ranked +recommendations into issue files that a technical writer or engineer with +project context can pick up and complete. Every file in the issues folder is +later uploaded to GitHub, so each file must be a complete, uploadable issue. + +This task creates standalone issues only. The `create-umbrella-issues` task +groups them after the writer has reviewed them. + +Sources. Use only these. Do not add issues from general knowledge of the +project; the analysis defines the scope. + +- `<work>/input/ai-ranked-recommendations.csv`: the ranked recommendations. +- `<work>/output/analysis/<project-slug>-<area>-recommendations.md`: context on + why each recommendation was made. +- `<work>/output/analysis/<project-slug>-<area>-comment.md`: only when you need + the problem statement behind a recommendation. +- The issue template: the headings under + [Project issue template](#project-issue-template), or the + [standalone issue template](#standalone-issue-template) by default. +- `https://github.com/kedacore/keda-docs/issues/1363`: an example issue from a + previous analysis. Match its scope and tone: a specific problem, a concrete + proposed change, and links to the affected pages. + +Ignore any sub-folders of `<work>/output/issues/`; they are archives of previous +runs. + +Procedure: + +1. Read the CSV and the recommendations files. Group the recommendations by + section and area. +2. Decide which recommendations are issues. A recommendation is an issue when it + describes a concrete problem or gap that a contributor can fix. Skip + recommendations that are optional improvements with minimal impact, that + depend on decisions the maintainers have not made, or that cannot be acted on + without information the analysis does not provide. Record every skipped + recommendation and the reason in the coverage report. +3. Scope every issue to about two to four hours of work for a technical writer + or engineer with project context. Split a larger recommendation into several + issues. Merge smaller recommendations into one issue when they address the + same pages or the same kind of change. +4. Give each issue a specific, imperative title, for example "Add a Getting + Started page that walks through the first deployment". A reader must + understand the task from the title alone, because umbrella issues list titles + only. +5. Number the issues as described in [Issue numbers](#issue-numbers), starting + at 9001. +6. Assign one priority label to every issue. Start from the CSV ranking without + its number prefix: `1 - Urgent` becomes `urgent`, `2 - High` becomes `high`, + and `3 - Backlog` becomes `backlog`. When an issue merges recommendations + with different rankings, use the highest. You may adjust a priority when + merging or splitting changes the impact; note each adjustment in the coverage + report. +7. Keep the number of issues well below the number of recommendations, to avoid + flooding the project. A good target is 40 to 60 issues. If you exceed that, + merge further. + +Issue body. Follow the issue template and keep its top-level headings in order. +Add subheadings within a section when they help, but do not add or remove +top-level headings. With the default template: + +- `## Description`: State the problem or gap, where it is, and why it matters to + users. Then describe the proposed change concretely enough that a contributor + can start without further research: which pages or files to add, edit, or + move, and what the result must contain. Use a bulleted list or a proposed + outline when the change has several parts. +- `## What you expected`: Describe what done looks like from the reader's point + of view, in one to three sentences. This is the acceptance criterion. +- `## URL`: Link to the affected pages under the published documentation URL and + name the source Markdown files. For issues that affect the project website or + a repository file, link to that page or file in its repository. +- `## Additional context`: Add anything else a contributor needs, such as + related issues by simulated number (for example, "Depends on #9003"), + decisions the maintainers must make first, or questions for the maintainers. + End with the [attribution sentence](#attribution-sentence). + +In every issue body: + +- Follow [Writing style](#writing-style). +- Do not mention the CSV, the priority ranking, or this file. The body is for + the project's contributors. + +Outputs: + +1. Issue files in `<work>/output/issues/`, formatted and named as described in + [Issue file conventions](#issue-file-conventions). Before writing, delete any + standalone and umbrella issue files left in that folder by a previous run, so + that the numbering restarts cleanly at 9001. Leave sub-folders alone. +2. A coverage report at `<work>/output/issues-coverage.md`, outside the issues + folder so that it is not uploaded as an issue. Overwrite any existing report. + Include, in this order: + - An `## Issues` table with the columns `Issue` (the simulated number, for + example `#9001`), `Title`, `Label`, and `File`. List every issue file here; + the `create-umbrella-issues` task reads this table. + - A `## Recommendation mapping` table that maps every CSV recommendation, + numbered by its row order, to the issue number that covers it, or to "Not + an issue" with a one-sentence reason. A recommendation may map to several + issues, and an issue may cover several recommendations. + - A `## Priority adjustments` table with the recommendation, its CSV ranking, + the label you assigned, and the reason. + - A `## Counts` list with the number of recommendations, recommendations + turned into issues, recommendations skipped, and issue files written. + +When finished, summarize the issues you created (number, title, and label). Ask +the writer to review the scope of each one before running +`create-umbrella-issues`. + +### Task: create-umbrella-issues + +Act as a senior technical writer on the CNCF TechDocs team. Group the standalone +issue files into umbrella issues, so that maintainers can track related work as +a unit and CNCF can report progress on the analysis. Each umbrella issue is +itself an uploadable issue file that links to its sub-issues. It does not repeat +their content. + +Run this task in a new session, after `create-issues` has finished and the +writer has reviewed the standalone issues. + +Sources: + +- The standalone issue files in `<work>/output/issues/`. Read every one. Ignore + sub-folders. +- `<work>/output/issues-coverage.md`: its `## Issues` table lists every issue + number, title, label, and file. Use it to look up titles and to find the next + free issue number. +- The [umbrella issue template](#umbrella-issue-template). +- `https://github.com/kedacore/keda-docs/issues/1361`: an example umbrella issue + from a previous analysis. Match its purpose: a short statement of what the + group achieves, followed by a list of sub-issues. Do not copy its "Use-Case:" + and "Specification:" heading prefixes. + +Do not read the recommendations CSV or the analysis files. The issue files are +the only input. If an issue is missing or wrongly scoped, tell the writer to fix +it with `create-issues` instead of working around it here. + +Procedure: + +1. Read the `## Issues` table and every standalone issue file. Note each issue's + number, title, label, and the pages or repository it touches. +2. Group the issues into themes. Group by the outcome a reader or maintainer + gets, not by the section or area an issue came from; a theme may span + sections. Typical themes are a new-user path, navigation and information + architecture, a feature-coverage gap, site configuration, accessibility, + contributor onboarding, community channels, and site ownership. +3. Size each umbrella at three to ten sub-issues. Split a theme that has more + than ten. Merge or drop a theme that has fewer than three. An issue belongs + to at most one umbrella. An issue that fits no theme stays standalone; list + such issues in the coverage report rather than forcing them into an umbrella. +4. Order the sub-issues in the sequence a contributor should work them: + prerequisites first, then by priority (`urgent`, `high`, `backlog`). +5. Assign each umbrella the highest priority label among its sub-issues. +6. Number the umbrellas by continuing the sequence, as described in + [Issue numbers](#issue-numbers). +7. Give each umbrella a title that names the outcome, for example "Create a + new-user getting started path". Do not use "Umbrella" in the title. + +Umbrella body. Follow the umbrella issue template and keep its two headings: + +- `## Umbrella Issue`: In one or two paragraphs, state the problem the theme + addresses, how the sub-issues together resolve it, and the order in which to + work them if it matters. End with the + [attribution sentence](#attribution-sentence), with "umbrella issue" in place + of "issue". +- `## Sub-issues`: A bulleted list with one line for each sub-issue, in work + order, formatted as described in [Issue numbers](#issue-numbers). Do not + describe the sub-issues; the link is enough. + +Follow [Writing style](#writing-style). Do not mention the CSV, the priority +ranking, or this file in the body. + +Outputs: + +1. Umbrella files in `<work>/output/issues/`, formatted and named as described + in [Issue file conventions](#issue-file-conventions). Before writing, delete + any umbrella files left by a previous run. Do not modify the standalone issue + files. +2. An updated `<work>/output/issues-coverage.md`: + - Append the umbrella issues to the `## Issues` table. + - Add a `## Umbrella issues` section after the `## Issues` table, with one + subsection for each umbrella that lists its number, title, label, and + sub-issue numbers. Follow it with a list of the standalone issues that + belong to no umbrella. + - Update the `## Counts` list with the number of umbrella issues and the + number of standalone issues that are not in an umbrella. + +When finished, summarize the umbrella issues you created (number, title, label, +and sub-issue count) and list any issues left outside an umbrella. Ask the +writer to review the grouping. + +## Issue file conventions + +Issues are a subset of the recommendations: those that identify a problem or a +gap. A separate process uploads every Markdown file directly inside +`<work>/output/issues/` to the project's issues repository. It ignores +sub-folders. To archive a previous run, move its files into a sub-folder, for +example `prev_1/`. + +### Front matter + +Every issue file starts with this front matter. Everything after it is the issue +description, written in GitHub-flavored Markdown. + +```yaml +--- +title: Add a user-facing troubleshooting page #9007 +labels: [high, help wanted] +repo: https://github.com/kubevirt/user-guide/issues +--- +``` + +- `title` becomes the GitHub issue title. Append the simulated issue number + after a space and `#`. +- `labels` becomes the GitHub issue labels. Include exactly one of `urgent`, + `high`, or `backlog`, and always include `help wanted`. Add no other labels. + The upload process maps them using the + [priority label mapping](#priority-label-mapping). The `help wanted` label + signals that maintainers seek community help; CNCF uses it to surface + documentation tasks on CLOTributor (`https://clotributor.dev`). +- `repo` is always the issues repository URL from + [Project context](#project-context). + +### Issue numbers + +Issue numbers simulate GitHub issue numbers so that files can refer to each +other before upload. They start at 9001 and increase by one for each issue, in +the order written. Umbrella issues continue the sequence: the first umbrella +takes the number after the highest number in the `## Issues` table of the +coverage report. + +In an umbrella issue, each sub-issue is one list item. The link text is the +sub-issue title and number, and the link target is the sub-issue file name, +relative to the umbrella file: + +<!-- markdownlint-disable MD013 --> + +```markdown +- [Add a user-facing troubleshooting page #9007](9007-add-troubleshooting-page.md) +``` + +<!-- markdownlint-enable MD013 --> + +The upload process replaces these references with real issue numbers and links, +so every title and number must match the sub-issue's front matter exactly. + +### Issue file names + +Name standalone issue files `<number>-<slug>.md` and umbrella issue files +`U-<number>-<slug>.md`. Build the slug from the issue title: + +- Use lowercase words separated by hyphens, with no punctuation, and no more + than about five words. +- Keep the action verb and the main subject noun. Drop articles, conjunctions, + and secondary clauses. +- Name the topic (for example, `navigation`, `footer`, or `virtctl`), not the + analysis area it came from. +- When the title lists several changes, use a collective noun such as + `improvements` or `updates`. +- Strip punctuation from version numbers, so that `v1.9` becomes `v19`. + +<!-- markdownlint-disable MD013 --> + +| Issue title | File name | +| --------------------------------------------------- | --------------------------------------- | +| Add a user-facing troubleshooting page #9007 | `9007-add-troubleshooting-page.md` | +| Move Release Notes out of the main navigation #9004 | `9004-move-release-notes.md` | +| Document the v1.9 VirtualMachineBackup API #9012 | `9012-document-v19-backup-api.md` | +| Create a new-user getting started path #9061 | `U-9061-create-getting-started-path.md` | + +<!-- markdownlint-enable MD013 --> + +### Attribution sentence + +End the `## Additional context` section of every standalone issue, and the +`## Umbrella Issue` section of every umbrella issue, with this sentence, adapted +as needed. Fill in the project name, the CNCF project page URL, and the +assistance-request issue from [Project context](#project-context). + +```markdown +This issue tracks recommended changes resulting from an analysis of the +<project name> documentation commissioned by [CNCF](<CNCF project page URL>). +The assistance request is tracked in +[cncf/techdocs#<NNN>](https://github.com/cncf/techdocs/issues/<NNN>). +``` + +### Standalone issue template + +```markdown +--- +title: <title> #<issue number> +labels: [<urgent | high | backlog>, help wanted] +repo: <issues repository URL> +--- + +## Description + +A clear and concise description of the problem or gap, where it is, and the +proposed change. + +## What you expected + +A clear and concise description of what done looks like for the reader. + +## URL + +Links to the affected documentation pages and source files. + +## Additional context + +Related issues, decisions the maintainers must make, and the attribution +sentence. +``` + +### Umbrella issue template + +```markdown +--- +title: <title> #<issue number> +labels: [<urgent | high | backlog>, help wanted] +repo: <issues repository URL> +--- + +## Umbrella Issue + +Describe the umbrella issue and how the sub-issues work to resolve it. + +## Sub-issues + +- [<sub-issue title> #<issue number>](<number>-<slug>.md) +``` + +## Where the outputs go in analysis.md + +The writer pastes the content of each output file, without its level-1 heading, +into the project's `analysis.md`. Use this table when the writer asks where an +output belongs, or asks you to assemble a draft. + +<!-- markdownlint-disable MD013 --> + +| Output file | Location in `analysis.md` | +| ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `<project-slug>-<section>-comment.md` | The ratings table replaces the placeholder table under the section's level-2 heading. The comment paragraphs go under the section's "Comments" heading, before the sentence that begins "The following sections contain brief assessments". | +| `<project-slug>-<area>-answers.md` | Under the area's heading in the section's "Comments" part, after the paragraph that ends "We evaluate on the following:". The answers replace the author's note that links to the evaluation questions. | +| `<project-slug>-<area>-comment.md` | After the area's answers, before the next heading. The writer copies the rating from the last line into the section ratings table if the section comment was not regenerated. | +| `<project-slug>-<area>-recommendations.md` | Under the area's heading in the section's "Recommendations" part. | + +<!-- markdownlint-enable MD013 --> + +TODO: This mapping follows the template structure proposed in cncf/techdocs#423. +Confirm it when that pull request merges. + +## Maintaining this file + +These conventions are for people and agents who edit the harness itself. They +carry over the maintenance conventions of the earlier `_github/` harness. + +- Keep data and procedure separate. Area and section names live only in the + tables under [Sections and areas](#sections-and-areas). Procedures live only + under [Tasks](#tasks). Change what is analyzed in the tables, and change how + it is analyzed in the tasks. Do not write an area's name into a task + procedure. +- Do not copy criteria questions into this file. `criteria.md` is the single + source. To change a question, edit `criteria.md`. +- When `criteria.md` renames, adds, or removes a questions heading, update the + "Questions heading" column of the Areas table in the same change. A heading + that does not match exactly stops the `answers` task. +- Adding an area requires one row in the Areas table and a questions heading in + `criteria.md`. Add an entry under + [Area-specific instructions](#area-specific-instructions) only if the area + needs one. +- Keep the tasks project-agnostic. Anything project-specific belongs in + [Project context](#project-context). Keep the KubeVirt values as the reference + example. +- Keep every task self-contained: its sources, procedure, and outputs appear + under its own heading. Shared rules belong in + [Conventions for all tasks](#conventions-for-all-tasks), + [Rules for every task](#rules-for-every-task), or + [Issue file conventions](#issue-file-conventions), and tasks link to them + instead of repeating them. +- When the layout of `<work>` or a file-naming rule changes, update the layout + block under [Working locations](#working-locations), the affected tasks, and + the "Workflow" and "Folder layout" sections of `index.md` together. +- Write in the voice this file asks of the agent: complete sentences, active + voice, imperative mood, and short paragraphs. +- Format this file with Prettier before committing. Run `npm run check:format` + from the root of the `cncf/techdocs` clone, and `npm run fix:format` to apply + the changes. +- `index.md` records the agent version and model that the harness was validated + with. Update it when the tasks are validated again. + +## Migration TODO + +These items remain before this file can replace the `_github/` folder. + +Decisions: + +- [ ] Decide the file name and location. The AGENTS.md convention uses the + upper-case name `AGENTS.md`, and agents load it from the repository root + or the working directory. This prototype is named `agents.md` and sits in + `docs/analysis/ai-harness/`, where no agent loads it automatically. +- [ ] Decide how the agent loads the file without adding it to the project + clone. Candidates: run the agent from a folder in the `<techdocs>` clone + that holds the file and grant it access to `<project>`; point the agent at + the file's folder with a setting such as Copilot CLI's + `COPILOT_CUSTOM_INSTRUCTIONS_DIRS`; or copy the one file to the root of + `<project>` and leave it untracked. The last option conflicts with any + `AGENTS.md` the project already has. +- [ ] Decide where `<work>` lives, and whether its `input/` and `output/` layout + stays. Then replace the "Working folder" field with a default. +- [ ] Decide where the filled-in [Project context](#project-context) lives. If + it stays in this file, each analysis needs its own edited copy. A separate + project file next to `analysis.md`, which this file tells the agent to + read, would keep this file unmodified and shared. +- [ ] Decide whether one file is the right size. All of this file is loaded into + every session, including the issue tasks during area runs. Some agents cap + the size of instruction files. An alternative is a short `AGENTS.md` that + holds the rules and tables and points to task files that are read on + demand. +- [ ] Decide whether the area IDs stay as they are. This prototype uses one ID + for each area. The earlier harness had a separate output-file stem that + differed for three areas (`new-user`, `new-contributor`, and + `communication-methods-docd`), so output file names change. +- [ ] Decide how a project supplies its own issue template now that the + templates are embedded here instead of being files. +- [ ] Decide how the writer invokes a task. Slash commands such as + `/information-architecture-full` and the `title` input variable are + Copilot prompt-file features and do not exist in AGENTS.md. This prototype + uses plain requests such as `run full information-architecture`. + +Dependencies on cncf/techdocs#423: + +- [ ] Confirm the questions headings in the Areas table against the merged + `criteria.md`. +- [ ] Confirm the criterion labels and section names against the merged + `analysis.md` template. The template names the third section "Website and + infrastructure"; this file keeps "Website & Infrastructure". +- [ ] Decide whether the "Other" questions under "Website" in `criteria.md` + (HTTPS access and working links), which have no questions heading, become + an area. +- [ ] Decide whether the "Other Metrics" table returns to the `section-comment` + task. +- [ ] Decide whether the reference sources for `agent-readiness` and the + instruction for `inclusive-language` move into `criteria.md`, which would + empty [Area-specific instructions](#area-specific-instructions). + +Remaining work: + +- [ ] Validate every task against a real project and compare the output with a + run of the `_github/` harness. Record the agent version and model in + `index.md`. +- [ ] Test the file with at least one agent other than Copilot CLI. The earlier + `full` engine asked for the "Claude default" output style; this file drops + that instruction to stay agent-neutral. +- [ ] Rewrite `index.md` for the new model: the toolchain, the pre-flight + checklist, the workflow steps, the upload process, and the folder layout + all describe copying `_github/` into the project. +- [ ] Move the KubeVirt example (`_github/examples/kubevirt/`) to a location + that survives the removal of `_github/`, and convert it to the + [Project context](#project-context) fields. +- [ ] Remove `_github/` (criteria files, engines, wrappers, templates, and + readme files) once this file is validated, and update the admin + instructions that describe the folder roles. +- [ ] Decide whether this file is published on the TechDocs site. Files under + `docs/` are built as pages; `_github/` was excluded by its underscore + prefix. +- [ ] Add the upload script that `index.md` describes, or document the upload + steps as a task here. diff --git a/docs/analysis/ai-harness/index.md b/docs/analysis/ai-harness/index.md new file mode 100644 index 00000000..e2ce0db0 --- /dev/null +++ b/docs/analysis/ai-harness/index.md @@ -0,0 +1,152 @@ +--- +title: AI Assisted Analysis Harness +--- + +# AI Assisted Analysis Harness + +This folder holds a reusable harness for running a CNCF TechDocs +[documentation analysis](../howto.md) with GitHub Copilot. The `_github/` folder +is a template for the target project's `.github/` folder: it contains Copilot +instructions, the analysis criteria broken into areas and sections, predefined +prompts for each step, issue templates, and input and output folders. The +harness is project-agnostic; everything project-specific lives in one section of +`copilot-instructions.md`. + +The harness was initially built for the KubeVirt analysis +([cncf/techdocs#343](https://github.com/cncf/techdocs/issues/343)). A worked +example of its project-specific section is in `_github/examples/kubevirt/`. + +## How it fits the analysis + +The harness is a human-driven, AI-assisted analysis tool. A writer still owns +the analysis. The harness has Copilot answer the [criteria](../criteria.md) +questions, draft comments and ratings, propose recommendations, rank them, and +write issue files, all in a consistent format that maps onto the +[analysis template](../templates/analysis.md). The writer reviews and edits +every output before it goes into `analysis.md` or the project's issue tracker. +For how Copilot should write, see "Analysis responses style" in +`_github/copilot-instructions.md`. + +## Validated toolchain + +The prompts were validated with: + +- GitHub Copilot CLI 1.0.x (`copilot --version`; 1.0.103 at the time of + writing), run from the root of the target project's documentation repository + so that `.github/copilot-instructions.md` is picked up automatically. +- Claude Fable 5.1 (`/model` in the CLI shows the current one). The `full` + engine asks for "Claude default" output style; other models work but may need + the style section tuned. +- Node.js and Prettier (`npm run check:format` in `cncf/techdocs`) for + formatting the generated Markdown before pasting it into `analysis.md`. +- Prompt files that rely on two Copilot prompt-file features: `agent: agent` in + the front matter and the `${input:title}` variable for the optional output + filename override. The prompts are invoked as slash commands + (`/information-architecture-full`), which requires the files to end in + `.prompt.md` and live under `.github/prompts/`. + +Record the version and model you used in the analysis document's front matter or +introduction so later runs can be compared. + +## Pre-flight checklist + +Complete these before the first prompt run: + +- [ ] The project has an assistance request issue in `cncf/techdocs`, and a + branch in `cncf/techdocs` with `docs/analysis/<NNNN>-<project>/` + scaffolded from the [templates](../templates/) (`analysis.md`, + `implementation.md`, `issues-list.md`). +- [ ] The target repository's `.github/` has no existing `prompts/` or + `copilot-instructions.md` that would conflict. If it does, merge rather + than overwrite, and keep the harness instructions at the top. +- [ ] The issues repository has the labels the "Priority label mapping" table + maps to (for KubeVirt: `priority/critical-urgent`, `priority/high`, + `priority/backlog`) plus `help wanted`. Ask the maintainers to create any + that are missing before uploading issues. +- [ ] You can run `copilot` in the repository and it reports the expected + version and model. +- [ ] You have maintainer agreement that the analysis work is not committed to + the project's repository. The harness writes only under `.github/input` + and `.github/output`; keep those changes on a local branch. + +## Workflow + +1. **Install the harness.** Copy `_github/` into the root of the target + project's documentation repository and rename it to `.github/`. Every prompt + references `.github/...` paths, so the rename is required. Do not copy + `examples/`. +2. **Fill in the project context.** Complete every `<!-- FILL IN -->` + placeholder in the "Current Repository" section of + `.github/copilot-instructions.md`, using `examples/kubevirt/` as a model. If + the project has its own documentation issue template, copy it to + `.github/templates/` with the four headings `Description`, + `What you expected`, `URL`, and `Additional context`, and name it in the + "Issue template file" field. +3. **Run the area prompts.** For each of the seventeen areas, run `/<area>-full` + (for example `/information-architecture-full`). Each run writes three files + to `.github/output/analysis/`: answers, comment, and recommendations. Start a + new session for each area so context from one area does not leak into the + next. Review each comment's `Rating:` line; it feeds the section tables. +4. **Run the section prompts.** Run `/project-documentation-comment`, + `/contributor-documentation-comment`, and `/website-infrastructure-comment`. + Each synthesizes its member areas' comment files into an overall comment with + a ratings table. +5. **Assemble `analysis.md`.** Paste the outputs into the analysis document + following the "Analysis.md and Output Files Mapping" table in + `copilot-instructions.md`. Drop the temporary H1 headings. Edit for accuracy + and voice; the writer's judgment is the final word on ratings. +6. **Rank the recommendations.** Run `/rank-recommendations`. It writes + `.github/input/ai-ranked-recommendations.csv`. Review the priorities and + re-run with corrections if needed. +7. **Create issues.** Run `/create-issues` and review the scope of each file in + `.github/output/issues/`. Then, in a new session, run + `/create-umbrella-issues`. Both prompts update + `.github/output/issues-coverage.md`, which maps every recommendation to an + issue and lists the issue numbers. +8. **Upload the issues.** Use the upload process described below. Record the + real issue numbers in `issues-list.md` and link the umbrella issues from the + assistance request issue. +9. **Finish the analysis documents.** Complete `implementation.md` and + `issues-list.md` from the coverage report, then open the pull request in + `cncf/techdocs`. + +## Upload process + +The issue prompts write one Markdown file per issue with YAML front matter +(`title`, `labels`, `repo`) followed by the GitHub-flavored Markdown body. The +upload step reads every `*.md` file in `.github/output/issues/` except +`readme.md` and sub-folders, and for each file: + +1. Creates a GitHub issue in `repo` with the front matter `title` (minus the + trailing `#<number>`) and the body. +2. Maps the harness labels to repository labels using the "Priority label + mapping" table and applies them. +3. Records the simulated number (`#9001`) against the real issue number. +4. After all standalone issues exist, creates the `U-*` umbrella issues, + rewriting each `[title #9001](9001-file.md)` link to the real issue URL so + the umbrella's sub-issue list resolves. + +For KubeVirt this was done with a `gh issue create` script run by the writer; +the script is not part of the harness yet. Until it is added here, write the +script per project following the steps above, and upload standalone issues first +so the umbrella links can be resolved. Keep the resulting mapping of simulated +to real numbers in `.github/input/` for the `issues-list.md` step. + +## Folder layout + +```text +ai-harness/ + index.md # This file + _github/ # Copy to <project repo>/.github/ + copilot-instructions.md # Run-time instructions; fill in "Current Repository" + criteria/ # Per-area questions and per-section membership + prompts/ # Area, section, and issue prompts + templates/ # doc-issue.md and umbrella-issue.md + input/ # ai-ranked-recommendations.csv and ad hoc inputs + output/analysis/ # Area and section outputs + output/issues/ # Standalone and umbrella issue files + examples/kubevirt/ # Worked example of the project-specific section +``` + +For the structure of the criteria and prompt files, the output file naming, and +the `analysis.md` mapping, see `_github/copilot-instructions.md`.