Put the whole family into the search index

.github/workflows/check.yml

@@ -106,6 +106,10 @@ jobs:
       # alone. A page that does not build is a page nobody can read.
       - name: build the site
         if: ${{ !cancelled() }}
+        # the search index lists the organisations' repositories and the
+        # framework's classes from the GitHub API (scripts/lib/search-family.mjs)
+        env:
+          GITHUB_TOKEN: ${{ github.token }}
         run: npm run docs:build
 
       # every link out of this deployment and into a neighbouring one on the

.github/workflows/deploy.yml

@@ -176,6 +176,10 @@ jobs:
       # THE SITE THAT IS PUBLISHED: VitePress's build, from this repository
       # alone.
       - name: Build the site
+        # the search index lists the organisations' repositories and the
+        # framework's classes from the GitHub API (scripts/lib/search-family.mjs)
+        env:
+          GITHUB_TOKEN: ${{ github.token }}
         run: npm run docs:build
 
       # The one gate that cannot run before the build, because what it judges

AGENTS.md

@@ -21,7 +21,8 @@ person reads the page. Do not put "as an AI, …" prose back into `docs/`.
 | `scripts/generate-api-reference.mjs` | Generates the client API reference — the block in `docs/resources/api.md` and `docs/public/api/client-api.json` — from `z2ui5_if_client` on the framework branch this site tracks (`main`); `--check` is the freshness gate |
 | `scripts/lib/client-interface.mjs` | Where `z2ui5_if_client` is fetched from (the ref comes from `lib/release.mjs`, shared with `check-api-names.mjs`; `A2UI5_HOME` reads the same ref out of a local clone of the framework instead, with `git show`, never its working tree) and the full parser `generate-api-reference.mjs` renders from; pinned by `test/client-interface.test.mjs` |
 | `scripts/check-version.mjs` | The release number in the bar's version dropdown, the deprecations page and the changelog, against the newest release tag of the framework |
-| `scripts/generate-search.mjs` | Builds `docs/public/search-index.json` — the pages of this site plus every entry in the three sample catalogues. This site's own bar searches with VitePress's local search; the index is published for the playground's search box, which fetches it from here. `scripts/lib/search-engine.js` is the matcher the playground carries a copy of, pinned with the index by `test/search.test.mjs`. Runs inside `docs:build`, so the deploy publishes it; `scripts/lib/search-index.mjs` is what goes in |
+| `scripts/generate-search.mjs` | Builds `docs/public/search-index.json` — the pages of this site, every entry in the three sample catalogues, and the rest of the family (below). This site's own bar searches with VitePress's local search; the index is published for the search box on the home page (https://abap2ui5.github.io/, abap2UI5/abap2UI5.github.io) and the playground's, which fetch it from here. `scripts/lib/search-engine.js` is the matcher both carry a copy of, pinned with the index by `test/search.test.mjs` and `test/search-family.test.mjs`. Runs inside `docs:build`, so the deploy publishes it; `scripts/lib/search-index.mjs` is what goes in |
+| `scripts/lib/search-family.mjs` | The rest of the family in that index: where to go (the tools, as data here), every repository of `abap2UI5` and `abap2UI5-addons` (GitHub API, `GITHUB_TOKEN` in CI), the essays, the page for decision makers and the addons (each from the `search.json` its site publishes - one shape, `{ site, url, entries: [{ title, text, headings, terms, url, links }] }`), the client API (from the committed `client-api.json`, each method linked to its line in `z2ui5_if_client`) and every class of the framework outside `src/99`. Every source may fail and costs its rows, never the build. Issues and discussions are not indexed: the box hands the query to GitHub's search for them (`elsewhere( )` in the matcher), so they are never a day old |
 | `scripts/lib/pages.mjs` | What a page of this site IS: the sidebar walk, its title, its opening sentences, its headings, its words. Shared by `generate-llms.mjs` and `generate-search.mjs` so a page added to the sidebar reaches both; `summarise( )` — the description of 150 pages and the note in `llms.txt`, which nobody writes and nobody proofreads — is pinned by `test/summarise.test.mjs` |
 | `docs/.vitepress/playground.mjs` | Decides which fenced ABAP example gets a **Run** button, and wraps the fence; `theme/playground.js` is the browser half |
 | `scripts/list-runnable.mjs` | The measurement's worklist: every fenced example that carries a Run button, out of the same `playground.mjs` that decides the button. `--json` adds each example's ABAP verbatim - what the button sends - so the measurement below can be driven rather than clicked |

scripts/generate-search.mjs

@@ -1,6 +1,8 @@
 #!/usr/bin/env node
-// The search index the bar reads: every page of this site, every sample in the
-// three catalogues, one JSON document at /docs/search-index.json.
+// The search index the boxes of the family read: every page of this site, every
+// sample in the three catalogues and the rest of the family (repositories,
+// tools, the sister sites, the API - scripts/lib/search-family.mjs), one JSON
+// document at /docs/search-index.json.
 //
 // Runs inside `docs:build`, before vitepress, so the file is in docs/public/
 // when the build copies it - the same arrangement as generate-llms.mjs, and
@@ -24,7 +26,8 @@ const OUT = path.join(DOCS, 'public', 'search-index.json');
 const index = await buildIndex(ROOT, { log: (line) => console.log(line) });
 
 const docs = index.entries.filter((e) => e.area === 'docs').length;
-const samples = index.entries.length - docs;
+const samples = index.entries.filter((e) => e.area === 'samples').length;
+const rest = index.entries.length - docs - samples;
 
 /* The floor. An index with no page in it is a broken walk - the sidebar moved,
  * or config.mjs stopped exporting one - and an empty search box that reports
@@ -40,4 +43,4 @@ fs.mkdirSync(path.dirname(OUT), { recursive: true });
 fs.writeFileSync(OUT, JSON.stringify(index));
 
 const kb = Math.round(fs.statSync(OUT).size / 1024);
-console.log(`search-index.json: ${docs} page(s) + ${samples} sample(s) (${kb} kB)`);
+console.log(`search-index.json: ${docs} page(s) + ${samples} sample(s) + ${rest} from the rest of the family (${kb} kB)`);

scripts/lib/search-engine.js

@@ -1,5 +1,5 @@
 /*
- * The matcher behind the search box in the bar — framework-free on purpose.
+ * The matcher behind the search boxes of the family — framework-free on purpose.
  *
  * Plain JavaScript with no imports, and this site's own: the index it reads
  * is /docs/search-index.json, generated by scripts/generate-search.mjs from
@@ -292,6 +292,21 @@ function nearest(entries, term) {
  * (above) and the list says so: `hits.relaxedTo` is the query that answered.
  */
 export function search(entries, query, { limit = 30 } = {}) {
+  /* A scope - `repo: popup` - narrows the entries before anything is scored,
+   * and a scope alone - `repo:` - lists its area, which is how a reader asks
+   * "what repositories are there". */
+  const scoped = parseScope(query);
+  if (scoped.area) {
+    entries = entries.filter((e) => e.area === scoped.area);
+    query = scoped.query;
+    if (!queryTerms(query).length) {
+      return entries
+        .slice()
+        .sort((a, b) => (a.title || '').localeCompare(b.title || ''))
+        .slice(0, limit)
+        .map((entry) => ({ entry, score: 0, heading: null, strong: true }));
+    }
+  }
   /* Eight words at most. Nobody narrows a list with a ninth, and a pasted
    * error message or a line of ABAP is thirty: every relaxing pass below
    * scored the whole index once per word, and the box froze for seconds on
@@ -476,3 +491,113 @@ export function recallQuery() {
   const q = record.q.trim();
   return q && q.length <= QUERY_MAX ? q : '';
 }
+
+/* ── SCOPES, JUMPS AND THE WAY OUT ──────────────────────────────────────────
+ *
+ * The index carries more than the manual and the samples since the home page
+ * got its box: where to go (`tools`), the repositories (`repos`), the sister
+ * sites (`insights`, `why`, `addons`) and the code (`api`) - see
+ * scripts/lib/search-family.mjs. Three things go with that, all of them data
+ * about the QUERY rather than about the ranking, so a box that does not use
+ * them (the playground's, today) loses nothing:
+ *
+ *   parseScope( )  `repo: popup`, `api: bind` - a word and a colon in front
+ *                  narrow the search to one area;
+ *   jumps( )       the query that NAMES something - `#1234`, a class name,
+ *                  a repository, a tool - answered with that thing, before
+ *                  any list;
+ *   elsewhere( )   the query handed to GitHub's own search, for what no index
+ *                  of ours carries: the code of every repository, the issues,
+ *                  the discussions. Links, not requests - the box fetches
+ *                  nothing from GitHub, and the answer is never a day old.
+ */
+
+/** The words a scope can be written as, and the area each means. */
+export const SCOPES = {
+  doc: 'docs', docs: 'docs',
+  sample: 'samples', samples: 'samples',
+  repo: 'repos', repos: 'repos',
+  api: 'api', code: 'api', class: 'api',
+  tool: 'tools', tools: 'tools', go: 'tools',
+  insight: 'insights', insights: 'insights', essay: 'insights',
+  why: 'why',
+  addon: 'addons', addons: 'addons',
+};
+
+/** `{ area, query }`: the area a leading `word:` names, and the rest. A word
+ *  that is no scope leaves the query as it is - `z2ui5:` is a search. */
+export function parseScope(query) {
+  const m = /^\s*([a-z]+):\s*(.*)$/is.exec(query || '');
+  const area = m && SCOPES[m[1].toLowerCase()];
+  return area ? { area, query: m[2] } : { area: null, query: query || '' };
+}
+
+const GITHUB = 'https://github.com';
+const FRAMEWORK = `${GITHUB}/abap2UI5/abap2UI5`;
+const enc = encodeURIComponent;
+
+/** GitHub's code search over both organisations. */
+export const codeSearchUrl = (q) => `${GITHUB}/search?q=${enc(`(org:abap2UI5 OR org:abap2UI5-addons) ${q}`)}&type=code`;
+
+/**
+ * What the query NAMES, if it names something: at most three targets, each
+ * `{ entry }` (a hit of the index) or `{ title, text, url }` (a place
+ * outside it). Empty when the query is a question rather than a name - the
+ * list answers those.
+ *
+ *   #1234, issue 1234, pr 1234   that issue or pull request of the framework
+ *   a class, a method, a repo    the entry whose `code` it is, whole
+ *   a tool or a repository name  the entry whose title it is, whole
+ *   z2ui5_… in no entry          the code search for it
+ */
+export function jumps(entries, query) {
+  const q = parseScope(query).query.trim();
+  const out = [];
+  const m = /^(?:#|(?:issue|pr|pull)\s*#?)(\d{1,6})$/i.exec(q);
+  if (m) out.push({ title: `#${m[1]}`, text: 'Issue or pull request of abap2UI5/abap2UI5', url: `${FRAMEWORK}/issues/${m[1]}` });
+
+  const low = normalise(q).replace(/\s+/g, ' ');
+  if (low.length >= 3) {
+    for (const e of entries) {
+      const code = e.code || '';
+      const named = code === low || code.split('/').pop() === low
+        || ((e.area === 'tools' || e.area === 'repos') && normalise(e.title) === low);
+      if (named) out.push({ entry: e });
+    }
+  }
+  if (!out.length && /^z2ui5_\w{3,}$/i.test(q)) {
+    out.push({ title: q, text: 'Not in the index - every repository’s code, on GitHub', url: codeSearchUrl(q) });
+  }
+  const seen = new Set();
+  return out.filter((j) => {
+    const url = j.entry ? j.entry.url : j.url;
+    if (seen.has(url)) return false;
+    seen.add(url);
+    return true;
+  }).slice(0, 3);
+}
+
+/** The query handed on to GitHub: `[{ label, url }]`, empty for an empty
+ *  query. Issues and discussions are never indexed (a nightly copy of them
+ *  is a day old exactly when it matters), so this is how they are searched. */
+export function elsewhere(query) {
+  const q = parseScope(query).query.trim();
+  if (!q) return [];
+  return [
+    { label: 'Code', url: codeSearchUrl(q) },
+    { label: 'Issues', url: `${GITHUB}/search?q=${enc(`org:abap2UI5 org:abap2UI5-addons ${q}`)}&type=issues` },
+    { label: 'Discussions', url: `${GITHUB}/search?q=${enc(`org:abap2UI5 ${q}`)}&type=discussions` },
+  ];
+}
+
+/** The actions beside a hit: what its entry carries (`links` - the source of
+ *  a sample, the README of a repository) and, for a repository, its code
+ *  searched for what was typed. */
+export function actions(entry, query = '') {
+  const out = Array.isArray(entry?.links) ? entry.links.filter((l) => l && l.label && l.url) : [];
+  const q = parseScope(query).query.trim();
+  if (entry?.area === 'repos' && entry.code && q && normalise(q) !== entry.code.split('/').pop()) {
+    out.push({ label: 'Search code', url: `${GITHUB}/search?q=${enc(`repo:${entry.code} ${q}`)}&type=code` });
+  }
+  return out;
+}