Skip to content

feat(dev): tc exploration: resolve app hash, no fabricated cases, manual trigger and local runner #16

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
yashcdev merged 1 commit into from
Oct 20, 2026
Merged

feat(dev): tc exploration: resolve app hash, no fabricated cases, manual trigger and local runner #16

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
92 changes: 78 additions & 14 deletions .github/workflows/tc-explore.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,30 @@
name: TC explore and generate test cases

# Runs after "Build release APK" completes successfully.
# Runs after "Build release APK" completes successfully, or manually (workflow_dispatch).
# Reads the tc-handoff artifact (bs:// app id + feature context), explores the app on a real
# Android device with Test Companion, and saves the generated cases to Test Management.
on:
workflow_run:
workflows: ["Build release APK"]
types: [completed]
workflow_dispatch:
inputs:
app:
description: "App to explore: bs://<id> or a custom id such as BStackBankLatest (default)"
required: false
default: "BStackBankLatest"
device:
description: "Device name (leave empty to let Test Companion choose)"
required: false
default: ""
os_version:
description: "OS version (only used together with a device)"
required: false
default: ""
instructions:
description: "Extra instructions appended to the feature context (optional)"
required: false
default: ""

concurrency:
group: tc-explore
Expand All @@ -23,33 +41,57 @@ env:
TM_PROJECT: "PR-79" # Test-Companion-Demo-Vision
TM_FOLDER_NAME: "Functional"
TM_FOLDER_ID: "60003370"
DEVICE: ${{ github.event.inputs.device }}
OS_VERSION: ${{ github.event.inputs.os_version }}

jobs:
explore:
if: github.event.workflow_run.conclusion == 'success'
if: github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success'
runs-on: ubuntu-latest
timeout-minutes: 40
timeout-minutes: 60
steps:
# --- Automatic path: take the app id and context from the build run ---
- name: Download handoff from the build run
if: github.event_name == 'workflow_run'
uses: actions/download-artifact@v4
with:
name: tc-handoff
path: handoff
run-id: ${{ github.event.workflow_run.id }}
github-token: ${{ secrets.GITHUB_TOKEN }}

# --- Manual path: use the repo's current feature context and the chosen app ---
- name: Checkout (manual run)
if: github.event_name == 'workflow_dispatch'
uses: actions/checkout@v4

- name: Read handoff
id: handoff
env:
EVENT: ${{ github.event_name }}
INPUT_APP: ${{ github.event.inputs.app }}
EXTRA: ${{ github.event.inputs.instructions }}
BROWSERSTACK_USERNAME: ${{ secrets.BROWSERSTACK_USERNAME }}
BROWSERSTACK_ACCESS_KEY: ${{ secrets.BROWSERSTACK_ACCESS_KEY }}
run: |
set -euo pipefail
test -s handoff/handoff.json || { echo "handoff.json missing"; exit 1; }
mkdir -p handoff
if [ "$EVENT" = "workflow_dispatch" ]; then
cp .github/tc-context/feature-context.md handoff/feature-context.md
APP="${INPUT_APP:-BStackBankLatest}"
BS_URL="$(node scripts/resolve-bs-app.js "$APP")"
PR_NUMBER="manual"
if [ -n "$EXTRA" ]; then printf '\n\n## Extra instructions\n%s\n' "$EXTRA" >> handoff/feature-context.md; fi
else
test -s handoff/handoff.json || { echo "handoff.json missing"; exit 1; }
BS_URL=$(node -p "require('./handoff/handoff.json').bs_url")
PR_NUMBER=$(node -p "require('./handoff/handoff.json').pr_number")
fi
test -s handoff/feature-context.md || { echo "feature-context.md missing"; exit 1; }
BS_URL=$(node -p "require('./handoff/handoff.json').bs_url")
PR_NUMBER=$(node -p "require('./handoff/handoff.json').pr_number")
case "$BS_URL" in bs://*) ;; *) echo "invalid bs_url in handoff"; exit 1;; esac
case "$BS_URL" in bs://*) ;; *) echo "invalid app id: $BS_URL"; exit 1;; esac
echo "bs_url=$BS_URL" >> "$GITHUB_OUTPUT"
echo "pr_number=$PR_NUMBER" >> "$GITHUB_OUTPUT"
echo "App: $BS_URL (PR #$PR_NUMBER)"
echo "App: $BS_URL (PR $PR_NUMBER), device: ${DEVICE:-auto} ${OS_VERSION}"

- name: Install tc (pinned)
run: |
Expand All @@ -73,15 +115,36 @@ jobs:
BROWSERSTACK_USERNAME: ${{ secrets.BROWSERSTACK_USERNAME }}
BROWSERSTACK_ACCESS_KEY: ${{ secrets.BROWSERSTACK_ACCESS_KEY }}
run: |
set -o pipefail
set -uo pipefail
CTX="$(cat handoff/feature-context.md)"
INSTRUCTIONS="Save every generated test case to Test Management project ${TM_PROJECT} (Test-Companion-Demo-Vision), inside the existing folder ${TM_FOLDER_NAME} (folder id ${TM_FOLDER_ID}). Do not create a new project and do not use the Automation folders.
Write manual, step-based test cases only (a title, preconditions, numbered steps, and one clear expected result per step). Do not generate automation scripts or file names.
Explore the feature below on the device, then derive the cases from what you actually observed.
You MUST explore the feature on the real device and base every test case on what you actually observed. If the device session cannot be started or is lost, STOP and report failure. Never write test cases from the description alone and never save unverified cases.

${CTX}"
tc app generate --app "$BS_URL" --platform android --auto-grant-permissions \
--save-to-tm --yolo --json "$INSTRUCTIONS" 2>&1 | tee tc-run.log

DEVICE_ARGS=()
if [ -n "${DEVICE:-}" ]; then DEVICE_ARGS+=(--device "$DEVICE"); fi
if [ -n "${DEVICE:-}" ] && [ -n "${OS_VERSION:-}" ]; then DEVICE_ARGS+=(--os-version "$OS_VERSION"); fi

# Start from a clean pending queue so nothing unverified can be approved by accident.
tc tm review --reject --all || true

# Retry only when the device session itself failed.
for attempt in 1 2 3; do
echo "=== Attempt $attempt (device: ${DEVICE:-auto}) ==="
tc app generate --app "$BS_URL" --platform android ${DEVICE_ARGS[@]+"${DEVICE_ARGS[@]}"} \
--auto-grant-permissions --save-to-tm --yolo --json "$INSTRUCTIONS" 2>&1 | tee "tc-run-$attempt.log"
code=${PIPESTATUS[0]}
if grep -qE 'Session liveness check failed|session crashed/terminated|No active Appium session|task_blocked|could not be started|no longer available on BrowserStack|not one of the devices provided' "tc-run-$attempt.log"; then
echo "Device session failed on attempt $attempt (exit $code). Discarding anything created without a device."
tc tm review --reject --all || true
continue
fi
exit "$code"
done
echo "Exploration did not complete after 3 attempts. Nothing was verified or saved."
exit 7

- name: Write run summary
if: always()
Expand All @@ -94,7 +157,8 @@ jobs:
echo "## Test Companion exploration"
echo "- Result: ${JOB_STATUS}"
echo "- App: ${BS_URL}"
echo "- PR: #${PR_NUMBER}"
echo "- PR: ${PR_NUMBER}"
echo "- Device: ${DEVICE:-auto} ${OS_VERSION}"
echo "- Test Management: project ${TM_PROJECT}, folder ${TM_FOLDER_NAME} (${TM_FOLDER_ID})"
} > tc-summary.md
cat tc-summary.md >> "$GITHUB_STEP_SUMMARY"
Expand All @@ -105,7 +169,7 @@ jobs:
with:
name: tc-explore-logs
path: |
tc-run.log
tc-run-*.log
tc-summary.md
if-no-files-found: ignore
retention-days: 7
37 changes: 37 additions & 0 deletions scripts/resolve-bs-app.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
#!/usr/bin/env node
// Resolves an App Automate app reference to a real bs://<hash> id.
// Usage: node scripts/resolve-bs-app.js <bs://hash | bs://CustomId | CustomId | user/CustomId>
// Env: BROWSERSTACK_USERNAME, BROWSERSTACK_ACCESS_KEY
// Prints the bs:// id on stdout. Exits 1 if the app cannot be found.
const { Buffer } = require('buffer');

async function main() {
const raw = (process.argv[2] || '').trim();
if (!raw) throw new Error('app reference is required');
const user = process.env.BROWSERSTACK_USERNAME;
const key = process.env.BROWSERSTACK_ACCESS_KEY;
if (!user || !key) throw new Error('BROWSERSTACK_USERNAME / BROWSERSTACK_ACCESS_KEY not set');

const ref = raw.replace(/^bs:\/\//, '');
// A real app id is a 40-char hex hash; use it as-is.
if (/^[0-9a-f]{40}$/i.test(ref)) {
console.log(`bs://${ref}`);
return;
}
const customId = ref.includes('/') ? ref.split('/').pop() : ref;
const res = await fetch(
`https://api-cloud.browserstack.com/app-automate/recent_apps/${encodeURIComponent(customId)}`,
{ headers: { Authorization: 'Basic ' + Buffer.from(`${user}:${key}`).toString('base64') } }
);
if (!res.ok) throw new Error(`Lookup failed for "${customId}": ${res.status} ${await res.text()}`);
const apps = await res.json();
if (!Array.isArray(apps) || apps.length === 0 || !apps[0].app_url)
throw new Error(`No uploaded app found for custom id "${customId}"`);
// The API lists the newest upload first.
console.log(apps[0].app_url);
}

main().catch((e) => {
console.error(e.message);
process.exit(1);
});
61 changes: 61 additions & 0 deletions scripts/run-tc-explore.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
#!/usr/bin/env bash
# Trigger the Test Companion exploration directly, without the app build/merge flow.
#
# Usage:
# ./scripts/run-tc-explore.sh ci [app] [device] [os_version] # runs the GitHub workflow (workflow_dispatch)
# ./scripts/run-tc-explore.sh local [app] [device] [os_version] # runs tc on THIS machine
#
# app bs://<hash>, or a custom id such as BStackBankLatest (default). Custom ids are
# resolved to the real bs://<hash> first; tc rejects a bare custom id.
# device optional, e.g. "Google Pixel 8". Empty = Test Companion chooses.
# os_version optional, only used together with a device
#
# Both modes use .github/tc-context/feature-context.md as the exploration prompt.
set -euo pipefail

MODE="${1:-ci}"
APP="${2:-BStackBankLatest}"
DEVICE="${3:-}"
OSV="${4:-}"

TM_PROJECT="PR-79"
TM_FOLDER_NAME="Functional"
TM_FOLDER_ID="60003370"
CTX_FILE=".github/tc-context/feature-context.md"
WORKFLOW="tc-explore.yml"

case "$MODE" in
ci)
command -v gh >/dev/null || { echo "gh CLI required"; exit 1; }
BRANCH="$(git rev-parse --abbrev-ref HEAD)"
echo "Dispatching $WORKFLOW (needs the workflow file on the default branch)"
gh workflow run "$WORKFLOW" -f app="$APP" -f device="$DEVICE" -f os_version="$OSV"
sleep 4
gh run list --workflow "$WORKFLOW" --limit 1
echo "Watch it with: gh run watch \$(gh run list --workflow $WORKFLOW --limit 1 --json databaseId -q '.[0].databaseId')"
;;
local)
command -v tc >/dev/null || { echo "tc not installed"; exit 1; }
[ -f "$CTX_FILE" ] || { echo "$CTX_FILE not found (run from the repo root)"; exit 1; }
: "${BROWSERSTACK_USERNAME:?set BROWSERSTACK_USERNAME (e.g. source .env)}"
: "${BROWSERSTACK_ACCESS_KEY:?set BROWSERSTACK_ACCESS_KEY (e.g. source .env)}"
APP="$(node scripts/resolve-bs-app.js "$APP")"
echo "App: $APP"
tc auth login >/dev/null
tc tm use browserstack >/dev/null
tc tm review --reject --all >/dev/null 2>&1 || true # no unverified leftovers
DEVICE_ARGS=()
[ -n "$DEVICE" ] && DEVICE_ARGS+=(--device "$DEVICE")
[ -n "$DEVICE" ] && [ -n "$OSV" ] && DEVICE_ARGS+=(--os-version "$OSV")
CTX="$(cat "$CTX_FILE")"
INSTRUCTIONS="Save every generated test case to Test Management project ${TM_PROJECT} (Test-Companion-Demo-Vision), inside the existing folder ${TM_FOLDER_NAME} (folder id ${TM_FOLDER_ID}). Do not create a new project and do not use the Automation folders.
Write manual, step-based test cases only (a title, preconditions, numbered steps, and one clear expected result per step). Do not generate automation scripts or file names.
You MUST explore the feature on the real device and base every test case on what you actually observed. If the device session cannot be started or is lost, STOP and report failure. Never write test cases from the description alone and never save unverified cases.

${CTX}"
tc app generate --app "$APP" --platform android ${DEVICE_ARGS[@]+"${DEVICE_ARGS[@]}"} \
--auto-grant-permissions --save-to-tm --yolo "$INSTRUCTIONS"
;;
*)
echo "Usage: $0 ci|local [app] [device] [os_version]"; exit 1;;
esac
Loading