-
Notifications
You must be signed in to change notification settings - Fork 2
added an ai using tutorial page with screenshots and a diagram #661
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,163 @@ | ||
| --- | ||
| title: Using AI in Testing | ||
| description: Learn how to use AI across a full testing cycle in Testomat.io. This guide walks you through enabling AI, connecting a custom AI provider, adding a requirement, generating tests and descriptions from it, analyzing a run, and chatting with your analytics. | ||
| type: article | ||
| url: https://docs.testomat.io/tutorials/using-ai-in-testing | ||
| head: | ||
| - tag: meta | ||
| attrs: | ||
| name: og:image | ||
| content: https://docs.testomat.io/_astro/New_b1BS3EKN_2024-08-29.DGrFkMLR_ZOOgn1.webp | ||
|
|
||
| - tag: meta | ||
| attrs: | ||
| name: keywords | ||
| content: AI in testing, Testomat.io, AI-powered features, AI Agents, AI requirements, generate tests from requirements, suggest test case description, custom AI provider, clusterize errors, analyze failed tests, chat with analytics, test coverage, test design, test management, software testing, QA | ||
| --- | ||
|
|
||
| Welcome! | ||
|
|
||
| This tutorial turns on Testomat.io's AI features and walks them through a full testing cycle: you start from a requirement, generate tests from it, describe them, then analyze the run and question your analytics. AI here is a helper that saves you typing and digging, not a replacement for your judgment. | ||
|
|
||
|  | ||
|
|
||
| What you will do: | ||
|
|
||
| 1. Enable AI for your company. | ||
| 2. Add a requirement to your project. | ||
| 3. Generate tests from the requirement. | ||
| 4. Generate a test description. | ||
| 5. Analyze a run with AI. | ||
| 6. Chat with Analytics. | ||
|
|
||
| ## Before you start | ||
|
|
||
| AI features are off by default. A Company Owner turns them on, and only your company's chosen AI provider sees the content you send. | ||
|
|
||
| :::note | ||
|
|
||
| Only enable AI if your test data is not sensitive. The AI provider processes the text you send it, so check your data privacy rules first. | ||
|
|
||
| ::: | ||
|
|
||
| ## Enable AI | ||
|
|
||
| To integrate and begin working with AI features in your Testomat.io account, follow these steps: | ||
|
|
||
| 1. Open **Company Settings**. | ||
| 2. Go to the **Administration** section. | ||
| 3. Turn on the **AI Features**. | ||
|
|
||
|  | ||
|
TetianaKhomenko marked this conversation as resolved.
|
||
|
|
||
| Once enabled, the AI actions appear across your project. See [AI settings](https://docs.testomat.io/management/company/administration/#ai) for the details. | ||
|
|
||
| ### Use your own AI provider | ||
|
|
||
| If your company prefers an AI provider it already trusts, Testomat.io can connect to a third-party service such as OpenAI, Anthropic, and others. You keep all the same AI features while choosing the provider that fits your security and compliance rules. | ||
|
|
||
| 1. In the AI settings, turn on the **Custom AI Provider** option. | ||
| 2. Select a provider from the dropdown. | ||
| 3. Enter the **API Key**. | ||
| 4. Enter the **Model** name. | ||
| 5. Enter **Max Tokens**, if your provider requires it. | ||
| 6. Click **Save Settings**. | ||
|
|
||
|  | ||
|
|
||
| :::note | ||
|
|
||
| Once a custom provider is connected, every AI request runs through it. Testomat.io does not fall back to a shared provider — if your provider is unavailable, the request fails and you are notified. See [Custom AI Provider](https://docs.testomat.io/management/company/administration/#custom-ai-provider) for the full reference. | ||
|
|
||
| ::: | ||
|
|
||
| ## Add a requirement | ||
|
|
||
| Requirements are what your tests exist to cover, so the cycle starts here. Link one to your project and AI can generate tests from it. | ||
|
|
||
| First, set up the integration for your requirement source. See [Jira](https://docs.testomat.io/integrations/issues-management/jira/#connecting-to-jira-project) or [Confluence](https://docs.testomat.io/integrations/issues-management/confluence) for the steps. Then add the requirement: | ||
|
|
||
| 1. Open the **Requirements** page. | ||
| 2. Click **+ New Requirements** or **+ Add Requirement**. | ||
| 3. Select your requirement source, for example Jira. | ||
| 4. Enter the source identifier, such as the **Jira Issue ID**. | ||
| 5. Click **Save**. | ||
|
|
||
|  | ||
|
|
||
| :::note | ||
|
|
||
| You can add a requirement to an ongoing project at any time using the same steps. See [AI-Requirements](https://docs.testomat.io/advanced/ai-powered-features/ai-requirements/) for other sources, including files and plain text. | ||
|
|
||
| ::: | ||
|
|
||
| ## Generate tests from the requirement | ||
|
|
||
| With the requirement linked, AI reads it and proposes the test cases that cover it, so you start from a draft instead of a blank suite. | ||
|
|
||
| 1. Open the **Requirements** page and click your requirement. | ||
| 2. On the **Summary** tab, click **Analyze Requirement**. | ||
| 3. Click **Add tests to {Suite_name} Suite**. | ||
| 4. Review the suggested cases and click **Add** on the ones you want. | ||
|
|
||
|  | ||
|
|
||
| Click **Suggest More Tests** if you need extra cases, and double-click any suggested title to edit it before adding. | ||
|
|
||
| :::note | ||
|
|
||
| Nothing is added automatically. You always choose which cases to keep. | ||
|
|
||
| ::: | ||
|
|
||
| ## Generate a test description | ||
|
|
||
| A test name tells you what to check, but not how. AI writes the description from the name, or improves the one you already have. | ||
|
|
||
| 1. Open a test case. | ||
| 2. Click **Suggest Description**. | ||
| 3. Review the result, edit it if needed, then save. | ||
|
|
||
|  | ||
|
|
||
| :::note | ||
|
|
||
| This works the same way in BDD projects. On the **Scenario description** tab, click **Suggest Description** and AI turns your Given/When/Then steps into a readable overview. | ||
|
|
||
| ::: | ||
|
|
||
| ## Analyze a run with AI | ||
|
|
||
| Once your tests have run, AI groups the failures so you read a handful of causes instead of a wall of errors. | ||
|
|
||
| 1. Go to the **Runs** page. | ||
| 2. Open a finished automated run. | ||
| 3. Click **Clusterize Errors**. | ||
|
|
||
| :::note | ||
|
|
||
| **Clusterize Errors** appears only on finished automated runs with 5 or more failures. | ||
|
|
||
| ::: | ||
|
|
||
| ## Chat with Analytics | ||
|
|
||
| Analytics answers questions you already know to ask. Chat with Analytics lets you ask in plain language instead of hunting through dashboards. | ||
|
|
||
| 1. Go to the **Analytics** page. | ||
| 2. Open **Chat with Analytics**. | ||
| 3. Ask your question and send it. | ||
|
|
||
| Ask things like which tests are most unstable, which areas fail most often, or how execution trends changed over time. | ||
|
|
||
| :::note | ||
|
|
||
| Metrics are calculated over a 30-day window unless you specify a different range. | ||
|
|
||
| ::: | ||
|
|
||
| ## Next steps | ||
|
|
||
| * Want to expand your AI capabilities? Try setting up [AI Agents](https://docs.testomat.io/advanced/ai-powered-features/ai-agents/). | ||
| * See every generative feature across suites, tests, and code in [AI-Powered Features](https://docs.testomat.io/advanced/ai-powered-features/ai-powered-features/). | ||
| * Drive your test design from product requirements with [AI-Requirements](https://docs.testomat.io/advanced/ai-powered-features/ai-requirements/). | ||
Binary file added
BIN
+120 KB
src/content/docs/tutorials/images/working-with-ai/1-ai-features-settings.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+113 KB
src/content/docs/tutorials/images/working-with-ai/2-custom-provider.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+66.8 KB
src/content/docs/tutorials/images/working-with-ai/3-add-requirement.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+107 KB
...ent/docs/tutorials/images/working-with-ai/4-generate-tests-from-requirement.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+91.7 KB
src/content/docs/tutorials/images/working-with-ai/5-suggest-description.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+69.3 KB
src/content/docs/tutorials/images/working-with-ai/ai-capabilities.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In fact we have so may AI features, maybe we shall try to show all of them grouped by "test design", "requirements", "reporting"?