diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS new file mode 100644 index 00000000..280bbd46 --- /dev/null +++ b/.github/CODEOWNERS @@ -0,0 +1 @@ +* @amitava82 diff --git a/src/content/docs/docs/alerts/alert-groups.mdx b/src/content/docs/docs/alerts/alert-groups.mdx index ffb91dc2..929cb388 100644 --- a/src/content/docs/docs/alerts/alert-groups.mdx +++ b/src/content/docs/docs/alerts/alert-groups.mdx @@ -130,7 +130,7 @@ Each tab shows how many items it holds, such as **Instances (7)**. - **Notifications:** every notification sent for the group, reminders included, with each channel's result and a link to where it landed: the Slack thread, the Jira ticket, the KloudMate Incidents incident, and so on. - **Audit:** a timeline of the group's history, from when it opened to when it resolved. It records each notification and reminder sent, notifications suppressed by a silence or skipped, and investigations. For Auto-correlated groups, it also logs when the engine opened the group and when it correlated each alert. -![Reminder on an alert group Audit tab](./images/alert-group-audit-reminder.png) +![The Audit tab of an alert group: a timeline from the group opening through each notification sent, including a reminder and a new instance](./images/alert-group-audit-tab.png) ### Instances diff --git a/src/content/docs/docs/alerts/images/alarm-detail-overview.png b/src/content/docs/docs/alerts/images/alarm-detail-overview.png index 9343f19c..35d7a40e 100644 Binary files a/src/content/docs/docs/alerts/images/alarm-detail-overview.png and b/src/content/docs/docs/alerts/images/alarm-detail-overview.png differ diff --git a/src/content/docs/docs/alerts/images/alert-group-audit-reminder.png b/src/content/docs/docs/alerts/images/alert-group-audit-reminder.png deleted file mode 100644 index c87f21f5..00000000 Binary files a/src/content/docs/docs/alerts/images/alert-group-audit-reminder.png and /dev/null differ diff --git a/src/content/docs/docs/alerts/images/alert-group-audit-tab.png b/src/content/docs/docs/alerts/images/alert-group-audit-tab.png new file mode 100644 index 00000000..6f6e0e22 Binary files /dev/null and b/src/content/docs/docs/alerts/images/alert-group-audit-tab.png differ diff --git a/src/content/docs/docs/alerts/images/instances-tab-dismiss.png b/src/content/docs/docs/alerts/images/instances-tab-dismiss.png index 4333b203..e3892b52 100644 Binary files a/src/content/docs/docs/alerts/images/instances-tab-dismiss.png and b/src/content/docs/docs/alerts/images/instances-tab-dismiss.png differ diff --git a/src/content/docs/docs/apm-and-tracing/apm-views/api-monitoring.mdx b/src/content/docs/docs/apm-and-tracing/apm-views/api-monitoring.mdx index 29549d7e..d38b520e 100644 --- a/src/content/docs/docs/apm-and-tracing/apm-views/api-monitoring.mdx +++ b/src/content/docs/docs/apm-and-tracing/apm-views/api-monitoring.mdx @@ -38,6 +38,13 @@ The protocol badge determines what a row can show: - **HTTP** rows carry a method chip and a full `2xx`/`3xx`/`4xx`/`5xx` status breakdown. - **RPC** rows (gRPC, Connect) have no HTTP method or status classes, so the **Status** column shows a dash (`—`). Judge these endpoints by their **error rate**, which is derived from span status and works for every protocol. +### No route and Unknown method rows + +Two kinds of HTTP row group requests that have no endpoint label of their own: + +- **No route:** requests the service handled without reporting a route. Typical causes are `404` responses, requests rejected before routing (authentication, CORS preflight `OPTIONS`), and proxies that do not record routes. Each service gets one such row per HTTP method. Open the row to see the individual requests and, where the span records one, their paths. +- **Unknown method:** requests with a non-standard HTTP method, such as scanner verbs (`PROPFIND`) or request lines the instrumentation could not read. Each service gets one such row. + :::tip The trend sparkline rolls up hourly, so it needs at least two hourly buckets to draw a line. ::: @@ -75,7 +82,7 @@ At the top, summary tiles show **Requests**, **Error rate**, and **p95 latency** The drill-down ends with a table of the endpoint's traces, so you can move from a metric to the exact requests behind it: -- Each row shows the start time, span, and duration, with a **View trace** action that opens the full trace in [Trace Detail](../../trace-detail/). +- Each row shows the start time, the request (its path, or the method name for RPC), the HTTP status, and the duration, with a **View trace** action that opens the full trace in [Trace Detail](../../trace-detail/). - Toggle **Failed only** to keep just the errored requests. This uses the same protocol-neutral error definition as the metrics, so it covers HTTP `5xx` and gRPC/Connect errors alike. - **Open in Traces** hands the current endpoint and time range to the [Trace Explorer](../../trace-explorer/) for deeper filtering and analysis. diff --git a/src/content/docs/docs/apm-and-tracing/apm-views/service-map.mdx b/src/content/docs/docs/apm-and-tracing/apm-views/service-map.mdx index f50055fa..409c4d97 100644 --- a/src/content/docs/docs/apm-and-tracing/apm-views/service-map.mdx +++ b/src/content/docs/docs/apm-and-tracing/apm-views/service-map.mdx @@ -12,10 +12,15 @@ You can use it to understand service dependencies, traffic flow, and operational The map is built from distributed traces, so both sources of tracing feed it: -- **[eBPF monitoring](../../../kloudmate-agent/ebpf-observability/)** captures traces at the kernel level with no code change, so an eBPF-covered service appears on the map on its own — on VMs, hosts, and containers. (On [Kubernetes](../../../kloudmate-agent/auto-instrumentation/kubernetes/), eBPF provides the network topology instead, and traces come from the SDK.) +- **[eBPF monitoring](../../../kloudmate-agent/ebpf-observability/)** captures traces at the kernel level with no code change, so an eBPF-covered service appears on the map on its own. This works on VMs, hosts, and containers, and for [Kubernetes](../../../kloudmate-agent/auto-instrumentation/kubernetes/) workloads set to **eBPF**. - **[Auto-instrumentation](../../../kloudmate-agent/auto-instrumentation/overview/)** and the OpenTelemetry SDK produce traces from inside the application. -Each service becomes a node under its `service.name`, and an edge is drawn wherever one service's trace calls another. A service needs a distinct `service.name` to appear as its own node — the agent gives an eBPF-monitored service the same name it carries once auto-instrumented, so it keeps one identity whether eBPF or the SDK is tracing it. +Each service becomes a node under its `service.name`, and an edge is drawn wherever one service's trace calls another. A service needs a distinct `service.name` to appear as its own node. The agent gives an eBPF-monitored service the same name it carries once auto-instrumented, so it keeps one identity whether eBPF or the SDK is tracing it. + +The map also draws calls where only one side is traced: + +- **Calls into an eBPF-monitored service from a caller that sends no telemetry**, such as a script or an uninstrumented service. The caller is named after its Kubernetes workload, or shown as `incoming` when it can't be identified. +- **Calls to a database.** Databases don't send traces, so the calling service's trace is enough to draw the edge. ## Overview @@ -25,7 +30,7 @@ Open **APM & Services** and select the **Service map** view. You can also open t Summary chips count the services, external dependencies, dependencies, services with errors, and erroring edges in the selected time range. A service or dependency counts as erroring when its error rate is 0.05% or higher. -The **Service map** view uses the same time range as the **List** view. Click **Refresh** to reload the map with the latest data, and turn on **Show external dependencies** to add the uninstrumented dependencies your services call, such as databases, queues, and third-party hosts. +The **Service map** view uses the same time range as the **List** view. Click **Refresh** to reload the map with the latest data, and turn on **Show external dependencies** to add the uninstrumented hosts your services call, such as third-party APIs, message queues, and managed databases outside your cluster. Databases running inside your cluster appear without the switch. The main area displays the **Service Dependency Graph**. Each service is represented as a node showing its name, protocol, request rate, and average latency. The directed edges between nodes represent the call flow and dependencies between services. diff --git a/src/content/docs/docs/database-monitoring/agent-jobs-and-awr.mdx b/src/content/docs/docs/database-monitoring/agent-jobs-and-awr.mdx index fe4db218..05cc225d 100644 --- a/src/content/docs/docs/database-monitoring/agent-jobs-and-awr.mdx +++ b/src/content/docs/docs/database-monitoring/agent-jobs-and-awr.mdx @@ -15,12 +15,15 @@ The **Agent jobs** tab lists the SQL Server Agent job runs in the time range, so Each run shows the job name, its status, when it ran, how long it took, and the message it logged. Expand a run to read the full message. -To collect job runs, turn on the **Agent jobs** coverage for the instance in the agent's database monitoring settings. The monitoring login needs the `SQLAgentReaderRole` role in the `msdb` database: +To collect job runs, turn on the **Agent jobs** coverage for the instance in the agent's database monitoring settings. The monitoring login needs read access to the job tables in the `msdb` database: ```sql USE msdb; CREATE USER [kloudmate] FOR LOGIN [kloudmate]; -ALTER ROLE SQLAgentReaderRole ADD MEMBER [kloudmate]; +GRANT SELECT ON dbo.sysjobs TO [kloudmate]; +GRANT SELECT ON dbo.sysjobhistory TO [kloudmate]; +GRANT SELECT ON dbo.sysjobactivity TO [kloudmate]; +GRANT EXECUTE ON dbo.agent_datetime TO [kloudmate]; ``` Replace `kloudmate` with your monitoring login. Skip the `CREATE USER` line if the login already has a user in `msdb`. diff --git a/src/content/docs/docs/database-monitoring/databases-list.mdx b/src/content/docs/docs/database-monitoring/databases-list.mdx index 01553c41..774f5b67 100644 --- a/src/content/docs/docs/database-monitoring/databases-list.mdx +++ b/src/content/docs/docs/database-monitoring/databases-list.mdx @@ -56,6 +56,6 @@ You can sort by instance name, the query columns, connections, and uptime. Insta 1. Click **Monitor a database**. 2. Choose the agent that will monitor the database. The agent must be able to reach the database: install it on the database's own host for a self-hosted database, or on a host with network access to a managed one. Agents in manual configuration mode can't be used here; switch the agent to managed in **Settings** first. 3. Choose the engine. The list shows the engines that agent supports. -4. Configure the database in the agent's database monitoring settings, which open with the agent and engine already selected. See [Database monitoring overview](../../kloudmate-agent/database-monitoring/overview/) for the coverages and credentials. +4. Configure the database in the agent's database monitoring settings, which open with the agent and engine already selected. See [the wizard steps](../../kloudmate-agent/database-monitoring/overview/#the-wizard) for the coverages and credentials. Data starts arriving within a minute of saving. diff --git a/src/content/docs/docs/database-monitoring/index.mdx b/src/content/docs/docs/database-monitoring/index.mdx index 3f1412e8..c1be198c 100644 --- a/src/content/docs/docs/database-monitoring/index.mdx +++ b/src/content/docs/docs/database-monitoring/index.mdx @@ -12,7 +12,7 @@ The **Databases** page lists every database the KloudMate agent monitors, with i ## Get your databases onto the page -A database appears on the **Databases** page after you set up monitoring for it in the KloudMate agent. You do this through a wizard: pick the engine, choose what to collect, and give the agent a monitoring user's credentials. See [Database monitoring overview](../kloudmate-agent/database-monitoring/overview/) for the setup, and [Engines and coverages](../kloudmate-agent/database-monitoring/engines-and-coverages/) for what each engine can collect. +A database appears on the **Databases** page after you set up monitoring for it in the KloudMate agent. You do this through a [wizard](../kloudmate-agent/database-monitoring/overview/#the-wizard): pick the engine, choose what to collect, and give the agent a monitoring user's credentials. See [Database monitoring overview](../kloudmate-agent/database-monitoring/overview/) for the setup, and [Engines and coverages](../kloudmate-agent/database-monitoring/engines-and-coverages/) for what each engine can collect. What you see on an instance depends on the **coverages** you turned on for it in the wizard. For example, **Top queries** and **Query patterns** need the **Query performance (samples + top queries)** coverage, and **Topology** needs **Replication topology & lag**. When a tab has no data because a coverage is off, the tab tells you which coverage to turn on. diff --git a/src/content/docs/docs/incident-management/index.mdx b/src/content/docs/docs/incident-management/index.mdx index 55cea03f..59a7a9ab 100644 --- a/src/content/docs/docs/incident-management/index.mdx +++ b/src/content/docs/docs/incident-management/index.mdx @@ -19,6 +19,7 @@ Use Incident Management to organize incident response workflows, route alerts to - [Routing Rules](./routing-rules/) - [Status Pages](./status-pages/) - [Slack Integration](./slack-integration/) +- [War Rooms](./war-rooms/) :::note **Two features share the name "Routing rules."** The [Routing Rules](./routing-rules/) listed here decide how an incoming alert becomes an incident: which service, escalation policy, and severity it gets, and whether to suppress it. To route alert *notifications* to channels like Slack, see [Alerts → Routing rules](../alerts/routing-rules/) instead. diff --git a/src/content/docs/docs/incident-management/war-rooms/_meta.json b/src/content/docs/docs/incident-management/war-rooms/_meta.json new file mode 100644 index 00000000..6cc89020 --- /dev/null +++ b/src/content/docs/docs/incident-management/war-rooms/_meta.json @@ -0,0 +1,4 @@ +{ + "label": "War Rooms", + "order": 12 +} diff --git a/src/content/docs/docs/incident-management/war-rooms/index.mdx b/src/content/docs/docs/incident-management/war-rooms/index.mdx new file mode 100644 index 00000000..4607a194 --- /dev/null +++ b/src/content/docs/docs/incident-management/war-rooms/index.mdx @@ -0,0 +1,81 @@ +--- +title: "Incident War Rooms" +description: "Open a dedicated Slack channel or Microsoft Teams chat for an incident, with the responders already in it and a call link attached." +sidebar: + label: "Overview" + order: 1 +--- + +Some incidents need a room, not another notification. A war room is a Slack channel or a Microsoft Teams group chat that belongs to one incident. KloudMate creates it, invites the responders and the on-call engineers, posts an incident card at the top, and attaches a conference call link. + +The card tracks the incident. It shows the current severity and status, and in Slack you can acknowledge, resolve, add responders, and leave a note without opening KloudMate. + +An incident gets one room. If a routing rule, a Slack button, and the incident page all ask for a room in the same minute, you still end up with one channel. + +## Open a war room + +- **From the incident.** Click **Start war room** on the **Incident War Room** panel. You can change the channel name, paste a call link, and invite extra people before it opens. +- **From Slack.** Click **Start war room** on the incident alert in your shared channel. +- **From a routing rule.** Turn on **Start War Room** in the rule's actions, and every incident it creates opens with a room. +- **From an escalation step.** Check **Start War Room on this step**, and the room opens when the incident escalates that far. + +Both automatic paths are covered in [Set up war rooms](./setup/#open-a-room-automatically). KloudMate creates the room in the background, so it appears a few seconds after the incident, not instantly. + +## What responders see in Slack + +The room opens as a private channel by default, and the first message is the incident card. + +The card leads with the incident title as a link back to KloudMate, then the severity, status, and service. Below that it lists everyone who was invited. Anyone KloudMate couldn't match to a Slack account is called out under **Could not add**, so you know to pull them in yourself. When the incident has a call link, the card carries a **Join Call** button, and a dial-in number when the call provider gives one. + +The channel topic repeats the severity, the title, the link to the incident, and the call link. It's there for whoever joins an hour in and scrolls to the top of nothing. + +### Act on the incident from the room + +The card's buttons change with the incident: + +| Button | What it does | +|---|---| +| **Acknowledge** / **Unack** | Acknowledges the incident, or takes the acknowledgement back. | +| **Resolve** / **Unresolve** | Resolves the incident, or reopens it. | +| **Add Responders** | Opens a picker of workspace members and attaches them to the incident. Each one is invited to the room. | +| **Note** | Opens a box for a note. It lands on the incident's **Notes** tab with your name against it. | +| **Run RCA** | Starts an [AI investigation](/kloudmate-assistant/investigations/) for the incident. | + +While an investigation runs, the card reads **RCA Running**. The findings arrive as a reply in the card's thread, with a link to the full investigation, and the card switches to **RCA Complete**. If the investigation stalls, the card reads **RCA Failed** after 15 minutes and **Run RCA** comes back so you can try again. + +KloudMate also replies in that thread as the incident moves: acknowledged or unacknowledged, resolved or reopened, assigned or unassigned, escalated, and re-notified, each stamped in UTC. The card and the topic are rewritten in place on every change, wherever the change came from. Resolve the incident in KloudMate and the Slack card follows. + +## What responders see in Microsoft Teams + +The room is a group chat owned by the Microsoft 365 account you connected. The card shows the same incident facts, the people who were invited, the ones who couldn't be added, and buttons for **Join Call** and **View Incident**. It's rewritten in place as the incident changes, the same as in Slack. + +:::note +Teams rooms don't carry the action buttons, the thread replies, or RCA. Acknowledge, resolve, and add notes from the incident page instead. Teams group chats also have no public or private setting, so **Private room** has no effect on them. +::: + +## The incident alert turns into a pointer + +Once a Slack war room exists, the incident alert in your shared channel stops carrying its own buttons and becomes a signpost: **Open war room** and **Open in KloudMate**. It isn't rewritten again after that, so read the room's card for the current state. That keeps the conversation in one place instead of splitting it between the alert thread and the room. + +## Follow the room from the incident page + +The **Incident War Room** panel on the incident shows the room as **Active** or **Archived**, links to the channel or chat, and lists the members. A member KloudMate couldn't match to a chat account is tagged **Unmapped**, and hovering it tells you why. + +**Conference call** sits beside it. Use the edit icon to add or replace the **Meeting link** and the **Dial-in** at any point while the incident is open, and KloudMate updates the card and the topic. A Developer can also end the room early with **Archive war room**, which posts the closing summary and archives the channel straight away. + +## When the incident resolves + +KloudMate marks the card resolved, then starts the clock on the archive delay, which is one hour unless you've changed it. When the delay runs out, it posts a closing summary and archives the Slack channel. A Teams chat stays in place, because group chats can't be archived, so the closing summary is what tells responders the room is done. Either way the history stays in your chat workspace. + +Set the delay to **Keep the room open** if you'd rather archive rooms yourself. + +:::caution +Reopening an incident doesn't cancel an archive that's already scheduled. If you unresolve an incident inside the delay window, the Slack channel is still archived on time, and starting the war room again returns the archived room. Unarchive the channel in Slack to carry on in it. +::: + +## Related + +- [Set up war rooms](./setup/): what to connect, which permissions it needs, and the defaults for every room. +- [Troubleshooting](./troubleshooting/): what to do when a room won't open or opens without a call. +- [Slack Integration](../slack-integration/): the install that war rooms use, and per-user account linking for ChatOps. +- [Routing Rules](../routing-rules/) and [Escalation Policies](../escalation-policy/): the two places a room can open on its own. diff --git a/src/content/docs/docs/incident-management/war-rooms/setup.mdx b/src/content/docs/docs/incident-management/war-rooms/setup.mdx new file mode 100644 index 00000000..a768aeab --- /dev/null +++ b/src/content/docs/docs/incident-management/war-rooms/setup.mdx @@ -0,0 +1,119 @@ +--- +title: "Set up war rooms" +description: "Connect Slack or Microsoft Teams, choose how the conference call is created, and set the defaults every war room opens with." +sidebar: + label: "Setup" + order: 2 +--- + +import { Steps } from '@astrojs/starlight/components'; + +War rooms run on the chat connection you already use for incident notifications. Point them at a Slack or Microsoft Teams channel, decide who gets invited, and optionally let KloudMate create a conference call for each incident. + +The settings live on **Incidents → Chat Ops**, under **Incident War Rooms**, below the chat connection they use. You need the **Developer** role to change them. Everyone else sees them read-only. + +## Before you start + +- Your plan includes Incident Management. +- Your workspace has a Slack or Microsoft Teams channel under **Incidents → Chat Ops**. See [Slack Integration](../../slack-integration/). +- For a conference call that's created per incident, you've connected Microsoft 365, Google Meet, or Zoom under **Settings → Connections**. + +## Set up Slack rooms + +Slack needs more permissions to open a war room than it does to post a notification. Creating a channel, reading a member's email address, and inviting people all sit behind permissions that a notifications-only install was never granted. + + +1. Open **Settings → Connections** and find your Slack connection. + +2. Click **Edit**, add **Workflows** under **Can be used for**, and save. Those are the permissions a war room needs. + +3. Click **Reconnect** beside the connection and approve the Slack install again. Approving again only adds permissions, so you don't lose what Slack already had. + +4. Open **Incidents → Chat Ops**, set **Chat provider** to **Slack**, and pick the channel under **Chat channel**. + + +KloudMate checks the channel's permissions as soon as you pick it. If anything is missing, a **Reconnect Slack to open war rooms** banner lists the exact scopes, and **Start war room** stays disabled until you fix it. These are the ones it checks: + +| Scope | Why it's needed | +|---|---| +| `chat:write` | Post and update the incident card. | +| `users:read` and `users:read.email` | Match a KloudMate user to their Slack account by email address. | +| `groups:write` | Create a private channel and invite people to it. | +| `channels:manage` | The same, for public channels. | + +## Set up Microsoft Teams rooms + +A Teams room is a group chat, created by the Microsoft 365 account you sign in with. That account owns every room, so use a service account rather than a person's. + + +1. Open **Settings → Connections** and connect **Microsoft 365 / Teams**. Sign in as the account that should own the rooms. + +2. Approve the permissions on the Microsoft consent screen. + +3. Add a Microsoft Teams channel under **Incidents → Chat Ops**. + +4. Set **Chat provider** to **Microsoft Teams** and pick that channel under **Chat channel**. + + +A Microsoft 365 connection is granted for **Notifications** and **Meetings** when you make it, which is what a Teams war room needs. The room and the call come from the same grant, so one consent screen covers creating the chat, posting to it, looking a member up by email address, and creating the online meeting. + +## Add a conference call + +**Conference call** decides where the **Join Call** button points: + +| Option | What happens | +|---|---| +| **No call** | No call link, unless whoever opens the room pastes one. | +| **The same link every time** | KloudMate uses your **Call link**, rendered per incident, plus an optional **Dial-in**. Use a standing bridge, or a link with the incident key in it, such as `https://meet.google.com/lookup/inc-{{ key }}`. | +| **A new meeting per incident** | KloudMate creates a fresh meeting on the connection you pick. | + +For a new meeting per incident, connect one of these under **Settings → Connections** and select it in the war room settings. The picker lists connections granted for **Meetings**, so a Slack or AWS connection never shows up there. + +### Microsoft Teams + +The Microsoft 365 connection you set up above also creates the meetings. Nothing else to connect. A Teams meeting comes with a dial-in number when your tenant has audio conferencing. + +### Google Meet + +Connect **Google Meet** and sign in as the Google Workspace account that should own the meeting spaces. The consent screen asks to create meeting spaces, plus your name and email address to label the connection. The meeting permission reaches only the spaces KloudMate creates itself, not the rest of your Workspace. Google Meet returns no dial-in number, so rooms show the **Join Call** button on its own. + +### Zoom + +Connect **Zoom** and install the KloudMate app. Every war room meeting is created as the account that installs it, so install it from a Licensed Zoom account: meetings hosted by a Basic account cut out after 40 minutes, which is not what you want halfway through an incident. Zoom rooms don't show a dial-in number. + +Removing the KloudMate app from Zoom revokes the connection. KloudMate erases the stored credential, marks the connection **Revoked**, and switches it off, so rooms open without a call link until the app is installed again. **Reconnect** beside the connection re-runs the install and brings it back, with the same connection id, so the war room settings keep pointing at it. + +:::caution +Account-level Zoom settings override what KloudMate asks for. If responders land in a waiting room or get turned away, check the account's join-before-host and authentication settings. +::: + +## Set the defaults for every room + +| Setting | What it controls | +|---|---| +| **Room name template** | The channel name or chat topic. Use `{{ number }}`, `{{ key }}`, `{{ title }}`, `{{ severity }}`, and `{{ service }}`. The default is `inc-{{ number }}-{{ title }}`. Slack names are lowercased and stripped to letters, numbers, hyphens, and underscores, up to 80 characters. A Teams topic can run to 50. | +| **Private room** | Opens Slack rooms as private channels, which people have to be invited to. Turn it off to open public channels anyone in the Slack workspace can join. | +| **Incident responders** | Invites whoever is assigned to the incident. | +| **On-call engineers** | Invites whoever is on call. Leave the schedule picker empty to follow the incident's escalation policy, or name specific [schedules](../../oncall/) to use those instead. | +| **Always invite** | People who join every room, on call or not. An incident commander, for example. | +| **After the incident resolves** | How long a resolved room stays open before KloudMate archives it. **Keep the room open** turns the automatic archive off. | + +If a name is already taken by another incident's channel, KloudMate adds a short suffix and uses that. A name you type yourself is never changed for you, so you're told it's taken instead. + +Anyone who has no chat account for their KloudMate email address is listed on the card under **Could not add**, and tagged **Unmapped** on the incident panel. The room still opens, and everyone else is still invited. + +## Open a room automatically + +Opening a room by hand is fine for the first few. After that, let the rule that creates the incident do it. + +**From a routing rule.** Add the **Start War Room** action to the rule. Every incident that rule creates opens with a room. The rule uses the workspace roster unless you turn on **Customize member roster for this rule**, which replaces it for incidents this rule creates. See [Routing Rules](../../routing-rules/). + +**From an escalation step.** Check **Start War Room on this step** in the step editor. The room opens when the incident reaches that step, and only on the policy's first pass, so a policy that repeats won't open a second room. See [Escalation Policies](../../escalation-policy/). + +Both paths are worth scoping tightly. A rule that matches everything opens a channel for every incident, and archived channels still count towards your Slack workspace limits. + +## Related + +- [War rooms overview](../): what responders see once a room is open. +- [Troubleshooting](../troubleshooting/): scope banners, connection states, and missing call links. +- [Connections](/platform/settings/connections/): where the Slack, Microsoft 365, Google Meet, and Zoom credentials live. diff --git a/src/content/docs/docs/incident-management/war-rooms/troubleshooting.mdx b/src/content/docs/docs/incident-management/war-rooms/troubleshooting.mdx new file mode 100644 index 00000000..75202036 --- /dev/null +++ b/src/content/docs/docs/incident-management/war-rooms/troubleshooting.mdx @@ -0,0 +1,93 @@ +--- +title: "War room troubleshooting" +description: "Fix the permission banner, a chat connection that needs attention, a missing Join Call button, and responders who couldn't be invited." +sidebar: + label: "Troubleshooting" + order: 3 +--- + +Most war room problems come down to the chat connection: it isn't allowed to create a channel, or its credential has gone stale. Both show up before an incident if you look, which is the point of the banner below. + +## "Reconnect Slack to open war rooms" + +KloudMate checks the chat channel's permissions when you pick it in the settings, and again when you open the **Start war room** dialog. When something is missing, it shows this banner with the exact scopes listed, and keeps **Start war room** disabled. + +Add **Workflows** to the Slack connection under **Settings → Connections**, then reconnect it. The [setup page](../setup/#set-up-slack-rooms) walks through it and lists what each scope is for. + +:::note +The check in the settings always asks for the private-channel scope, `groups:write`. If you open public rooms, it can report `groups:write` as missing while your rooms open fine. Check **Private room** before acting on that one line. +::: + +For Microsoft Teams, the missing scopes come from the Microsoft 365 connection. Reconnect it and approve `Chat.Create`, `ChatMessage.Send`, `User.ReadBasic.All`, and `OnlineMeetings.ReadWrite`. + +## The chat connection needs attention + +If the connection is listed as anything other than **Verified**, war rooms are the first thing to break, because opening one takes more permissions than posting a message. + +| Status | What it means for war rooms | +|---|---| +| **Needs reconnect** | The grant is dead or too narrow. Rooms won't open until you reconnect. Hover the status for the reason. | +| **Revoked** | The provider refused the credential the last time something used it. Reconnect or re-enter it. | +| **Degraded** | The last check failed. Rooms may still open, since the check and the room use different calls. **Re-check** it. | +| **Disabled** | Someone switched the connection off. Nothing opens until it's back on. | + +**Re-check** can clear **Revoked**, but only a reconnect clears **Needs reconnect**. The full list of statuses and reconnect reasons is on [Manage a connection](/platform/settings/connections/manage-connections/#reconnect). + +## A rule should have opened a room and didn't + +Rooms open in the background, so give it a few seconds before you call it broken. + +If nothing appears, open the incident and click **Start war room**. The dialog runs the same permission check and shows you the reason: a scope gap in the banner, a channel name that's taken on the name field, or a credential error in a message. Fixing it there fixes the automatic path too. + +Also worth checking: the room only opens on an escalation step's first pass, and a routing rule only opens one for incidents it creates. Neither opens a room for an incident that already existed. + +## A connection isn't in the meeting picker + +**A new meeting per incident** lists connections granted for **Meetings**. Microsoft 365, Google Meet, and Zoom carry that grant when you connect them, so a connection that's missing from the picker was either made before this existed or had the grant removed. Open it under **Settings → Connections**, click **Edit**, and add **Meetings** under **Can be used for**. If it's then marked **Needs reconnect**, reconnect it. + +Saving a connection that can't create meetings is refused with `connection '' is not connected for meetings`. + +## The room opened without a Join Call button + +A call provider that fails doesn't hold up the room. KloudMate opens the channel, invites everyone, and leaves the call off the card. + +Open the incident, click the edit icon on **Conference call**, and paste a **Meeting link**. The card and the topic pick it up. Then look at why the provider failed: + +- The connection selected under **A new meeting per incident** was deleted, switched off, or is **Needs reconnect** or **Revoked**. Removing the KloudMate app at Zoom's end leaves the connection in exactly that state. +- The connection lost its **Meetings** grant, so the setting no longer points at anything usable. + +KloudMate doesn't retry the meeting for an open room, so adding the link by hand is the fix for the incident in front of you. + +## Zoom calls cut out after 40 minutes + +The KloudMate app was installed from a Basic Zoom account, and every war room meeting is created as the account that installed it. The Zoom connection shows **Degraded** with the account type in its error, something like `ops@example.com is a Zoom type 1 account`. + +Install the app again from a Licensed account. Meetings created after that carry no time limit. + +## Someone wasn't invited + +KloudMate matches people by email address: the address on their KloudMate account has to match their Slack or Teams account. When it can't find a match, that person is listed under **Could not add** on the card and tagged **Unmapped** on the incident panel, with the reason on hover. + +Two things cause nearly all of these. The addresses genuinely differ, which you fix at one end or the other. Or the connection can't read email addresses at all, which is the `users:read.email` scope for Slack and `User.ReadBasic.All` for Teams, and shows up as the banner above. + +Invite them to the channel yourself for the incident at hand. Once they're matched, KloudMate remembers the pairing for later rooms. + +## That name is taken + +You get this on the name field when you've typed a name another channel already holds. Choose another, or clear the field and let KloudMate name the room from your template. Generated names aren't rejected: KloudMate adds a short suffix and opens the room. + +## The room was archived while we were still using it + +An archive scheduled when the incident resolved still runs if you reopen the incident in the meantime. Unarchive the channel in Slack to carry on in it. Starting the war room again returns the same archived room rather than creating a second one. + +If this keeps happening, raise the delay under **After the incident resolves**, or set it to **Keep the room open** and archive rooms yourself with **Archive war room** on the incident panel. + +## A Teams room is missing things a Slack room has + +Teams rooms have no action buttons, no thread replies, and no RCA. Acknowledge, resolve, and add notes from the incident page. Microsoft Graph also can't archive a group chat, so a closed Teams room gets a closing summary and nothing else: KloudMate stops posting to it, but the chat stays in your Teams workspace. **Private room** doesn't apply to group chats either. + +## Related + +- [Set up war rooms](../setup/): the permissions each provider needs. +- [War rooms overview](../): what the card shows and what each button does. +- [Connections](/platform/settings/connections/): connection statuses, re-check, and reconnect. diff --git a/src/content/docs/docs/infrastructure/hosts.mdx b/src/content/docs/docs/infrastructure/hosts.mdx index 82eaf5ba..a441d006 100644 --- a/src/content/docs/docs/infrastructure/hosts.mdx +++ b/src/content/docs/docs/infrastructure/hosts.mdx @@ -56,6 +56,8 @@ Displays interactive time-series charts for the resource. - **Hosts:** View detailed charts for CPU, Memory, Disk I/O, Network traffic, and System Load. - **Containers:** View container-specific CPU and Memory utilization charts. +![The Metrics tab of a host, with tiles for memory, CPU, file system, inode, and disk IO, and charts for memory usage, CPU utilisation, CPU idle, and average CPU load](./images/hosts-metrics-tab.png) + ### Logs Tab Provides immediate access to the logs generated by the selected resource. - You can search, filter, and analyze logs specific to that host or container directly within the drawer. @@ -70,7 +72,7 @@ Each group shows how many of its resources are alerting. Hover over a resource's ![The Topology tab of a host, listing the databases and services that run on it](./images/hosts-topology-tab.png) -Linking a host to its EC2 instance needs KloudMate agent 2.2.0 or later. If an AWS account is connected and the host runs an older agent, the tab asks you to [update the agent](../../kloudmate-agent/management/#updating-the-agent). +Linking a host to its EC2 instance needs KloudMate agent 2.2.0 or later. If an AWS account is connected and the host runs an older agent, the tab asks you to [upgrade the agent](../../kloudmate-agent/upgrade/). ## Getting Started diff --git a/src/content/docs/docs/infrastructure/images/hosts-metrics-tab.png b/src/content/docs/docs/infrastructure/images/hosts-metrics-tab.png new file mode 100644 index 00000000..0a4a35c0 Binary files /dev/null and b/src/content/docs/docs/infrastructure/images/hosts-metrics-tab.png differ diff --git a/src/content/docs/docs/infrastructure/images/hosts-topology-tab.png b/src/content/docs/docs/infrastructure/images/hosts-topology-tab.png index f113cf54..eaa292e6 100644 Binary files a/src/content/docs/docs/infrastructure/images/hosts-topology-tab.png and b/src/content/docs/docs/infrastructure/images/hosts-topology-tab.png differ diff --git a/src/content/docs/docs/infrastructure/inventory.mdx b/src/content/docs/docs/infrastructure/inventory.mdx index a915f2f3..24edea7b 100644 --- a/src/content/docs/docs/infrastructure/inventory.mdx +++ b/src/content/docs/docs/infrastructure/inventory.mdx @@ -38,11 +38,22 @@ The **All resources** tab lists each resource with its **Name**, **Type**, **Whe **Where** shows where the resource lives: - A cloud resource shows its region and account. -- A Kubernetes object shows its cluster and namespace. +- A Kubernetes object shows its cluster, and its namespace when it has one. - A database shows the address and port that clients connect to. - Hosts and services leave it empty. -Resources that the agent reports leave the list when they haven't been seen for 2 hours. Cloud resources stay until a sync finds they're gone. Inventory doesn't list containers, network endpoints, or storage mounts. +Inventory doesn't list containers, network endpoints, or storage mounts. + +## How often Inventory updates + +Inventory checks each source on a schedule, not all the time. **Last seen** shows when a check last found the resource, not when the resource last sent data. + +- **Resources the agent reports** (hosts, Kubernetes objects, services, and databases): checked once an hour, so their **Last seen** can be up to an hour old. +- **AWS resources:** synced every 30 minutes in the account's [primary regions](../../platform/settings/data-sources/#primary-regions) and about once a day in other regions. Use [Sync now](../../platform/settings/data-sources/#sync-now) to sync an account right away. + +The page doesn't refresh by itself. Reload it to see the latest values. + +Resources that the agent reports leave the list when they haven't been seen for 2 hours. A resource that stops sending data leaves about 2 to 3 hours later. Cloud resources stay until a sync finds they're gone. ## Open a resource diff --git a/src/content/docs/docs/infrastructure/kubernetes/images/kubernetes-jobs-tab.png b/src/content/docs/docs/infrastructure/kubernetes/images/kubernetes-jobs-tab.png index d0e95e47..c87de565 100644 Binary files a/src/content/docs/docs/infrastructure/kubernetes/images/kubernetes-jobs-tab.png and b/src/content/docs/docs/infrastructure/kubernetes/images/kubernetes-jobs-tab.png differ diff --git a/src/content/docs/docs/infrastructure/kubernetes/images/kubernetes-map-tab.png b/src/content/docs/docs/infrastructure/kubernetes/images/kubernetes-map-tab.png new file mode 100644 index 00000000..85b2ceef Binary files /dev/null and b/src/content/docs/docs/infrastructure/kubernetes/images/kubernetes-map-tab.png differ diff --git a/src/content/docs/docs/infrastructure/kubernetes/images/kubernetes-nodes-tab.png b/src/content/docs/docs/infrastructure/kubernetes/images/kubernetes-nodes-tab.png index c2cbbb0d..d8801ff9 100644 Binary files a/src/content/docs/docs/infrastructure/kubernetes/images/kubernetes-nodes-tab.png and b/src/content/docs/docs/infrastructure/kubernetes/images/kubernetes-nodes-tab.png differ diff --git a/src/content/docs/docs/infrastructure/kubernetes/images/kubernetes-reporting-filter.png b/src/content/docs/docs/infrastructure/kubernetes/images/kubernetes-reporting-filter.png index 8e1ed7a2..6d97d145 100644 Binary files a/src/content/docs/docs/infrastructure/kubernetes/images/kubernetes-reporting-filter.png and b/src/content/docs/docs/infrastructure/kubernetes/images/kubernetes-reporting-filter.png differ diff --git a/src/content/docs/docs/infrastructure/kubernetes/images/kubernetes-storage-tab.png b/src/content/docs/docs/infrastructure/kubernetes/images/kubernetes-storage-tab.png index 41641971..c9be9758 100644 Binary files a/src/content/docs/docs/infrastructure/kubernetes/images/kubernetes-storage-tab.png and b/src/content/docs/docs/infrastructure/kubernetes/images/kubernetes-storage-tab.png differ diff --git a/src/content/docs/docs/infrastructure/kubernetes/images/kubernetes-topology-tab.png b/src/content/docs/docs/infrastructure/kubernetes/images/kubernetes-topology-tab.png index b66c82fc..39876cfb 100644 Binary files a/src/content/docs/docs/infrastructure/kubernetes/images/kubernetes-topology-tab.png and b/src/content/docs/docs/infrastructure/kubernetes/images/kubernetes-topology-tab.png differ diff --git a/src/content/docs/docs/infrastructure/kubernetes/images/kubernetes-workloads-tab.png b/src/content/docs/docs/infrastructure/kubernetes/images/kubernetes-workloads-tab.png index 7150e07e..cec8dfb0 100644 Binary files a/src/content/docs/docs/infrastructure/kubernetes/images/kubernetes-workloads-tab.png and b/src/content/docs/docs/infrastructure/kubernetes/images/kubernetes-workloads-tab.png differ diff --git a/src/content/docs/docs/infrastructure/kubernetes/index.mdx b/src/content/docs/docs/infrastructure/kubernetes/index.mdx index a10688ab..c5f444d1 100644 --- a/src/content/docs/docs/infrastructure/kubernetes/index.mdx +++ b/src/content/docs/docs/infrastructure/kubernetes/index.mdx @@ -102,6 +102,8 @@ The map draws workloads and Services that sent data in the last 2 hours, up to 2 The lines come from network flow data that the KloudMate agent collects with eBPF, so the nodes need a kernel that supports it. See [eBPF requirements](../../kloudmate-agent/ebpf-observability/#deployment-modes-and-requirements). Without flow data, the map draws no lines and notes that the cluster sends no traffic data. +![The Map tab in Kubernetes, with one box per namespace, their workload and Service counts, alerting counts, and lines between namespaces that talk to each other](./images/kubernetes-map-tab.png) + ## See what a resource connects to Open a node, pod, Deployment, StatefulSet, or DaemonSet, and select the **Topology** tab, the last tab in the drawer. It lists what the resource connects to, grouped by relation, such as **Talks with** for network traffic and **Runs here** for what runs on a node. diff --git a/src/content/docs/docs/issues/images/issue-details.png b/src/content/docs/docs/issues/images/issue-details.png new file mode 100644 index 00000000..f77bc8a0 Binary files /dev/null and b/src/content/docs/docs/issues/images/issue-details.png differ diff --git a/src/content/docs/docs/issues/images/issues-1.png b/src/content/docs/docs/issues/images/issues-1.png deleted file mode 100644 index 932f59c2..00000000 Binary files a/src/content/docs/docs/issues/images/issues-1.png and /dev/null differ diff --git a/src/content/docs/docs/issues/images/issues-2.png b/src/content/docs/docs/issues/images/issues-2.png deleted file mode 100644 index 92b9a4fe..00000000 Binary files a/src/content/docs/docs/issues/images/issues-2.png and /dev/null differ diff --git a/src/content/docs/docs/issues/images/issues-3.png b/src/content/docs/docs/issues/images/issues-3.png deleted file mode 100644 index 6ae049de..00000000 Binary files a/src/content/docs/docs/issues/images/issues-3.png and /dev/null differ diff --git a/src/content/docs/docs/issues/images/issues-4.png b/src/content/docs/docs/issues/images/issues-4.png deleted file mode 100644 index e5a2f0ac..00000000 Binary files a/src/content/docs/docs/issues/images/issues-4.png and /dev/null differ diff --git a/src/content/docs/docs/issues/images/issues-list.png b/src/content/docs/docs/issues/images/issues-list.png new file mode 100644 index 00000000..63340b7b Binary files /dev/null and b/src/content/docs/docs/issues/images/issues-list.png differ diff --git a/src/content/docs/docs/issues/index.mdx b/src/content/docs/docs/issues/index.mdx index 26247084..db279883 100644 --- a/src/content/docs/docs/issues/index.mdx +++ b/src/content/docs/docs/issues/index.mdx @@ -1,7 +1,6 @@ --- title: Issues -description: Learn how KloudMate automatically discovers, aggregates, and helps - you resolve errors and warnings across your services. +description: Find, investigate, and resolve the errors your logs, traces, and real user monitoring report, grouped into issues. editUrl: true head: [] template: doc @@ -14,83 +13,127 @@ pagefind: true draft: false --- -KloudMate automatically discovers errors from your application logs and traces and aggregates them in the Issues section. This gives you a centralised view of all errors and warnings detected across your services, so you can track, investigate, and resolve them efficiently. +KloudMate groups the errors in your logs, traces, and real user monitoring (RUM) into issues. Each issue is one kind of error, and every time that error happens again, it adds an event to the issue. Use the Issues list to see what's failing right now, then open an issue to find out who it hit, what led up to it, and where in your code it was thrown. -To navigate to the Issues screen, click **Issues** in the left navigation menu. +To open the list, click **Issues** in the left navigation menu. -## Issues Dashboard +## Find the issues that need attention -The **Issues** screen displays all detected issues in a list. +![Issue list filtered to RUM issues, with the Trend, Events, Last seen, and Assigned columns](./images/issues-list.png) -![Issues dashboard showing the list of all unresolved issues](./images/issues-1.png) +The figures in the list cover different time ranges. The event count is the total since the issue was first seen, while the trend and the count under it cover the last 24 hours. For example, an issue with 5,000 events and none in the last 24 hours has probably stopped happening. Sort by **Last seen** to find what's failing now. -Each issue shows the source, last seen time, occurrence count, and assigned user. +For a RUM issue, the users and sessions cover the last 14 days. The release shown is the latest event's release, which tells you whether the error still happens in your most recent deploy. -Use the **Assignee**, **Service**, **Source**, and **Error type** filters to narrow down the list. +An issue is marked **New** for 24 hours after it was first seen. -Issues are organised across three tabs — **All Unresolved**, **Ignored**, and **Resolved** — so you can focus on what's active or review previously handled issues. +Search matches both the error message and the error type. For example, searching for `TypeError` finds every issue of that type, whatever its message. -Click **Notification settings** at the top right to configure when you receive notifications for new or recurring issues. +To resolve or ignore several issues at once, select them in the list and choose the action. A selection covers only the current page, and changing the tab, a filter, the sort order, or the page clears it. -Click **Explore Error Patterns** at the top right to identify recurring error patterns across your services. +## Investigate an issue -## Issue Details +![RUM issue details: tags, summary tiles, actions, context bar, event navigator, stack trace, Tags, and Affected users](./images/issue-details.png) -Click on any issue to open its detail view. +Log, trace, and RUM issues all show how often the error happens and its stack trace. The rest of a RUM issue comes from what the RUM SDKs report with each error: the user, the session, and what happened before it. Users and sessions, **Before the error**, **Tags**, and **Affected users** appear on RUM issues only. -![Issue detail page showing error message, metadata, occurrence chart, stacktrace, and infrastructure context](./images/issues-2.png) +For a structured log, the title is the readable message from the log record. -The detail page shows: +### What the numbers count -- The full **error message** at the top -- **Service**, **source** (Logs or Traces), **first seen**, **last seen**, and total **occurrence count** -- An **Occurrences over time** chart showing how frequently the issue has occurred across the selected time range -- A **Stacktrace** section with the error type and the full call stack for pinpointing the origin of the error -- An **Infrastructure** section showing the service name and Kubernetes context — cluster, container, deployment, namespace, node, pod, and replicaset details +The figures at the top of an issue cover the last 14 days. The exceptions are the total event count and **First seen**, which go back to the first event. The trend counts events per UTC day, so a day's bar may not line up with your local calendar day. -### Related Errors +**Users** counts people, not sessions. Each signed-in user counts once, and so does each browser whose visitor never signed in. Anonymous browsers are counted for events from browser SDK 2.7.0 and later. For events from older versions, only users identified with `setUser()` are counted. -The **Related Errors** section at the bottom lists individual occurrences of the issue with timestamps, service names, and direct **Show Logs** links for deeper investigation into each occurrence. +On a RUM issue, **First seen** and **Last seen** also show the release of the first and latest events. The first event's release is shown only when that event happened in the last 14 days. -![Infrastructure context and Related Errors table with timestamps and Show Logs links](./images/issues-3.png) +### Handled and unhandled errors -### Actions +A RUM issue is marked **Handled** or **Unhandled** for the event you're viewing: -From the issue detail page you can take the following actions using the action bar: +- **Handled** means your app caught the error and reported it, with `recordException()` or `console.error()`. +- **Unhandled** means nothing in your app caught it, such as an uncaught exception or an unhandled promise rejection. -![Action bar showing Mark as resolved, Ignore, Ask Assistant, Create in Jira, and Comments](./images/issues-4.png) +The flag needs browser SDK 2.8.0 or later. Events from older versions aren't marked either way. -- **Mark as resolved** — removes the issue from the Unresolved list. -- **Ignore** — mutes future notifications for this issue if it reoccurs. -- **Ask Assistant** — automatically analyses the issue and provides a concise summary, probable root cause, and recommended actions for faster resolution. -- **Create in Jira** — creates a Jira ticket directly from the issue for tracking in your existing workflow. -- **Assign** — assign the issue to one or more team members using the assign icon on the right. Assigned users are notified when an issue is assigned to them. -- **Comments** — add comments using the comment icon on the right. You can mention other users in comments, and all participating users receive notifications for any updates on the issue. +### One event at a time -You can also mark issues as resolved or ignored in bulk by selecting multiple issues from the main list and applying the action collectively. +The context, stack trace, and **Before the error** on a RUM issue all describe a single event. The page opens on the latest event and moves to newer ones as they arrive. Use **Oldest**, **Newest**, and the arrows to look at any other event from the last 14 days. After you step back to an older event, the page stays on that event. -## Issue Statuses +A log or trace issue shows its latest event from the last 6 days. -- **Unresolved** — an issue that has been detected and is yet to be reviewed or addressed. -- **Resolved** — an issue that has been reviewed and marked as resolved. It will no longer appear in the Unresolved list. -- **Ignored** — an issue that has been intentionally muted. No notifications will be sent if the same issue reoccurs in the future. +### Read the stack trace -## Notification Preferences +For a browser issue, each frame is mapped through the [source maps](../rum/source-maps/) you uploaded, so you see your original file, line, and code instead of the minified bundle. Switch to **Raw** to see the stack as the browser reported it. -Click **Notification settings** on the Issues dashboard to configure when you receive alerts. You can choose to be notified for: +A frame that couldn't be mapped carries a badge with the reason, such as **no source map** or **ambiguous release**. [Troubleshoot unmapped frames](../rum/source-maps/#troubleshoot-unmapped-frames) explains each badge and how to fix it. -- **New Issues** — issues detected for the first time. -- **New and Regression Issues** — issues detected for the first time, as well as issues that had previously occurred and have reappeared. +For a log or trace issue, the stack is shown as it was reported. -## User Permissions +### See what led up to the error -Access to issue actions is determined by the user's assigned role: +**Before the error** lists what happened in the session during the 60 seconds before a RUM error: page loads and route changes, clicks, requests with their status and duration, console warnings and errors, custom events sent with `addEvent()`, long tasks, and earlier errors. For example, a `POST /api/cart` that returned `500` two seconds before a `TypeError` usually explains the error. -- **Owner and Developer** — can mark issues as resolved or ignored, assign issues to team members, create Jira tickets, and leave comments. -- **Viewer** — can assign issues to team members and leave comments, but cannot mark issues as resolved or ignored. +The trail needs browser SDK 2.7.0 or later. The SDK keeps personal data out of it: + +- URLs are recorded without their query string or fragment. +- A click records the label of the button or link, such as `click "Pay now"`, never the content of a form field. +- A click inside an element with the `km-mask` class, or with the session recorder's block or ignore class (`km-block` and `km-ignore` by default), records no text. The same applies to a control that contains one of these elements. + +To keep click labels out of the trail, set `captureClickText` to `false`. To turn the trail off, set `breadcrumbs.enabled` to `false`: + +```javascript +KloudMateRum.init({ + endpoint: 'https://otel.kloudmate.com:4318', + rumAccessToken: 'YOUR_PUBLIC_API_KEY', + applicationName: 'my-app', + breadcrumbs: { captureClickText: false }, +}); +``` + +### Find out who was affected + +**Tags** breaks down a RUM issue's events from the last 14 days by release, browser, OS, device, page, country, and environment. A value near 100% narrows down the cause. For example, `Safari 100%` under **Browser** points at a Safari-only bug, and a single release points at that deploy. + +**Affected users** lists the 10 sessions the issue hit most recently. A visitor who wasn't signed in is listed as **Anonymous** with the start of the browser's visitor ID, so you can tell two anonymous visitors apart. If the session was recorded, its replay starts 5 seconds before the error. See [Record sessions for replay](../rum/instrumentation-guide/web/#record-sessions-for-replay) to turn recording on. + +The event list holds up to 100 of the latest events: from the last 14 days for a RUM issue, or from the last 6 days for a log or trace issue. + +## Resolve, ignore, or assign an issue + +- **Mark as resolved** moves the issue to **Resolved**. If the error happens again, the issue reopens. +- **Ignore** stops notifications for the issue and moves it to **Ignored**, out of the unresolved list. Click it again to stop ignoring the issue. +- **Ask Assistant** opens the [KloudMate Assistant](../kloudmate-assistant/) with a prompt about the issue already written. Nothing is sent until you send it. +- **Create in Jira** needs a [Jira integration](../platform/settings/notification-channels/#jira). +- Team members are notified when you assign an issue to them. Everyone who comments on an issue is notified of later comments, and you can mention other users in a comment. + +## How long issues are kept + +An issue is deleted after 15 days without an event, whether it's unresolved, resolved, or ignored. Its comments and assignments are deleted with it. + +If the error happens again after that, it starts a new issue. The new issue is unresolved, marked **New**, and triggers a new-issue notification, even if you had ignored the old one. + +## Notification preferences + +Click **Notification settings** on the Issues list to choose when you're notified: + +- **New Issues**: issues detected for the first time. +- **New and Regression Issues**: issues detected for the first time, as well as issues that had previously occurred and have reappeared. + +To send these notifications to specific channels for specific services, see [Notification routing](./notification-routing/). + +Click **Explore Error Patterns** to find recurring error patterns across your services. + +## User permissions + +- **Owner and Developer**: resolve and ignore issues, one at a time or in bulk, assign issues, create Jira tickets, and comment. +- **Viewer**: assign issues and comment, but not resolve or ignore them. ## Related Resources - [Notification routing](./notification-routing/) +- [Source maps](../rum/source-maps/) +- [Session detail](../rum/session-detail/) +- [Browser SDK reference](../rum/instrumentation-guide/sdk-reference/) - [Routing Rules](../alerts/routing-rules/) - [Notification Channels](../platform/settings/notification-channels/) diff --git a/src/content/docs/docs/kloudmate-agent/auto-instrumentation/gitops-clusters.mdx b/src/content/docs/docs/kloudmate-agent/auto-instrumentation/gitops-clusters.mdx index fef65878..cefeaf51 100644 --- a/src/content/docs/docs/kloudmate-agent/auto-instrumentation/gitops-clusters.mdx +++ b/src/content/docs/docs/kloudmate-agent/auto-instrumentation/gitops-clusters.mdx @@ -34,7 +34,7 @@ Annotation tracking is the default from Argo CD 3.0. On an older version, set `a ## Check whether a workload is externally managed -Go to **Settings → Agents**, open the cluster agent's actions menu (⋮), and select **Configure**. On the **APM** tab, the **Application instrumentation** card shows **Managed by Argo CD** or **Managed by Flux** on each workload that one of them manages. Other workloads work as usual, even on a cluster that runs Argo CD. +Go to **Settings → Agents**, open the cluster agent's actions menu (⋮), and select **Configure**. On the **APM** tab, the **Application instrumentation** card shows **Managed by Argo CD** or **Managed by Flux** on a managed workload once you set it to **eBPF** or **SDK** and the agent checks in. A workload left at **Off** doesn't carry the label, but a banner at the top of the card tells you when Argo CD runs in the cluster. Other workloads work as usual, even on a cluster that runs Argo CD. ## Turn on eBPF mode @@ -71,6 +71,7 @@ Keep these limits in mind: - **Every pod in the namespace gets injected**, including short-lived jobs. - **One runtime per namespace.** The annotation sets a single language. If a namespace holds a Java service and a Python service, this covers only one of them. - **.NET runtime options come from the pod only.** If a .NET workload needs one, put it on that workload's pod template. +- **The workloads show Off in Application instrumentation.** The agent didn't add the annotation, so it doesn't track these workloads, but their traces still reach KloudMate. ## Commit the annotation @@ -132,6 +133,16 @@ helm upgrade kloudmate-release kloudmate/km-kube-agent \ --set patchExternallyManagedWorkloads=true ``` +If Argo CD installs the agent, as in [Install the agent on a GitOps cluster](#install-the-agent-on-a-gitops-cluster), set the value in the agent's Application instead. Argo CD reverts a `helm upgrade` to the values in the Application: + +```yaml +spec: + source: + helm: + valuesObject: + patchExternallyManagedWorkloads: true +``` + This value applies to the whole cluster. If an Application doesn't have the settings above, Argo CD reverts the annotation and the Application's pods keep restarting. With the value on, workloads that Argo CD manages no longer show **Managed by Argo CD** in **Application instrumentation**. They show their usual instrumentation status instead. These steps are for Argo CD. On Flux, exclude the same annotations from drift correction before you turn on `patchExternallyManagedWorkloads`, or Flux reverts the agent's changes. @@ -162,7 +173,17 @@ spec: instrumentation.opentelemetry.io/inject-java: km-agent/km-agent-instrumentation-crd ``` -Argo CD and Flux compare Deployments, not the pods those Deployments create, so a policy that annotates pods never shows up as drift. The OpenTelemetry documentation suggests the same pattern, and KloudMate needs no configuration for it. As with the namespace annotation, running pods get the annotation when they're recreated. +Argo CD and Flux compare Deployments, not the pods those Deployments create, so a policy that annotates pods never shows up as drift. The OpenTelemetry documentation suggests the same pattern. As with the namespace annotation, running pods get the annotation when they're recreated, and the workloads show **Off** in **Application instrumentation** while their traces still reach KloudMate. + +The OpenTelemetry Operator injects the SDK from its own admission webhook, which Kubernetes calls once per pod. That webhook has to run after Kyverno's, or it never sees the annotation. km-kube-agent charts released after 2.5.1 order the webhooks this way. On an older chart, set the order yourself: + +```bash +helm upgrade kloudmate-release kloudmate/km-kube-agent \ + --namespace km-agent --reuse-values \ + --set opentelemetry-operator.admissionWebhooks.namePrefix=zz +``` + +Without it, pods get the annotation but no SDK, and nothing reports an error. ## Install the agent on a GitOps cluster @@ -170,7 +191,7 @@ Install the agent with Helm, as shown in the [Kubernetes installation guide](../ To manage the agent through Argo CD instead, set these on its Application: -- **`ignoreDifferences` on the agent's own ConfigMaps.** The agent writes each collector's live configuration into the `/data/agent-daemonset.yaml` and `/data/agent-deployment.yaml` keys of `km-agent-configmap-daemonset` and `km-agent-configmap-deployment`. Argo CD renders the chart without reading the cluster, so without this setting, every sync resets the collectors to the configuration in the chart. +- **`ignoreDifferences` on the agent's own ConfigMaps, with `RespectIgnoreDifferences=true`.** The agent writes each collector's live configuration into the `/data/agent-daemonset.yaml` and `/data/agent-deployment.yaml` keys of `km-agent-configmap-daemonset` and `km-agent-configmap-deployment`. Argo CD renders the chart without reading the cluster, so it sees that configuration as drift. `ignoreDifferences` stops Argo CD from reporting it, and `RespectIgnoreDifferences=true` stops a sync from writing the chart's configuration over it. Without the sync option, any change to the Application, such as a new chart version or a Helm value, resets the collectors and drops the cluster's eBPF and SDK selections until the next **Apply** in KloudMate. - **The destination namespace set to `km-agent`.** The chart always creates the Instrumentation resource in `km-agent`, and the inject annotation points to it as `km-agent/km-agent-instrumentation-crd`. - **`SkipDryRunOnMissingResource=true`** in the sync options. On the first sync, Argo CD applies the Instrumentation resource in the same wave that installs the OpenTelemetry Operator's CRDs, and the dry run fails without this option. @@ -181,6 +202,7 @@ spec: syncPolicy: syncOptions: - SkipDryRunOnMissingResource=true + - RespectIgnoreDifferences=true ignoreDifferences: - group: "" kind: ConfigMap diff --git a/src/content/docs/docs/kloudmate-agent/database-monitoring/credentials.mdx b/src/content/docs/docs/kloudmate-agent/database-monitoring/credentials.mdx index c84c81d6..b82be59a 100644 --- a/src/content/docs/docs/kloudmate-agent/database-monitoring/credentials.mdx +++ b/src/content/docs/docs/kloudmate-agent/database-monitoring/credentials.mdx @@ -1,49 +1,75 @@ --- title: "Database credentials" -description: "The three credential modes for KloudMate agent database monitoring: environment reference, plaintext inline, and operator-managed secrets, and how to rotate them." +description: "Provide a database monitoring user's password to the KloudMate agent with an environment variable or an entered value, and rotate it safely." sidebar: order: 5 --- import { LinkCard, CardGrid } from '@astrojs/starlight/components'; -Database monitoring needs a monitoring user's credentials to read from the database. The agent supports three ways to provide them, so you can keep secrets out of your configuration. The automatic eBPF view of your queries needs no credentials at all. For the general flow, see the [Database monitoring overview](../overview/). +Database monitoring connects as a monitoring user, so the agent needs that user's password. You can keep the password on the host and out of KloudMate, or type it into the wizard. Monitoring from eBPF and application traces needs no credentials. For the general flow, see the [Database monitoring overview](../overview/). -## The three credential modes +## Choose how to provide the password -### Environment reference (default) +The **Password** field in the wizard has two modes. MongoDB Atlas uses the same two modes for its **Atlas private key**. -The credential is stored as a reference, not a value. The agent reads it from an environment variable on the host at runtime, so the plaintext value never reaches KloudMate or the stored configuration. This is the default and the recommended mode. +### Environment variable (default) -The agent also handles escaping for you, so a password with special characters works in a connection string without you having to encode it. +The agent reads the password from an environment variable on its host. KloudMate stores only the variable's name, so the password never leaves the host. Use this mode in production. -### Plaintext inline +The field shows the variable to create. The agent handles escaping, so passwords containing `@`, `:`, `/`, `?`, `#`, or spaces work without encoding. -You can enter the value directly. It is then stored in the configuration. The interface shows a clear "stored unencrypted" warning, because the value lives in the configuration and on disk. Use this only when you accept that trade-off. +### Enter value -### Operator-managed +You type the password into the wizard, and it's written into the agent's configuration in plain text. Use this only for testing. -The credential is still an environment reference, but you supply the environment variable yourself rather than through the wizard. This suits secrets that come from an external secret manager. How you supply the variable depends on the platform: +When you edit a database with an entered password, the field shows it masked. Leave it empty to keep the stored password, or type a new one to replace it. A stored password can be replaced but not cleared, so switch to **Environment variable** to remove it. -- **On a virtual machine:** set `KM_SECRET_` for the agent, for example through the systemd `EnvironmentFile`. -- **On Kubernetes:** put `KM_SECRET_` in a Kubernetes Secret and point the Helm release at it with the `dbMonitoring.secretName` value. See [Database monitoring on Kubernetes](../on-kubernetes/). +## How variable names are built -## Rotation +The variable name is `KM_SECRET_`, then the instance name or, without one, the engine, then the field name. It's uppercased, and any character other than a letter, digit, or underscore becomes `_`. -The reference modes store only a reference. So you rotate a database password by updating the environment variable or Kubernetes Secret on the host. The agent's configuration does not change. +| Instance name | Engine | Variable | +|---|---|---| +| (none) | PostgreSQL | `KM_SECRET_POSTGRESQL_PASSWORD` | +| `Orders DB` | PostgreSQL | `KM_SECRET_ORDERS_DB_PASSWORD` | +| (none) | MongoDB Atlas | `KM_SECRET_MONGODBATLAS_PRIVATE_KEY` | -:::note -On Kubernetes, the collector reads the Secret when its pod starts, so a rotated password takes effect on the next pod restart. See the ordering note in [Database monitoring on Kubernetes](../on-kubernetes/). +Set the **Instance name** before you create the variable, because it changes the name. Once a database is saved, renaming it keeps the variable it already uses. + +:::caution +Two databases on one agent that produce the same variable name share one password, and the wizard warns you when that happens. This happens with two unnamed databases of the same engine, or names such as `DB 1` and `DB-1`. Give each database a distinct instance name to keep their passwords separate. ::: +Databases saved before this naming scheme may use an older variable such as `KM_SECRET_postgresql`. It keeps working. + +## Set the variable on a host + +The agent must see the variable in its own environment, not only in your shell. + +- **Linux (systemd):** add `KM_SECRET_POSTGRESQL_PASSWORD=YOUR_PASSWORD` to an `EnvironmentFile` referenced by the agent's service, or to a drop-in with `Environment=`, then restart the agent. This also suits a password written by an external secret manager. +- **Windows:** set it as a system environment variable, then restart the KloudMate Agent service. +- **Docker:** pass it to the agent container with `-e` or an env file. +- **Kubernetes:** put it in a Kubernetes Secret and set `dbMonitoring.secretName`. See [Database monitoring on Kubernetes](../on-kubernetes/#provide-credentials-with-a-kubernetes-secret). + +## Rotate a password + +With **Environment variable**, change the password on the database, update the variable, and restart the agent, or the collector pod on Kubernetes. The configuration in KloudMate doesn't change. + +With **Enter value**, change the password on the database, then edit the database in the wizard and type the new password. + +## Passwords in Advanced overrides + +The **Advanced** YAML override has no secret storage. Anything you type there, including a password, is written into the agent configuration in plain text. Keep credentials in the **Password** field. + ## Grants -Whichever mode you use, the monitoring user needs read access to the engine's statistics. Some coverages need specific grants too, for example `GRANT pg_monitor` on PostgreSQL. The wizard shows the exact setup steps for the coverages you selected. See [Engines and coverages](../engines-and-coverages/). +Whichever mode you use, the monitoring user needs the grants for the coverages you turn on. The wizard shows them under **Setup required**, and each engine's page lists them. The agent doesn't test the password or grants when you save. A mistake shows up as errors in the agent log and empty tabs in the Databases module. See [Engines and coverages](../engines-and-coverages/). ## Next steps - + diff --git a/src/content/docs/docs/kloudmate-agent/database-monitoring/engines-and-coverages.mdx b/src/content/docs/docs/kloudmate-agent/database-monitoring/engines-and-coverages.mdx index bf7eea01..abc1f7a2 100644 --- a/src/content/docs/docs/kloudmate-agent/database-monitoring/engines-and-coverages.mdx +++ b/src/content/docs/docs/kloudmate-agent/database-monitoring/engines-and-coverages.mdx @@ -1,53 +1,96 @@ --- title: "Engines and coverages" -description: "The database engines the KloudMate agent monitors and the metrics, query performance, locks, and logs available for each engine." +description: "Every database engine the KloudMate agent monitors, the coverages you can turn on for each, and what each coverage needs on the database." sidebar: order: 2 --- import { LinkCard, CardGrid } from '@astrojs/starlight/components'; -You choose what to monitor for a database as a set of **coverages**. This page lists the engines the agent monitors and the coverages available for each. +A **coverage** is one area of data the agent collects from a database, such as standard metrics, query performance, or a schema catalog. You pick coverages per database in the wizard. This page lists the coverages for each engine, what they collect, what they need, and how to set up the database for them. -## Coverage types +## How coverages work -- **Standard metrics:** the database's health and capacity metrics, such as connections, cache hit ratio, replication lag, buffers, and table and index statistics. You can collect a basic set or a full set. -- **Query performance:** statistics for the most expensive queries across all clients, so you can find slow and heavy queries. -- **Locks:** lock and blocking information. -- **Query logs:** query activity captured as log records. -- **Server and slow logs:** the database's own log entries, including slow queries. +- **Standard metrics** is always on. Check **Collect all optional metrics** to collect the receiver's full metric set. +- **Every other coverage is off** until you turn it on. +- **Setup steps** appear in the wizard when you turn on a coverage that needs them, such as a grant or an extension. The wizard doesn't test them when you save, so run them first. A missing grant shows up as an empty tab in the Databases module and as errors in the agent log. +- **Passwords** go in the wizard's **Password** field, or in the environment variable it names. See [Database credentials](../credentials/). +- **Log file coverages** read files on the database host. They're available on a virtual machine and hidden on Kubernetes. +- **The coverage list comes from the agent**, so an older agent offers fewer coverages. -## Engine and coverage matrix +## Engine support at a glance -| Engine | Standard metrics | Query performance | Locks | Query logs | Logs | -|---|---|---|---|---|---| -| PostgreSQL | Yes | Yes | Yes | No | Server logs | -| MySQL | Yes | Yes | No | No | Slow log | -| SQL Server | Yes | Yes | No | Yes | No | -| Oracle | Yes | Yes | No | Yes | No | -| MongoDB | Yes | No | No | No | No | -| MongoDB Atlas | Yes | No | No | No | No | -| Redis | Yes | No | No | No | No | -| Elasticsearch | Yes | No | No | No | No | -| ClickHouse | Yes | No | No | No | No | -| SAP HANA | Yes | No | No | No | No | -| Snowflake | Yes | Yes | No | No | No | +| Engine | Metrics | Query performance | Schema | Topology | Logs | Also available | +|---|---|---|---|---|---|---| +| [PostgreSQL](#postgresql) | Yes | Yes | Yes | Yes | Server logs | Locks, vacuum, bloat, replication health | +| [MySQL and MariaDB](#mysql-and-mariadb) | Yes | Yes | Yes | Yes | Slow query log | Locks | +| [SQL Server](#sql-server) | Yes | Yes | Yes | Availability groups | Query log | Agent jobs, backups, index health, memory, sessions, TempDB, waits | +| [Oracle](#oracle) | Yes | Yes | Yes | CDB/PDB, RAC, Data Guard | Query log | AWR, RMAN backups, locks | +| [MongoDB](#mongodb) | Yes | Yes | — | — | — | — | +| [MongoDB Atlas](#mongodb-atlas) | Yes | — | — | — | Host, audit, and access logs | — | +| [Redis](#redis) | Yes | — | — | — | — | — | +| [ClickHouse](#clickhouse) | Yes | — | — | Yes | Query log | Instance info and core metrics | +| [Elasticsearch](#elasticsearch) | Yes | — | — | — | — | — | +| [SAP HANA](#sap-hana) | Yes | — | — | — | — | — | +| [Snowflake](#snowflake) | Yes | — | — | — | Top queries | — | -:::note -Database log coverages are available on a virtual machine. On Kubernetes, the agent collects database metrics and query coverages. See [Database monitoring on Kubernetes](../on-kubernetes/). -::: +## PostgreSQL + +Every coverage connects as a monitoring user with the `pg_monitor` role. + +| Coverage | Collects | Needs | +|---|---|---| +| **Standard metrics** | Connections, cache hit ratio, replication lag, WAL and background writer, buffers, table and index statistics | `pg_monitor` | +| **Instance info** | Version, role, and key settings | — | +| **Query performance (samples + top queries)** | Top queries from `pg_stat_statements`, live samples from `pg_stat_activity`, and optionally execution plans | `pg_stat_statements` in `shared_preload_libraries` (restart), then `CREATE EXTENSION` in each monitored database. Plans also need `pg_read_all_data` and an **Advanced** override. | +| **Locks & blocking** | Lock counts by mode and blocked sessions | — | +| **Schema catalog** | Tables, columns, indexes, and constraints in the schemas you list | — | +| **Replication topology & lag** | Primary or standby role, standby roster, lag and backlog per link | Turn on for the primary | +| **Replication health (slots, WAL retention)** | Replication slot state and retained WAL | — | +| **Vacuum health & wraparound** | Wraparound headroom and per-table vacuum recency | — | +| **Table bloat & vacuum blockers** | Dead tuples per table and sessions blocking vacuum | — | +| **Server logs (JSON, PostgreSQL 15+)** | Slow queries and errors from `jsonlog` files | `log_destination = 'jsonlog'`. VM only. | +| **Logs (stderr, PostgreSQL <15)** | The same, from stderr log files | `log_destination = 'stderr'`. VM only. | + +Only one of the two log coverages can be on. **Schema catalog**, **Table bloat**, and per-table vacuum stats cover the database set in the **Database** field. + +### Set up the database + +#### Create a monitoring user + +Every PostgreSQL coverage connects as a monitoring user with the built-in `pg_monitor` role: + +```sql +CREATE USER kmmonitor WITH PASSWORD 'YOUR_PASSWORD'; +GRANT pg_monitor TO kmmonitor; +``` + +#### Enable pg_stat_statements + +**Query performance** reads `pg_stat_statements`: + +1. Add `pg_stat_statements` to `shared_preload_libraries` in `postgresql.conf`. This needs a PostgreSQL restart. +2. In each database you monitor, create the extension: -## Setup steps + ```sql + CREATE EXTENSION pg_stat_statements; + ``` -Some coverages need a one-time setup step on the database. The wizard shows you the step when you select that coverage. For example: +`pg_stat_statements.max` (default `5000`) caps how many distinct statements PostgreSQL tracks. If your workload has many distinct statements, raise it. Evicted statements show up as gaps in **Top queries**. -- PostgreSQL query performance needs the `pg_stat_statements` extension enabled, which needs a PostgreSQL restart. -- Query performance and query-log coverages need the monitoring user to have the right grants, such as `GRANT pg_monitor` on PostgreSQL or the equivalent view permissions on SQL Server and Oracle. -- Log coverages may need a log setting turned on, such as the MySQL slow query log. +#### Enable server logs -If a prerequisite is missing, the agent reports it and the other coverages keep working. +For **Server logs (JSON, PostgreSQL 15+)**, set these in `postgresql.conf`, then run `SELECT pg_reload_conf();`: -### Keep a `postgres` database on managed PostgreSQL +```ini +logging_collector = on +log_destination = 'jsonlog' +log_min_duration_statement = 250 +``` + +For **Logs (stderr, PostgreSQL <15)**, use `log_destination = 'stderr'` instead. Any `log_line_prefix` works. In both cases, set **Log file glob** in the wizard to where PostgreSQL writes its logs. `log_min_duration_statement` is the slow-query threshold in milliseconds. + +#### Keep a `postgres` database on managed PostgreSQL The agent connects to the database named `postgres` to read server-wide statistics and query performance. These statistics cover every database on the server, so the data you see still comes from your application databases. @@ -64,14 +107,506 @@ GRANT CONNECT ON DATABASE postgres TO kmmonitor; Replace `kmmonitor` with your monitoring user. KloudMate doesn't write anything to this database. You don't need to restart the agent; data appears on the next collection. -## Filtering database logs +## MySQL and MariaDB + +MariaDB uses the MySQL coverages. + +| Coverage | Collects | Needs | +|---|---|---| +| **Standard metrics** | Buffer pool, commands, handlers, InnoDB, connections, replication | `PROCESS`, `REPLICATION CLIENT`, `SELECT` | +| **Instance info** | Version, flavor, read-only state, binlog, key settings, and `max_connections` | — | +| **Query performance (samples + top queries)** | Samples and top queries from `performance_schema`, and optionally execution plans | `performance_schema=ON`, larger digest and SQL text lengths, `SELECT` on `performance_schema` | +| **Locks & blocking** | Lock counts and blocked transactions | `performance_schema=ON`, `SELECT` on `performance_schema` | +| **Schema catalog** | Tables, columns, indexes, and constraints | **Schemas (databases) to scan** is required | +| **Replication topology & lag** | Source or replica role, Group Replication roster, applier lag | `performance_schema=ON`, `REPLICATION CLIENT` | +| **Slow query log** | Slow queries with query time, lock time, and rows | `slow_query_log=ON`. VM only. | + +### Set up the database + +#### Create a monitoring user + +Every MySQL coverage connects as one monitoring user: + +```sql +CREATE USER 'kmmonitor'@'%' IDENTIFIED BY 'YOUR_PASSWORD'; +GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'kmmonitor'@'%'; +``` + +:::caution[Use one host part for the user] +A MySQL account is a user name plus a host part, and a `GRANT` to an account that doesn't exist fails on MySQL 8. The wizard's steps create the user as `'kmmonitor'@'localhost'` but grant later privileges to `'kmmonitor'@'%'`. Pick one host part and use it in every statement: `'%'` when the agent connects over the network, as in the examples on this page. +::: + +#### Enable performance_schema + +**Query performance**, **Locks & blocking**, and **Replication topology & lag** read `performance_schema`. **Standard metrics** also reads connection counts and uptime from it. + +1. Add these settings under `[mysqld]` in `my.cnf`, then restart MySQL. None of them can be changed at runtime. + + ```ini + [mysqld] + performance_schema = ON + performance_schema_max_digest_length = 4096 + performance_schema_max_sql_text_length = 4096 + ``` + +2. Give the monitoring user read access to it: + + ```sql + GRANT SELECT ON performance_schema.* TO 'kmmonitor'@'%'; + ``` + +The two length settings stop long statements from being cut off in **Top queries** and **Running queries**. + +:::note[MariaDB] +MariaDB ships with `performance_schema` off. Until you turn it on, the query tabs stay empty and connection counts and uptime are missing, even though the instance still shows as available. +::: + +#### Turn on the slow query log + +For **Slow query log**, turn on the log and find where MySQL writes it: + +```sql +SET GLOBAL slow_query_log = ON; +SET GLOBAL long_query_time = 1; +SHOW VARIABLES LIKE 'slow_query_log_file'; +``` + +`long_query_time` is the slow-query threshold in seconds. `SET GLOBAL` lasts until the next restart, so add the same settings to `my.cnf` to keep them. In the wizard, make sure **Slow log path** matches the file the last statement returns. The default glob, `/var/{log,lib}/mysql/*slow*.log`, covers the usual locations. + +## SQL Server + +Most coverages need `VIEW SERVER STATE` on the monitoring login. + +| Coverage | Collects | Needs | +|---|---|---| +| **Standard metrics** | Connections, batches, buffer cache, page life expectancy, locks, transactions, wait stats | `VIEW SERVER STATE` | +| **Instance info** | Version, edition, Always On state, CPUs, memory settings | `VIEW SERVER STATE` | +| **Query performance (samples + top queries)** | Samples and top queries from the DMVs, and optionally XML showplans | `VIEW SERVER STATE` (or `VIEW SERVER PERFORMANCE STATE` on 2022+), `VIEW ANY DATABASE` | +| **Query log (active + top queries)** | Active queries and the most CPU-expensive statements, as log records | `VIEW SERVER STATE` | +| **Schema catalog** | Tables, columns, indexes, and foreign keys in the connected database | — | +| **Replication topology & lag** | Availability group roles, replica roster, commit lag, send and redo backlog | `VIEW SERVER STATE`, `VIEW ANY DEFINITION` | +| **Agent jobs** | SQL Server Agent job runs, outcomes, and messages | A user in `msdb` with `SELECT` on `sysjobs`, `sysjobhistory`, and `sysjobactivity`, and `EXECUTE` on `agent_datetime`. Not on Azure SQL Database. | +| **Backup tracking** | Time since each database's last full, differential, and log backup | `SELECT` on `msdb.dbo.backupset` and `backupmediafamily` | +| **Index health** | Missing index suggestions, index usage, fragmentation | `VIEW SERVER STATE` | +| **Memory & plan cache** | Memory clerks, plan cache by type, memory grants | `VIEW SERVER STATE` | +| **Session breakdown** | Sessions by login, application, and status; request states | `VIEW SERVER STATE` | +| **TempDB usage** | TempDB space by consumer and file | `VIEW SERVER STATE` | +| **Waits, scheduler & file I/O** | Top wait types, scheduler queues, per-file I/O latency | `VIEW SERVER STATE` | + +### Set up the database + +The wizard's setup steps don't include creating the login, so start there. + +#### Create a monitoring login + +Every SQL Server coverage connects as a monitoring login. Create it and give it `VIEW SERVER STATE`, which Standard metrics already needs to read connections and uptime even though the wizard only lists the grant for other coverages: + +```sql +CREATE LOGIN kmmonitor WITH PASSWORD = 'YOUR_PASSWORD'; +GRANT VIEW SERVER STATE TO kmmonitor; +``` + +`VIEW SERVER STATE` also covers **Instance info**, **Waits, scheduler & file I/O**, **Session breakdown**, **Memory & plan cache**, **TempDB usage**, **Index health**, and **Query log**. + +#### Query performance + +**Query performance** also needs to see every database: + +```sql +GRANT VIEW ANY DATABASE TO kmmonitor; +``` + +On SQL Server 2022 and later, the wizard offers `VIEW SERVER PERFORMANCE STATE` as the narrower alternative to `VIEW SERVER STATE` for this coverage. + +#### Grant access to msdb + +**Agent jobs** and **Backup tracking** read tables in `msdb`, which needs a database user for the login there. Create the user once, then add the grants for the coverages you use: + +```sql +USE msdb; +CREATE USER kmmonitor FOR LOGIN kmmonitor; + +-- Agent jobs +GRANT SELECT ON dbo.sysjobs TO kmmonitor; +GRANT SELECT ON dbo.sysjobhistory TO kmmonitor; +GRANT SELECT ON dbo.sysjobactivity TO kmmonitor; +GRANT EXECUTE ON dbo.agent_datetime TO kmmonitor; + +-- Backup tracking +GRANT SELECT ON dbo.backupset TO kmmonitor; +GRANT SELECT ON dbo.backupmediafamily TO kmmonitor; +``` + +#### Replication topology + +**Replication topology & lag** reads the `sys.dm_hadr_*` DMVs and the availability group catalog views: + +```sql +GRANT VIEW ANY DEFINITION TO kmmonitor; +``` + +Without `VIEW ANY DEFINITION`, SQL Server returns no rows from the catalog views rather than an error, so the group looks like it has no members. Logins aren't synchronized across replicas, so create the login and grants on each replica you monitor. + +#### Schema catalog + +**Schema catalog** scans the database named in the wizard's **Database** field, which defaults to `master`. Set it to your application database, and set **Schemas to scan** (default `dbo`). The monitoring login needs a user in that database and permission to see its table definitions, for example through `VIEW ANY DEFINITION`. + +## Oracle + +Every coverage connects with the **Service name** you enter, which selects a PDB or the CDB root. Monitor the root and each PDB as separate databases with distinct instance names. + +| Coverage | Collects | Needs | +|---|---|---| +| **Standard metrics** | Sessions, tablespace usage, wait classes, executions, PGA and SGA, hard parses | `CREATE SESSION`, `SELECT_CATALOG_ROLE` | +| **Instance info** | Version, open mode, Data Guard role, archive mode, sizing parameters | — | +| **Query performance (samples + top queries)** | Samples, top queries, and session waits from `V$` views, and optionally execution plans | `SELECT` on `V$SQL`, `V$SQL_PLAN`, `V$SQL_PLAN_STATISTICS_ALL`, `V$SESSION`, `V$SESSION_EVENT`, `V$LOCK`, `V$CONTAINERS`, `DBA_OBJECTS`, `DBA_PROCEDURES` | +| **Query log (active sessions)** | Active sessions and the SQL text they run | `SELECT` on `V$SESSION`, `V$SQL` | +| **Schema catalog** | Tables, columns, indexes, and constraints for the owners you list | Oracle 12.2 or later. **Schemas (owners) to scan** is required. | +| **Locks & blocking** | Lock counts and blocked sessions | `SELECT` on `V$LOCK`, `V$SESSION` | +| **Container topology (CDB / PDB)** | Root or PDB role and the container roster | Turn on for the CDB root | +| **RAC topology** | RAC instances and their status | `SELECT` on `V$DATABASE`, `GV$INSTANCE` | +| **Data Guard topology & lag** | Primary and standbys, and apply lag | `SELECT` on `V$DATABASE`, `V$DATAGUARD_CONFIG`, `V$DATAGUARD_STATS` | +| **Backup tracking (RMAN)** | Time since the last RMAN backup of each type | `SELECT` on `V$RMAN_BACKUP_JOB_DETAILS` | +| **AWR load profile & instance efficiency** | Load profile and instance efficiency percentages | Diagnostics Pack | +| **AWR time model & wait classes** | DB time breakdown and wait classes as average active sessions | Diagnostics Pack | +| **AWR wait events, enqueues & services** | Named wait events, enqueues, and service time | Diagnostics Pack | +| **AWR top SQL & plan history** | Top SQL by elapsed time with plan hash, per snapshot | Diagnostics Pack | +| **AWR I/O & segment statistics** | I/O by function and tablespace, top segments | Diagnostics Pack | +| **AWR active session history** | ASH activity by wait class, SQL, module, and object | Diagnostics Pack | + +:::caution +The AWR coverages read `DBA_HIST_*` views, which require the Oracle Diagnostics Pack, an Enterprise Edition option. Reading them records Diagnostics Pack usage in `DBA_FEATURE_USAGE_STATISTICS`. Before you turn one on, check that the pack is enabled: + +```sql +SELECT value FROM v$parameter WHERE name = 'control_management_pack_access'; +``` + +`DIAGNOSTIC` or `DIAGNOSTIC+TUNING` means it's enabled. `NONE`, the Standard Edition default, means don't turn on AWR coverages. See [Agent jobs and AWR](../../../database-monitoring/agent-jobs-and-awr/) for what they show. +::: + +### Set up the database + +#### Create a monitoring user + +Every Oracle coverage connects as a monitoring user with `CREATE SESSION` and `SELECT_CATALOG_ROLE`. Connect as `SYS`, or another user allowed to grant these, and run: + +```sql +CREATE USER kmmonitor IDENTIFIED BY "YOUR_PASSWORD"; +GRANT CREATE SESSION TO kmmonitor; +GRANT SELECT_CATALOG_ROLE TO kmmonitor; +``` + +`SELECT_CATALOG_ROLE` covers the dictionary and `V$` views the coverages read, including `V$SQLSTATS`, the `DBA_*` views behind **Schema catalog**, and the `DBA_HIST_*` views behind AWR. The wizard doesn't show this step, but every coverage depends on it. + +Use the same user in the wizard's **Monitoring user** field, and the same password in **Password** or in the environment variable it names. See [Database credentials](../credentials/). + +:::caution[Container databases] +In a PDB, run the statements above while connected to that PDB. In the CDB root, Oracle only allows common users, whose names start with `C##`. To monitor the root, create a common user instead and use its name in **Monitoring user** and in every grant below: + +```sql +CREATE USER C##KMMONITOR IDENTIFIED BY "YOUR_PASSWORD" CONTAINER=ALL; +GRANT CREATE SESSION TO C##KMMONITOR CONTAINER=ALL; +GRANT SELECT_CATALOG_ROLE TO C##KMMONITOR CONTAINER=ALL; +``` + +::: + +#### Grant access for specific coverages + +The wizard lists these object grants when you turn on the coverages that need them. They overlap with `SELECT_CATALOG_ROLE`, and they're what you need if your security policy grants individual views instead of the role. Oracle takes one object per `GRANT SELECT` statement. + +**Query performance:** + +```sql +GRANT SELECT ON V_$SQL TO kmmonitor; +GRANT SELECT ON V_$SQLSTATS TO kmmonitor; +GRANT SELECT ON V_$SQL_PLAN TO kmmonitor; +GRANT SELECT ON V_$SQL_PLAN_STATISTICS_ALL TO kmmonitor; +GRANT SELECT ON V_$SESSION TO kmmonitor; +GRANT SELECT ON V_$SESSION_EVENT TO kmmonitor; +GRANT SELECT ON V_$LOCK TO kmmonitor; +GRANT SELECT ON V_$CONTAINERS TO kmmonitor; +GRANT SELECT ON DBA_OBJECTS TO kmmonitor; +GRANT SELECT ON DBA_PROCEDURES TO kmmonitor; +``` + +**Backup tracking (RMAN):** + +```sql +GRANT SELECT ON V_$RMAN_BACKUP_JOB_DETAILS TO kmmonitor; +``` + +**Data Guard topology & lag:** + +```sql +GRANT SELECT ON V_$DATABASE TO kmmonitor; +GRANT SELECT ON V_$DATAGUARD_CONFIG TO kmmonitor; +GRANT SELECT ON V_$DATAGUARD_STATS TO kmmonitor; +``` + +**RAC topology:** + +```sql +GRANT SELECT ON V_$DATABASE TO kmmonitor; +GRANT SELECT ON GV_$INSTANCE TO kmmonitor; +``` + +**Locks & blocking:** + +```sql +GRANT SELECT ON V_$LOCK TO kmmonitor; +GRANT SELECT ON V_$SESSION TO kmmonitor; +``` + +**Query log (active sessions):** + +```sql +GRANT SELECT ON V_$SESSION TO kmmonitor; +GRANT SELECT ON V_$SQL TO kmmonitor; +``` + +**Instance info**, **Container topology**, **Schema catalog**, and the AWR coverages need nothing beyond `SELECT_CATALOG_ROLE`. + +## MongoDB + +| Coverage | Collects | Needs | +|---|---|---| +| **Standard metrics** | Operations, connections, memory, cache, document and collection stats, replication | Leave the user empty if the instance has no authentication | +| **Query performance (samples + top queries)** | Live samples from `currentOp` and top slow queries from the profiler | The `clusterMonitor` role. Profiling level 1 on each monitored database is recommended. | + +Set **Disable TLS** for a self-hosted MongoDB without TLS. It's off by default, and without it the connection expects a verified certificate. + +### Set up the database + +#### Grant the clusterMonitor role + +The agent connects as the **Monitoring user** (default `kmmonitor`). Give that user the built-in `clusterMonitor` role. Run this in the database where the user is defined, which is usually `admin`: + +```javascript +db.grantRolesToUser("kmmonitor", [{ role: "clusterMonitor", db: "admin" }]) +``` + +If the user is defined in a database other than the one the agent connects to, set **Auth database** in the wizard. Leave **Monitoring user** empty if the instance has no authentication. For the password, see [Database credentials](../credentials/). + +#### Enable database profiling + +Top queries come from the database profiler. Turn it on in each database you want to monitor: + +```javascript +db.setProfilingLevel(1) +``` + +Level `1` records operations slower than MongoDB's `slowms` threshold (100 ms by default). Without profiling, the agent reads slow operations from MongoDB's diagnostic log instead. That log keeps only about 1,024 recent entries, so on a busy server slow operations can rotate out before the agent reads them, and **Top queries** misses them. + +#### Connect with or without TLS + +**Disable TLS** is off by default, so the agent expects TLS and verifies the server certificate. For a self-hosted MongoDB with no TLS configured, turn on **Disable TLS**, or the connection fails. For a self-signed certificate, turn on **Skip certificate verification** instead. + +**Auth mechanism** can stay unset for username and password authentication. Set it only for certificate (`MONGODB-X509`), AWS IAM (`MONGODB-AWS`), or Kerberos (`GSSAPI`) authentication. + +## MongoDB Atlas + +Atlas is monitored through the Atlas Admin API, not a database connection. + +| Coverage | Collects | Needs | +|---|---|---| +| **MongoDB Atlas (cloud)** | Project, cluster, process, and disk metrics | An Atlas API key with the **Project Read Only** role, with the agent host's IP on the key's access list | +| **Logs (host, audit, access)** | mongod, audit, and access logs for one project | **Atlas project name**. Audit logs also need database auditing enabled in the project. | + +The logs coverage uses the API, so it's also available on Kubernetes. + +### Set up Atlas + +#### Create an Atlas API key + +1. In Atlas, open **Project Settings → Access Manager → API Keys** and create an API key with the **Project Read Only** role. +2. Add the public IP address of the host running the KloudMate Agent to the key's access list. Atlas rejects requests from any other address. +3. Copy the public key and the private key. Atlas shows the private key only once. + +The **Logs** coverage uses the same key and role. + +#### Fill in the wizard + +- **Atlas public key:** the public part of the API key. +- **Atlas private key:** by default, the wizard shows the environment variable the agent reads it from, such as `KM_SECRET_MONGODBATLAS_PRIVATE_KEY`. Set that variable on the agent host. See [Database credentials](../credentials/). +- **Metric granularity:** how finely Atlas aggregates each measurement, as an ISO-8601 duration. The default `PT1M` is one minute. +- **Atlas project name** (Logs only): the project to pull logs from. Logs are scoped to one project, unlike metrics. + +#### Collect audit logs + +**Collect audit logs** is off by default. Before you turn it on, enable auditing for the project in Atlas under **Project Settings → Advanced → Database Auditing**. + +## Redis + +| Coverage | Collects | Needs | +|---|---|---| +| **Standard metrics** | Commands, memory, keyspace, clients, replication | Nothing for an instance without authentication. Set **ACL username** only for a Redis 6+ ACL user. | + +### Connect to Redis + +Redis needs no grants or database-side setup. In the wizard, fill in: + +- **Endpoint:** `host:port` of the Redis server, such as `localhost:6379`. +- **Password:** leave it empty if the server has no authentication. See [Database credentials](../credentials/). +- **ACL username:** only for a Redis 6 or later ACL user. Leave it empty if the server uses `requirepass` alone. + +If the server uses TLS with a self-signed certificate, turn on **Skip certificate verification**. **Disable TLS** connects without TLS at all. + +## ClickHouse + +Standard metrics come from ClickHouse's Prometheus endpoint. The other coverages connect over SQL with their own host, port, and user. + +| Coverage | Collects | Needs | +|---|---|---| +| **Standard metrics (Prometheus endpoint)** | Memory, queries, merges, replication, parts | The `` endpoint enabled in `config.xml`, on port `9363` by default | +| **Instance info & core metrics** | Version, uptime, connections, database size, query counters | `SELECT` on `system.metrics`, `system.events`, `system.parts`, `system.server_settings` | +| **Query log (top queries by duration)** | Top queries from `system.query_log`, as log records | `SELECT` on `system.query_log` | +| **Replication topology & lag** | Cluster roster and replication delay | `SELECT` on `system.clusters`, `system.replicas` | + +:::caution +Turn on **Instance info & core metrics** for every ClickHouse instance. Without it, the instance doesn't appear in the Databases list. +::: + +### Set up ClickHouse + +ClickHouse coverages use two separate connections. **Standard metrics** reads the Prometheus endpoint over HTTP. The other coverages run SQL against ClickHouse's native protocol port, with their own **Host**, **Port**, **Database**, **Monitoring user**, and **Password** fields. + +#### Expose the Prometheus endpoint + +In ClickHouse's `config.xml`, add a `` block: + +```xml + + /metrics + 9363 + true + true + true + +``` + +Keep `metrics`, `events`, and `asynchronous_metrics` all set to `true`, since the Metrics tab reads all three. + +In the wizard, set **Prometheus endpoint** to the host and port of this endpoint, such as `10.0.0.12:9363`. It's `localhost:9363` by default. If the endpoint is served over TLS, set **Scrape scheme** to `https`, and check **Skip certificate verification** for a self-signed certificate. + +#### Connect over SQL + +Set **Host** and **Port** to ClickHouse's native protocol interface, not the Prometheus port. **Port** defaults to `9000`, and **Monitoring user** defaults to `kmmonitor`. + +Grant the monitoring user read access to the system tables each coverage reads. The wizard shows the grants for the coverages you turn on. Replace `kmmonitor` if your user has a different name. + +**Instance info & core metrics:** + +```sql +GRANT SELECT ON system.metrics TO kmmonitor; +GRANT SELECT ON system.events TO kmmonitor; +GRANT SELECT ON system.parts TO kmmonitor; +GRANT SELECT ON system.server_settings TO kmmonitor; +``` + +**Replication topology & lag:** + +```sql +GRANT SELECT ON system.clusters TO kmmonitor; +GRANT SELECT ON system.replicas TO kmmonitor; +``` + +**Query log:** + +```sql +GRANT SELECT ON system.query_log TO kmmonitor; +``` + +## Elasticsearch + +| Coverage | Collects | Needs | +|---|---|---| +| **Standard metrics** | Cluster health, node JVM, OS, and file system, indices, shards, thread pools, circuit breakers | Leave the user empty when security is disabled. Use an `https://` endpoint for TLS. | + +### Connect to the cluster + +1. Set **Endpoint** to the base URL of an Elasticsearch node, including the scheme and port, such as `http://10.0.0.20:9200`. The default is `http://localhost:9200`. +2. If security is enabled on the cluster, enter a **Monitoring user** and **Password**. The user needs read access to the `_cluster/health` and `_nodes/stats` APIs. If security is disabled, leave **Monitoring user** empty. +3. For a cluster that serves HTTPS, use an `https://` endpoint. Check **Skip certificate verification** if the node uses a self-signed certificate. + +One endpoint is enough for the whole cluster. Cluster health comes from the node you point at, and node statistics arrive as one series per node, which most panels add together into a cluster value. + +## SAP HANA + +| Coverage | Collects | Needs | +|---|---|---| +| **Standard metrics** | Connections, memory, CPU, disk, service and host status, transactions, volume I/O | A monitoring user on the SQL port (`30015` by default) | + +### Connect to SAP HANA + +1. Set **Endpoint** to the host and SQL port of the HANA database, such as `10.0.0.30:30015`. The default is `localhost:30015`. The SQL port depends on your instance number and whether you connect to a tenant database, so check it in your HANA system before you save. +2. Enter a **Monitoring user** and **Password**. Both are required. The user needs read access to HANA's system monitoring views. +3. Decide on TLS. **Disable TLS** is on by default, which matches a self-hosted HANA without TLS configured. Turn it off to connect over TLS. The server's certificate isn't verified. + +## Snowflake + +Snowflake is queried through `SNOWFLAKE.ACCOUNT_USAGE`, which lags real time and uses warehouse credits. The collection interval defaults to 300 seconds for that reason. + +| Coverage | Collects | Needs | +|---|---|---| +| **Warehouse & storage metrics** | Warehouse credit usage and per-database storage | `IMPORTED PRIVILEGES` on the `SNOWFLAKE` database and `USAGE` on the warehouse | +| **Top queries (QUERY_HISTORY)** | Top queries by elapsed time, as log records, with about 45 minutes of latency | The same grants | + +:::note +Elasticsearch, SAP HANA, and Snowflake coverages don't send the instance heartbeat the Databases list uses, so these databases don't appear there yet. Their metrics are available in Explore and in dashboards. +::: + +### Set up Snowflake + +#### Grant access to ACCOUNT_USAGE + +The monitoring user's role needs imported privileges on the `SNOWFLAKE` database and usage on the warehouse that runs the monitoring queries: + +```sql +GRANT IMPORTED PRIVILEGES ON DATABASE SNOWFLAKE TO ROLE YOUR_ROLE; +GRANT USAGE ON WAREHOUSE YOUR_WAREHOUSE TO ROLE YOUR_ROLE; +``` + +Replace `YOUR_ROLE` and `YOUR_WAREHOUSE` with your role and warehouse names. + +#### Fill in the connection + +| Field | What to enter | +|---|---| +| **Account identifier** | Your Snowflake account identifier, such as `xy12345.us-east-1` | +| **Warehouse** | The warehouse that runs the monitoring queries. Required. | +| **Role** | The role you granted access to. It defaults to `ACCOUNTADMIN`; enter a dedicated role if you created one. | +| **User** and **Password** | The monitoring user. Both are required. | +| **Database** and **Schema** | Leave them at `SNOWFLAKE` and `ACCOUNT_USAGE`. | + +For how to store the password, see [Database credentials](../credentials/). + +#### Keep the collection interval coarse + +**Collection interval** defaults to `300s` for Snowflake, instead of the `60s` other engines use. Each collection runs queries on the warehouse you named, and those queries consume credits. A shorter interval raises that cost without giving you fresher data, because `ACCOUNT_USAGE` itself lags. + +A collection also resumes the warehouse if it's suspended, so use a small warehouse for monitoring. + +## Execution plans + +PostgreSQL, MySQL and MariaDB, SQL Server, and Oracle can send each statement's execution plan for [query details](../../../database-monitoring/query-performance/#explain-plan). Check **Collect execution plans** on the **Query performance** coverage. A new statement can take a few collection intervals before its first plan appears. + +| Engine | Also needed | Notes | +|---|---|---| +| PostgreSQL | `GRANT pg_read_all_data TO kmmonitor;`, because running `EXPLAIN` needs read access to your tables. Then open **Advanced**, choose **Query performance**, and set `top_query_collection.max_explain_each_interval: 1000`. | — | +| MySQL and MariaDB | Nothing. The monitoring user already has `SELECT` on your tables. | Plans for statements with subqueries or `UNION` can't be displayed yet. | +| SQL Server | Nothing. | Each plan is the full XML showplan, often tens to hundreds of KB per statement, so expect a noticeable increase in data volume. | +| Oracle | Nothing. Plans come from `V$SQL_PLAN_STATISTICS_ALL`, which the Oracle grants already cover. | — | + +## Filter database logs -Database logs carry a `km.db.coverage` attribute set to the coverage they came from, such as `query-performance`, `server-logs`, or `slow-log`. You can filter database logs by this attribute in Log Explorer. +Database log records carry `km.db.coverage`, set to the coverage they came from. Filter on it in Log Explorer, for example `query-performance`, `query-log`, `top-queries`, `schema`, `agent-jobs`, `index-health`, `slow-log`, `server-logs`, `stderr-logs`, or `logs`. ## Next steps - + + diff --git a/src/content/docs/docs/kloudmate-agent/database-monitoring/images/advanced-query-performance.png b/src/content/docs/docs/kloudmate-agent/database-monitoring/images/advanced-query-performance.png new file mode 100644 index 00000000..42ccfd40 Binary files /dev/null and b/src/content/docs/docs/kloudmate-agent/database-monitoring/images/advanced-query-performance.png differ diff --git a/src/content/docs/docs/kloudmate-agent/database-monitoring/images/agent-databases-tab.png b/src/content/docs/docs/kloudmate-agent/database-monitoring/images/agent-databases-tab.png new file mode 100644 index 00000000..ac3cf185 Binary files /dev/null and b/src/content/docs/docs/kloudmate-agent/database-monitoring/images/agent-databases-tab.png differ diff --git a/src/content/docs/docs/kloudmate-agent/database-monitoring/images/wizard-add-database.png b/src/content/docs/docs/kloudmate-agent/database-monitoring/images/wizard-add-database.png index 2d6d8c16..a981f8f1 100644 Binary files a/src/content/docs/docs/kloudmate-agent/database-monitoring/images/wizard-add-database.png and b/src/content/docs/docs/kloudmate-agent/database-monitoring/images/wizard-add-database.png differ diff --git a/src/content/docs/docs/kloudmate-agent/database-monitoring/on-kubernetes.mdx b/src/content/docs/docs/kloudmate-agent/database-monitoring/on-kubernetes.mdx index 33280cba..4c5e1dfa 100644 --- a/src/content/docs/docs/kloudmate-agent/database-monitoring/on-kubernetes.mdx +++ b/src/content/docs/docs/kloudmate-agent/database-monitoring/on-kubernetes.mdx @@ -1,44 +1,50 @@ --- title: "Database monitoring on Kubernetes" -description: "How the KloudMate agent monitors databases running in your Kubernetes cluster, using a Kubernetes Secret for credentials and managed mode." +description: "Monitor databases in your Kubernetes cluster from the KloudMate cluster collector, with credentials in a Kubernetes Secret." sidebar: order: 4 --- import { Steps, LinkCard, CardGrid } from '@astrojs/starlight/components'; -On Kubernetes, the agent monitors the databases running in your cluster. The flow is the same as on a virtual machine, with two Kubernetes differences. Credentials come from a Kubernetes Secret, and database monitoring needs managed mode. For the general model, see the [Database monitoring overview](../overview/). +On Kubernetes, the cluster collector monitors the databases in your cluster. The flow is the same as on a virtual machine, with three differences: the agent must be in managed mode, passwords come from a Kubernetes Secret, and log file coverages aren't available. For the general model, see the [Database monitoring overview](../overview/). -## One monitor per cluster +## One collector per cluster -On Kubernetes, one collector in the cluster monitors your databases. Each database is monitored once for the whole cluster, not once per node. +Database monitoring runs in the cluster's Deployment collector, which runs once per cluster. Each database is monitored once, not once per node. -## Discovery prefills the endpoint +## Discovery fills in the endpoint -The agent finds databases running in your cluster. It prefills the wizard with their in-cluster address, `..svc.cluster.local:`. Use this address. It is the name your applications connect to, so the database's metrics and traces line up. Databases outside the cluster are not discovered, so add them manually. +The agent discovers databases exposed through Kubernetes Services and lists them under **Discovered in this cluster**. Clicking **Monitor** fills in the engine, the instance name, and the endpoint `..svc.cluster.local:`. Keep that address. It's the name your applications connect to, so the database's metrics and traces line up. + +Each discovered database stays listed until a database with the same endpoint is monitored. Databases outside the cluster aren't discovered, so add them with **Add database**. ## Managed mode is required -Database monitoring needs the agent in **managed mode**. On Kubernetes, use the "Configure databases" action on the agent. If the agent is still in manual mode, it prompts you to switch. See the [configuration model](../../concepts/config-model/). +In **Settings → Agents**, open the cluster's agent and go to its **Databases** tab. If the agent is in manual mode, click **Switch to managed**. In managed mode, KloudMate generates the agent's configuration from its integrations and replaces any manual configuration. See the [configuration model](../../concepts/config-model/). ## Available coverages -On Kubernetes, the agent collects database metrics and query coverages: standard metrics, query performance, locks, and query logs. Database log coverages are available on a virtual machine, not on Kubernetes. See [Engines and coverages](../engines-and-coverages/). +Every coverage is available on Kubernetes except those that read log files on the database host: **Server logs (JSON, PostgreSQL 15+)**, **Logs (stderr, PostgreSQL <15)**, and the MySQL **Slow query log**. API-based coverages such as MongoDB Atlas logs still work. See [Engines and coverages](../engines-and-coverages/). ## Provide credentials with a Kubernetes Secret -Credentials use an environment reference backed by a Kubernetes Secret you provide. Set it up before you configure any database, because the collector reads the Secret when its pod starts. +The wizard defaults to reading the password from an environment variable, and shows the variable's name, such as `KM_SECRET_POSTGRESQL_PASSWORD`. On Kubernetes, that variable comes from a Secret you create. Set it up before you configure a database, because the collector reads the Secret when its pod starts. -1. **Create a Kubernetes Secret** with a key of the form `KM_SECRET_` for each database credential. For example, a key `KM_SECRET_postgresql` holding the monitoring user's password. +1. **Find the variable name.** Open the wizard, choose the engine, and set the **Instance name** you'll use. The **Password** field shows the variable to create. The name is built from the instance name, or the engine when there isn't one, plus the field, for example `KM_SECRET_ORDERS_DB_PASSWORD` for an instance named `Orders DB`. + +2. **Create a Secret** with that key: ```bash kubectl create secret generic km-db-creds \ --namespace km-agent \ - --from-literal=KM_SECRET_postgresql='' + --from-literal=KM_SECRET_POSTGRESQL_PASSWORD='YOUR_PASSWORD' ``` -2. **Point the Helm release at the Secret** with the `dbMonitoring.secretName` value, and upgrade the release. The cluster collector then reads the Secret through `envFrom`. + Add one key per database password. You can keep them all in one Secret. + +3. **Point the Helm release at the Secret** and upgrade. The cluster collector loads it with `envFrom`. ```bash helm upgrade kloudmate-release kloudmate/km-kube-agent \ @@ -46,41 +52,40 @@ Credentials use an environment reference backed by a Kubernetes Secret you provi --set dbMonitoring.secretName=km-db-creds ``` -3. **Configure the database** in the wizard, referencing the key by name. The password stays in the cluster and never reaches KloudMate. +4. **Save the database** in the wizard. The password stays in the cluster and never reaches KloudMate. :::caution -Because the collector reads the Secret at pod startup, create the Secret and run the Helm upgrade before you configure a database. If you rotate a password without changing the configuration, the collector keeps the old value until its pod restarts. +The collector reads the Secret when its pod starts. If you add a key or rotate a password after the pod started, restart the collector pod for the change to take effect. ::: ## Worked example: PostgreSQL -1. **Create a monitoring role** in PostgreSQL and grant it the built-in `pg_monitor` role: +1. **Create the monitoring user** in PostgreSQL: ```sql - CREATE ROLE kmmonitor LOGIN PASSWORD ''; + CREATE USER kmmonitor WITH PASSWORD 'YOUR_PASSWORD'; GRANT pg_monitor TO kmmonitor; ``` -2. **Store the password** in a Kubernetes Secret under `KM_SECRET_postgresql` and set `dbMonitoring.secretName`, as above. - -3. **Configure PostgreSQL** in the wizard with the Service DNS endpoint (prefilled from discovery), the `kmmonitor` user, and an environment reference to the secret. Choose the coverages you want, such as standard metrics, locks, and query performance. - -4. **Enable query performance prerequisites, if used.** Query performance reads `pg_stat_statements`. Add it to `shared_preload_libraries` and create the extension, which needs a PostgreSQL restart: +2. **Enable `pg_stat_statements`** if you'll use query performance. Add it to `shared_preload_libraries`, restart PostgreSQL, then in each monitored database run: ```sql - -- postgresql.conf: shared_preload_libraries = 'pg_stat_statements' (then restart) CREATE EXTENSION pg_stat_statements; ``` + +3. **Store the password** under `KM_SECRET_POSTGRESQL_PASSWORD` in a Secret and set `dbMonitoring.secretName`, as above. + +4. **Monitor the discovered database.** Click **Monitor** next to it under **Discovered in this cluster**, turn on **Query performance**, **Locks & blocking**, and any other coverages you want, enter `kmmonitor` as the user, and click **Save and apply**. -After you apply the configuration, the cluster collector starts collecting from PostgreSQL. Its metrics carry the same database identity as the application spans that hit it. +The cluster collector starts collecting within a minute. Its metrics carry the same database identity as the application spans that call it. ## Next steps - + diff --git a/src/content/docs/docs/kloudmate-agent/database-monitoring/on-vm.mdx b/src/content/docs/docs/kloudmate-agent/database-monitoring/on-vm.mdx index 8b0fcd0b..b118297d 100644 --- a/src/content/docs/docs/kloudmate-agent/database-monitoring/on-vm.mdx +++ b/src/content/docs/docs/kloudmate-agent/database-monitoring/on-vm.mdx @@ -1,43 +1,72 @@ --- title: "Database monitoring on a VM" -description: "How the KloudMate agent monitors databases on a Linux or Windows host, covering databases on the host and reachable remote databases." +description: "Set up database monitoring from a KloudMate agent on a Linux or Windows host, for databases on the host or reachable from it." sidebar: order: 3 --- import { Steps, LinkCard, CardGrid } from '@astrojs/starlight/components'; -On a virtual machine, the agent monitors databases running on the host or reachable from it. For the general flow, see the [Database monitoring overview](../overview/). +An agent on a virtual machine can monitor databases running on the same host, and databases on other hosts it can reach over the network. This page walks you through adding one. For how coverages and the wizard work, see the [Database monitoring overview](../overview/). ## What you can monitor -- **Databases on the host:** a database listening on the host is discovered and prefilled in the wizard. -- **Reachable remote databases:** you can also monitor a database on another host by entering its endpoint and credentials in the wizard. +- **Databases on the host.** The agent discovers databases running on the host, including in containers, and lists them under **Discovered on this host**. +- **Remote databases.** A database on another host, or a managed service such as Amazon RDS, can be monitored from any agent that can reach it. Add it with **Add database** and enter its endpoint. -On a virtual machine, you can collect the full set of coverages, including database logs. See [Engines and coverages](../engines-and-coverages/). +Every coverage is available on a VM, including log file coverages such as PostgreSQL server logs and the MySQL slow query log. Log file coverages read files on the agent's host, so they only work for a database on that host. ## How local databases are identified -A database on the host often listens on `localhost`. That address is not unique across your servers. So the agent identifies the database by the host's routable address instead, which keeps its metrics and traces lined up. See [Telemetry identity](../../reference/telemetry-identity/) for how to build dashboards on a database's server address. +A database on the host often listens on `localhost`, which isn't unique across your servers. The agent identifies it by the host's routable address instead, so its metrics and traces line up. A database the agent reaches on the same host also gets host CPU, memory, and disk counted toward its health. See [Telemetry identity](../../reference/telemetry-identity/). ## Set up monitoring -1. **Open the wizard** from a discovered database, or add one manually with its endpoint. +1. **Make sure the agent is in managed config mode.** In **Settings → Agents**, open the agent. If the **Databases** tab asks you to switch to managed, click **Switch to managed**. Manual edits to the collector configuration are replaced. -2. **Choose coverages** for the depth you want. See [Engines and coverages](../engines-and-coverages/). +2. **Open the wizard.** On the **Databases** tab, click **Monitor** next to a discovered database, or **Add database** for one that wasn't discovered. You can also start from **Databases → Monitor a database**. -3. **Provide credentials** for the monitoring user, using an environment reference by default. See [Database credentials](../credentials/). +3. **Choose coverages.** **Standard metrics** is always on. Turn on the others you need. See [Engines and coverages](../engines-and-coverages/). -4. **Apply the setup steps** the wizard shows, such as creating the monitoring role and granting it the right permissions. +4. **Run the setup steps** the wizard shows on the database, such as creating the monitoring user and its grants. -5. **Save.** The agent applies the configuration and starts collecting. +5. **Fill in the connection.** Enter the endpoint, such as `localhost:5432`. Discovery fills in the engine and instance name but not the endpoint. Then enter the monitoring user and choose how to provide the password. See [Database credentials](../credentials/). + +6. **Click Save and apply.** Data starts arriving within a minute. + + +:::note +After you add one database of an engine, other discovered databases of the same engine disappear from **Discovered on this host**. To monitor a second PostgreSQL on the same host, use **Add database** and give it a distinct **Instance name**. +::: + +## Worked example: PostgreSQL on the host + + +1. **Create the monitoring user** in PostgreSQL: + + ```sql + CREATE USER kmmonitor WITH PASSWORD 'YOUR_PASSWORD'; + GRANT pg_monitor TO kmmonitor; + ``` + +2. **Enable `pg_stat_statements`** for query performance. Add it to `shared_preload_libraries` in `postgresql.conf`, restart PostgreSQL, then run this in each database you monitor: + + ```sql + CREATE EXTENSION pg_stat_statements; + ``` + +3. **Set the password variable** for the agent. With the default environment variable mode and no instance name, the wizard reads `KM_SECRET_POSTGRESQL_PASSWORD`. On Linux, add it to the agent's environment file and restart the agent. See [Database credentials](../credentials/#set-the-variable-on-a-host). + +4. **Open the wizard** from the discovered PostgreSQL database. Turn on **Query performance**, **Instance info**, and **Schema catalog**, enter `localhost:5432` as the endpoint and `kmmonitor` as the user, and click **Save and apply**. +The instance appears in the [Databases module](../../../database-monitoring/) within a minute. + ## Next steps - + diff --git a/src/content/docs/docs/kloudmate-agent/database-monitoring/overview.mdx b/src/content/docs/docs/kloudmate-agent/database-monitoring/overview.mdx index 95b74832..bd2177f1 100644 --- a/src/content/docs/docs/kloudmate-agent/database-monitoring/overview.mdx +++ b/src/content/docs/docs/kloudmate-agent/database-monitoring/overview.mdx @@ -1,48 +1,121 @@ --- title: "Database monitoring overview" -description: "How the KloudMate agent monitors your databases: automatic query visibility from eBPF plus opt-in health metrics, query performance, and logs." +description: "How the KloudMate agent monitors your databases: where to set it up, how coverages and credentials work, and what happens when you save." sidebar: order: 1 --- import { LinkCard, CardGrid } from '@astrojs/starlight/components'; -Database monitoring gives you a full picture of your databases. You set it up through a wizard, not hand-written configuration. It works alongside the automatic eBPF view of your queries. +Database monitoring connects the KloudMate agent to your databases with a monitoring user, and collects their health metrics, query performance, sessions, schema, topology, and logs. You set it up per database through a wizard, and the agent generates the collector configuration for you. The results appear in the [Databases module](../../../database-monitoring/). ## Two views of your databases -The agent shows your databases in two ways that work together: +The agent sees your databases in two ways that work together: -- **From eBPF, automatically and with no credentials.** Database Activity Monitoring (DAM) shows the queries your applications run against the database, with their timing. These appear as part of your distributed traces and the service map. This turns on by default. See [eBPF observability](../../ebpf-observability/). -- **From the database, when you opt in.** Database monitoring reads the database's own health metrics (connections, cache hit ratio, replication lag, and more), query performance, and logs. This needs a monitoring user's credentials. +- **From your applications, with no credentials.** eBPF and application traces capture the queries your services run against a database, with their timing. These appear in distributed traces, the service map, and the **Calling services** tab of the Databases module. See [eBPF observability](../../ebpf-observability/). +- **From the database itself, when you opt in.** Database monitoring reads the database's own statistics: connections, cache hit ratio, replication lag, top queries, active sessions, and more. This needs a monitoring user. -The two views line up on a shared database identity, so a slow query you see in a trace matches the server's health metrics for the same database. See [Telemetry identity](../../reference/telemetry-identity/). +Both views share a database identity, so a slow query in a trace lines up with the server metrics for the same database. See [Telemetry identity](../../reference/telemetry-identity/). + +## Before you start + +- **The agent must be in managed config mode.** Database monitoring is generated into the managed collector configuration, on every platform. If the agent is in manual mode, the **Databases** tab offers to switch it. Switching regenerates the configuration from the agent's integrations and replaces manual edits. See the [configuration model](../../concepts/config-model/). +- **The agent must reach the database.** For a self-hosted database, install the agent on the same machine. For a managed database such as Amazon RDS, Cloud SQL, or MongoDB Atlas, install it on a host that can reach the database over the network. +- **The engines and coverages you're offered come from the agent itself**, so they depend on the agent version. Upgrade the agent to get newer coverages. + +## Where to set it up + +You can open the wizard from two places: + +- **Databases → Monitor a database.** Choose the agent, then the engine. This is the quickest path when you're adding a database for the first time. +- **Settings → Agents.** Open the agent and go to its **Databases** tab. It lists the databases the agent discovered and not yet monitored, with a **Monitor** button on each, and the databases already monitored, with **Edit** and **Remove**. Use **Add database** for a database that wasn't discovered. + +Once you add one database of an engine on a host, the other discovered databases of that engine drop off the **Discovered on this host** list. Add further databases of the same engine with **Add database**. ## Coverages -You choose what to monitor for a database as a set of **coverages**, such as standard metrics, query performance, locks, or logs. You pick the coverages you want, and the agent sets up the collection. See [Engines and coverages](../engines-and-coverages/) for what each engine supports. +You choose what to collect for a database as a set of **coverages**. **Standard metrics** is always on. Every other coverage, such as **Query performance**, **Schema catalog**, or **Replication topology & lag**, is off until you turn it on. + +Under **Standard metrics**, check **Collect all optional metrics** to collect the receiver's full metric set instead of the basic one. You get more detail, at a higher series count. Several panels in the Databases module need it. + +See [Engines and coverages](../engines-and-coverages/) for every coverage by engine. ## The wizard -You set up monitoring for a database through a wizard, usually starting from a database the agent already found: +The wizard is one form, top to bottom: + +1. **Database engine.** Skipped when you started from a discovered database or from the Databases module, since the engine is already chosen. +2. **What to collect.** Turn coverages on and off. +3. **Setup required.** The grants, extensions, and database settings the selected coverages need, ready to copy. Run them on the database before you save. +4. **Connection.** The fields your coverages need: endpoint, monitoring user, password, and coverage-specific settings such as scan intervals or schemas to scan. **Instance name** is optional, but set it when you monitor more than one database of the same engine on one endpoint, such as an Oracle CDB root and its PDBs. It keeps their data apart. +5. **Advanced.** An optional YAML fragment per coverage, merged into that coverage's generated receiver configuration. Use it for settings the form doesn't expose, such as TLS options. It isn't validated against a schema, and a password typed here is stored in plain text. + +Click **Save and apply**. The change reaches the agent within a minute. + +![The "Add database monitoring" wizard for MySQL, with Query performance turned on, the setup steps it needs, and the connection fields with the credential mode selector (environment variable or entered value).](./images/wizard-add-database.png) + +## What's checked when you save + +The wizard checks that required fields are filled in and that any **Advanced** override is a YAML mapping. It doesn't connect to the database, so it can't test your password or grants. + +- **A wrong password or a missing grant** shows up as errors in the agent log, and as empty tabs in the Databases module. If a tab stays empty, run the setup steps again and check the agent log. +- **A database the agent can't generate** is skipped on its own. The other databases and the rest of your telemetry keep working. Skipped integrations and the reason are listed on the agent's **Collector config** tab. + +## Edit or remove a database + +Monitored databases are edited from the agent that monitors them, not from the Databases module. Go to **Settings → Agents**, open the agent, and go to its **Databases** tab. + +![An agent's Databases tab listing its monitored databases, each with a feature count, an Edit (pencil) button, and a Remove (trash) button.](./images/agent-databases-tab.png) + +Click **Edit** on a monitored database to change its coverages, connection, or overrides. Stored passwords stay in place unless you replace them. To turn on a coverage you haven't enabled yet, expand the database's feature list and follow **Edit to enable**. + +Click **Remove** to stop monitoring a database. The change applies within a minute. + +## Reduce query performance volume + +The **Query performance** coverage sends two kinds of log events. **Query samples** (`db.server.query_sample`) are snapshots of the sessions on the database at each collection. **Top queries** (`db.server.top_query`) are the most expensive statements since the previous collection. On a busy database, these can add up to thousands of events an hour. + +To send fewer, edit the database, expand **Advanced**, choose the **Query performance** coverage, and set these keys: + +| Engine | Top queries per collection | Top query interval | Sessions per sample | +|---|---|---|---| +| PostgreSQL | `top_query_collection.top_n_query` (200) | `top_query_collection.collection_interval` (1m) | `query_sample_collection.max_rows_per_query` (1000) | +| MySQL and MariaDB | `top_query_collection.top_query_count` (200) | `top_query_collection.collection_interval` (1m) | `query_sample_collection.max_rows_per_query` (100) | +| SQL Server | `top_query_collection.top_query_count` (250) | `top_query_collection.collection_interval` (1m) | `query_sample_collection.max_rows_per_query` (100) | +| Oracle | `top_query_collection.top_query_count` (200) | `top_query_collection.collection_interval` (1m) | `query_sample_collection.max_rows_per_query` (100) | + +Defaults are in parentheses. For example, this sends the top 50 PostgreSQL statements every 5 minutes instead of the top 200 every minute, and samples at most 200 sessions: + +```yaml +top_query_collection: + top_n_query: 50 + collection_interval: 5m +query_sample_collection: + max_rows_per_query: 200 +``` + +![The Advanced section of the edit wizard with the Query performance coverage selected and the YAML above in the editor.](./images/advanced-query-performance.png) + +- **Query samples follow the standard metrics interval.** The receivers don't support a separate interval for samples. Don't set a top-level `collection_interval` in this override: query performance shares a receiver with standard metrics, so it would slow those down too. +- **Connection poolers produce many idle samples.** An idle session is sampled once after each query it finishes, which is how short queries are captured at all. Behind a pooler such as PgBouncer, most connections are idle at any moment, so most samples are idle sessions. Samples can't be filtered by session state, so lower `max_rows_per_query` to send fewer of them. +- **The sample row limit isn't a ranking.** When there are more sessions than `max_rows_per_query`, the sessions that get sampled aren't chosen by activity or duration. -1. **Pick the engine.** The agent lists databases it discovered, so the engine and endpoint are often prefilled. -2. **Choose coverages.** Turn on the coverage areas you want, such as standard metrics, query performance, or locks. -3. **Provide credentials.** Enter the monitoring user and its secret. By default, the secret is stored as a reference and never leaves the host. See [Database credentials](../credentials/). -4. **Follow the setup steps.** The wizard shows the exact grants and prerequisites each coverage needs, ready to copy. -5. **Review and apply.** The wizard previews what will be collected, and on save the agent starts collecting. +## PII detection -![The "Add database monitoring" wizard for a discovered MySQL instance, showing coverage toggles (query performance, slow query log) and the credential mode selector (environment variable or entered value).](./images/wizard-add-database.png) +Database query logs and database spans pass through the agent's PII detection before they leave the host. Records that contain personal data are tagged with `km.pii.detected`, `km.pii.critical`, and `km.pii.categories`. The matched values themselves aren't copied into these attributes. -## Safe to apply +The agent's configuration page shows the status of the on-host machine learning classifier: **ML active** when it's running, **ML not responding** when it stopped reporting, **ML not installed**, or **PII detection off**. -The agent validates the setup before applying it. It handles each database on its own, so one bad database setting cannot affect the others or your other telemetry. +:::note +Query samples and top queries carry normalized statements with literal values replaced. The coverages that can carry raw SQL are the query log coverages for Oracle, SQL Server, and ClickHouse, Snowflake top queries, and database log file coverages such as the MySQL slow query log. +::: ## Where to run it - + - - + + diff --git a/src/content/docs/docs/kloudmate-agent/ebpf-observability.mdx b/src/content/docs/docs/kloudmate-agent/ebpf-observability.mdx index 56a295e1..e8ea726b 100644 --- a/src/content/docs/docs/kloudmate-agent/ebpf-observability.mdx +++ b/src/content/docs/docs/kloudmate-agent/ebpf-observability.mdx @@ -16,7 +16,7 @@ eBPF is part of the agent's automatic monitoring. On a managed-mode agent, it tu When eBPF monitoring is active, the agent collects: - **RED metrics** (Rate, Errors, and Duration) for your services, with no instrumentation. -- **Service dependencies**, used to build the visual service map, including peer-to-peer latency. +- **Request traces** for every call an eBPF-monitored service makes or receives. These draw the [service map](../../apm-and-tracing/apm-views/service-map/), with the latency of each call. A caller that sends no telemetry of its own still appears, named after its Kubernetes workload, or as `incoming` when the agent can't identify it. - **Network topology:** a host-to-host map of Layer 4 flows across your fleet. See [eBPF network topology](../ebpf-network-topology/). - **Database Activity Monitoring (DAM):** kernel-level capture of database queries on the wire, with no credentials. - **Distributed context propagation:** the agent reads and injects HTTP trace context (`traceparent`), so requests stay on one trace across services. diff --git a/src/content/docs/docs/kloudmate-agent/installation/kubernetes-agent.mdx b/src/content/docs/docs/kloudmate-agent/installation/kubernetes-agent.mdx index 73b713b9..f227f9fd 100644 --- a/src/content/docs/docs/kloudmate-agent/installation/kubernetes-agent.mdx +++ b/src/content/docs/docs/kloudmate-agent/installation/kubernetes-agent.mdx @@ -142,6 +142,40 @@ This example defines two tolerations to match the taints on the cluster nodes, s - If your cluster has no taints, omit these parameters. The agent pods then schedule onto untainted nodes by default. ::: +## Setting Resources and Priority for Agent Pods + +You can set the CPU and memory requests and limits of each agent container, and a priority class for the agent pods, with Helm values. The defaults are the values the chart has always used, so an existing install behaves the same until you change them. + +On clusters with small nodes, for example 2 vCPU, some agent pods can stay **Pending** with `Insufficient cpu`. Without a priority class they can't preempt other pods, so those nodes go unmonitored. The node agent typically uses only a few millicores of CPU and about 100Mi of memory, so lowering its requests is safe. A priority class lets the agent pods schedule ahead of lower-priority workloads. The cluster agent and fleet manager run as single-replica Deployments and can be tuned the same way. + +| Value | What it sets | Default | +|---|---|---| +| `daemonset.resources` | Node agent (DaemonSet) container | Requests 100m CPU / 256Mi, limits 2 CPU / 1Gi | +| `daemonset.initContainer.resources` | DaemonSet sysctl-tuning init container | Requests 10m / 16Mi, limits 100m / 32Mi | +| `daemonset.priorityClassName` | PriorityClass of the DaemonSet pods | Not set | +| `clusterAgent.resources` | Cluster agent (Deployment) container | Requests 150m / 384Mi, limits 2 / 1Gi | +| `clusterAgent.priorityClassName` | PriorityClass of the cluster agent pod | Not set | +| `configUpdater.resources` | Fleet manager container | Requests 100m / 128Mi, limits 500m / 512Mi | +| `configUpdater.priorityClassName` | PriorityClass of the fleet manager pod | Not set | + +A `priorityClassName` is added to a pod only when you set it. These values need the latest km-kube-agent chart, so run `helm repo update` before you upgrade. + +Example Helm command that lowers the requests of the node agent and the cluster agent, and gives their pods a priority class: + +```bash +helm upgrade kloudmate-release kloudmate/km-kube-agent --namespace km-agent --reuse-values \ +--set daemonset.resources.requests.cpu=10m \ +--set daemonset.resources.requests.memory=128Mi \ +--set daemonset.priorityClassName=system-node-critical \ +--set clusterAgent.resources.requests.cpu=100m \ +--set clusterAgent.resources.requests.memory=256Mi \ +--set clusterAgent.priorityClassName=system-cluster-critical +``` + +:::note +`system-node-critical` and `system-cluster-critical` are built-in Kubernetes priority classes, and they let the agent preempt other pods. `system-node-critical` suits the per-node DaemonSet, and `system-cluster-critical` suits the cluster-wide Deployment. For a gentler option, create your own PriorityClass and use its name. +::: + ## APM Setup Instructions There are two ways to enable **Application Performance Monitoring (APM)** in your cluster: diff --git a/src/content/docs/docs/kloudmate-agent/management.mdx b/src/content/docs/docs/kloudmate-agent/management.mdx index 193a0b61..37d65746 100644 --- a/src/content/docs/docs/kloudmate-agent/management.mdx +++ b/src/content/docs/docs/kloudmate-agent/management.mdx @@ -176,9 +176,9 @@ The agent is light. On a virtual machine or host, it runs as a single process an On Kubernetes, usage is split across the per-node collector and a few cluster-wide components. See [Resource usage and sizing](../platform-notes/kubernetes/#resource-usage-and-sizing) for per-component figures and suggested CPU and memory requests and limits. -## Updating the agent +## Upgrade the agent -Update the agent by upgrading its package on the host or the Helm release on Kubernetes, the same way you installed it. The bundled instrumentation agents are versioned with the KloudMate agent, so they update together. See the [installation guides](../installation/linux-agent/). +Upgrade an agent from its page in the dashboard with one click, or rerun its install command. See [Upgrade the agent](../upgrade/). ## Best Practices diff --git a/src/content/docs/docs/kloudmate-agent/reference/metric-catalogs.mdx b/src/content/docs/docs/kloudmate-agent/reference/metric-catalogs.mdx index 7d182214..b1f87fdc 100644 --- a/src/content/docs/docs/kloudmate-agent/reference/metric-catalogs.mdx +++ b/src/content/docs/docs/kloudmate-agent/reference/metric-catalogs.mdx @@ -15,7 +15,7 @@ Host metrics are the system-level metrics the agent collects: CPU, memory, disk, ## eBPF monitoring metrics -eBPF monitoring produces Rate, Errors, and Duration (RED) metrics for your services and the data behind the service map: request rate, error rate, request duration, and the connections between services that draw the map. It also captures the database queries your applications run, for Database Activity Monitoring. These power the service map and the prebuilt RED views, which is where you read them. See [eBPF observability](../../ebpf-observability/). +eBPF monitoring produces Rate, Errors, and Duration (RED) metrics for your services, and request traces that draw the service map. It also captures the database queries your applications run, for Database Activity Monitoring. You read this data in the service map and the prebuilt RED views. See [eBPF observability](../../ebpf-observability/). ## Database metrics diff --git a/src/content/docs/docs/kloudmate-agent/upgrade.mdx b/src/content/docs/docs/kloudmate-agent/upgrade.mdx new file mode 100644 index 00000000..d9973f89 --- /dev/null +++ b/src/content/docs/docs/kloudmate-agent/upgrade.mdx @@ -0,0 +1,106 @@ +--- +title: "Upgrade the agent" +description: "Upgrade the KloudMate agent from the dashboard with one click, or manually on Linux, Windows, Docker, and Kubernetes." +sidebar: + order: 13 +--- + +You can upgrade agents running version 2.5.0 or later from the dashboard with one click. The agent downloads the new release, verifies it, installs it, and rolls itself back if the new version doesn't come up healthy. When one-click upgrade isn't available for an agent, upgrade it manually with the same command you installed it with. + +The bundled instrumentation agents are versioned with the KloudMate agent, so they upgrade together. + +## Upgrade from the dashboard + +1. Open **Settings → Agents**. An agent with a newer release has an **Update available** tag in the **Agent Version** column. +2. Click the agent to open its page. +3. In the **Agent version** card, click **Upgrade to vX.Y.Z** and confirm. + +The agent picks up the request on its next check-in, within a minute, and most upgrades finish in one to two minutes. A Kubernetes cluster takes longer, because the node agents restart one node at a time. + +### What happens during an upgrade + +- The agent downloads the new release from GitHub and checks it against a signing key built into the agent. A package that fails the check is never installed. +- Before installing, the agent also downloads the package for the version it's running now, so it always has a way back. +- After installing, the new version has 5 minutes to start and check in. If it doesn't, the previous version is reinstalled, and the agent's page shows why the upgrade failed. + +On Kubernetes, Helm rolls the release back if the new pods don't become ready. On Docker, the agent's container is recreated with the same settings on the new image, and the old container is started again if the new one doesn't come up. + +:::note +One-click upgrade needs agent version **2.5.0 or later**. Upgrade an older agent [manually](#upgrade-manually) once, and later releases can then be applied from the dashboard. +::: + +### When the Upgrade button is disabled + +Hover over the button to see why. The usual causes and fixes: + +| Cause | Fix | +|---|---| +| Remote upgrades are turned off on the host. | Remove the setting described in [Turn off one-click upgrade on a host](#turn-off-one-click-upgrade-on-a-host), or upgrade manually. | +| The host can't reach the release download host. | Allow outbound HTTPS to `github.com`, `release-assets.githubusercontent.com`, and `objects.githubusercontent.com`. See [Before you start](../installation/before-you-start/). | +| The Helm chart was installed with `selfUpdate.enabled=false`. | Set it to `true` with `helm upgrade`, or upgrade manually. | +| The cluster is managed by Argo CD or Flux. | Upgrade the chart through your GitOps repository. | +| The container is part of a Docker Compose project or a Swarm service. | Change the image in the Compose file or service definition, and redeploy. | +| The container doesn't run the official `ghcr.io/kloudmate/km-agent` image, or doesn't have the Docker socket mounted. | Recreate the container with the KloudMate Docker install command. | +| The Docker host is arm64. | Upgrade manually. One-click upgrade for Docker agents works on amd64 hosts. | + +Agents on Amazon ECS can't be upgraded from the dashboard. Redeploy the task with the new image. + +### Turn off one-click upgrade on a host + +To block dashboard upgrades on a Linux or Windows host, set `KM_ALLOW_REMOTE_UPGRADE=false` in the agent's environment and restart it. On Linux: + +```bash +sudo systemctl edit kmagent +``` + +Add these lines, save, and run `sudo systemctl restart kmagent`: + +```ini +[Service] +Environment=KM_ALLOW_REMOTE_UPGRADE=false +``` + +On Kubernetes, install or upgrade the chart with `--set selfUpdate.enabled=false`. + +## Upgrade manually + +Run the install command again. The install script always fetches the latest release, so the command you used to install the agent also upgrades it. Copy the install command for your platform from **Settings → Agents → Add Agent**, using the same API key and collector endpoint as the existing agent. + +### Linux + +Run the Linux install command on the host. The script downloads the latest `.deb` or `.rpm` package, installs it over the old one, and restarts the `kmagent` service. + +Your existing `/etc/kmagent/agent.yaml` is kept as it is, including any settings you've added by hand. + +### Windows + +Download the latest installer from **Settings → Agents → Add Agent → Windows** and run it from an elevated (Administrator) PowerShell prompt: + +```powershell +.\kmagent-setup.exe /VERYSILENT /SUPPRESSMSGBOXES /NORESTART +``` + +The installer stops the `kmagent` service, replaces the files, and starts the service again. Your existing `C:\ProgramData\kmagent\agent.yaml` is kept. Pass `/APIKEY=YOUR_API_KEY /ENDPOINT=https://otel.kloudmate.com:4318` only if you want to replace that file with a new API key and endpoint. + +### Docker + +Run the Docker install command on the host. The script pulls `ghcr.io/kloudmate/km-agent:latest`, removes the old `km-agent` container, and starts a new one. When it asks about additional log directories, give the same answers as the first time, or those directories won't be mounted into the new container. + +### Kubernetes + +Refresh the chart repository, then upgrade the release. `--reset-then-reuse-values` keeps the values you installed with and picks up the new chart's defaults (it needs Helm 3.14 or later): + +```bash +helm repo update +helm upgrade kloudmate-release kloudmate/km-kube-agent \ + --namespace km-agent \ + --reset-then-reuse-values +``` + +On an older Helm, rerun the Helm command from the install wizard with `helm install` changed to `helm upgrade --install`, and pass the same `--set` flags you installed with. + +Each chart version ships the matching agent image, so upgrading the chart rolls every agent pod to the new version. + +## Confirm the upgrade + +Open **Settings → Agents** and check the **Agent Version** of the upgraded agent. If it still shows the old version or the agent goes offline, see [Troubleshooting](../troubleshooting/). diff --git a/src/content/docs/docs/kloudmate-assistant/images/investigations-2.png b/src/content/docs/docs/kloudmate-assistant/images/investigations-2.png index 93933ed3..9bccd882 100644 Binary files a/src/content/docs/docs/kloudmate-assistant/images/investigations-2.png and b/src/content/docs/docs/kloudmate-assistant/images/investigations-2.png differ diff --git a/src/content/docs/docs/kloudmate-assistant/images/mcp-integrations-1.jpeg b/src/content/docs/docs/kloudmate-assistant/images/mcp-integrations-1.jpeg deleted file mode 100644 index 70334a10..00000000 Binary files a/src/content/docs/docs/kloudmate-assistant/images/mcp-integrations-1.jpeg and /dev/null differ diff --git a/src/content/docs/docs/kloudmate-assistant/images/mcp-integrations-1.png b/src/content/docs/docs/kloudmate-assistant/images/mcp-integrations-1.png deleted file mode 100644 index 308c7143..00000000 Binary files a/src/content/docs/docs/kloudmate-assistant/images/mcp-integrations-1.png and /dev/null differ diff --git a/src/content/docs/docs/kloudmate-assistant/images/mcp-integrations-tab.png b/src/content/docs/docs/kloudmate-assistant/images/mcp-integrations-tab.png new file mode 100644 index 00000000..fa4e7d03 Binary files /dev/null and b/src/content/docs/docs/kloudmate-assistant/images/mcp-integrations-tab.png differ diff --git a/src/content/docs/docs/kloudmate-assistant/images/mcp-integrations-tools.png b/src/content/docs/docs/kloudmate-assistant/images/mcp-integrations-tools.png new file mode 100644 index 00000000..2d9f6b78 Binary files /dev/null and b/src/content/docs/docs/kloudmate-assistant/images/mcp-integrations-tools.png differ diff --git a/src/content/docs/docs/kloudmate-assistant/images/mcp-oauth-consent.png b/src/content/docs/docs/kloudmate-assistant/images/mcp-oauth-consent.png new file mode 100644 index 00000000..2b982d33 Binary files /dev/null and b/src/content/docs/docs/kloudmate-assistant/images/mcp-oauth-consent.png differ diff --git a/src/content/docs/docs/kloudmate-assistant/index.mdx b/src/content/docs/docs/kloudmate-assistant/index.mdx index 2f1b2f0c..1feb277a 100644 --- a/src/content/docs/docs/kloudmate-assistant/index.mdx +++ b/src/content/docs/docs/kloudmate-assistant/index.mdx @@ -27,5 +27,3 @@ Open **Assistant** to review investigations, track AI credit usage, and manage t - [Chat Modes](./chat-modes/) - [Prompts, Skills, and MCP Integrations](./settings/) - [Usage](./usage/) - -![Assistant module tabs](./images/assistant-module-tabs.png) diff --git a/src/content/docs/docs/kloudmate-assistant/mcp-server.mdx b/src/content/docs/docs/kloudmate-assistant/mcp-server.mdx index 9c12d6e2..6dd9a0d6 100644 --- a/src/content/docs/docs/kloudmate-assistant/mcp-server.mdx +++ b/src/content/docs/docs/kloudmate-assistant/mcp-server.mdx @@ -1,6 +1,6 @@ --- title: "KloudMate MCP Server" -description: "Connect your AI agents and coding tools to KloudMate over MCP to query logs, traces, metrics, and alerts in plain language." +description: "Connect Claude, Cursor, VS Code, and other AI tools to KloudMate over MCP to query logs, traces, metrics, and alerts in plain language." sidebar: label: "MCP Server" order: 5 @@ -8,67 +8,190 @@ sidebar: import { Tabs, TabItem, Steps, LinkCard, CardGrid } from '@astrojs/starlight/components'; -The KloudMate MCP server lets your AI coding tools and agents pull observability data straight from your workspace. Point a client like Claude Code, Cursor, or VS Code at it, and you can ask about errors, latency, traces, and firing alerts in plain language, without leaving your editor or opening the KloudMate UI. +The KloudMate MCP server lets your AI tools and agents read observability data from your workspace. Connect a client such as Claude, Cursor, or VS Code, and you can ask about errors, latency, traces, and firing alerts in plain language without opening the KloudMate UI. -This is the mirror image of the [MCP integrations](../settings/#mcp-integrations) you connect *into* the Assistant. There, KloudMate is the client calling out to other tools. Here, KloudMate is the server, and your agent is the client calling in. The connection is built on the [Model Context Protocol](https://modelcontextprotocol.io/), an open standard for giving language models access to external tools and data. - -After you connect, your agent can chain KloudMate's query tools to investigate an incident, answer a question about a service, or run a repeatable workflow you've defined. +This is the reverse of the [MCP integrations](../settings/#mcp-integrations) you connect to the Assistant. There, KloudMate calls out to other tools. Here, your agent calls in to KloudMate. The connection uses the [Model Context Protocol](https://modelcontextprotocol.io/), an open standard for giving language models access to tools and data. ## Connection details -The server is a single remote endpoint. You authenticate with a KloudMate [Personal API Key](../../platform/settings/api-keys/#personal-api-keys) passed as a bearer token, and set the workspace you want to read in an `x-workspace-id` header. - | Setting | Value | |---|---| | **Endpoint** | `https://api.kloudmate.com/mcp` | | **Transport** | Streamable HTTP | -| **Authentication** | `Authorization: Bearer YOUR_API_KEY` | -| **Workspace** | `x-workspace-id: YOUR_WORKSPACE_ID` | +| **Authentication** | OAuth sign-in, or a Personal API Key with an `x-workspace-id` header | -A personal key carries your access to every workspace you belong to, so the connection needs the `x-workspace-id` header to know which one to read. Find the workspace ID in the workspace's URL in the KloudMate app. To point the same key at a different workspace, change that header. One key works for all of them. +**Sign in with OAuth** when your client supports it. The client needs only the endpoint URL, and you sign in to KloudMate in your browser and pick a workspace. Claude, ChatGPT, Cursor, VS Code, and MCP Inspector all support OAuth for remote MCP servers. + +**Use a Personal API Key** for scripts, and for clients that can't sign in with OAuth. You send the key and a workspace ID as request headers. ## Before you start You'll need: -- **A KloudMate Personal API Key.** From the account menu (top-right), open **API keys** and click **Create Key**. It's tied to your account and reads whatever you can read. Copy it when it's shown (it's shown only once) and keep it handy for your client's config. Any signed-in user can create one; you don't need the **Developer** role. See [API Keys](../../platform/settings/api-keys/#personal-api-keys) for the full walkthrough. -- **The workspace ID** you want to query. Find it in the workspace's URL in the KloudMate app; it goes in the `x-workspace-id` header. -- **An MCP-capable client.** Any tool that supports remote MCP servers over Streamable HTTP with custom headers works. The steps below cover Claude Code, Cursor, VS Code, and Claude Desktop. +- **A workspace whose plan includes the MCP server.** If the plan doesn't include it, you can't allow the connection on the consent page, and requests made with an API key are refused. +- **An MCP client that supports remote servers over Streamable HTTP.** The steps below cover Claude Code, Cursor, VS Code, and Claude on the web and desktop. -:::caution -Your API key carries your KloudMate access. Treat it like a password: don't paste it into shared files or commit it to version control. The client configs below show how to keep it in an environment variable or a prompted secret instead of hardcoding it. If a key leaks, delete it from the **Personal API Keys** screen in your account menu. -::: +## Connect with OAuth sign-in + +Add `https://api.kloudmate.com/mcp` to your client as a remote MCP server. You don't need a key or any headers. The first time the client connects, it opens KloudMate in your browser: + + + +1. Sign in to KloudMate, if you aren't signed in already. -## Connect your client +2. On the consent page, pick the workspace the app can read. -Pick your tool and follow the steps. Swap `YOUR_API_KEY` for the key you just created, and `YOUR_WORKSPACE_ID` for the workspace you want to query. +3. Click **Allow**. Your browser goes back to the app, and the app is connected. + + + +![The KloudMate consent page asking whether to allow Claude Code to use KloudMate, with a workspace picker and Cancel and Allow buttons](./images/mcp-oauth-consent.png) + +Click **Allow** only if you started the sign-in yourself. A sign-in expires if you don't finish it within 10 minutes. If that happens, connect again from the app. + +The connection reads only the workspace you picked, so you don't send a workspace ID with each request. To read a different workspace, sign out of the server in your client, sign in again, and pick the other workspace. -Add the server from your terminal with the `claude mcp add` command: +Add the server from your terminal: ```bash -claude mcp add --transport http kloudmate https://api.kloudmate.com/mcp \ - --header "Authorization: Bearer YOUR_API_KEY" \ - --header "x-workspace-id: YOUR_WORKSPACE_ID" +claude mcp add --transport http kloudmate https://api.kloudmate.com/mcp ``` -By default the server is available in the current project only. Add `--scope user` before the server name to make it available across all your projects: +By default, the server is available in the current project only. Add `--scope user` before the server name to make it available in all your projects: ```bash -claude mcp add --transport http --scope user kloudmate https://api.kloudmate.com/mcp \ +claude mcp add --transport http --scope user kloudmate https://api.kloudmate.com/mcp +``` + +Start Claude Code, run `/mcp`, select `kloudmate`, and authenticate. Claude Code opens KloudMate in your browser. You can also sign in from your terminal with `claude mcp login kloudmate`, and sign out with `claude mcp logout kloudmate`. + + + + +Create or edit `~/.cursor/mcp.json` (all projects) or `.cursor/mcp.json` (this project only), and add the server: + +```json +{ + "mcpServers": { + "kloudmate": { + "url": "https://api.kloudmate.com/mcp" + } + } +} +``` + +Cursor starts the sign-in when it connects to the server. If `kloudmate` shows that it needs authentication, click **Connect** on it. + + + + +Create `.vscode/mcp.json` in your workspace: + +```json +{ + "servers": { + "kloudmate": { + "type": "http", + "url": "https://api.kloudmate.com/mcp" + } + } +} +``` + +Start the server from the **Start** action above the entry, or run **MCP: List Servers** from the Command Palette. When VS Code asks whether the server can authenticate, click **Allow**, then sign in to KloudMate in your browser. The tools become available to agent mode in Copilot Chat. + + + + +These steps work on claude.ai and in Claude Desktop. + + + +1. Open **Customize > Connectors**. + +2. Click **+ Add**, then **Add custom connector**. + +3. Enter `KloudMate` as the name and `https://api.kloudmate.com/mcp` as the URL, then click **Continue**. + +4. Keep the detected authentication settings and click **Continue**. + +5. Under **Authentication**, choose **Sign in now**, then click **Add**. Claude opens KloudMate so you can sign in. + + + +To use the connector in a chat, click **+**, then **Connectors**, and turn on KloudMate. + +On a Team or Enterprise plan, an Owner adds the connector under **Organization settings > Connectors**. Each member then clicks **Connect** on it under **Customize > Connectors** and signs in to KloudMate with their own account. + + + + +Add a custom connector in ChatGPT with `https://api.kloudmate.com/mcp` as its URL and **OAuth** as its authentication. ChatGPT opens KloudMate so you can sign in. + +If ChatGPT asks how to register the client, either method works: + +- **Client ID Metadata Document (CIMD)**: ChatGPT identifies itself with its published metadata document. +- **Dynamic Client Registration (DCR)**: ChatGPT registers itself with KloudMate the first time it connects. + + + + +For any other client that supports OAuth for remote MCP servers, such as MCP Inspector, enter the endpoint URL and sign in the same way. + +### Client registration + +Most clients register themselves, so you don't need this section to connect one. It's for you if you're building a client, or if your client asks which registration method or authentication to use. + +KloudMate supports two ways for a client to register: + +| Method | How the client identifies itself | Token endpoint authentication | +|---|---|---| +| **Client ID Metadata Document (CIMD)** | Its `client_id` is an `https` URL to its own metadata document, which KloudMate fetches and caches for a day. | `none`, or `private_key_jwt` when the document names its signing keys: an inline `jwks`, or an `https` `jwks_uri` on the document's own host. | +| **Dynamic Client Registration (DCR)** | It registers at `https://api.kloudmate.com/oauth/register` and gets a `client_id`. | `none`, `client_secret_basic`, or `client_secret_post`. | + +Every client uses the authorization code flow with PKCE (`S256`). The scopes are `mcp`, and `offline_access` for a refresh token. + +A client that authenticates with `private_key_jwt` sends a client assertion (RFC 7523) to the token endpoint. Sign it with `RS256` or `ES256`, set `iss` and `sub` to the `client_id`, set `aud` to `https://api.kloudmate.com/oauth/token`, and keep it to 5 minutes or less. Each assertion works once. + +A client discovers all of this from two documents: + +- `https://api.kloudmate.com/.well-known/oauth-protected-resource/mcp`: the MCP server's protected resource metadata (RFC 9728). +- `https://api.kloudmate.com/.well-known/oauth-authorization-server`: the authorization server metadata (RFC 8414). + +## Connect with an API key + +Use a Personal API Key for scripts, and for clients that can't sign in with OAuth. Send the key as a bearer token, and the workspace ID in an `x-workspace-id` header: + +| Header | Value | +|---|---| +| `Authorization` | `Bearer YOUR_API_KEY` | +| `x-workspace-id` | `YOUR_WORKSPACE_ID` | + +To create a key, open **API keys** from the account menu (top-right) and click **Create Key**. Copy the key when it's shown, because it's shown only once. Any signed-in user can create one, and you don't need the **Developer** role. See [API Keys](../../platform/settings/api-keys/#personal-api-keys) for details. + +A personal key works for every workspace you belong to in your organization, so the `x-workspace-id` header picks the workspace each request reads. Find the workspace ID in the workspace's URL in the KloudMate app. + +:::caution +Your API key carries your KloudMate access. Treat it like a password: don't paste it into shared files or commit it to version control. The configs below keep it in an environment variable or a prompted secret instead. If a key leaks, delete it from the **Personal API Keys** screen in your account menu. +::: + + + + +```bash +claude mcp add --transport http kloudmate https://api.kloudmate.com/mcp \ --header "Authorization: Bearer YOUR_API_KEY" \ --header "x-workspace-id: YOUR_WORKSPACE_ID" ``` -Run `/mcp` inside Claude Code to confirm `kloudmate` shows as connected and to see the tools it exposes. +Add `--scope user` before the server name to make the server available in all your projects. Run `/mcp` inside Claude Code to confirm that `kloudmate` is connected. -Create or edit `~/.cursor/mcp.json` (global, all projects) or `.cursor/mcp.json` (this project only), and add the server: - ```json { "mcpServers": { @@ -83,12 +206,12 @@ Create or edit `~/.cursor/mcp.json` (global, all projects) or `.cursor/mcp.json` } ``` -The `${env:KLOUDMATE_API_KEY}` reference reads the key from your environment so it never lands in the file. Set `KLOUDMATE_API_KEY` in your shell, then open **Cursor Settings → MCP** to confirm `kloudmate` connects. You can also paste the key directly in place of the reference, but keep that file out of any repository. +The `${env:KLOUDMATE_API_KEY}` reference reads the key from your environment, so it never lands in the file. Set `KLOUDMATE_API_KEY` in your shell profile before you start Cursor. -Create `.vscode/mcp.json` in your workspace. The `inputs` block prompts you for the key the first time the server starts and stores it securely, so the key stays out of the file: +The `inputs` block prompts you for the key the first time the server starts and stores it securely, so the key stays out of the file: ```json { @@ -113,145 +236,87 @@ Create `.vscode/mcp.json` in your workspace. The `inputs` block prompts you for } ``` -Start the server from the **Start** action that appears above the entry, then enter your key when prompted. The tools become available to agent mode in Copilot Chat. - - - - -Claude Desktop's custom-connector UI is built around OAuth and has no field for a bearer token, so connect through the config file using the [`mcp-remote`](https://github.com/geelen/mcp-remote) bridge instead. - -Open `claude_desktop_config.json` (**Settings → Developer → Edit Config**) and add: - -```json -{ - "mcpServers": { - "kloudmate": { - "command": "npx", - "args": [ - "-y", - "mcp-remote", - "https://api.kloudmate.com/mcp", - "--header", - "Authorization:${AUTH_HEADER}", - "--header", - "x-workspace-id:YOUR_WORKSPACE_ID" - ], - "env": { - "AUTH_HEADER": "Bearer YOUR_API_KEY" - } - } - } -} -``` - -The `Bearer ` token goes in the `AUTH_HEADER` environment variable, and the `--header` argument has no spaces around the colon. That split is deliberate: it sidesteps a known bug where Claude Desktop mangles spaces inside `args` when it launches `npx`. Quit Claude Desktop fully and reopen it, because it reads the config only at startup. Then check that `kloudmate` appears in the tools menu. - -The config file lives at: - -- **macOS:** `~/Library/Application Support/Claude/claude_desktop_config.json` -- **Windows:** `%APPDATA%\Claude\claude_desktop_config.json` - -Using a different client? Point it at `https://api.kloudmate.com/mcp` with an `Authorization: Bearer YOUR_API_KEY` header and an `x-workspace-id: YOUR_WORKSPACE_ID` header. Any client that speaks Streamable HTTP and lets you set request headers can connect. - ## Verify the connection -Once the server is connected, your client lists the KloudMate tools and your agent can call them. The quickest check is to ask a question that needs live data: +After you connect, ask a question that needs live data: > Which alerts are firing in KloudMate right now? -If the connection is healthy, the agent calls a KloudMate tool and answers from your workspace. If it reports an authentication failure, the key is likely wrong, deleted, or missing the `Bearer ` prefix. Recheck the header and try again. +If the connection works, the agent calls a KloudMate tool and answers from your workspace. -## Available tools - -Every tool is read-only. The server queries your telemetry; it never creates, edits, or deletes KloudMate resources. Tools accept relative time ranges like `now-1h`, `now-15m`, or `now`, as well as ISO-8601 timestamps, and default to the last hour when you don't give one. - -**Issues and alerts** - -- `search_issues`: errors and warnings aggregated from logs and traces. -- `search_alert_rules`, `search_alert_states`, `search_alert_states_history`: list alert rules, see what's firing now, and review how states changed over time. -- `search_alert_groups`, `get_alert_group_details`: find alert groups and see a single group's alert instances, routing rule, and any root-cause investigation attached to it. +- **The app asks you to sign in again.** An app renews its access every hour on its own. If it goes 30 days without renewing, or you lose access to the workspace, the renewal fails and you need to sign in again. +- **Requests fail with an access error.** You no longer have access to the workspace the connection reads. KloudMate checks your access on every request. +- **An API key fails to authenticate.** The key is wrong or deleted, or the `Authorization` header is missing the `Bearer ` prefix. +- **An API key request is refused for a missing workspace.** Add the `x-workspace-id` header. -**Incidents** - -- `search_incidents`, `get_incident_details`: find open incidents, then look up an incident by its key, such as `INC-42`, to read its responders, notes, alerts, and activity timeline. -- `get_incident_metrics`: incident counts by status, plus MTTA and MTTR. - -**Service health (APM)** - -- `get_service_error_count`, `get_service_requests_count_and_avg_latency`: error counts, throughput, and average latency for a service. -- `get_service_to_service_requests_count`, `get_service_to_service_failed_requests_count`, `get_service_to_service_request_duration`: request volume, failures, and latency between services, to see which services a problem affects. +## Available tools -**SLOs** +Every tool is read-only. The tools read your telemetry and never create, edit, or delete anything in KloudMate. -- `search_slos`, `get_slo_details`: list SLOs with their latest compliance and remaining error budget, and read a single SLO's SLI definition, compliance history, and burn-rate alerts. -- `search_slo_burn_rate_alerts`: burn-rate alerts across every SLO, with firing alerts listed first. +A time window is a duration that ends now, such as `30m`, `6h`, or `2d`, or two ISO 8601 times joined by `/`. Each tool has its own default window. When a question mentions a service, host, or other resource, the tool looks it up first. If the name matches several resources, or none, the tool returns the candidates instead of reading data. -**Logs, traces, and spans** +**Logs, traces, and metrics** -- `search_logs`, `search_traces`, `search_spans`: query telemetry with filters. -- `search_trace_groups`: trace count, p95 duration, and error count for each service and operation, to find the busiest and slowest endpoints before you open individual traces. -- `get_log_details`, `get_span_details`, `get_trace_spans`, `get_trace_logs`: pull the full record for a single log line, span, or trace. -- `find_cloudwatch_log_groups`, `search_cloudwatch_logs`: find the CloudWatch log groups for an AWS resource and read their events directly from your connected AWS accounts, even for logs that aren't sent to KloudMate. +- `search_logs`: logs filtered by any field, with counts by severity over time, common patterns, and the fields that vary. It can also return the log lines, and it searches Kubernetes events too. +- `search_traces`: spans filtered by any field, with counts, errors, and p50, p95, and p99 latency, plus error patterns and the slowest spans. It can also return traces or a single trace. +- `aggregate_telemetry`: counts and percentiles over logs or traces, grouped by any field, such as errors by service or p95 duration by endpoint. +- `query_metrics`: find metrics, list a label's values, read series against the previous window, or run PromQL. For a single resource, it reads every metric the resource has, such as a pod's restarts and limits or a host's load. +- `explore_fields`: the fields that logs, traces, or metrics carry, with their common values, or the values of one field with counts. -**Schema discovery** +**Alerts, errors, and changes** -- `discover_dataset_fields`, `discover_field_values`: find which fields and values exist before building a filter. -- `discover_metrics`, `discover_metric_label_values`: find metric names and the values their labels take. +- `check_alerts`: alert groups and their alerts, with what fired and when, the rule and its condition, the latest values, silences, and maintenance windows. It can also read a single alert rule, or the rules firing for a resource. Without a target, it lists the open alert groups. +- `search_error_patterns`: error patterns from logs, traces, and RUM, with the service, severity, first and last seen times, how often they occurred compared with the previous window, state, assignees, and comments. +- `check_changes`: what changed in a window. It compares the values of the fields you choose, such as a version, an image, or a host, against the previous window, and lists Kubernetes Normal events. +- `find_entities`: the service, host, workload, or other resource a name refers to, with its kind, when it last reported, and its firing alerts. It can also show what the resource depends on and what depends on it. -**Aggregations and queries** +**Kubernetes** -- `run_aggregation_query`, `run_metric_aggregation_query`: grouped counts, percentiles, and timeseries over logs and traces, or over metrics. -- `run_multiple_log_trace_aggregations`, `run_multiple_metric_aggregations`: run several aggregations in one call. +- `check_kubernetes`: Kubernetes objects as the Kubernetes pages list them, problems first. That covers pods with their status, restarts, and usage against limits; workloads against their desired state; and jobs, persistent volume claims, and nodes. +- `inspect_kubernetes`: one workload, pod, or node in depth, with its state against desired, its pods, a deployment's ReplicaSets, and the events on it. Use it to find out why a workload is down or a pod keeps restarting. +- `search_kubernetes_events`: Kubernetes events by reason and object, warnings first, such as back-offs, failed scheduling, image pulls, probe failures, OOM kills, and rollouts. -**Profiling** +**Reliability and user impact** -- `discover_profiles`: list the services that send profiles and the profile types each service reports, such as CPU, allocated memory, or goroutines. -- `analyze_profile_hotspots`, `analyze_profile_trend`, `compare_profile_windows`: find the functions using the most CPU or memory, track a profile type over time to spot a memory leak, and compare two time windows, such as before and after a deploy. +- `check_slos`: each SLO's objective, compliance, remaining error budget, and burn-rate alerts over the SLO's own window. +- `check_synthetics`: whether each synthetic monitor is up or down, with its last run, success rate, outages, time to recover, and SSL expiry. +- `check_incidents`: open incidents; one incident's timeline, notes, responders, and tickets; status page posts; and who is on call now. +- `check_user_impact`: real user monitoring insights such as error spikes and slow pages, LCP and app cold start, error and crash rates, and a comparison between releases. **Dashboards** -- `search_dashboards`, `get_dashboard_panels`, `run_dashboard_panel_query`: find a dashboard, inspect its panels, and run a panel's query. - -**Kubernetes** - -- `list_kubernetes_resources`, `search_kubernetes_events`: inspect clusters, namespaces, deployments, and pods, and read recent cluster events. - -**Synthetics** - -- `search_synthetic_monitors`, `search_synthetic_incidents`: check uptime and browser monitors and the incidents they've raised. +- `read_dashboards`: find dashboards, list a dashboard's panels, or read a panel's current values against its thresholds. ## Example prompts -Once connected, ask in plain language. The agent picks the right tools and fills in the queries: +After you connect, ask in plain language. The agent picks the tools and fills in the queries: - *"What are the top errors in the `checkout` service over the last hour?"* - *"Show me the slowest traces for `api-gateway` in the past 15 minutes."* - *"Which alerts are firing right now, and what changed in the half hour before they fired?"* +- *"Why does the `payments` pod keep restarting?"* - *"Pull the logs for trace `4bf92f3577b34da6` and summarize where it failed."* -- *"Compare request volume between the `frontend` and `cart` services this morning."* ## Build custom workflows -Your agent can chain these tools and combine them with whatever else it can reach, such as your codebase, a runbook, or your deployment history. You can turn an investigation that takes several screens in the KloudMate UI into a repeatable workflow. +Your agent can combine these tools with anything else it can reach, such as your codebase, a runbook, or your deployment history. That lets you turn an investigation that spans several KloudMate screens into a repeatable prompt. -A triage workflow, for example, might pull the currently firing alerts, scope the affected services with the service-to-service tools, gather the relevant traces and logs, and hand you a single summary with a probable cause. Save it as a reusable command or prompt in your tool of choice, and the next on-call run is one sentence instead of twenty clicks. +For example, a triage prompt might read the open alert groups, check what changed before they fired, pull the related traces and logs, and return a summary with a probable cause. Save it as a reusable command or prompt in your tool, and run it at the start of each on-call investigation. ## Security and scope -A few things worth keeping in mind: - -- **Read-only.** The server can't change anything in your workspace. Connecting it can't cause an outage or alter a dashboard. -- **Scoped to you.** The key carries your own account's permissions, and the `x-workspace-id` header picks which workspace each request reads. It can't reach a workspace you can't. -- **Keep the key secret.** Prefer the environment-variable or prompted-secret options above over pasting the key into a file. Add config files that contain a key to `.gitignore`. -- **Delete when needed.** If a key is exposed or no longer used, delete it from the **Personal API Keys** screen in your account menu. Any client using it stops working immediately. +- **Read-only.** The tools can't change anything in your workspace. +- **Your permissions, one workspace.** Every connection acts with your own permissions. An OAuth connection reads only the workspace you picked when you signed in. An API key connection reads the workspace in its `x-workspace-id` header. KloudMate checks your access on every request, so a connection can't read a workspace you can't. +- **Disconnect from the app.** To disconnect an app that signed in with OAuth, remove the KloudMate server or connector in that app. KloudMate doesn't list the apps you've connected. +- **Keep API keys secret.** Prefer the environment variable or prompted secret options above over pasting a key into a file, and add any config file that contains a key to `.gitignore`. If a key is exposed or no longer used, delete it from the **Personal API Keys** screen. Any client using it stops working immediately. ## Related - + diff --git a/src/content/docs/docs/kloudmate-assistant/settings.mdx b/src/content/docs/docs/kloudmate-assistant/settings.mdx index 6bf83d8a..aeb97230 100644 --- a/src/content/docs/docs/kloudmate-assistant/settings.mdx +++ b/src/content/docs/docs/kloudmate-assistant/settings.mdx @@ -64,54 +64,60 @@ This opens the Import Skill Template dialog, which lists a set of curated templa ## MCP Integrations -**MCP (Model Context Protocol) integrations** connect the Assistant to external systems such as OAuth providers or your own custom MCP servers. Open **Assistant → MCP Integrations** to connect and manage them. After connecting an integration, you configure which of its tools are enabled and whether the Assistant must request approval before running any of them. +Connect a tool server to give the Assistant tools from another system, such as GitHub, Atlassian, Sentry, Google Cloud, AWS, or an MCP server your team runs. Open **Assistant → MCP Integrations** to connect a server and choose which of its tools the Assistant can use. -![The MCP Integrations tab under Assistant: Connected integrations with Name, Visibility, and Status columns, and Available integrations with Connect buttons](./images/mcp-integrations-1.png) +![The MCP Integrations tab under Assistant, listing tool servers with their provider, what they can be used for, status, visibility, and owner](./images/mcp-integrations-tab.png) -### Connected Integrations +Each tool server is a [connection](/platform/settings/connections/) that can be used for **AI**, so it also appears under **Settings → Connections**. For editing, re-checking, reconnecting, and deleting a server, see [Manage a connection](/platform/settings/connections/manage-connections/). -The Connected Integrations section lists all integrations you have already configured. For each integration, you can see its display name, whether it is scoped privately to your account or shared across the workspace, and its current connection status. +### Connect a tool server -:::info - A status of **Error** indicates the integration requires attention. Use the **Validate** button to re-test the connection, or the menu to edit or remove the integration. +Anyone in the workspace can connect a tool server from this tab. You don't need the **Developer** role. - If you see a **Reconnect required** status, the integration’s authorization has expired and needs to be refreshed. Click **Reconnect** to re-authenticate. -::: +1. Click **Connect** and pick a provider. +2. Fill in the **Connection name**. +3. Fill in the provider's fields, if it has any, and click **Connect**. For a provider that uses OAuth, such as Sentry or Google Cloud, you then approve the access on the provider's own page. + +For an MCP server that isn't in the list, pick **Custom MCP Server** and enter its URL. Under **Authentication**, choose how KloudMate authenticates to the server: **None**, **Bearer token**, **Basic auth**, **API key**, or **OAuth**. With **OAuth**, KloudMate saves the connection as **Pending** first and then opens the server's own sign-in page. You don't need a client ID. + +A tool server you connect here can be used only by the Assistant. + +### Private and shared tool servers + +A new tool server is **Private**. Only you can use it, so the Assistant uses it only in your own chats. That fits a personal token, such as your own GitHub account. -### Available Integrations +To give everyone in the workspace the same server, a **Developer** connects it and then clicks **Share with workspace** in its row menu. A shared server belongs to the workspace, so only Developers can change it. A workflow's [Ask KloudMate Assistant](/workflows/actions/#ask-kloudmate-assistant) step can use only shared servers. -The Available Integrations section lists pre-built connectors you can activate. +Without the **Developer** role, you can edit, re-check, and delete your own private tool servers, but you can't share them. -Click **Connect** next to any provider to begin the connection flow. +### One tool server per provider -### Connecting a Custom MCP Server +The Assistant uses one tool server per provider, such as one Atlassian server, so it never has to guess which one to act with. Custom MCP servers and AWS accounts are the exception: the Assistant uses all of them. -To connect an MCP server that is not listed among the available providers, click **Connect custom server**. +If you can use two servers of the same provider, the Assistant uses one that you connected before one that someone else connected, then the oldest. So two people in a workspace can be on different servers. The other one shows **Not used by the Assistant**. Point to that label to see which server the Assistant uses instead. To use the other one, switch off the one in use, or ask a workspace admin to. -![The Connect Custom MCP Server dialog with Integration name, MCP server URL, Headers, Visibility, and Status fields](./images/mcp-integrations-1.jpeg) +### Choose the tools the Assistant can use -This opens the **Connect Custom MCP Server** dialog where you configure the following: +Open a server's row menu and click **Tools…** to see the tools the server offers and set how the Assistant uses each one. Click **Save** when you're done. -- **Integration name:** A friendly label for the integration. -- **MCP server URL:** The publicly accessible endpoint for your MCP server. You enter it only for a custom server. Managed providers don't ask for a URL. -- **Headers:** Optional HTTP headers to be sent with every MCP request. Use this for API key or token-based authentication, tenant or workspace routing, or any other custom metadata the server requires. -- **Visibility:** Choose **Private** so only you can use the integration, or **Shared** to share it across the workspace. Keep integrations private for personal credentials, testing environments, or dev and staging connections, and share them when they are organization-managed and intended for the whole team. -- **Status:** The master on/off switch for the integration. Set it to **Disabled** during incidents, provider outages, security reviews, or when you want to deprecate the integration without permanently deleting it. +![The Tools dialog for a custom MCP server, with quick actions, a search box, and each tool's labels, Enabled switch, and Approval mode](./images/mcp-integrations-tools.png) -After completing the form, click **Save** to connect the integration. +**Enabled** decides whether the Assistant can call a tool at all. Keep the list short: with more than 10 tools enabled, the Assistant can respond more slowly. -### Tool-Level Settings +**Approval mode** decides whether the Assistant asks you before it runs a tool: -After connecting an integration, you can configure each of its individual tools. For every tool exposed by the integration, you can control: +| Approval mode | What the Assistant does | +|---|---| +| **Default** | Runs a tool labeled **Read** without asking, and asks you before it runs any other tool. | +| **Ask before run** | Asks you every time. | +| **Auto approve** | Runs the tool without asking. | -- **Enabled:** Determines whether the Assistant is allowed to call the tool at all. Disable individual tools you do not need to reduce the Assistant’s scope of action. -- **Approval mode:** Determines whether the Assistant must ask for confirmation before running the tool. Setting it to **Default** uses the platform’s built-in behavior. Setting it to **Ask before run** requires the Assistant to confirm with you every time it wants to invoke the tool. Setting it to **Auto approve** allows the tool to run without any confirmation prompt. +The labels come from the server's own description of each tool. **Read** means the tool only reads data, **Write** means it can change something, **Destructive** means it can delete, and **External** means it calls systems outside the server. Because the server sets them, treat the labels as hints. -Tools may also carry labels that hint at their behavior. +The quick actions above the list change every tool at once. For example, **Enable read only** turns on every tool labeled **Read**, and **Auto approve read only** sets each of them to **Auto approve**. -- A **Read** label indicates the tool is intended to be read-only and carries lower risk. -- A **Write** label indicates the tool may perform destructive or state-changing actions and should be used with care. +In a workflow's **Ask KloudMate Assistant** step, nobody can approve a tool. The Assistant uses only the enabled tools labeled **Read** there, whatever their approval mode. -:::info - When deciding on approval mode, it is advisable to apply stricter settings such as **Ask before run** for Write tools and for any tool with access to sensitive or production systems. +:::caution +**Auto approve** lets the Assistant run a **Write** or **Destructive** tool without asking you. Use it only for tools you'd let the Assistant run unchecked, and keep **Ask before run** on any tool that can change a production system. ::: diff --git a/src/content/docs/docs/logs/log-explorer/index.mdx b/src/content/docs/docs/logs/log-explorer/index.mdx index f95ffe53..dcff128d 100644 --- a/src/content/docs/docs/logs/log-explorer/index.mdx +++ b/src/content/docs/docs/logs/log-explorer/index.mdx @@ -63,6 +63,12 @@ Use **Stream** to watch logs as they are ingested into KloudMate. ![image](./images/log-explorer-7.jpeg) +## Search Archived Logs + +If your organization archives logs to S3 with [Cold Storage](../../platform/settings/cold-storage/), you can search logs that are older than your log retention. When the selected time range starts before your retention, Log Explorer offers to rehydrate it. To search a rehydration, click **Live logs** and select it under **Rehydrated**. + +See [Rehydrate archived logs](../../platform/settings/cold-storage/rehydrate-logs/) for how rehydration works and what it costs. + ## Related Resources - [Log Query Syntax](./syntax-for-log-querying/) diff --git a/src/content/docs/docs/opentelemetry/data-sampling.mdx b/src/content/docs/docs/opentelemetry/data-sampling.mdx index 36379dcb..0901326b 100644 --- a/src/content/docs/docs/opentelemetry/data-sampling.mdx +++ b/src/content/docs/docs/opentelemetry/data-sampling.mdx @@ -9,6 +9,13 @@ In a high-volume environment, you may not want to send every trace and log to Kl The OpenTelemetry Collector provides several processors for sampling data. The two most common approaches are **Head Sampling** (making a sampling decision at the beginning of a trace) and **Tail Sampling** (making a decision after all spans for a trace have been collected). +## Where to add the configuration + +- **KloudMate agent:** add the processor as a [custom config override](../../kloudmate-agent/custom-config-override/) on the agent's **Configuration** page. In the pipeline, list only the sampler, for example `processors: [probabilistic_sampler]`. The agent places it before `batch` and keeps the rest of the managed pipeline. The override page has ready-to-paste versions of [percentage sampling](../../kloudmate-agent/custom-config-override/#sample-a-percentage-of-traces-and-logs) and [tail sampling](../../kloudmate-agent/custom-config-override/#keep-error-and-slow-traces-sample-the-rest). +- **Your own OpenTelemetry Collector:** add the processor to the Collector's configuration file and restart the Collector. The examples below show the complete pipeline for this case. See [Configure the Collector](../using-opentelemetry-collector/) for the rest of the file. + +If the agent instruments your services, you can also lower trace volume at the source with a per-service [sampling rate](../../kloudmate-agent/auto-instrumentation/sampling/), which needs no collector configuration. + ## Probabilistic Sampling (Head Sampling) To reduce volume with a uniform rate, use the `probabilistic_sampler` processor, which randomly drops a set percentage of your data. @@ -85,5 +92,6 @@ service: - [Official Docs: Probabilistic Sampler Processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/probabilisticsamplerprocessor) - [Official Docs: Tail Sampling Processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor) +- [Custom config override for the KloudMate agent](../../kloudmate-agent/custom-config-override/) - [Mask and Drop Sensitive Data](../masking-sensitive-data/) - [Configure the Collector](../using-opentelemetry-collector/) diff --git a/src/content/docs/docs/platform/settings/api-keys.mdx b/src/content/docs/docs/platform/settings/api-keys.mdx index 000f4bd5..c194b1cb 100644 --- a/src/content/docs/docs/platform/settings/api-keys.mdx +++ b/src/content/docs/docs/platform/settings/api-keys.mdx @@ -10,7 +10,7 @@ import { Steps } from '@astrojs/starlight/components'; KloudMate has two kinds of API key, each for a different job: -- **Personal API Keys** authenticate *you*. Use one to read your data through the KloudMate API or the [MCP server](../../../kloudmate-assistant/mcp-server/). The key carries your own account's permissions. +- **Personal API Keys** authenticate *you*. Use one to read your data through the KloudMate API, or to connect a script or tool to the [MCP server](../../../kloudmate-assistant/mcp-server/). The key carries your own account's permissions. - **Ingest Keys** authenticate a *workspace's data pipeline*. Your OpenTelemetry collectors, SDKs, and the KloudMate agent use one to send logs, metrics, and traces into a single workspace. ## Personal API Keys @@ -31,10 +31,10 @@ Authorization: Bearer YOUR_API_KEY x-api-key: YOUR_API_KEY ``` -This is the key to use for the [KloudMate MCP server](../../../kloudmate-assistant/mcp-server/) and for any script or service that queries your observability data. Because it carries your access, treat it like a password. Anyone holding it can act as you. +Use it for any script or service that queries your observability data. It also connects the [KloudMate MCP server](../../../kloudmate-assistant/mcp-server/) to a client that can't sign in with OAuth. A client that supports OAuth, such as Claude, Cursor, or VS Code, can sign in to the MCP server instead, without a key. Because the key carries your access, treat it like a password. Anyone holding it can act as you. :::note -A personal key spans every workspace you can reach, so a tool that reads from **one** workspace at a time, like the MCP server, also needs you to name the target workspace on each request. The [MCP server](../../../kloudmate-assistant/mcp-server/) setup shows how with the `x-workspace-id` header. +A personal key spans every workspace you can reach, so a tool that reads from **one** workspace at a time, like the MCP server, also needs the target workspace on each request. The [MCP server](../../../kloudmate-assistant/mcp-server/#connect-with-an-api-key) setup shows how to send it in the `x-workspace-id` header. ::: ## Ingest Keys diff --git a/src/content/docs/docs/platform/settings/cold-storage/_meta.json b/src/content/docs/docs/platform/settings/cold-storage/_meta.json new file mode 100644 index 00000000..3a3cebd8 --- /dev/null +++ b/src/content/docs/docs/platform/settings/cold-storage/_meta.json @@ -0,0 +1,4 @@ +{ + "label": "Cold Storage", + "order": 9.6 +} diff --git a/src/content/docs/docs/platform/settings/cold-storage/archive-logs-to-s3.mdx b/src/content/docs/docs/platform/settings/cold-storage/archive-logs-to-s3.mdx new file mode 100644 index 00000000..76e59200 --- /dev/null +++ b/src/content/docs/docs/platform/settings/cold-storage/archive-logs-to-s3.mdx @@ -0,0 +1,145 @@ +--- +title: "Archive logs to S3" +description: "Connect an S3 bucket in your AWS account so KloudMate exports the workspace's logs to it every hour as Parquet." +sidebar: + label: "Archive logs to S3" + order: 2 +--- + +import { Steps } from '@astrojs/starlight/components'; + +Connect an S3 bucket in your AWS account, and KloudMate exports the workspace's logs to it every hour as Parquet files. You can then [rehydrate](../rehydrate-logs/) any archived range to search it in KloudMate, or query the files directly with your own tools. You need the organization **Owner** role to set this up. + +Archiving is set up per workspace, so connect a bucket for each workspace you want to archive. + +## Before you start + +- **An existing S3 bucket.** KloudMate doesn't create the bucket. If you don't have one yet, create it in S3 first, and note its region: you'll need to select that exact region in KloudMate. +- **No Object Lock on the KloudMate folder.** KloudMate writes under `/kloudmate/` and rewrites an hour's files when it re-exports that hour, so Object Lock must not apply to that folder. +- **The KMS key ARN, if the bucket uses SSE-KMS.** KloudMate writes with the bucket's default encryption. If that's SSE-KMS, have the key's ARN ready. It must be a key ARN (`arn:aws:kms:::key/`), not an alias. +- **Permission to create a CloudFormation stack that creates an IAM role.** The stack needs the `CAPABILITY_IAM` acknowledgment. +- **A log retention of at least 7 days.** KloudMate doesn't export anything for an organization whose log retention is shorter. + +The bucket, region, and prefix can't be changed after you connect. To move the archive somewhere else, [replace the bucket](#replace-the-bucket). + +## Connect the bucket + + + +1. Open **Settings → Cold Storage** and, on the **Archive** tab, click **Connect S3 bucket**. + +2. Enter the bucket details, then click **Continue**: + + - **Bucket name**: the existing bucket, for example `my-log-archive`. + - **Bucket region**: the bucket's own region. + - **Prefix (optional)**: a folder inside the bucket, without leading or trailing slashes, for example `observability/logs`. KloudMate writes under `/kloudmate/`, or under `kloudmate/` at the bucket root if you leave it empty. + - **KMS key ARN (optional)**: only if the bucket encrypts with SSE-KMS. + - **Backfill logs already stored**: on by default. It also exports the logs already in the workspace's retention, not only new ones. Turn it off to start the archive from now. + + ![The Connect S3 bucket dialog on the Bucket step, with Bucket name, Bucket region, and Prefix](./images/connect-bucket.png) + +3. Click **Open AWS Console**. The CloudFormation quick-create page opens in the bucket's region, with the template, bucket, prefix, KMS key, and external ID filled in. Leave **ExternalId** as it is. + + If a `kloudmate-archive` stack already exists in that region, see [If a kloudmate-archive stack already exists](#if-a-kloudmate-archive-stack-already-exists) instead. + +4. In the AWS console, acknowledge that the stack creates IAM resources and click **Create stack**. When the stack finishes, copy the **RoleArn** value from its **Outputs** tab. Back in KloudMate, click **Continue**. + +5. Paste the ARN into **Role ARN** and click **Verify**. + + Verify does what the export will do: it assumes the role, writes a small test file to `/kloudmate/_verify/check.txt`, reads it back, and lists the folder. If it fails, the message tells you what to fix. See [Troubleshoot Cold Storage](../troubleshooting/). + +6. When verification passes, click **Done**. + + + +The first hour is exported about 2 hours after it ends. With backfill on, the older hours are exported after that, which can take a while for a workspace that stores a lot of logs. The **Coverage** table shows how far it has got. + +## What the role can do + +The stack creates one IAM role and nothing else. The role trusts KloudMate's AWS account, and only when the request carries your workspace ID as the external ID. Its permissions cover only `/kloudmate/` in your bucket: + +| Permission | Used to | +| --- | --- | +| `s3:PutObject` | Write the Parquet files and the verification file | +| `s3:GetObject` | Read files back for verification and rehydration | +| `s3:ListBucket`, limited to the prefix | List the folder | +| `s3:AbortMultipartUpload`, `s3:ListMultipartUploadParts` | Complete or clean up large uploads | +| `s3:RestoreObject` | Restore objects from Glacier storage classes. Rehydration doesn't use it yet. | +| `kms:GenerateDataKey`, `kms:Decrypt` | Encrypt and decrypt with your KMS key. Added only if you entered a KMS key ARN. | + +The role has no delete permission, so KloudMate can't remove anything from your bucket, including files it wrote. Archived files are removed only by you or by the bucket's lifecycle rules. + +## If a kloudmate-archive stack already exists + +A stack name can be used only once per region, so the quick-create link fails if an earlier attempt left a `kloudmate-archive` stack there. What to do depends on whether that stack's bucket holds archived logs you still want to rehydrate: + +- **It doesn't**, for example an attempt that never verified. Update the existing stack instead of creating a new one. In CloudFormation, select **kloudmate-archive**, click **Update**, choose **Use current template**, set **BucketName**, **Prefix**, and, if you use one, **KmsKeyArn** to the new bucket's values, and finish the update. The role ARN stays the same, so paste it into KloudMate and verify. +- **It does.** Keep that stack, since rehydrating from the old bucket still uses its role. On the quick-create page, change **Stack name** to something new, such as `kloudmate-archive-2`, and create the stack. + +## What gets written + +Each hour of logs is written to its own folder: + +```text +/kloudmate/v1/workspace=/signal=logs/dt=/hour=/part-.parquet +``` + +- Dates and hours are in UTC. A busy hour is split across several `part-.parquet` files. +- The `dt=` and `hour=` folders are Hive-style partitions, so Amazon Athena, AWS Glue, and other Parquet tools can query the archive and skip the hours they don't need. +- The layout is stable. You can point Athena tables and lifecycle rules at it. New columns may be added to the files over time, but existing paths don't change. +- An hour with no logs gets no files. + +### When logs are exported + +- Each hour is exported about 2 hours after it ends, so logs that arrive a little late are included. +- Once a day, KloudMate checks the last 3 days and re-exports any hour whose log count has changed since it was exported. Logs that arrive more than 3 days late aren't archived. +- With backfill on, the archive also covers the logs that were already in the workspace's retention when you connected. With it off, the archive starts from the hour you connected. + +## Check archive status + +The **Archive** tab shows the connected bucket and its state: + +- **Archiving**: verified and exporting. +- **Not working**: KloudMate can't write to the bucket. Exports are paused, and the reason is shown with its fix. See [Troubleshoot Cold Storage](../troubleshooting/). +- **Setup incomplete**: the role hasn't been verified yet. Click **Finish setup** to continue where you left off. +- **Disabled**: exports are turned off. + +![The Archive tab: the connected bucket marked Archiving, a previous bucket, and the Coverage table](./images/archive-tab.png) + +### Coverage + +The **Coverage** table shows the export status of each UTC day for the last 30, 90, or 400 days: + +- **Archived**: every hour with logs has been exported. +- **In progress**: some hours are still waiting to be exported. +- **No logs**: the day had no logs. +- **Failed**: some hours failed to export. KloudMate retries them once a day while they're still within the workspace's retention. +- **Expired**: some hours left retention before they could be exported. They aren't in the archive. + +A day shows its worst status, so one failed hour marks the whole day as **Failed**. + +## Manage the bucket + +**Replace…**, **Disable**, and **Delete** are in the bucket's **More options** menu. + +### Verify or change the role + +When the bucket shows **Not working**, fix the cause in AWS and click **Verify**. Exports resume when it passes. Click **Change role** to use a different role ARN. + +### Replace the bucket + +Click **Replace…** to connect a different bucket, region, or prefix. It's available while exports are enabled. You go through the same steps as a new connection. New hours are exported to the new bucket, and the old one moves to **Previous buckets**. + +![The Replace bucket dialog on the Create the role step, with the Open AWS Console button](./images/create-role-step.png) + +You can still rehydrate logs from a previous bucket as long as its role keeps working, so leave the old stack as it is. If the new stack goes in the same region, give it a new name, as described in [If a kloudmate-archive stack already exists](#if-a-kloudmate-archive-stack-already-exists). + +### Disable or enable exports + +Click **Disable** to stop exports. When you click **Enable** again, KloudMate exports the hours it missed, as long as they're still within the workspace's retention. You can still rehydrate the logs already archived while exports are disabled. + +### Delete the archive + +Click **Delete** to stop archiving the workspace and remove the bucket connection. KloudMate also deletes its record of every hour it exported, to this bucket and to any previous ones, and without that record those logs can't be rehydrated. Rehydrations that are still loading are cancelled. Ready ones stay searchable until they expire. + +The files stay in your bucket, because KloudMate has no permission to delete them. You can still query them with your own tools. To remove the role too, delete the CloudFormation stack in AWS. diff --git a/src/content/docs/docs/platform/settings/cold-storage/images/archive-tab.png b/src/content/docs/docs/platform/settings/cold-storage/images/archive-tab.png new file mode 100644 index 00000000..1d7716e7 Binary files /dev/null and b/src/content/docs/docs/platform/settings/cold-storage/images/archive-tab.png differ diff --git a/src/content/docs/docs/platform/settings/cold-storage/images/connect-bucket.png b/src/content/docs/docs/platform/settings/cold-storage/images/connect-bucket.png new file mode 100644 index 00000000..3ab6f5f0 Binary files /dev/null and b/src/content/docs/docs/platform/settings/cold-storage/images/connect-bucket.png differ diff --git a/src/content/docs/docs/platform/settings/cold-storage/images/create-role-step.png b/src/content/docs/docs/platform/settings/cold-storage/images/create-role-step.png new file mode 100644 index 00000000..d8c287a5 Binary files /dev/null and b/src/content/docs/docs/platform/settings/cold-storage/images/create-role-step.png differ diff --git a/src/content/docs/docs/platform/settings/cold-storage/images/log-explorer-rehydration.png b/src/content/docs/docs/platform/settings/cold-storage/images/log-explorer-rehydration.png new file mode 100644 index 00000000..5ed2e3a0 Binary files /dev/null and b/src/content/docs/docs/platform/settings/cold-storage/images/log-explorer-rehydration.png differ diff --git a/src/content/docs/docs/platform/settings/cold-storage/images/rehydrate-dialog.png b/src/content/docs/docs/platform/settings/cold-storage/images/rehydrate-dialog.png new file mode 100644 index 00000000..d2b2ed53 Binary files /dev/null and b/src/content/docs/docs/platform/settings/cold-storage/images/rehydrate-dialog.png differ diff --git a/src/content/docs/docs/platform/settings/cold-storage/images/rehydrate-estimate.png b/src/content/docs/docs/platform/settings/cold-storage/images/rehydrate-estimate.png new file mode 100644 index 00000000..c24d2dbb Binary files /dev/null and b/src/content/docs/docs/platform/settings/cold-storage/images/rehydrate-estimate.png differ diff --git a/src/content/docs/docs/platform/settings/cold-storage/images/rehydrations-tab.png b/src/content/docs/docs/platform/settings/cold-storage/images/rehydrations-tab.png new file mode 100644 index 00000000..d876442e Binary files /dev/null and b/src/content/docs/docs/platform/settings/cold-storage/images/rehydrations-tab.png differ diff --git a/src/content/docs/docs/platform/settings/cold-storage/index.mdx b/src/content/docs/docs/platform/settings/cold-storage/index.mdx new file mode 100644 index 00000000..d8057d34 --- /dev/null +++ b/src/content/docs/docs/platform/settings/cold-storage/index.mdx @@ -0,0 +1,55 @@ +--- +title: "Cold Storage" +description: "Archive a workspace's logs to your own S3 bucket, and load them back into KloudMate when you need to search them." +sidebar: + label: "Overview" + order: 1 +--- + +Cold Storage keeps a copy of your logs in an S3 bucket in your own AWS account, long after they've left KloudMate's log retention. When you need older logs again, you rehydrate a time range: KloudMate loads it back so you can search it in the Log Explorer for 7 or 14 days. + +Use it when you need logs older than your plan's retention, for example to: + +- keep a year or more of logs at S3 storage prices +- answer an audit or compliance request +- investigate an incident or security event that started before your retention window + +Cold Storage archives logs only. Traces and metrics aren't archived. It's available on paid plans. + +## How it works + +1. **Archive.** An organization Owner connects an S3 bucket for the workspace. Every hour, KloudMate exports that hour's logs to the bucket as Parquet files. The files belong to you: they stay in your bucket, and you can query them with your own tools, such as Amazon Athena. +2. **Rehydrate.** When you need logs from the archive, pick a time range and, optionally, the services you're interested in. You see an estimate of the size, load time, and charge before anything starts. +3. **Search.** In the Log Explorer, switch the source from **Live logs** to the rehydration and search it like any other logs. When it expires, KloudMate removes the rehydrated logs. The files in your bucket aren't affected. + +## Where to find it + +Open **Settings → Cold Storage**. + +- **Archive**: connect the bucket, check that exports are working, and see which days are in the archive. Only organization Owners can use this tab. +- **Rehydrations**: start a rehydration, follow its progress, and see when each one expires. + +In the Log Explorer, the **Live logs** button switches between live logs and the rehydrations you can search. See [Search a rehydration](./rehydrate-logs/#search-a-rehydration-in-the-log-explorer). + +## Who can do what + +| Action | Owner | Developer | Viewer | +| --- | --- | --- | --- | +| Connect, verify, replace, disable, or delete the bucket | Yes | No | No | +| See the archive status and coverage | Yes | No | No | +| See rehydrations and search them in the Log Explorer | Yes | Yes | Yes | +| Start, cancel, or delete a rehydration | Yes | Yes | No | + +**Owner** is the organization role. **Developer** and **Viewer** are workspace roles. + +## Pricing + +- **Archiving** has no KloudMate charge. AWS bills your account for storing the files in S3. +- **Rehydration** costs $0.10 per GB, measured on the volume the workspace originally ingested for the hours you load, not on the compressed Parquet size. For example, loading a day in which the workspace ingested 50 GB of logs costs $5.00. The charge appears on your KloudMate bill as **Log Rehydration**. +- **Reading files out of S3** during a rehydration is billed by AWS as data transfer. The estimate shows the size that will be read from S3, but its charge covers only KloudMate's part, not AWS's. + +## Next steps + +- [Archive logs to S3](./archive-logs-to-s3/): connect a bucket and start exporting. +- [Rehydrate archived logs](./rehydrate-logs/): load a time range back and search it. +- [Troubleshoot Cold Storage](./troubleshooting/): fix a bucket that stopped working or a rehydration that failed. diff --git a/src/content/docs/docs/platform/settings/cold-storage/rehydrate-logs.mdx b/src/content/docs/docs/platform/settings/cold-storage/rehydrate-logs.mdx new file mode 100644 index 00000000..47e49ef9 --- /dev/null +++ b/src/content/docs/docs/platform/settings/cold-storage/rehydrate-logs.mdx @@ -0,0 +1,133 @@ +--- +title: "Rehydrate archived logs" +description: "Load a time range of archived logs from your S3 bucket back into KloudMate and search it in the Log Explorer for 7 or 14 days." +sidebar: + label: "Rehydrate archived logs" + order: 3 +--- + +import { Steps } from '@astrojs/starlight/components'; + +Rehydration loads a time range from your S3 archive back into KloudMate, so you can search logs that are older than your retention in the Log Explorer. A rehydration stays searchable for 7 or 14 days, then KloudMate removes it. + +You need the **Owner** or **Developer** role to start, cancel, or delete a rehydration. Everyone in the workspace, including Viewers, can see rehydrations and search them. + +Only hours that have already been [archived](../archive-logs-to-s3/) can be loaded. The **Coverage** table on the **Archive** tab shows which days are in the archive. + +## Start a rehydration + +You can start one from either place: + +- **Settings → Cold Storage → Rehydrations**: click **Rehydrate**. +- **Log Explorer**: when the selected time range starts before your log retention, a notice offers **Rehydrate**, and the **Live logs** menu shows **Rehydrate this range…**. Both fill in the time range you're looking at. + + + +1. Set the **Range**. One rehydration can cover up to 400 days. For a longer period, start several. KloudMate loads and charges whole UTC hours, so a range that starts at 10:30 includes the whole 10:00 hour. + +2. Optionally, enter **Services** as a comma-separated list, for example `checkout, payments`. Leave it empty to load every service. + +3. Choose how long to **Keep for**: **7 days** or **14 days**. + +4. Enter a **Rehydration name**, for example `Checkout errors, 12 Oct`. It identifies the rehydration in the list and in the Log Explorer. + +5. Enter a **Reason** that says why you need these logs. It's required, and it's shown with the rehydration and in its emails. + + ![The Rehydrate logs dialog with a range, Keep for set to 7 days, a name, and a reason filled in](./images/rehydrate-dialog.png) + +6. Click **Estimate**. Check the estimate, then click **Rehydrate** to start. + + + +The estimate shows the rows and size that will load, how long loading will take, when it will start, how long it stays searchable, and the **Charge**. **Size** shows both the volume originally ingested, which the charge is based on, and the amount read from S3. + +Filtering by service narrows what you load and search, but it doesn't reduce the estimate or the charge. Both cover every service in each hour, because the archive is read an hour at a time. + +![The estimate review, with Rows, Size, Load time, Queue, and Charge](./images/rehydrate-estimate.png) + +## Limits + +The estimate checks these limits and tells you which one a request breaks: + +- **Range**: at most 400 days per rehydration. +- **Size**: at most 1,000 GB of ingested logs per rehydration. Split a larger range into several rehydrations. +- **Active rehydrations**: at most 2 per organization, across all its workspaces. A rehydration counts while it's queued, loading, or paused, and after it's ready until it expires or you delete it. +- **Overlap**: a new rehydration can't overlap the range of an active one in the same workspace. +- **Archived hours**: if nothing in the range has been exported yet, there's nothing to load. +- **Capacity**: if KloudMate can't load the range right now, try a shorter range or try again later. + +## How long it takes + +Rehydration isn't instant. The estimate's **Load time** and **Queue** rows show roughly how long it will take and when it will start. Treat them as a guide, since the actual time can be longer. + +You can search a rehydration as soon as its first hour has loaded. While it's still loading, the Log Explorer shows how many hours have loaded so far. + +When a rehydration is ready, or if it fails, KloudMate emails the person who requested it and the organization's Owners. + +## Rehydration status + +The **Rehydrations** tab lists every rehydration in the workspace, newest first, with its status: + +| Status | What it means | +| --- | --- | +| **Queued** | Waiting for its turn. The **Progress** column shows how many are ahead. | +| **Loading** | Loading hour by hour. Searchable once the first hour has loaded. | +| **Paused** | Waiting for capacity to load the rest. It resumes on its own, and the hours already loaded stay searchable. | +| **Ready** | Finished, and searchable until it expires. If some hours failed to load, **Progress** shows how many. Hover over the count to see why. | +| **Failed** | No hour could be loaded. Hover over the status to see why. | +| **Cancelled** | Stopped before it finished. It's no longer searchable. | +| **Expired** | Past its keep-for period. It's no longer searchable. | + +![The Rehydrations tab with a Ready rehydration, showing its range, progress, and expiry, and Open in Log Explorer](./images/rehydrations-tab.png) + +## Search a rehydration in the Log Explorer + + + +1. Open the rehydration in either of these ways: + + - On the **Rehydrations** tab, click **Open in Log Explorer**. + - In the Log Explorer, click **Live logs** and select the rehydration under **Rehydrated**. + +2. Search and filter as you would with live logs. The time range is set to the rehydration's range, and a banner shows the range and when it expires. Only logs from this rehydration are searched. + +3. Click **Back to live** when you're done. + + + +Rehydrated logs are searchable only in the Log Explorer. Dashboards, alerts, and other parts of KloudMate keep reading live data. Streaming and **Jump to Now** aren't available while you're reading a rehydration, since it covers a fixed range in the past. + +![The Log Explorer reading a rehydration named audit, with the banner showing its range, when it expires, and Back to live](./images/log-explorer-rehydration.png) + +## Expiry + +The keep-for period is counted from when loading starts plus the estimated load time, that is, from about when loading finishes. The **Expires** column shows the exact time. After that, the rehydration shows **Expired**, and its logs are removed from KloudMate. + +Expiry doesn't touch the files in your bucket. To search the same range again, start a new rehydration. It's charged again. + +## Cancel or delete a rehydration + +Open the rehydration's **More options** menu on the **Rehydrations** tab: + +- **Cancel** stops a rehydration that's queued, loading, or paused. The logs loaded so far are no longer searchable. +- **Delete** removes a rehydration and its logs from KloudMate right away. Deleting a ready rehydration frees its place in the organization's limit of 2 active rehydrations. + +Either way, the hours that already loaded are still charged. + +## Pricing + +Rehydration costs $0.10 per GB, measured on the volume the workspace originally ingested for each hour that loads. Each hour is charged as it loads, so hours that fail to load aren't charged. The charge appears on your KloudMate bill as **Log Rehydration**. + +AWS bills your account separately for the data read out of S3. That cost isn't part of the estimate's **Charge**, but the estimate shows the size read from S3, so you can work it out from AWS's data transfer pricing. + +## Logs in Glacier storage classes + +Rehydration reads the archived files directly from your bucket. It can't read files that a lifecycle rule has moved to S3 Glacier Flexible Retrieval or S3 Glacier Deep Archive, or to the Archive Access or Deep Archive Access tier of S3 Intelligent-Tiering. Files in S3 Standard, Glacier Instant Retrieval, and the other storage classes load directly. + +The estimate doesn't check storage classes, so it still counts those hours in its rows and charge. When loading reaches one of them, that hour fails right away with a reason saying its files are in a Glacier storage class. You aren't charged for it, and the rest of the range keeps loading. + +To search those hours, restore their files in S3, then rehydrate the range again. A new rehydration can't overlap an active one in the same workspace, so delete the first rehydration or wait for it to expire. + +:::tip +If you might want to rehydrate logs later, keep them in S3 Standard or Glacier Instant Retrieval. Glacier Instant Retrieval costs less to store and loads without a restore. +::: diff --git a/src/content/docs/docs/platform/settings/cold-storage/troubleshooting.mdx b/src/content/docs/docs/platform/settings/cold-storage/troubleshooting.mdx new file mode 100644 index 00000000..c788ba16 --- /dev/null +++ b/src/content/docs/docs/platform/settings/cold-storage/troubleshooting.mdx @@ -0,0 +1,104 @@ +--- +title: "Troubleshoot Cold Storage" +description: "Fix an S3 archive bucket that fails verification or stops working, and find out why a rehydration hour didn't load." +sidebar: + label: "Troubleshooting" + order: 4 +--- + +When KloudMate can't write to your bucket, the **Archive** tab marks it **Not working** and shows the reason, including what to change. Verify shows the same reasons during setup. + +## Verification and export errors + +After you fix the cause in AWS, click **Verify** on the **Archive** tab. Exports resume when verification passes. + +### The bucket does not exist + +KloudMate couldn't find a bucket with that name. KloudMate doesn't create the bucket, so create it in S3 in the region you selected, then verify again. Also check the bucket name for typos. + +### The bucket is not in the selected region + +The bucket exists, but in a different region from the one you selected. The region can't be changed on an existing connection, so click **Replace…** and connect the same bucket again with its own region. + +### KloudMate could not assume the role + +KloudMate's request to assume the role was refused. Check that: + +- the **Role ARN** you pasted is the **RoleArn** output of the stack +- the stack finished creating, and the role still exists +- the role's trust policy still has your workspace ID as the external ID. The stack sets it from the **ExternalId** parameter, so recreate the stack from the **Open AWS Console** link if that parameter was changed. + +To use a different role, click **Change role** and paste its ARN. + +### The role was assumed, but S3 refused access + +The role works, but S3 refused to let it write, read, or list under `/kloudmate/`. + +The usual cause is a role from an earlier attempt whose stack still names a different bucket or prefix. In CloudFormation, update the `kloudmate-archive` stack with **Use current template**, set **BucketName** and **Prefix** to the bucket and prefix you connected in KloudMate, and verify again. The role ARN stays the same. + +If that stack's old bucket holds archived logs you still want to rehydrate, don't update it. Create a new stack under another name instead, as described in [If a kloudmate-archive stack already exists](../archive-logs-to-s3/#if-a-kloudmate-archive-stack-already-exists). + +If the stack's values are already correct, look for something else denying access: + +- a bucket policy with a `Deny` that covers the role or the prefix +- an AWS Organizations service control policy or a permissions boundary that limits the role + +### The KMS key refused KloudMate access + +The bucket encrypts with SSE-KMS, and the key didn't let the role encrypt or decrypt. Set the stack's **KmsKeyArn** parameter to the ARN of the bucket's default encryption key, and check that the key policy allows the role to use the key. Then verify again. + +### Could not check the bucket + +A message that starts with "Could not check the bucket" means verification hit a problem that isn't a setting on your side, such as a temporary AWS error. Wait a few minutes and click **Verify** again. + +## A bucket that stops working later + +KloudMate can lose access after setup, for example when someone deletes the role or the bucket, edits the bucket policy, or changes the KMS key policy. When that happens: + +- Exports pause, and the bucket shows **Not working** with the reason. +- KloudMate emails the Owner who connected the bucket, with a link to **Settings → Cold Storage**. +- A rehydration that's loading from that bucket stops. The hours it hadn't loaded yet fail with the same reason, and the hours already loaded stay searchable. + +Fix the cause using the sections above, then click **Verify**. Exports resume and catch up on the hours that were missed, as long as they're still within the workspace's log retention. Hours that leave retention before then show as **Expired** in **Coverage** and aren't archived, so fix a broken bucket as soon as you can. + +## Failed or expired days in Coverage + +- **Failed**: some hours on that day failed to export. KloudMate retries them once a day while they're within retention. If the bucket shows **Not working**, fix that first. +- **Expired**: some hours left the workspace's log retention before they could be exported. They can't be recovered. + +## Rehydration problems + +If the estimate refuses a request, it says which limit the request breaks. See [Limits](../rehydrate-logs/#limits). + +To see why individual hours failed, hover over the failed count in the **Progress** column on the **Rehydrations** tab. + +To retry hours that failed, start a new rehydration that covers them. A new rehydration can't overlap an active one in the same workspace, so delete the first one or wait for it to expire. + +### This hour's files are in a Glacier storage class + +A lifecycle rule moved the hour's files to a storage class that KloudMate can't read, such as S3 Glacier Flexible Retrieval or Deep Archive. Restore the files in S3, then retry the hour. See [Logs in Glacier storage classes](../rehydrate-logs/#logs-in-glacier-storage-classes). + +### This hour's archived files are no longer in the bucket + +The hour's files were deleted from the bucket, for example by a lifecycle rule that expires old objects. KloudMate can't load that hour. + +### The hour could not be loaded + +KloudMate tried to load the hour several times and couldn't. Retry the hour later. + +### The archived files for this hour changed after they were written + +Files under that hour's folder were added, removed, or replaced after KloudMate exported it. KloudMate loads an hour only if its files are exactly as it wrote them. Don't add or edit files under `/kloudmate/`. + +### An hour fails with a bucket error + +If an hour fails with one of the [verification errors](#verification-and-export-errors), KloudMate lost access to the bucket during the load. Fix the cause and verify the bucket, then retry the hours that didn't load. + +### The Log Explorer can't search a rehydration + +- **You can search this rehydration once its first hour loads**: it's still queued, or its first hour is loading. +- **You can no longer search this rehydration**: it expired, was cancelled, or was deleted. Start a new one to search the range again. + +## Disabling or deleting keeps your files + +KloudMate's role has no delete permission, so disabling, replacing, or deleting the archive never removes files from your bucket. To remove archived logs, delete them in S3 or set up a lifecycle rule. When you no longer need to rehydrate from a bucket, delete its CloudFormation stack to remove the role. diff --git a/src/content/docs/docs/platform/settings/connections/atlassian.mdx b/src/content/docs/docs/platform/settings/connections/atlassian.mdx new file mode 100644 index 00000000..e8e00a4c --- /dev/null +++ b/src/content/docs/docs/platform/settings/connections/atlassian.mdx @@ -0,0 +1,119 @@ +--- +title: "Atlassian" +description: "Connect the KloudMate Assistant to Jira and Confluence through Atlassian's MCP server, with a scoped API token or a service account API key." +sidebar: + label: "Atlassian" + order: 5 +--- + +import { Steps } from '@astrojs/starlight/components'; + +An Atlassian connection lets the [KloudMate Assistant](../../../../kloudmate-assistant/) read and search Jira and Confluence through Atlassian's Rovo MCP server. It takes one API token, which is one of these: + +- **A personal API token.** You create it in your Atlassian account. The assistant acts as you. +- **A service account API key.** An Atlassian admin creates it for a service account. The assistant acts as that account. + +Both need agent-interface scopes. A token without them still connects and lists the tools, but Atlassian refuses every tool call, and the connection shows **Degraded**. + +## Before you start + +An Atlassian organization admin must allow API tokens for the MCP server. In Atlassian Administration, go to **Rovo** > **Rovo MCP server** > **Authentication** and turn on **Allow API token authentication**. Until then, Atlassian refuses every token and key. + +## Scopes + +A personal API token and a service account API key take the same scopes. Select every scope you need when you create the token, because Atlassian doesn't let you change the scopes of an existing token. + +| To let the assistant | Select | +|---|---| +| Find your Atlassian sites. The connection check needs this. | `read:me`, `read:account` | +| Read and search Jira | `read:jira:agent-interface`, `search:jira:agent-interface` | +| Read and search Confluence | `read:confluence:agent-interface`, `search:confluence:agent-interface` | +| Create and edit Jira issues, add comments, and move issues between statuses | `write:jira:agent-interface` | +| Create and edit Confluence pages | `write:confluence:agent-interface` | + +Select only the `agent-interface` scopes. Classic scopes such as `read:jira-work` don't work with the MCP server. Leave out the `write:` scopes if the assistant must only read. + +## Create a personal API token + + + +1. Open [API tokens](https://id.atlassian.com/manage-profile/security/api-tokens) in your Atlassian account. + +2. Click **Create API token with scopes**. Don't use **Create API token**: a token without scopes can't call any tool. + +3. Enter a name and an expiry date. + +4. Select the **Atlassian MCP V2** app. + +5. Select the [scopes](#scopes) for what the assistant does. + +6. Create the token and copy it. Atlassian shows it only once. + + + +## Create a service account API key + +Only an Atlassian organization admin can do this. + + + +1. In [Atlassian Administration](https://admin.atlassian.com), select your organization. + +2. Select **Directory** > **Service accounts**. If there is no service account yet, create one. + +3. Select the service account, then select **Create credentials** > **API token** > **Next**. + +4. Enter a name and an expiry date, then select **Next**. The key can last from 1 to 365 days. + +5. Select the [scopes](#scopes), then select **Next** > **Create**. + +6. Copy the key. Atlassian shows it only once. + + + +### Give the service account access + +The scopes set what the key can do. The service account's own access sets where it can do it. The assistant can do only what both allow. + +| For the assistant to | Give the service account | +|---|---| +| Read and search Jira | Access to Jira, and a role in each project that lets it browse the project | +| Create and edit issues, add comments, and move issues between statuses | Also the project permissions to create issues, edit issues, add comments and transition issues | +| Read and search Confluence | Access to Confluence, and permission to view each space | +| Create and edit pages | Also permission to add and edit pages in each space | + +A personal API token needs none of this. It has the same access as the account that created it. + +## Connect it in KloudMate + + + +1. Go to **Settings** > **Connections**, or **Assistant** > **MCP Integrations**, and click **Connect**. + +2. Select **Atlassian**. + +3. Paste the personal API token or the service account API key into **API token**. + +4. Click **Connect**. + + + +KloudMate then checks the connection. It lists the tools and calls `getAccessibleAtlassianResources`, a read-only tool that lists the Atlassian sites the credential can reach. When that call works, the connection shows **Verified**. + +The Assistant uses one Atlassian connection. If there are two, the one it doesn't use shows **Not used by the Assistant**. See [One tool server per provider](../../../../kloudmate-assistant/settings/#one-tool-server-per-provider). + +## If the connection shows Degraded + +To see why the last check failed, point to **Degraded** in the connections list. + +| Atlassian says | What to do | +|---|---| +| The session token is missing the scope claim | The token has no scopes. Create a token with **Create API token with scopes**, then put it in the connection with [Edit](../manage-connections/#edit). | +| Insufficient scopes … Required: [read:me, read:account] | The token can't list your Atlassian sites. Create a token that has the scopes in the message, then put it in the connection with [Edit](../manage-connections/#edit). Atlassian doesn't let you change the scopes of an existing token. | +| You don't have permission to connect via API token | The Atlassian organization doesn't allow API tokens for the MCP server. Ask an organization admin to turn it on, as described in [Before you start](#before-you-start). | + +## Related + +- [Providers](../providers/) +- [Manage connections](../manage-connections/) +- [Configure authentication via API token](https://developer.atlassian.com/cloud/rovo-mcp/guides/configuring-authentication-via-api-token/) (Atlassian) diff --git a/src/content/docs/docs/platform/settings/connections/connect-an-account.mdx b/src/content/docs/docs/platform/settings/connections/connect-an-account.mdx index d497e323..8de31614 100644 --- a/src/content/docs/docs/platform/settings/connections/connect-an-account.mdx +++ b/src/content/docs/docs/platform/settings/connections/connect-an-account.mdx @@ -6,19 +6,21 @@ sidebar: order: 2 --- -You need the **Developer** role to connect an account. Open **Settings → Connections** and click **Connect**. +You need the **Developer** role to connect an account. Open **Settings → Connections** and click **Connect**. To connect a tool server for the Assistant without that role, use [Assistant → MCP Integrations](/kloudmate-assistant/settings/#connect-a-tool-server). The picker lists only the providers your organization can connect. If a provider you expect is missing, see [Providers](../providers/#providers-in-your-plan). -![The Connect dialog listing providers such as Atlassian, AWS, Azure, Custom MCP Server, GitHub, Jira, OAuth 2.0, and Slack, each with chips for what it can be used for](./images/connect-provider-picker.png) +![The Connect dialog listing providers such as Atlassian, AWS, Azure, Custom MCP Server, GitHub, Google Workspace, Jira, Microsoft 365 / Teams, and OAuth 2.0, each with chips for what it can be used for](./images/connect-provider-picker.png) Pick a provider and fill in the **Connection name**. How you finish depends on the provider. -A connection made here starts with the provider's usual capability: **Notifications** for Slack and Jira Cloud; **Workflows** for AWS, Azure, a custom MCP server, and OAuth 2.0; and **AI** for the assistant tools. Change what it's used for, and who can use it, afterward under [Edit](../manage-connections/#edit). +A connection made here starts with the provider's usual capability: **Notifications** for Slack and Jira Cloud; **Notifications** and **Meetings** for Microsoft 365 / Teams; **Meetings** for Google Workspace and Zoom; **Workflows** for AWS, Azure, a custom MCP server, and OAuth 2.0; and **AI** for the assistant tools. Change what it's used for, and who can use it, afterward under [Edit](../manage-connections/#edit). ## Connect through OAuth -Slack, Jira Cloud, Sentry, and the Google Cloud providers send you to the provider's own site. Click **Connect** and approve the access there. KloudMate adds the connection when you're done. +Slack, Jira Cloud, Microsoft 365 / Teams, Google Workspace, Sentry, and the Google Cloud providers send you to the provider's own site. Click **Connect** and approve the access there. KloudMate adds the connection when you're done. + +Zoom works the same way, except that you install the KloudMate app in Zoom instead of approving a consent screen. Install it from a Licensed Zoom account, because every war room meeting is hosted by the account that installed the app. The connection records the account you authorized, so you can tell two Slack installs or two Google accounts apart. @@ -26,7 +28,9 @@ The connection records the account you authorized, so you can tell two Slack ins AWS, Azure, GitHub, Atlassian, a custom MCP server, and a generic OAuth 2.0 API take their credentials in a form. Fill in the provider's fields and click **Connect**. -Where a provider supports more than one method, choose one under **Authentication**. AWS offers **Assume Role** (recommended) or **Access keys**; a custom MCP server offers no auth, a bearer token, or an API key in a named header. +Where a provider supports more than one method, choose one under **Authentication**. For example, a custom MCP server offers **None**, **Bearer token**, **Basic auth**, **API key**, or **OAuth**. + +A custom MCP server with **OAuth** is saved as **Pending** first, and then KloudMate opens the server's own sign-in page. KloudMate finds the server's sign-in, registers itself as a client, and uses PKCE, so you don't need a client ID. If the sign-in doesn't finish, the row stays **Pending** with an **Authorize** button beside it. KloudMate checks the credential as soon as you save. If the check fails, it still saves the connection, so you don't lose what you typed, and marks it **Degraded**. Click **Edit** beside it and fix the values. @@ -42,15 +46,15 @@ That matters most for app-event triggers. A Slack account connected from Setting ## Visibility -Every connection is **Workspace** or **Private**. +Every connection is **Workspace** or **Private**. A connection that can be used only for **AI** starts **Private**. Every other connection starts **Workspace**. -**Workspace** is how every connection starts, and it's almost always the right choice. The connection belongs to the workspace, so anyone with the **Developer** role can use and manage it, and it keeps working when whoever connected it leaves. +A **Workspace** connection belongs to the workspace. Anyone in the workspace can use it, anyone with the **Developer** role can manage it, and it keeps working when whoever connected it leaves. -**Private** restricts a connection to its creator. Only a connection whose one capability is **AI** can be private, so use it for a personal GitHub or Google account that you want the assistant to use on your behalf and nobody else's. +**Private** restricts a connection to its creator. Only a connection whose one capability is **AI** can be private, so use it for a personal GitHub or Google account that you want the Assistant to use on your behalf and nobody else's. A private connection is invisible to everyone else: it's absent from their list, and they can't use it even if they know its id. -To make a connection private, set **Visibility** under **Edit**. To share a private connection with the workspace, use **Share with workspace** on its row menu. +To make a connection private, set **Visibility** under **Edit**. To share a private connection with the workspace, use **Share with workspace** on its row menu. Only a **Developer** who created the connection can share it. ## Related diff --git a/src/content/docs/docs/platform/settings/connections/images/connect-provider-picker.png b/src/content/docs/docs/platform/settings/connections/images/connect-provider-picker.png index 328bf4d4..bab0ad2c 100644 Binary files a/src/content/docs/docs/platform/settings/connections/images/connect-provider-picker.png and b/src/content/docs/docs/platform/settings/connections/images/connect-provider-picker.png differ diff --git a/src/content/docs/docs/platform/settings/connections/images/connections-list.png b/src/content/docs/docs/platform/settings/connections/images/connections-list.png index f6588355..3c7d666c 100644 Binary files a/src/content/docs/docs/platform/settings/connections/images/connections-list.png and b/src/content/docs/docs/platform/settings/connections/images/connections-list.png differ diff --git a/src/content/docs/docs/platform/settings/connections/index.mdx b/src/content/docs/docs/platform/settings/connections/index.mdx index 203a9e46..1ff42c79 100644 --- a/src/content/docs/docs/platform/settings/connections/index.mdx +++ b/src/content/docs/docs/platform/settings/connections/index.mdx @@ -6,7 +6,7 @@ sidebar: order: 1 --- -A connection is one credential for one account: an AWS role, a Slack bot install, an Atlassian grant, an MCP server token. Connect the account once, and every workflow step, notification channel, and dashboard query that needs it uses the same connection. +A connection is one credential for one account: a pair of AWS access keys, a Slack bot install, an Atlassian grant, an MCP server token. Connect the account once, and every workflow step, notification channel, and dashboard query that needs it uses the same connection. When a credential changes, you rotate it once. Before you delete a connection, KloudMate lists everything that still uses it, and one switch takes it out of service everywhere. @@ -24,6 +24,7 @@ Each connection is granted for particular uses, and KloudMate checks the grant e |---|---| | **Workflows** | Workflow steps can use this connection, and events from its account can start a workflow. | | **Notifications** | Notification channels can send through it. | +| **Meetings** | [War rooms](/incident-management/war-rooms/) can create a conference call for each incident through it. | | **Data source** | Dashboards and alert rules can query through it. | | **AI** | AI assistants can call it as a tool. | @@ -53,7 +54,7 @@ When a connection isn't working, a banner at the top of the page says how many, A row that needs attention has its fix beside its status: **Authorize** for an approval that never finished, **Reconnect** for a provider you approve at its own site, or **Edit** for one whose credential you type in. -![The Connections list: each row shows its provider, its Can be used for capability badges, and its status, with one row marked Needs reconnect beside a Reconnect action](./images/connections-list.png) +![The Connections list: each row shows its provider, its Can be used for capability badges, and its status, with one row marked Needs reconnect beside a Reconnect action and a second Atlassian and a second GitHub connection marked Not used by the Assistant](./images/connections-list.png) ## In this section diff --git a/src/content/docs/docs/platform/settings/connections/manage-connections.mdx b/src/content/docs/docs/platform/settings/connections/manage-connections.mdx index 305426f6..4981f828 100644 --- a/src/content/docs/docs/platform/settings/connections/manage-connections.mdx +++ b/src/content/docs/docs/platform/settings/connections/manage-connections.mdx @@ -6,7 +6,7 @@ sidebar: order: 4 --- -Every action here is on the row menu in **Settings → Connections**, and all of them need the **Developer** role. A row that needs attention also has its fix beside its status: **Authorize**, **Reconnect**, or **Edit**. +Every action here is on the row menu in **Settings → Connections**. They need the **Developer** role, except on a private connection you created for the Assistant, as [Who can manage a connection](#who-can-manage-a-connection) explains. A row that needs attention also has its fix beside its status: **Authorize**, **Reconnect**, or **Edit**. ## Edit @@ -14,7 +14,7 @@ Every action here is on the row menu in **Settings → Connections**, and all of Rotate a credential the same way: type the new value and save. To keep a stored secret, leave its field blank. When you save a changed credential or setting, KloudMate checks the connection against the provider again. -You can't change the provider or the authentication method after you create a connection, so an AWS connection made with access keys can't switch to a role. Connect a new one instead. +You can't change the provider or the authentication method after you create a connection. For example, a custom MCP server that uses a bearer token can't switch to OAuth. Connect a new one instead. :::caution Changing **Can be used for** can break things that are already working. Removing a capability stops every use that depends on it at the next call. Adding one can mark the connection **Needs reconnect** until you approve the extra permissions, and you can't add a capability the provider is no longer offered for. @@ -26,9 +26,13 @@ Changing **Can be used for** can break things that are already working. Removing A connection that fails the check becomes **Degraded**. That's a warning, not an outage: workflow steps and dashboard queries still run through it. App-event triggers are the exception, and pause until it verifies again. +To see why the last check failed, point to **Degraded** in the connections list. + ## Reconnect -**Reconnect** re-runs the OAuth approval and rotates the credential in place. The connection keeps its id, so every workflow, channel, and dashboard that uses it keeps working. Slack, Jira Cloud, Sentry, the Google Cloud providers, and an OAuth 2.0 connection that uses the authorization-code grant have it. For the other providers, use **Edit** instead. While a connection's first approval is unfinished, the action is called **Authorize**. +**Reconnect** re-runs the OAuth approval and rotates the credential in place. The connection keeps its id, so every workflow, channel, and dashboard that uses it keeps working. Slack, Jira Cloud, Sentry, the Google Cloud providers, a custom MCP server that uses **OAuth**, and an OAuth 2.0 connection that uses the authorization-code grant have it. For the other providers, use **Edit** instead. While a connection's first approval is unfinished, the action is called **Authorize**. + +A custom MCP server's sign-in belongs to its server URL. If you change the URL, reconnect the connection before anything uses it again. **Needs reconnect** shows one of these reasons when you hover over it: @@ -75,6 +79,8 @@ A **workspace** connection belongs to the workspace, so anyone with the **Develo Only the creator can edit or rotate a **private** connection. An organization **Owner** can still delete one, so a workspace isn't stuck when someone leaves. +You don't need the **Developer** role to manage a private connection you created that's used only for **AI**, such as your own GitHub token for the Assistant. You can edit, re-check, reconnect, and delete it. Sharing it with the workspace, or adding another capability, needs the **Developer** role. + When someone leaves the organization or the workspace, their workspace connections keep working and lose their owner. KloudMate deletes their private connections. ## Related diff --git a/src/content/docs/docs/platform/settings/connections/providers.mdx b/src/content/docs/docs/platform/settings/connections/providers.mdx index 38955da5..645a418b 100644 --- a/src/content/docs/docs/platform/settings/connections/providers.mdx +++ b/src/content/docs/docs/platform/settings/connections/providers.mdx @@ -12,12 +12,12 @@ Each provider decides how you supply its credential and what a connection to it | Provider | Credential | Can be used for | |---|---|---| -| **AWS** | Assume Role with a role ARN and external ID (recommended), or an access key pair. A default region comes with it, and a workflow step can override it. | Workflows, AI | +| **AWS** | Access keys: an access key ID and a secret access key. A default region comes with it, and a workflow step can override it. | Workflows, AI | | **Azure** | A service principal: directory (tenant) id, application (client) id, client secret, and a default subscription id. | Workflows | Both allow more than one connection, so a production account and a staging account are two rows. -The IAM policy or Azure role assignment on the credential decides what a step can do with it. There's no second permission list inside KloudMate, so narrow the role itself. Gate risky steps with an [approval](/workflows/approvals-and-input/). +The IAM policy on the AWS keys, or the Azure role assignment, decides what a step can do with the credential. There's no second permission list inside KloudMate, so narrow the policy or the role itself, and create a separate connection with its own keys for each job. Gate risky steps with an [approval](/workflows/approvals-and-input/). :::note This isn't the same as the AWS account you connect for inventory and metrics. That one lives under [Settings → Data Sources](/platform/settings/data-sources/) and doesn't appear in the Connections list. Connect AWS here when a workflow step needs to call an AWS API, or when the AI assistant needs AWS as a tool. @@ -34,23 +34,35 @@ One Slack install serves alert notifications, incident ChatOps, the [Slack workf One Jira credential can access several Atlassian sites, which is why every Jira action and trigger starts with a site picker. +## Meetings + +These create the conference call for an incident [war room](/incident-management/war-rooms/). The war room settings list only connections that can be used for **Meetings**. + +| Provider | Credential | Can be used for | +|---|---|---| +| **Microsoft 365 / Teams** | Microsoft sign-in, as the service account that should own the rooms and meetings. | Notifications, Meetings | +| **Google Workspace** | Google sign-in, as the Workspace account that should own the meeting spaces. | Meetings | +| **Zoom** | An install of the KloudMate app, from a Licensed Zoom account. | Meetings | + +One Microsoft 365 connection covers both the Teams war room chat and its meeting. Meetings hosted by a Basic Zoom account end after 40 minutes, so install the Zoom app from a Licensed account. See [Set up war rooms](/incident-management/war-rooms/setup/#add-a-conference-call) for what each provider adds to a room. + ## Tools and custom endpoints | Provider | Credential | Can be used for | |---|---|---| -| **Custom MCP Server** | A server URL, a transport of Streamable HTTP or SSE, and either no auth, a bearer token, or an API key in a header you name. Extra headers can be sent alongside, for a server that wants a routing header as well as a credential. | Workflows | +| **Custom MCP Server** | A server URL, a transport of Streamable HTTP or SSE, and one **Authentication** option: **None**, **Bearer token**, **Basic auth** (a username and a password or API token), **API key** (sent in a header you choose), or **OAuth** (sign in on the server's own page). Extra headers can be sent alongside, for a server that wants a routing header as well as a credential. | Workflows, AI | | **OAuth 2.0** | Your own authorize and token URLs, scopes, client id, and client secret, with either the client-credentials or the authorization-code grant. | Workflows | Use **OAuth 2.0** to give an [HTTP Request](/workflows/actions/#http-request) step a credential that's refreshed for you and never stored in the workflow definition. ## AI assistant tools -These connect an MCP server that the [KloudMate Assistant](/kloudmate-assistant/) can call as a tool. They can be used for **AI** only, which also makes them the only connections that can be [private](../connect-an-account/#visibility). +These connect an MCP server that the [KloudMate Assistant](/kloudmate-assistant/) can call as a tool. They can be used for **AI** only. Any connection that's used only for **AI** can be [private](../connect-an-account/#visibility), including an AWS or Custom MCP Server connection. | Provider | Credential | |---|---| | **GitHub** | A GitHub token. | -| **Atlassian** | An Atlassian API token. | +| **Atlassian** | A scoped personal API token or a service account API key. See [Atlassian](../atlassian/). | | **Sentry** | OAuth. | | **Google Cloud BigQuery** | Google OAuth. | | **Google Cloud Compute Engine** | Google OAuth. | @@ -59,7 +71,9 @@ These connect an MCP server that the [KloudMate Assistant](/kloudmate-assistant/ | **Google Cloud Monitoring** | Google OAuth. | | **Google Cloud Resource Manager** | Google OAuth. | -AWS can also be used for **AI**, with an access key pair. See the cloud table above. +An Atlassian personal API token needs the agent-interface scopes for the tools you want, such as `read:jira:agent-interface`. A token with only REST API scopes can't call tools. Before either kind of Atlassian credential works, an Atlassian organization admin must allow API tokens under **Atlassian Administration > Rovo > Rovo MCP server > Authentication**. + +AWS and Custom MCP Server can also be used for **AI**. See the tables above. ## Providers in your plan diff --git a/src/content/docs/docs/platform/settings/images/workspaces-create-step1.png b/src/content/docs/docs/platform/settings/images/workspaces-create-step1.png index 2a13aec3..ac938897 100644 Binary files a/src/content/docs/docs/platform/settings/images/workspaces-create-step1.png and b/src/content/docs/docs/platform/settings/images/workspaces-create-step1.png differ diff --git a/src/content/docs/docs/platform/settings/images/workspaces-create-step2-aws.png b/src/content/docs/docs/platform/settings/images/workspaces-create-step2-aws.png index 520ffebe..96ea8a2f 100644 Binary files a/src/content/docs/docs/platform/settings/images/workspaces-create-step2-aws.png and b/src/content/docs/docs/platform/settings/images/workspaces-create-step2-aws.png differ diff --git a/src/content/docs/docs/platform/settings/images/workspaces-create-step2-methods.png b/src/content/docs/docs/platform/settings/images/workspaces-create-step2-methods.png index 0979a755..8416d024 100644 Binary files a/src/content/docs/docs/platform/settings/images/workspaces-create-step2-methods.png and b/src/content/docs/docs/platform/settings/images/workspaces-create-step2-methods.png differ diff --git a/src/content/docs/docs/platform/settings/images/workspaces-create-step2-otel.png b/src/content/docs/docs/platform/settings/images/workspaces-create-step2-otel.png index 79430dc0..ff7662be 100644 Binary files a/src/content/docs/docs/platform/settings/images/workspaces-create-step2-otel.png and b/src/content/docs/docs/platform/settings/images/workspaces-create-step2-otel.png differ diff --git a/src/content/docs/docs/platform/settings/images/workspaces-create-step3-verify.png b/src/content/docs/docs/platform/settings/images/workspaces-create-step3-verify.png index 7940bc12..fafa94c7 100644 Binary files a/src/content/docs/docs/platform/settings/images/workspaces-create-step3-verify.png and b/src/content/docs/docs/platform/settings/images/workspaces-create-step3-verify.png differ diff --git a/src/content/docs/docs/platform/settings/images/workspaces-list.png b/src/content/docs/docs/platform/settings/images/workspaces-list.png index 20c134b9..2664d822 100644 Binary files a/src/content/docs/docs/platform/settings/images/workspaces-list.png and b/src/content/docs/docs/platform/settings/images/workspaces-list.png differ diff --git a/src/content/docs/docs/platform/settings/images/workspaces-rename-delete.png b/src/content/docs/docs/platform/settings/images/workspaces-rename-delete.png index 62649b82..34d9c9c5 100644 Binary files a/src/content/docs/docs/platform/settings/images/workspaces-rename-delete.png and b/src/content/docs/docs/platform/settings/images/workspaces-rename-delete.png differ diff --git a/src/content/docs/docs/platform/settings/images/workspaces-switcher.png b/src/content/docs/docs/platform/settings/images/workspaces-switcher.png index c06a2c3a..8439cd07 100644 Binary files a/src/content/docs/docs/platform/settings/images/workspaces-switcher.png and b/src/content/docs/docs/platform/settings/images/workspaces-switcher.png differ diff --git a/src/content/docs/docs/platform/settings/index.mdx b/src/content/docs/docs/platform/settings/index.mdx index 5ea45c79..b4c33da1 100644 --- a/src/content/docs/docs/platform/settings/index.mdx +++ b/src/content/docs/docs/platform/settings/index.mdx @@ -20,5 +20,6 @@ The Settings area is where you manage workspaces, users, security, billing, and - [Remove an AWS Account](./deleting-your-aws-account-from-kloudmate/) - [Notification Channels](./notification-channels/) - [Connections](./connections/) +- [Cold Storage](./cold-storage/) - [Subscription & Billing](./subscription-billing/) - [Usage Report](./usage-report/) diff --git a/src/content/docs/docs/rum/instrumentation-guide/index.mdx b/src/content/docs/docs/rum/instrumentation-guide/index.mdx index 97ed5579..8771b0ff 100644 --- a/src/content/docs/docs/rum/instrumentation-guide/index.mdx +++ b/src/content/docs/docs/rum/instrumentation-guide/index.mdx @@ -53,6 +53,7 @@ Keys live under **Settings → Ingest Keys**, where you can add one, edit its al - Install and configure the SDK: [Web](./web/), [Android](./android/), [iOS](./ios/), or [React Native](./react-native/). - Record what a visitor accomplished, not only how the app behaved: [Send custom events](./custom-events/). +- Show your original code in browser error stacks instead of minified bundles: [Source maps](../source-maps/). - Look up an option or an attribute: [SDK reference](./sdk-reference/). *** diff --git a/src/content/docs/docs/rum/instrumentation-guide/web.mdx b/src/content/docs/docs/rum/instrumentation-guide/web.mdx index c001cd22..45dd5012 100644 --- a/src/content/docs/docs/rum/instrumentation-guide/web.mdx +++ b/src/content/docs/docs/rum/instrumentation-guide/web.mdx @@ -120,4 +120,5 @@ There's nothing to configure. Loading the current bundle is enough, and the `/v2 - [Send custom events](../custom-events/) - [SDK reference](../sdk-reference/) +- [Source maps](../../source-maps/) - [Correlate RUM traces with OpenTelemetry backends](../../correlate-rum-traces-with-opentelemetry-backends/) diff --git a/src/content/docs/docs/rum/source-maps.mdx b/src/content/docs/docs/rum/source-maps.mdx new file mode 100644 index 00000000..19cc25f5 --- /dev/null +++ b/src/content/docs/docs/rum/source-maps.mdx @@ -0,0 +1,300 @@ +--- +title: "Source maps" +description: "Upload your build's source maps so RUM error stacks show your original files, lines, and source code instead of minified bundles." +sidebar: + order: 6 +--- + +import { Steps, Tabs, TabItem } from '@astrojs/starlight/components'; + +Production JavaScript is minified, so a browser error's stack points at a position such as `index-4f2a.js:1:84210`. Upload the source maps from each build, and the stack trace on the issue page and in the error drawer shows the original file, line, and column instead, along with the lines of source around it. + +You upload the maps from your build, usually in CI, with the `@kloudmate/sourcemaps` package. They're stored in your workspace and never served publicly. Stacks are mapped when you open an error, not when it's reported, so maps you upload after a deploy also apply to errors that build has already reported. + +A map is kept for 90 days after it was last used. Mapping a stack counts as a use, and so does an upload from a later build that contains the same map. Source maps work with the browser SDK and are included in every plan. + +## Create an upload key + +Uploads authenticate with a **Backend** ingest key (`sk_…`). The Frontend key your app sends RUM data with won't work, and neither will a personal API key (`uk_…`). + + + +1. In the workspace your app sends RUM data to, go to **Settings → Ingest Keys** and click **Add New**. + +2. Choose **Ingest Key - Backend**, add a note such as `sourcemaps-ci`, and click **Submit**. + +3. Copy the key and store it as a CI secret named `KM_SOURCEMAPS_TOKEN`. The key is shown only once. + + + +Maps are stored per workspace, so create the key in the same workspace as the app. A key from another workspace uploads the maps where your errors never look for them. + +:::caution +A Backend key can send any telemetry into your workspace from anywhere. Keep it in your CI secret store, and never commit it or put it in client code. +::: + +Then pass it to the build step. For example, in GitHub Actions: + +```yaml +- run: npm run build + env: + KM_SOURCEMAPS_TOKEN: ${{ secrets.KM_SOURCEMAPS_TOKEN }} + KM_RELEASE: ${{ github.sha }} # the same value as the SDK's `version` +``` + +`KM_RELEASE` is optional. See [How a frame finds its map](#how-a-frame-finds-its-map) for when it matters. + +## Install the package + +Add `@kloudmate/sourcemaps` as a dev dependency, so every build runs the version you pinned: + +```bash +npm i -D @kloudmate/sourcemaps +``` + +The package needs Node.js 18.17 or later and is ESM-only. A CommonJS config can `require()` it on Node.js 20.19, 22.12, or later. + +## Upload from your build + +Use the plugin for your bundler if there is one. For any other bundler, run the CLI after the build. + +### Vite and Rollup + +```ts +// vite.config.ts +import { defineConfig } from 'vite'; +import { kloudmateSourcemaps } from '@kloudmate/sourcemaps/vite'; + +export default defineConfig({ + build: { sourcemap: 'hidden' }, + plugins: [kloudmateSourcemaps({ app: 'shop-web' })], +}); +``` + +`sourcemap: 'hidden'` writes the maps without adding a `sourceMappingURL` comment to your scripts, so no deployed script points at a map that isn't there. The same plugin works in a Rollup config. + +### webpack and Rspack + +```js +// webpack.config.mjs or rspack.config.mjs +import { kloudmateSourcemaps } from '@kloudmate/sourcemaps/webpack'; + +export default { + mode: 'production', + devtool: 'hidden-source-map', + plugins: [kloudmateSourcemaps({ app: 'shop-web' })], +}; +``` + +### What the plugins do + +On a production build, each plugin: + +1. Injects a debug ID into every script and its map. +2. Uploads the maps KloudMate doesn't already have. Rebuilding unchanged code uploads nothing. +3. Removes every `.map` file from the build output, including `.map.gz` and `.map.br` copies, so your source is never deployed. + +Development builds, watch mode, Vite SSR builds, and library builds are left alone. If the upload fails, the plugin logs the error and the build carries on. Set `failOnError: true` to fail the build instead. + +:::caution +The maps are removed even when `KM_SOURCEMAPS_TOKEN` isn't set, so a build made without the token has no maps left to upload. Set `deleteAfterUpload: false` to keep them, for example in a local build you want to inspect. +::: + +### Plugin options + +Both plugins take the same options. + +| Option | Default | Description | +| --- | --- | --- | +| `app` | None, required | Must equal the SDK's `applicationName` exactly. | +| `release` | `KM_RELEASE` | Should equal the SDK's `version`. Frames with a debug ID match without it. | +| `urlPrefix` | Vite's `base` or webpack's `output.publicPath`, else `~/` | The URL path the output directory is served from. `/shop/`, `~/shop`, and `https://cdn.example.com/shop/` all mean the same thing, because `~` stands for any origin. A `publicPath` of `auto` counts as `~/`. | +| `deleteAfterUpload` | `true` | Removes every `.map` file from the output, whether or not it was uploaded. | +| `failOnError` | `false` | Fails the build when KloudMate is unreachable or refuses the upload. | +| `token` | `KM_SOURCEMAPS_TOKEN` | The upload key. Use the environment variable rather than writing the key into config. | +| `url` | `KM_SOURCEMAPS_URL`, else `https://api.kloudmate.com` | The KloudMate API base URL. | + +### Any other bundler: the CLI + +For Next.js, Angular, esbuild, or any bundler without a plugin, run the CLI after the build. Run `inject` first, then `upload`: + +```bash +npx @kloudmate/sourcemaps inject ./dist +npx @kloudmate/sourcemaps upload ./dist --app shop-web --release "$APP_VERSION" --delete-after-upload +``` + +`inject` adds a debug ID to every script in the directory that has a map, and records it in the map. It changes the built scripts, so run it before any step that hashes them, such as subresource integrity or a service worker manifest. `upload` pairs each script with its map through its `sourceMappingURL` comment, or a `.map` file next to it. + +Unlike the plugins, the CLI keeps the maps unless you pass `--delete-after-upload`. + +:::caution +Always call the CLI by its scoped name, `@kloudmate/sourcemaps`, so `npx` runs the version installed in your project. Any other name could resolve to an unrelated npm package, which would then run in your CI with access to your token and your maps. +::: + +#### CLI reference + +```text +npx @kloudmate/sourcemaps inject +npx @kloudmate/sourcemaps upload --app [--release ] [--url-prefix ] + [--delete-after-upload] [--url ] [--strict] +``` + +The key always comes from `KM_SOURCEMAPS_TOKEN`. Without it, `upload` sends nothing. + +| Flag | Description | +| --- | --- | +| `--app` | Required for `upload`. Must equal the SDK's `applicationName` exactly. | +| `--release` | Should equal the SDK's `version`. Defaults to `KM_RELEASE`. | +| `--url-prefix` | The URL path `` is served from, such as `/shop/` or `https://cdn.example.com/shop/`. Defaults to `~/`, the site root. | +| `--delete-after-upload` | Deletes every `.map` file in ``, whether or not it was uploaded, even when no key is set. | +| `--url` | The KloudMate API base URL. Defaults to `KM_SOURCEMAPS_URL`, else `https://api.kloudmate.com`. | +| `--strict` | Exits non-zero when the upload fails. Without it, failures are logged and the command exits 0, so an upload problem never breaks the build. | + +#### Recipes + + + + +Turn on browser source maps. This works for both webpack and Turbopack builds: + +```js +// next.config.js +module.exports = { productionBrowserSourceMaps: true }; +``` + +Then run the CLI on `.next/static`, which is served under `/_next/static`: + +```jsonc +// package.json +"build": "next build && npx @kloudmate/sourcemaps inject .next/static && npx @kloudmate/sourcemaps upload .next/static --app shop-web --url-prefix /_next/static --delete-after-upload" +``` + +With a `basePath`, or a path in `assetPrefix`, add it to the prefix, for example `--url-prefix /shop/_next/static`. Don't use `experimental.sri`, because its hashes are computed before `inject` changes the files. + + + + +In `angular.json`, turn on hidden script maps for the production configuration: + +```jsonc +"configurations": { + "production": { + "sourceMap": { "scripts": true, "styles": false, "hidden": true, "vendor": true } + } +} +``` + +Then run the CLI on the browser output: + +```jsonc +// package.json +"build": "ng build && npx @kloudmate/sourcemaps inject dist/shop-web/browser && npx @kloudmate/sourcemaps upload dist/shop-web/browser --app shop-web --release \"$APP_VERSION\" --delete-after-upload" +``` + +Projects on the older `browser` builder output to `dist/shop-web`, without the `browser` folder. If the app is served under a `baseHref` such as `/shop/`, add `--url-prefix /shop/`. + +If you use `@angular/service-worker`, regenerate its manifest after `inject`, or the service worker rejects the changed files. Run this from the project root, adding the `baseHref` as a third argument if the app has one: + +```bash +ngsw-config dist/shop-web/browser ngsw-config.json +``` + + + + +Build with `sourcemap: 'external'`, which writes the maps without pointing the scripts at them: + +```js +await esbuild.build({ + entryPoints: ['src/main.ts'], + bundle: true, + minify: true, + sourcemap: 'external', + outdir: 'dist', +}); +``` + +Then run `inject` and `upload` on `dist`, as shown above. + + + + +## How a frame finds its map + +Each frame in a stack is matched to an uploaded map by its debug ID when it has one, and otherwise by app, release, and path. + +**By debug ID.** The plugins and `inject` write a debug ID into each script and its map. `@kloudmate/rum-web` 2.6.0 and later reports the debug ID of every script in an error's stack, so each frame matches the map from the exact build that produced it, whatever its release or path. The `/v2/` CDN script picks up new versions on its own. If you install the SDK from npm, upgrade to 2.6.0 or later. + +**By app, release, and path.** A frame with no debug ID is matched on all three: + +- **App:** the `app` you upload with must equal the SDK's [`applicationName`](../instrumentation-guide/sdk-reference/#initoptions) exactly. +- **Release:** the `release` you upload with should equal the SDK's `version`. When several uploads hold a file at the same path, the one from the error's release is used. Without a matching release, the frame maps only if every upload for that path holds the same map. +- **Path:** the URL path the script was served from, on any origin, without the query string. An upload is named from the URL prefix plus the file's path in the output directory, so if you serve your scripts from a subpath such as `/shop/`, set `urlPrefix` or `--url-prefix` to match. + +A frame falls back to this match when the SDK is older than 2.6.0 or the maps were uploaded with the CLI but never injected. With the CLI, it also applies to an error thrown while a script's top-level code first runs, because that happens before the script registers its debug ID. + +Content-hashed file names such as `index-4f2a.js` change whenever their code does, so release matters most for files with fixed names, such as `main.js`. + +### Recommended setup + +- **Build hidden source maps:** `build.sourcemap: 'hidden'` in Vite, `devtool: 'hidden-source-map'` in webpack and Rspack. Your scripts then carry no `sourceMappingURL` comment that points at a map. +- **Keep the maps out of the deploy:** The plugins delete them by default. With the CLI, pass `--delete-after-upload`. +- **Upload in CI:** Run the upload in the same job that builds what you deploy, with the key as a CI secret. The maps then always come from the build that ships. +- **Set a release:** Set the SDK's `version`, and pass the same value as `release` or `KM_RELEASE`. + +## Self-hosted and other regions + +The package uploads to `https://api.kloudmate.com` by default. If your workspace uses a different API host, such as a [KloudMate Infinity](../../platform/infinity/) deployment, set `KM_SOURCEMAPS_URL` in the build, or pass `url` to the plugin or `--url` to the CLI: + +```bash +npx @kloudmate/sourcemaps upload ./dist --app shop-web --url https://api.kloudmate.example.com +``` + +If the build log says `the KloudMate sourcemaps API is not available at `, the URL is wrong or that deployment doesn't serve the source maps API. + +## Troubleshoot unmapped frames + +A frame that couldn't be mapped carries a badge in the stack trace that names the reason. + +### no source map + +No uploaded map matches this file. Frames from third-party scripts you don't build, such as an analytics tag, always show this badge. For your own scripts, check these in order: + +1. The build uploaded maps. A successful upload logs a line such as `[@kloudmate/sourcemaps] shop-web@1.4.0: 12 maps uploaded, 0 already stored`. If the log says `KM_SOURCEMAPS_TOKEN is not set`, add the secret to the build. If it says `no source maps to upload`, turn on source maps in your bundler. A `401` means the key isn't a Backend ingest key. +2. `app` equals the SDK's `applicationName` exactly. +3. The upload key belongs to the workspace your app sends RUM data to. +4. The URL prefix matches the path your scripts are served from. This only matters for frames without a debug ID. +5. The map hasn't expired. A map unused for 90 days is deleted, so rebuild and upload it again. + +### ambiguous release + +Several uploads hold a different map for this path, and none of them was uploaded with the error's release. Set the SDK's `version`, and pass the same value as `release` when you upload. + +### map unavailable + +The map is recorded, but it couldn't be read from storage at that moment. This is temporary, so reopen the error shortly. + +### invalid map + +The stored file isn't a valid source map. Check that your bundler writes valid source map JSON, then rebuild and upload again. An upload skips any map KloudMate already holds byte for byte, so uploading the same file again won't change the result. + +### position not in map + +The map has no entry for this line and column, which means it was built from different code than the script that threw. Upload the maps from the build you deployed, ideally in the same CI job. + +### no source + +The frame maps to its original file, line, and column, but the map doesn't embed the source, so no lines can be shown. Most bundlers embed it by default. It goes missing when it's turned off, for example by webpack's `nosources-source-map` devtool or esbuild's `sourcesContent: false`. + +### KloudMate SDK + +The frame is inside the KloudMate RUM SDK, not your code, so there's nothing to fix. + +*** + +## Related Resources + +- [Web setup](../instrumentation-guide/web/) +- [SDK reference](../instrumentation-guide/sdk-reference/) +- [The Errors tab](../rum-interface/#errors) +- [Ingest Keys](../../platform/settings/api-keys/#ingest-keys) diff --git a/src/content/docs/docs/visualize-data/dashboards/kloudmate-metrics.mdx b/src/content/docs/docs/visualize-data/dashboards/kloudmate-metrics.mdx index 5762da32..30c5967b 100644 --- a/src/content/docs/docs/visualize-data/dashboards/kloudmate-metrics.mdx +++ b/src/content/docs/docs/visualize-data/dashboards/kloudmate-metrics.mdx @@ -22,18 +22,22 @@ All usage metrics have the following attributes: ## **Service Map Metrics** -| Name | Description | Unit | -| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- | -| **KloudMateServiceMapRequests** | Number of requests between a source and target service. | Count | -| **KloudMateServiceMapFailedRequests** | Number of failed requests between a source and target service. A request is considered failed if either the source span or target span's status.code is 2. | Count | -| **KloudMateServiceMapDuration** | Duration of requests between source and target. | Nanoseconds | +These metrics count the calls behind each edge of the [service map](../../../apm-and-tracing/apm-views/service-map/). KloudMate derives them from your traces, including traces from eBPF monitoring. + +| Name | Description | Unit | +| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | ----------- | +| **KloudMateServiceMapRequests** | Number of requests from a source to a target. | Count | +| **KloudMateServiceMapFailedRequests** | Number of failed requests from a source to a target. A request counts as failed if a span on either side of the call has `status.code` 2. | Count | +| **KloudMateServiceMapDuration** | Duration of requests from a source to a target, as a histogram. | Nanoseconds | All service map metrics have the following attributes: -- `source.service.name`: Source service name. -- `target.service.name`: Target service name. -- `source.isError`: A boolean which is true if the source span's status.code is 2. -- `target.isError`: A boolean which is true if the target span's status.code is 2. +- `source_service_name`: The calling service. A caller that sends no telemetry is named after its Kubernetes workload, or `incoming` when it can't be identified. +- `target_service_name`: The called service, or the host name of a dependency that sends no telemetry, such as a database. +- `source_span_kind`, `target_span_kind`: The span kind on each side of the call, such as `Client` and `Server`. +- `source_system`, `target_system`: The kind of call on each side: `http`, `db`, `messaging`, `service` (RPC), or `application`. +- `edge_id`: The edge, as `->`. +- `is_external`: `true` when the target is outside your cluster, such as a third-party API. The service map hides these edges unless **Show external dependencies** is on. *** diff --git a/src/content/docs/docs/workflows/actions.mdx b/src/content/docs/docs/workflows/actions.mdx index 9add52c1..e7cc3eca 100644 --- a/src/content/docs/docs/workflows/actions.mdx +++ b/src/content/docs/docs/workflows/actions.mdx @@ -109,7 +109,7 @@ The step returns AWS's response under `result`, so a later step reads a field as Changing the connection clears the region, changing the service clears the action and the parameters, and changing the action clears the parameters. -The IAM policy on the credential decides what the step is allowed to do. There's no second permission list in KloudMate, so narrow the role itself. Put an [approval](../approvals-and-input/) in front of anything that changes a resource. +The IAM policy on the credential decides what the step is allowed to do. There's no second permission list in KloudMate, so narrow the IAM policy itself. Put an [approval](../approvals-and-input/) in front of anything that changes a resource. ### Azure @@ -287,7 +287,7 @@ The extracted fields are at the top level of the step output, so a later step re Runs the [KloudMate Assistant](/kloudmate-assistant/) on a prompt and returns its answer. It's the same Assistant you chat with, so it can query your telemetry. For example, use it to look into an alert before the workflow posts about it. - **Prompt** says what to find out or do. The Assistant can't ask you questions, so say which service and time range to look at, and what you want back. -- **MCP Integrations** is optional. Pick integrations to give the Assistant their tools, which it runs without asking for approval. Only integrations shared with the workspace are listed. Tools set to **Ask before run** are off. Manage the tools in [Assistant → MCP Integrations](/kloudmate-assistant/settings/#tool-level-settings). +- **MCP Integrations** is optional. Pick integrations to give the Assistant their tools. Only integrations shared with the workspace are listed. Nobody can approve a tool during a run, so the Assistant uses only the enabled tools labeled **Read**, and runs them without asking. Manage the tools in [Assistant → MCP Integrations](/kloudmate-assistant/settings/#choose-the-tools-the-assistant-can-use). For an alert trigger, a prompt could look like this: diff --git a/src/content/docs/docs/workflows/variables.mdx b/src/content/docs/docs/workflows/variables.mdx index cca9d275..2827e283 100644 --- a/src/content/docs/docs/workflows/variables.mdx +++ b/src/content/docs/docs/workflows/variables.mdx @@ -89,4 +89,4 @@ When you [import a workflow](../build-a-workflow/#export-and-import), the import - [Templating](../templating/) for using a variable in a field. - [Storage](../storage/) for values a run writes, rather than values you set. -- [Connections](/platform/settings/connections/) for account credentials, such as an AWS role or a Slack install. +- [Connections](/platform/settings/connections/) for account credentials, such as AWS access keys or a Slack install. diff --git a/src/content/docs/guides/llm/_meta.json b/src/content/docs/guides/llm/_meta.json index 4aad6589..22a0a885 100644 --- a/src/content/docs/guides/llm/_meta.json +++ b/src/content/docs/guides/llm/_meta.json @@ -1,4 +1,4 @@ { "label": "LLM & AI", - "order": 3 + "order": 4 } diff --git a/src/content/docs/guides/llm/anthropic/_meta.json b/src/content/docs/guides/llm/anthropic/_meta.json new file mode 100644 index 00000000..3188ced1 --- /dev/null +++ b/src/content/docs/guides/llm/anthropic/_meta.json @@ -0,0 +1,4 @@ +{ + "label": "Anthropic", + "order": 2 +} diff --git a/src/content/docs/guides/llm/anthropic/send-claude-api-traces.mdx b/src/content/docs/guides/llm/anthropic/send-claude-api-traces.mdx new file mode 100644 index 00000000..1df64de5 --- /dev/null +++ b/src/content/docs/guides/llm/anthropic/send-claude-api-traces.mdx @@ -0,0 +1,220 @@ +--- +title: "Send Claude API Traces to KloudMate" +description: "Send OpenTelemetry traces of your Claude Messages API requests, tool uses, and Managed Agents sessions to KloudMate, from the Claude Console or the Claude SDK." +sidebar: + label: "Claude API and SDK" + order: 1 +--- + +import { Tabs, TabItem } from '@astrojs/starlight/components'; + +The Claude API and the Claude SDK can emit an OpenTelemetry span for each Messages API request, model run, tool use, and Managed Agents turn. Send these spans to KloudMate to debug failed or slow requests, track token usage and latency, and see each Claude call inside your application's own traces. + +You can send traces from two places, and you can use both at once: + +- **API tracing** sends spans from the Claude API itself. You add a destination in the Claude Console, and every request in the workspaces it covers is traced, from any client, with no code change. It's the only way to see each model run inside a request, server tools such as web search, and Managed Agents turns. +- **SDK tracing** sends spans from the Claude SDK in your application, through your own OpenTelemetry exporter. It also traces the client tools that the SDK's tool runner runs, and it nests each Claude call under your application's spans. + +To monitor Claude Code, Claude Desktop, or Cowork instead, see [Send Claude Code and Cowork Telemetry to KloudMate](../send-claude-code-telemetry/). + +:::note +OpenTelemetry export for the Claude API is in beta. Span and attribute names can change between versions, so check the [Claude OpenTelemetry docs](https://platform.claude.com/docs/en/manage-claude/opentelemetry) if a query or dashboard stops matching. +::: + +## Prerequisites + +- A KloudMate workspace API key, from **Settings → Ingest Keys**. +- For API tracing, the **admin** role in your Claude Console organization. With the developer role, you can view destinations but not add them. +- For SDK tracing, the [KloudMate agent](../../../../kloudmate-agent/installation/) on the machine that runs your application, or your API key to send to KloudMate directly. + +## Send API traces from the Claude Console + +The Claude API sends spans over OTLP/HTTP to a public HTTPS endpoint on port 443, so the destination points at KloudMate directly, not at a local KloudMate agent. + +1\. In the Claude Console, open the [OpenTelemetry settings page](https://platform.claude.com/settings/telemetry-destinations) and click **Add destination**. + +2\. Fill in the destination: + +| Field | Value | +| --- | --- | +| **Provider** | **Custom** | +| **Name** | `KloudMate` | +| **OTLP endpoint URL** | `https://otel.kloudmate.com`. Don't add `/v1/traces`, because the API adds it. | +| **Request headers** | Name `Authorization`, value your KloudMate API key, with no `Bearer` prefix. | +| **Content mode** | **Metadata only** or **Full content**. See [Choose a content mode](#choose-a-content-mode). | +| **Workspaces** | **All**, or the Claude workspaces you want to trace. | + +3\. Click **Add destination**. + +4\. In the destination's **More actions** menu, click **Test**. The Console sends an `anthropic.test` span and shows whether it was delivered. + +5\. Send a Messages API request from a workspace the destination covers, for example from the [Console playground](https://platform.claude.com/playground). Within about a minute, the destination's **Delivery** column shows **Healthy**. + +6\. In KloudMate, go to **Traces** and filter by the service `claude-platform`. Every span from the Claude API has this service name. + +You can also add the destination with the Admin API instead of the Console. Use an Admin API key in `ANTHROPIC_API_KEY`: + +```bash +curl --fail-with-body -sS "https://api.anthropic.com/v1/organizations/telemetry_destinations" \ + -H "x-api-key: $ANTHROPIC_API_KEY" \ + -H "anthropic-version: 2023-06-01" \ + -H "anthropic-beta: telemetry-destinations-2026-08-11" \ + -H "content-type: application/json" \ + -d '{ + "endpoint_url": "https://otel.kloudmate.com", + "headers": {"Authorization": "YOUR_KLOUDMATE_API_KEY"}, + "traces": {"content_mode": "metadata_only"}, + "workspaces": {"type": "all"} + }' +``` + +## Send SDK traces from your application + +The Claude SDK emits its spans through the tracer provider that your application registers, so your Claude calls stay the same. Register the tracer provider before the first call you want to trace. + +1\. Install the Claude SDK and the OpenTelemetry SDK with an OTLP exporter: + + + + +```bash +pip install --upgrade anthropic opentelemetry-sdk opentelemetry-exporter-otlp-proto-http +``` + + + + +```bash +npm install @anthropic-ai/sdk@latest @opentelemetry/sdk-node @opentelemetry/exporter-trace-otlp-proto +``` + + + + +2\. Set the service name and the endpoint. The exporter reads both from the environment: + +```bash +export OTEL_SERVICE_NAME="my-claude-app" +export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318" +export ANTHROPIC_API_KEY="YOUR_CLAUDE_API_KEY" +``` + +`http://localhost:4318` is the KloudMate agent on the same machine, which forwards the spans to KloudMate with your workspace's API key. To send to KloudMate directly instead, set the endpoint to `https://otel.kloudmate.com:4318` and add your key: + +```bash +export OTEL_EXPORTER_OTLP_HEADERS="Authorization=YOUR_KLOUDMATE_API_KEY" +``` + +3\. Register a tracer provider with the OTLP exporter, then call Claude as usual: + + + + +```python +from anthropic import Anthropic +from opentelemetry import trace +from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter +from opentelemetry.sdk.trace import TracerProvider +from opentelemetry.sdk.trace.export import BatchSpanProcessor + +provider = TracerProvider() +provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter())) +trace.set_tracer_provider(provider) + +client = Anthropic() + +message = client.messages.create( + model="claude-opus-5-5", + max_tokens=1024, + messages=[{"role": "user", "content": "Hello, Claude"}], +) +print(message.content) +``` + + + + +```ts +import Anthropic from "@anthropic-ai/sdk"; +import { NodeSDK } from "@opentelemetry/sdk-node"; +import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto"; + +const sdk = new NodeSDK({ traceExporter: new OTLPTraceExporter() }); +sdk.start(); + +const client = new Anthropic(); + +const message = await client.messages.create({ + model: "claude-opus-5-5", + max_tokens: 1024, + messages: [{ role: "user", content: "Hello, Claude" }], +}); +console.log(message.content); + +// Flush the spans before a short script exits. +await sdk.shutdown(); +``` + + + + +4\. Run the application. In KloudMate, go to **Traces** and filter by the service `my-claude-app`. Each Claude call appears as an `anthropic.messages.create` span. + +The C#, Go, Java, PHP, and Ruby SDKs work the same way. For their setup, see [Set up OpenTelemetry with the Claude SDK](https://platform.claude.com/docs/en/manage-claude/opentelemetry/sdk). + +:::caution +If your Node.js application also uses `@opentelemetry/auto-instrumentations-node`, create the client with `new Anthropic({ openTelemetry: { propagation: false } })`. Otherwise, the undici instrumentation adds a second `traceparent` header to each request, and the trace context becomes invalid. +::: + +## Connect Claude API spans to your traces + +With both API tracing and SDK tracing on, call the SDK from inside one of your own spans. The SDK sends the trace context to the Claude API in the `traceparent` header, so the API spans nest under the SDK span, which nests under yours. In KloudMate, the request then reads as one trace, from your handler down to each model run and tool use. + +```python +tracer = trace.get_tracer("my-claude-app") + +with tracer.start_as_current_span("answer_question"): + client.messages.create( + model="claude-opus-5-5", + max_tokens=1024, + messages=[{"role": "user", "content": "What's the weather in Paris?"}], + ) +``` + +To group the calls of one conversation, pass a conversation ID to the SDK. It's written to `gen_ai.conversation.id` on every span, so you can filter by it in KloudMate. The ID is exported in both content modes, so don't put personal data or secrets in it. + +## Choose a content mode + +The content mode sets whether spans include prompts and responses: + +- **Metadata only** is the default. Spans carry IDs, timing, status, token counts, and the names of your agents, tools, and MCP servers. +- **Full content** adds the system prompt, the input messages, and the model's response, in `gen_ai.system_instructions`, `gen_ai.input.messages`, and `gen_ai.output.messages`. Use it to read full conversations and see why a run failed or was slow. + +For API tracing, set it on the destination. For SDK tracing, set `ANTHROPIC_OPEN_TELEMETRY_TRACES_CONTENT_MODE=content`. + +With **Full content**, prompts and responses are stored in KloudMate under your workspace's retention settings, so check that this meets your organization's requirements before you turn it on. Each span holds up to 0.75 MB of content by default, and longer content is shortened and marked with `anthropic.content.truncated`. + +## Read the spans + +| Span | Emitted by | What it covers | +| --- | --- | --- | +| `anthropic.messages.create` | SDK (`CLIENT`) and API (`SERVER`) | One Messages API request | +| `anthropic.model_request` | API | One model run inside a request | +| `anthropic.tool_use`, `anthropic.mcp_tool_use` | API | A server tool or an MCP connector tool that the API runs | +| `anthropic.custom_tool_use` | SDK | A client tool that the SDK's tool runner runs | +| `anthropic.session.turn` | API | One Managed Agents turn | + +Token usage is in `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens`, and `gen_ai.usage.input_tokens` already includes prompt cache reads and writes. A request's usage appears on more than one span, so when you chart tokens, sum only the `anthropic.messages.create` spans from one source. If you use both API and SDK tracing, sum the `SERVER` spans. + +For every span and attribute, see the [Claude OpenTelemetry span reference](https://platform.claude.com/docs/en/api/opentelemetry/spans). For what KloudMate shows on an LLM trace, see [Agentic Workflow Observability](/llm-observability/ai-trace-observability/). + +## Troubleshooting + +| Problem | What to check | +| --- | --- | +| The **Delivery** column shows **Failing** with status 401 or 403 | The `Authorization` header doesn't hold a valid KloudMate API key. Edit the destination and enter the key again, with no `Bearer` prefix. | +| The **Delivery** column shows **Failing** with status 404 | The **OTLP endpoint URL** has a path. Set it to `https://otel.kloudmate.com`. | +| The **Delivery** column shows **Disabled** | The API disables a destination after 24 hours of failed deliveries. Fix the cause, then click **Enable** in the destination's **More actions** menu. | +| No API spans in KloudMate | The destination must cover the Claude workspace that the request ran in. Spans can take a minute to arrive. | +| No SDK spans in KloudMate | Register the tracer provider before the first Claude call, and check that `ANTHROPIC_OPEN_TELEMETRY` isn't set to `false`. For a short script, flush the spans before it exits. | +| Each request appears twice | The SDK span and the API span are both named `anthropic.messages.create`. Tell them apart by span kind or by service name. | diff --git a/src/content/docs/guides/llm/anthropic/send-claude-code-telemetry.mdx b/src/content/docs/guides/llm/anthropic/send-claude-code-telemetry.mdx new file mode 100644 index 00000000..32339a64 --- /dev/null +++ b/src/content/docs/guides/llm/anthropic/send-claude-code-telemetry.mdx @@ -0,0 +1,166 @@ +--- +title: "Send Claude Code and Cowork Telemetry to KloudMate" +description: "Send OpenTelemetry metrics, events, and traces from Claude Code in the terminal, the IDE, and Claude Desktop, and events from Cowork, to KloudMate, for one developer or a whole Team or Enterprise organization." +sidebar: + label: "Claude Code and Cowork" + order: 2 +--- + +import { Tabs, TabItem } from '@astrojs/starlight/components'; + +Claude Code and Cowork can export OpenTelemetry data to KloudMate, so you can track sessions, token usage, cost, tool use, and errors for every developer in your organization. Claude Code exports metrics, events, and traces. Cowork, in Claude Desktop, exports events. + +| Where Claude runs | Where you configure it | `service.name` in KloudMate | +| --- | --- | --- | +| Claude Code in the terminal or an IDE | `~/.claude/settings.json`, or managed settings for a team | `claude-code` | +| Claude Desktop, **Code** tab | The same settings as Claude Code | `claude-code-desktop` | +| Claude Desktop, **Cowork** tab | **Organization settings → Cowork** on claude.ai | `cowork` | + +To trace your own application's calls to the Claude API instead, see [Send Claude API Traces to KloudMate](../send-claude-api-traces/). + +## Prerequisites + +- A KloudMate workspace API key, from **Settings → Ingest Keys**. When you roll the configuration out to a team, the key is copied to every developer's machine, so create a dedicated ingest key for it. +- To configure a team, the **Owner** or **Primary Owner** role in a Claude for Teams or Claude for Enterprise organization. + +## Send Claude Code telemetry from your machine + +Claude Code reads its telemetry settings from environment variables. Put them in the `env` block of `~/.claude/settings.json`, so they apply to every Claude Code session on your machine, including the IDE extensions and the **Code** tab in Claude Desktop: + +```json +{ + "env": { + "CLAUDE_CODE_ENABLE_TELEMETRY": "1", + "OTEL_METRICS_EXPORTER": "otlp", + "OTEL_LOGS_EXPORTER": "otlp", + "OTEL_EXPORTER_OTLP_PROTOCOL": "http/protobuf", + "OTEL_EXPORTER_OTLP_ENDPOINT": "https://otel.kloudmate.com:4318", + "OTEL_EXPORTER_OTLP_HEADERS": "Authorization=YOUR_KLOUDMATE_API_KEY" + } +} +``` + +Replace `YOUR_KLOUDMATE_API_KEY` with your KloudMate API key. KloudMate expects the raw key in the `Authorization` header, with no `Bearer` prefix. + +If the [KloudMate agent](../../../../kloudmate-agent/installation/) runs on the same machine, set `OTEL_EXPORTER_OTLP_ENDPOINT` to `http://localhost:4318` and leave out `OTEL_EXPORTER_OTLP_HEADERS`. The agent forwards the data with your workspace's API key. + +Claude Code reads the settings when a session starts, so restart Claude Code, or start a new session in Claude Desktop. Within a minute, the `claude_code.session.count` metric and the `claude_code.user_prompt` events appear in KloudMate under the service `claude-code`, or `claude-code-desktop` for a **Code** tab session. + +:::tip +You can also `export` the same variables in your shell before you run `claude`. Claude Desktop reads only a few variables from your shell profile, so use `settings.json` for the **Code** tab. +::: + +### Turn on traces + +Traces are in beta. Add these variables to the same `env` block to export a trace for each prompt, with a span for each model request and each tool use: + +```json +"CLAUDE_CODE_ENHANCED_TELEMETRY_BETA": "1", +"OTEL_TRACES_EXPORTER": "otlp" +``` + +Each trace starts at a `claude_code.interaction` span for one prompt. Under it, `claude_code.llm_request` spans carry the model, token counts, time to first token, and the `gen_ai.*` attributes, and `claude_code.tool` spans show each tool call, including the time spent waiting for your approval. + +## Roll out to your team + +As an Owner of a Claude for Teams or Claude for Enterprise organization, you can deliver the configuration to every developer from claude.ai, with no change on their machines. + +1\. On claude.ai, go to [**Organization settings → Claude Code → Managed settings**](https://claude.ai/admin-settings/claude-code). + +2\. Add the telemetry settings, with a resource attribute that identifies the team: + +```json +{ + "env": { + "CLAUDE_CODE_ENABLE_TELEMETRY": "1", + "CLAUDE_CODE_ENHANCED_TELEMETRY_BETA": "1", + "OTEL_METRICS_EXPORTER": "otlp", + "OTEL_LOGS_EXPORTER": "otlp", + "OTEL_TRACES_EXPORTER": "otlp", + "OTEL_EXPORTER_OTLP_PROTOCOL": "http/protobuf", + "OTEL_EXPORTER_OTLP_ENDPOINT": "https://otel.kloudmate.com:4318", + "OTEL_EXPORTER_OTLP_HEADERS": "Authorization=YOUR_KLOUDMATE_API_KEY", + "OTEL_RESOURCE_ATTRIBUTES": "department=engineering,team.id=platform" + } +} +``` + +3\. Save. Claude Code picks up the settings the next time each developer starts it. Because the settings set an export endpoint, Claude Code asks each developer to approve them once, in their next interactive terminal session. + +Managed settings take precedence over each developer's own settings, so developers can't point the export somewhere else. They reach sessions that sign in with the organization's Team or Enterprise account. A session that uses an API key from the Claude Console, Amazon Bedrock, Google Cloud, or Microsoft Foundry doesn't fetch them. + +For those sessions, or if you manage devices with MDM, deploy the same JSON as a `managed-settings.json` file instead: + + + + +`/Library/Application Support/ClaudeCode/managed-settings.json`, or the `com.anthropic.claudecode` managed preferences domain in a configuration profile. + + + + +`/etc/claude-code/managed-settings.json` + + + + +`C:\Program Files\ClaudeCode\managed-settings.json`, or a `REG_SZ` value named `Settings` under `HKLM\SOFTWARE\Policies\ClaudeCode` that holds the JSON. + + + + +`OTEL_RESOURCE_ATTRIBUTES` adds its keys to every metric and event, so you can group cost and usage by team in KloudMate. Values can't contain spaces. + +## Send Cowork events + +Cowork exports events, not metrics or traces, and it has its own settings. It's available on the Team and Enterprise plans, and needs Claude Desktop 1.1.4173 or later. + +1\. On claude.ai, go to **Organization settings → Cowork**. + +2\. Fill in the OpenTelemetry fields: + +| Field | Value | +| --- | --- | +| **OTLP endpoint** | `https://otel.kloudmate.com:4318` | +| **OTLP protocol** | HTTP/protobuf | +| **OTLP headers** | `Authorization=YOUR_KLOUDMATE_API_KEY` | + +3\. Save. Cowork reads the settings when a session starts, so the events arrive from the next Cowork session each user starts. + +In KloudMate, Cowork events are logs with the service `cowork`. They include `user_prompt`, `assistant_response`, `api_request` with the model, tokens, and estimated cost, `tool_result`, `tool_decision`, and `api_error`. A shared `prompt.id` links all the events from one prompt. + +## Choose what content is exported + +By default, Claude Code and Cowork export metadata only: counts, durations, models, tokens, cost, and tool names. Prompt text, responses, and tool arguments are redacted. To include them in Claude Code, add these variables to the `env` block: + +| Variable | Adds | +| --- | --- | +| `OTEL_LOG_USER_PROMPTS=1` | Prompt text, and model response text unless you set `OTEL_LOG_ASSISTANT_RESPONSES=0` | +| `OTEL_LOG_TOOL_DETAILS=1` | Tool arguments, such as Bash commands and file paths, and MCP server and tool names | +| `OTEL_LOG_TOOL_CONTENT=1` | Tool output, on trace spans | + +For Cowork, content is set on each device with the `otlpContentCapture` key of the [Claude Desktop managed configuration](https://claude.com/docs/cowork/monitoring#content-capture). + +Even with metadata only, events carry `user.email` when developers sign in with a Claude account, and Cowork events carry the paths of the folders a user connects. Content you turn on is stored in KloudMate under your workspace's retention settings. + +## What you get in KloudMate + +| Signal | Examples | +| --- | --- | +| Metrics | `claude_code.session.count`, `claude_code.token.usage` by `type` and `model`, `claude_code.cost.usage` in USD, `claude_code.lines_of_code.count`, `claude_code.commit.count`, `claude_code.pull_request.count`, `claude_code.active_time.total` | +| Events, as logs | `claude_code.user_prompt`, `claude_code.api_request`, `claude_code.api_error`, `claude_code.tool_result`, `claude_code.tool_decision` | +| Traces | `claude_code.interaction`, `claude_code.llm_request`, `claude_code.tool` | + +Metrics and events carry `user.account_uuid`, `user.email`, `organization.id`, and `session.id`, so you can break cost and token usage down by developer. Claude Code sends metrics with delta temporality every 60 seconds, which KloudMate reads as-is. Cost values are estimates. For billing, use the Claude Console. + +For every metric, event, and attribute, see [Claude Code monitoring](https://code.claude.com/docs/en/monitoring-usage) and [Cowork monitoring](https://claude.com/docs/cowork/monitoring). + +## Troubleshooting + +| Problem | What to check | +| --- | --- | +| No data from Claude Code | Check that `CLAUDE_CODE_ENABLE_TELEMETRY` is `1` and `OTEL_EXPORTER_OTLP_PROTOCOL` is set, because it has no default. Run `claude --debug-file /tmp/claude.log` and search the log for `[3P telemetry]` errors. | +| Export errors with status 401 | `OTEL_EXPORTER_OTLP_HEADERS` doesn't hold a valid KloudMate API key, or the value has a `Bearer` prefix. | +| Managed settings don't apply | Run `claude doctor` and read the `Managed settings (remote)` line. Exporter changes apply only when Claude Code restarts. | +| No data from the Claude Desktop **Code** tab | Put the variables in `~/.claude/settings.json` or managed settings, not in your shell profile. Filter KloudMate by the service `claude-code-desktop`, not `claude-code`. | +| No Cowork events | Start a new Cowork session after you save, and check that Claude Desktop is version 1.1.4173 or later. | diff --git a/src/content/docs/guides/opentelemetry/_meta.json b/src/content/docs/guides/opentelemetry/_meta.json index 964e6009..c5fa266e 100644 --- a/src/content/docs/guides/opentelemetry/_meta.json +++ b/src/content/docs/guides/opentelemetry/_meta.json @@ -1,4 +1,4 @@ { "label": "OpenTelemetry Instrumentation", - "order": 2 -} \ No newline at end of file + "order": 3 +} diff --git a/src/content/docs/guides/opentelemetry/java.mdx b/src/content/docs/guides/opentelemetry/java.mdx index d7bee294..eaf412e6 100644 --- a/src/content/docs/guides/opentelemetry/java.mdx +++ b/src/content/docs/guides/opentelemetry/java.mdx @@ -1,156 +1,187 @@ --- title: "Instrument a Java App" -description: "Learn how to instrument a sample Java application with OpenTelemetry to send data to KloudMate." +description: "Instrument a Java application with the OpenTelemetry Java agent, with no code change, and send its traces, metrics, and logs to KloudMate. Covers a JAR, Apache Tomcat, and Docker." --- -In this guide, we will walk you through the process of setting up and using OpenTelemetry in Java. You will learn how to instrument a simple Java application to produce traces, metrics, and logs. +import { Tabs, TabItem } from '@astrojs/starlight/components'; -**Step 1: Prerequisites** +The OpenTelemetry Java agent instruments a Java application at launch time, with no code change. It attaches to the JVM, traces the frameworks and libraries your application uses, and sends traces, metrics, and logs to KloudMate. You configure it with environment variables. -Before diving into OpenTelemetry, be sure that you have the following installed locally: +## Instrument a Java application -- Java JDK -- [Gradle](https://gradle.org/) +1\. Download the agent JAR. It contains the agent and every automatic instrumentation: -**Step 2: Example Application** - -For this tutorial, we will be using a basic [Spring Boot](https://spring.io/guides/gs/spring-boot/) application. However, OpenTelemetry Java is compatible with other web frameworks like Apache Wicket and Play as well. Feel free to adapt the instructions to your preferred framework. +```bash +curl -L -o opentelemetry-javaagent.jar \ + https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar +``` -**Step 3: Installation** +Earlier versions are on the [releases page](https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases). -1\. Set up an environment in a new directory called `java-simple`. Create a file `build.gradle.kts` in that directory and add the following content to it. +2\. Set the variables that attach the agent and send its data to KloudMate: -```java -plugins { - id("java") - id("org.springframework.boot") version "3.0.6" - id("io.spring.dependency-management") version "1.1.0" -} +```bash +export JAVA_TOOL_OPTIONS="-javaagent:/path/to/opentelemetry-javaagent.jar" +export OTEL_SERVICE_NAME="my-java-app" +export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318" +``` -sourceSets { - main { - java.setSrcDirs(setOf(".")) - } -} +Replace `/path/to` with the folder you downloaded the JAR to, and `my-java-app` with the name you want the service to have in KloudMate. -repositories { - mavenCentral() -} +:::note +`http://localhost:4318` is the [KloudMate agent](../../../kloudmate-agent/installation/) on the same machine. It receives the data and forwards it to KloudMate with your workspace's API key, so the application needs no key of its own. Install the agent first. +::: -dependencies { - implementation("org.springframework.boot:spring-boot-starter-web") -} +3\. Start your application as usual, from the same shell: +```bash +java -jar my-app.jar ``` -2\. Create another file called `DiceApplication.java` in the same directory and add the following content to it: +You can also attach the agent on the command line instead of through `JAVA_TOOL_OPTIONS`: `java -javaagent:/path/to/opentelemetry-javaagent.jar -jar my-app.jar`. + +4\. Check that the agent loaded. The startup output shows lines from `otel.javaagent`, including the agent version. Send a few requests to your application, and the service appears in KloudMate within a minute. -```java -package otel; +## Instrument an app on Apache Tomcat -import org.springframework.boot.SpringApplication; -import org.springframework.boot.Banner; -import org.springframework.boot.autoconfigure.SpringBootApplication; +The Java agent works the same way for an application that runs inside Tomcat, with no code change. The difference is where you attach it: Tomcat starts the JVM, so you add the agent to Tomcat's startup options instead of to a `java -jar` command. The agent then traces every request the server handles. OpenTelemetry tests it on Tomcat 7.0 to 10.1 and TomEE 7.0 to 9.1. -@SpringBootApplication -public class DiceApplication { - public static void main(String[] args) { - SpringApplication app = new SpringApplication(DiceApplication.class); - app.setBannerMode(Banner.Mode.OFF); - app.run(args); - } -} +1\. Download the agent to a path that Tomcat can read, for example `/opt/opentelemetry`: +```bash +sudo mkdir -p /opt/opentelemetry +sudo curl -L -o /opt/opentelemetry/opentelemetry-javaagent.jar \ + https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar ``` -3\. Create another file called `RollController.java` in the same directory and add the following content to it: - -```java -package otel; - -import java.util.Optional; -import java.util.concurrent.ThreadLocalRandom; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import org.springframework.web.bind.annotation.GetMapping; -import org.springframework.web.bind.annotation.RequestParam; -import org.springframework.web.bind.annotation.RestController; - -@RestController -public class RollController { - private static final Logger logger = LoggerFactory.getLogger(RollController.class); - - @GetMapping("/rolldice") - public String index(@RequestParam("player") Optional player) { - int result = this.getRandomNumber(1, 6); - if (player.isPresent()) { - logger.info("{} is rolling the dice: {}", player.get(), result); - } else { - logger.info("Anonymous player is rolling the dice: {}", result); - } - return Integer.toString(result); - } - - public int getRandomNumber(int min, int max) { - return ThreadLocalRandom.current().nextInt(min, max + 1); - } -} +2\. Add the agent and the KloudMate settings to Tomcat's startup options. Where they go depends on how Tomcat was installed: + + + + +Create or edit `/bin/setenv.sh`. Tomcat reads it on every start. +```bash +CATALINA_OPTS="$CATALINA_OPTS -javaagent:/opt/opentelemetry/opentelemetry-javaagent.jar" +export OTEL_SERVICE_NAME="my-tomcat-app" +export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318" ``` -4\. Run the application using the following command then open http\://localhost:8080/rolldice in your web browser to ensure it is working. + + -```java -gradle assemble -java -jar ./build/libs/java-simple.jar +For a Tomcat installed from a package, add the agent to `JAVA_OPTS` in `/etc/tomcat*/tomcat*.conf`, with the settings as `-D` system properties on the same line. + +```bash +JAVA_OPTS="$JAVA_OPTS -javaagent:/opt/opentelemetry/opentelemetry-javaagent.jar -Dotel.service.name=my-tomcat-app -Dotel.exporter.otlp.endpoint=http://localhost:4318" ``` -**Step 4: Instrumentation** + + + +Create or edit `\bin\setenv.bat`: -Now we are going to use Java agent to automatically instrument the application at launch time. We will configure the Java agent using environment variables. +```bat +set CATALINA_OPTS=%CATALINA_OPTS% -javaagent:"C:\opentelemetry\opentelemetry-javaagent.jar" +set OTEL_SERVICE_NAME=my-tomcat-app +set OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 +``` -1\. Download the [opentelemetry-javaagent.jar](https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar) from [Releases](https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases) of the `opentelemetry-java-instrumentation` repository. The JAR file contains the agent and all automatic instrumentation packages: +If Tomcat runs as a Windows service, open `\bin\tomcat*w.exe`, go to the **Java** tab, and add these lines to **Java Options**, one per line: ```txt -https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases +-javaagent:C:\opentelemetry\opentelemetry-javaagent.jar +-Dotel.service.name=my-tomcat-app +-Dotel.exporter.otlp.endpoint=http://localhost:4318 ``` + + -2\. Set and export variables that specify the Java agent JAR and a [console exporter.](https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk-extensions/autoconfigure/README.md#logging-exporter) +Replace `my-tomcat-app` with the name you want the service to have in KloudMate. -```txt -export JAVA_TOOL_OPTIONS="-javaagent:PATH/TO/opentelemetry-javaagent.jar" \ - OTEL_TRACES_EXPORTER=otlp \ - OTEL_METRICS_EXPORTER=otlp \ - OTEL_LOGS_EXPORTER=otlp \ - OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \ - OTEL_EXPORTER_OTLP_ENDPOINT=https://otel.kloudmate.com:4318 \ - OTEL_EXPORTER_OTLP_HEADERS="Authorization=" - +3\. Restart Tomcat. On Linux, for a downloaded install: + +```bash +/bin/shutdown.sh +/bin/startup.sh ``` +For a package install, restart the service instead, for example `sudo systemctl restart tomcat`. + +4\. Check that the agent loaded. Tomcat's log (`/logs/catalina.out` on Linux) shows lines from `otel.javaagent`, including the agent version, when the server starts. Send a few requests to your application, and the service appears in KloudMate within a minute. + :::note - The exporter URL mentioned above, along with the KloudMate API-Key as the Authorization header will send the data to KloudMate. +One Tomcat runs every deployed web application in a single JVM, so all of them report under the one service name you set. To see applications as separate services, run them in separate Tomcat instances. ::: -Replace `PATH/TO` above, with your path to the JAR file. +## Instrument a Java app in Docker + +For a containerized application, add the agent to the image and set the service name and endpoint when the container starts, so the same image works in every environment. -3\. Run your application using the following command: + + -```java -$ java -jar ./build/libs/java-simple.jar -... +Add the agent to your `Dockerfile` and attach it with `JAVA_TOOL_OPTIONS`, which every JVM reads at startup: + +```dockerfile +FROM eclipse-temurin:21-jre +WORKDIR /app +COPY target/my-app.jar app.jar +ADD https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar /otel/opentelemetry-javaagent.jar +ENV JAVA_TOOL_OPTIONS="-javaagent:/otel/opentelemetry-javaagent.jar" +ENTRYPOINT ["java", "-jar", "app.jar"] +``` + + + + +Start from the official Tomcat image, deploy your WAR, and attach the agent through `CATALINA_OPTS`: + +```dockerfile +FROM tomcat:10.1-jre21 +COPY target/my-app.war /usr/local/tomcat/webapps/ROOT.war +ADD https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar /otel/opentelemetry-javaagent.jar +ENV CATALINA_OPTS="-javaagent:/otel/opentelemetry-javaagent.jar" ``` -Note the output from the `otel.javaagent`. + + -4\. Send a request using `curl` from another terminal. You will see trace & log output from the server and client. +Build the image, then set the service name and endpoint when you run it: -```java -curl localhost:8080/rolldice +```bash +docker build -t my-java-app . +docker run --network host \ + -e OTEL_SERVICE_NAME=my-java-app \ + -e OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 \ + my-java-app ``` -5\. Stop the server process to see an output of all the metrics collected. +Inside a container, `localhost` is the container itself. `--network host` puts the container on the host's network, so `localhost:4318` reaches the KloudMate agent on the host, and the application's ports open directly on the host without `-p`. On Docker Desktop for Mac or Windows, use `http://host.docker.internal:4318` as the endpoint instead. + +With Docker Compose, set `network_mode: host` on the service and the same two variables under its `environment:` key. + +:::tip +The `latest` download URL gives you a different agent version each time the image is rebuilt. For repeatable builds, pin a version instead: `https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v/opentelemetry-javaagent.jar`. The versions are listed on the [releases page](https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases). +::: + +If the KloudMate agent is installed on the host or in your Kubernetes cluster, it can attach the Java agent to your services for you. See [Java auto-instrumentation with the KloudMate agent](../../../kloudmate-agent/auto-instrumentation/java/). + +## Configure the agent + +Every setting can be given in three ways. If the same setting is given more than once, the first one in this list wins: + +- **A system property** on the JVM command line, for example `-Dotel.service.name=my-service`. +- **An environment variable**, for example `OTEL_SERVICE_NAME=my-service`. To get a variable's name from a property name, uppercase it and replace every `.` and `-` with `_`. +- **A properties file**, whose path you give in `-Dotel.javaagent.configuration-file` or `OTEL_JAVAAGENT_CONFIGURATION_FILE`. This keeps the settings out of the startup script: + + ```properties + otel.service.name=my-tomcat-app + otel.exporter.otlp.endpoint=http://localhost:4318 + ``` + +The agent sends traces, metrics, and logs over OTLP with `http/protobuf` by default, so the endpoint above is all it needs to reach the KloudMate agent. For every other option, see the OpenTelemetry [agent configuration](https://opentelemetry.io/docs/zero-code/java/agent/configuration/) reference, and check which frameworks and libraries are traced on the [supported libraries](https://opentelemetry.io/docs/zero-code/java/agent/supported-libraries/) page. ## Java Metrics @@ -172,7 +203,3 @@ curl localhost:8080/rolldice | jvm\_memory\_pool\_used | The current memory pool memory usage | | jvm\_memory\_pool\_committed | The amount of memory that is guaranteed to be available for the memory pool | | jvm\_threads\_count | The current number of threads | - -*** - -*Source URL for the example application:* [Getting Started with OpenTelemetry Java](https://opentelemetry.io/docs/languages/java/getting-started/) diff --git a/src/content/docs/guides/profiling/_meta.json b/src/content/docs/guides/profiling/_meta.json index 9e9af091..283a0c77 100644 --- a/src/content/docs/guides/profiling/_meta.json +++ b/src/content/docs/guides/profiling/_meta.json @@ -1,4 +1,4 @@ { "label": "Profiling", - "order": 4 + "order": 5 } diff --git a/src/content/docs/guides/traces/_meta.json b/src/content/docs/guides/traces/_meta.json new file mode 100644 index 00000000..f9ca750d --- /dev/null +++ b/src/content/docs/guides/traces/_meta.json @@ -0,0 +1,4 @@ +{ + "label": "Traces", + "order": 2 +} diff --git a/src/content/docs/guides/traces/images/cloudflare-add-traces-destination.png b/src/content/docs/guides/traces/images/cloudflare-add-traces-destination.png new file mode 100644 index 00000000..c860b072 Binary files /dev/null and b/src/content/docs/guides/traces/images/cloudflare-add-traces-destination.png differ diff --git a/src/content/docs/guides/traces/images/cloudflare-trace-explorer.png b/src/content/docs/guides/traces/images/cloudflare-trace-explorer.png new file mode 100644 index 00000000..f65fc05a Binary files /dev/null and b/src/content/docs/guides/traces/images/cloudflare-trace-explorer.png differ diff --git a/src/content/docs/guides/traces/images/cloudflare-trace-settings.png b/src/content/docs/guides/traces/images/cloudflare-trace-settings.png new file mode 100644 index 00000000..cc1d2a37 Binary files /dev/null and b/src/content/docs/guides/traces/images/cloudflare-trace-settings.png differ diff --git a/src/content/docs/guides/traces/send-cloudflare-traces.mdx b/src/content/docs/guides/traces/send-cloudflare-traces.mdx new file mode 100644 index 00000000..6a7a49a4 --- /dev/null +++ b/src/content/docs/guides/traces/send-cloudflare-traces.mdx @@ -0,0 +1,110 @@ +--- +title: "Send Cloudflare Traces to KloudMate" +description: "Export production request traces from Cloudflare to KloudMate over OTLP/HTTP and view them in Trace Explorer." +sidebar: + label: "Cloudflare Traces" + order: 2 +--- +Export production request traces from Cloudflare to KloudMate with the OpenTelemetry Protocol (OTLP) over HTTP. + +Cloudflare creates the traces from requests that pass through the configured domain. KloudMate receives those traces and displays them in Trace Explorer. + +## Prerequisites + +- A KloudMate workspace. +- Permission to create an **Ingest Key — Backend** in that workspace. +- A Cloudflare account with the target domain configured for Cloudflare Traces. +- The domain traffic proxied through Cloudflare. +- Permission to create an OpenTelemetry destination in Cloudflare. + +For Cloudflare's own prerequisites and domain configuration, see the [Cloudflare Traces documentation](https://developers.cloudflare.com/observability/traces/). + +## Step 1: Create a KloudMate backend ingest key + +In KloudMate, open the workspace that should receive the traces. + +1. Go to **Settings → Ingest Keys**. +2. Select **Add New**. +3. Choose **Ingest Key — Backend**. +4. Add a note, for example `cloudflare-traces`. +5. Create the key and copy it. + +The key is shown only once. Store it securely, and keep it out of source control and screenshots. + +Backend ingest keys authenticate telemetry sent by servers, collectors, and integrations. A frontend ingest key won't work for Cloudflare trace export. See [API Keys](/platform/settings/api-keys/). + +## Step 2: Create a Cloudflare OpenTelemetry destination + +In the Cloudflare dashboard, go to **Observability → OpenTelemetry → Add Destination** and configure it as follows. + +| Field | Value | +| --- | --- | +| Destination type | `Traces` | +| OTLP Traces Endpoint | `https://otel.kloudmate.com:4318/v1/traces` | +| Destination name | `kloudmate-traces` | +| Header name | `Authorization` | +| Header value | `Bearer ` | + +The destination name can be anything, as long as it is easy to identify when you assign it to a domain. + +![The Cloudflare Add New Destination dialog, with Traces selected, the KloudMate OTLP endpoint, and an Authorization header](./images/cloudflare-add-traces-destination.png) + +:::note +Port `4318` is the OTLP/HTTP endpoint. Don't use the OTLP/gRPC endpoint on port `4317` for Cloudflare's OpenTelemetry export. If your account uses a different regional endpoint, use the one KloudMate gives you for that account. +::: + +Select **Save**, then confirm that the destination is enabled. + +## Step 3: Assign the destination to a domain + +Cloudflare destinations are account-level resources, so creating one doesn't export traces from a domain on its own. + +Assign `kloudmate-traces` to the target domain in Cloudflare's trace settings, turn on tracing for the domain, and set the sampling and export options you want. A typical configuration is: + +- **Tracing:** enabled +- **Export destination:** `kloudmate-traces` +- **Default sample rate:** whatever suits your traffic volume + +![Cloudflare trace settings with tracing enabled, a 100% sample rate, and kloudmate-traces selected as the export destination](./images/cloudflare-trace-settings.png) + +Cloudflare owns the domain-level collection, sampling, persistence, and routing settings. KloudMate receives the OTLP payload after Cloudflare exports it. For the current dashboard steps, see [Configure Cloudflare Traces](https://developers.cloudflare.com/observability/traces/configuration/). + +## Step 4: Generate traffic through the domain + +Send a request to the proxied domain: + +```bash +curl -i https://your-domain.example/ +``` + +To generate several test requests: + +```bash +for i in $(seq 1 10); do + curl -sS -o /dev/null "https://your-domain.example/?trace_test=$i" +done +``` + +An ordinary request is enough to trigger trace collection once the domain is enabled and sampling selects the request. There is no separate trace payload to send from the client. + +The request has to reach Cloudflare. DNS-only records, traffic that bypasses Cloudflare, and requests excluded by sampling rules produce no exported trace. + +## Step 5: Verify delivery in Cloudflare + +Return to the Cloudflare destination list and confirm that `kloudmate-traces` is enabled and reporting delivery activity. + +## Step 6: Find the traces in KloudMate + +1. Open **Traces**, or **APM & Services → Traces**. +2. Select the workspace where you created the backend ingest key. +3. Set the time range to cover the test request. +4. Search for the request path, such as `GET /`. +5. Where available, use the Cloudflare Ray ID or the request timestamp to pick out a specific request. + +![KloudMate Trace Explorer listing Cloudflare request traces, with the edge_processing service and their span counts and durations](./images/cloudflare-trace-explorer.png) + +## Troubleshooting + +- **Cloudflare reports `Error`:** check that the endpoint is `https://otel.kloudmate.com:4318/v1/traces` and that the `Authorization` header carries `Bearer ` from the right workspace. +- **No traces in KloudMate:** check that Trace Explorer is on the workspace and time range that go with that ingest key. +- **Only a test trace appears:** search again after real traffic has reached the domain. Sampling and domain-side settings are covered in [Configure Cloudflare Traces](https://developers.cloudflare.com/observability/traces/configuration/). diff --git a/src/content/docs/guides/workflows/_meta.json b/src/content/docs/guides/workflows/_meta.json index 6dfd97a2..8b314d7f 100644 --- a/src/content/docs/guides/workflows/_meta.json +++ b/src/content/docs/guides/workflows/_meta.json @@ -1,4 +1,4 @@ { "label": "Workflows", - "order": 5 + "order": 6 } diff --git a/src/content/docs/guides/workflows/stop-dev-instances-at-night.mdx b/src/content/docs/guides/workflows/stop-dev-instances-at-night.mdx index 7b7279e2..7379fc06 100644 --- a/src/content/docs/guides/workflows/stop-dev-instances-at-night.mdx +++ b/src/content/docs/guides/workflows/stop-dev-instances-at-night.mdx @@ -34,7 +34,7 @@ The workflows pass information to each other through [Storage](/workflows/storag ## Step 1: Let the AWS connection stop and start the instances -Add this policy to the IAM role that your AWS connection uses. Replace `ACCOUNT_ID` with your AWS account ID: +Attach this policy to the IAM user whose access keys your AWS connection uses. Replace `ACCOUNT_ID` with your AWS account ID: ```json { @@ -59,7 +59,7 @@ Add this policy to the IAM role that your AWS connection uses. Replace `ACCOUNT_ The condition limits stopping and starting to instances tagged as development, so the connection can't stop a production instance even if a step asks it to. -If the instances' EBS volumes are encrypted with a customer managed KMS key, the role also needs `kms:CreateGrant` on that key. Without it, an instance goes back to **Stopped** a few seconds after the morning workflow starts it. +If the instances' EBS volumes are encrypted with a customer managed KMS key, the IAM user also needs `kms:CreateGrant` on that key. Without it, an instance goes back to **Stopped** a few seconds after the morning workflow starts it. ## Step 2: Import the workflows diff --git a/src/content/docs/guides/workflows/temporary-ssh-access.mdx b/src/content/docs/guides/workflows/temporary-ssh-access.mdx index 775e279a..9373894a 100644 --- a/src/content/docs/guides/workflows/temporary-ssh-access.mdx +++ b/src/content/docs/guides/workflows/temporary-ssh-access.mdx @@ -37,7 +37,7 @@ Form: Request SSH access to the bastion ## Step 1: Let the AWS connection change the security group -Add this policy to the IAM role that your AWS connection uses. Replace `REGION`, `ACCOUNT_ID`, and the security group ID with your own values: +Attach this policy to the IAM user whose access keys your AWS connection uses. Replace `REGION`, `ACCOUNT_ID`, and the security group ID with your own values: ```json { diff --git a/src/content/docs/guides/workflows/weekly-reliability-report.mdx b/src/content/docs/guides/workflows/weekly-reliability-report.mdx index 3b1fcd16..11fe94f4 100644 --- a/src/content/docs/guides/workflows/weekly-reliability-report.mdx +++ b/src/content/docs/guides/workflows/weekly-reliability-report.mdx @@ -14,8 +14,7 @@ The finished workflow has these steps: ```text Schedule: Mondays at 09:00 ├─ Collect the week's data Parallel -│ ├─ Requests and latency by service MCP Tool Call -│ ├─ Errors by service MCP Tool Call +│ ├─ Traffic and errors by service MCP Tool Call │ ├─ Open incidents MCP Tool Call │ └─ SLOs MCP Tool Call ├─ Write the report AI Prompt @@ -32,7 +31,7 @@ Schedule: Mondays at 09:00 1. Create a [personal API key](/platform/settings/api-keys/#a-personal-key) and copy it. The workflow reads whatever the key's owner can read, so create it from an account that can see every service you want in the report. 2. Copy the ID of this workspace from its URL in the KloudMate app. It starts with `ws_`. -3. Open **Workflows → Connections**, click **Connect**, and pick **Custom MCP Server**. Name the connection `KloudMate MCP`, and fill in the form: +3. Open **Workflows → Connections**, click **Connect**, and pick **Custom MCP Server**. Set **Connection name** to `KloudMate MCP`, and fill in the form: | Field | Value | |---|---| @@ -50,7 +49,7 @@ The personal key works for every workspace its owner belongs to, and `x-workspac 1. Copy the YAML below. 2. Open **Workflows**, click **Import**, paste the YAML, and click **Import**. 3. Click **Open workflow**, and set these before you publish: - - On the four MCP Tool Call steps, pick your `KloudMate MCP` connection. + - On the three MCP Tool Call steps, pick your `KloudMate MCP` connection. - On **Post the report**, pick your Slack connection and the channel. - On the trigger, set **Timezone** to your team's timezone, such as `Europe/Berlin`. @@ -78,25 +77,16 @@ spec: - - id: traffic type: action action: mcp.call - display_name: Requests and latency by service + display_name: Traffic and errors by service connection_id: $input: kloudmate_mcp with: - tool: get_service_requests_count_and_avg_latency + tool: search_traces arguments: - from: now-7d - to: now - - - id: errors - type: action - action: mcp.call - display_name: Errors by service - connection_id: - $input: kloudmate_mcp - with: - tool: get_service_error_count - arguments: - from: now-7d - to: now + groupBy: + - serviceName + window: 7d + view: detail - - id: incidents type: action action: mcp.call @@ -104,9 +94,9 @@ spec: connection_id: $input: kloudmate_mcp with: - tool: search_incidents + tool: check_incidents arguments: - limit: 50 + view: detail - - id: slos type: action action: mcp.call @@ -114,10 +104,9 @@ spec: connection_id: $input: kloudmate_mcp with: - tool: search_slos + tool: check_slos arguments: - enabled: true - limit: 50 + view: detail - id: write type: action action: ai.prompt @@ -127,18 +116,15 @@ spec: Write this week's reliability report for the engineering team, as a Slack message. Use Slack formatting: *bold* for headings and short bullet lists. Keep it under 250 words, and don't include links. Include these sections: - 1. Traffic and errors: the five busiest services, with their request count and error rate. Error rate is the service's errors divided by its requests, as a percentage. + 1. Traffic and errors: the five busiest services, with their request count, error rate, and p95 latency. 2. Open incidents: how many are open, and the three highest-severity ones by title. - 3. SLOs at risk: every SLO that is breached or has less than 25% of its error budget left. + 3. SLOs at risk: every enabled SLO that is breached or has less than 20% of its error budget left. Skip disabled SLOs. - Use only the numbers in the data below. If a section has no data, say so in one line. Latency in the data is in nanoseconds, so show it in milliseconds. + Use only the numbers in the data below. If a section has no data, say so in one line. - Requests and average latency by service: + Traffic and errors by service: {{ steps.traffic.output.result }} - Errors by service: - {{ steps.errors.output.result }} - Open incidents: {{ steps.incidents.output.result }} @@ -167,32 +153,33 @@ inputs: ### Collect the week's data -A **Parallel** step runs four **MCP Tool Call** steps at the same time, one in each branch. Each step calls one tool on the KloudMate MCP server: +A **Parallel** step runs three **MCP Tool Call** steps at the same time, one in each branch. Each step calls one tool on the KloudMate MCP server: | Step | Tool | Arguments | |---|---|---| -| **Requests and latency by service** | `get_service_requests_count_and_avg_latency` | `from` is `now-7d`, and `to` is `now` | -| **Errors by service** | `get_service_error_count` | `from` is `now-7d`, and `to` is `now` | -| **Open incidents** | `search_incidents` | `limit` is `50` | -| **SLOs** | `search_slos` | `enabled` is `true`, and `limit` is `50` | +| **Traffic and errors by service** | `search_traces` | `groupBy` is `serviceName`, `window` is `7d`, and `view` is `detail` | +| **Open incidents** | `check_incidents` | `view` is `detail` | +| **SLOs** | `check_slos` | `view` is `detail` | + +Pick the tool from the **Tool** list, and the step builds the **Arguments** form from the tool's own input schema. A `window` of `7d` covers the seven days before the step runs, so the same step reads the past week every Monday. -Pick the tool from the **Tool** list, and the step builds the **Arguments** form from the tool's own input schema. The tools accept a relative time such as `now-7d`, so the same step reads the last seven days every week. `search_incidents` returns the incidents that aren't resolved yet, and `search_slos` returns each SLO with its latest compliance and error budget. +`search_traces` grouped by `serviceName` returns each service's requests, errors, error rate, and p50, p95, and p99 latency, busiest service first. `check_incidents` without a target returns the incidents that aren't resolved yet, with their severity. `check_slos` returns every SLO, worst first, with its compliance and remaining error budget. It includes disabled SLOs, so the prompt tells the model to skip them. The `detail` view returns more rows than the default `summary` view, which lists only the three busiest services. -Every tool on this server only reads data, so the workflow can't change anything in KloudMate. Each step returns what the tool answered as `result`, so the next step reads it as `{{ steps.errors.output.result }}`. See [MCP Tool Call](/workflows/actions/#mcp-tool-call). +Every tool on this server only reads data, so the workflow can't change anything in KloudMate. Each tool answers with a text summary, and the step returns it as `result`, so the next step reads it as `{{ steps.traffic.output.result }}`. See [MCP Tool Call](/workflows/actions/#mcp-tool-call). ### Write the report -An **AI Prompt** step writes the report. The **Prompt** gives the model the structure to follow and the rules for the numbers, and then the data from the four tools: +An **AI Prompt** step writes the report. The **Prompt** gives the model the structure to follow and the rules for the numbers, and then the data from the three tools: ```liquid -Requests and average latency by service: +Traffic and errors by service: {{ steps.traffic.output.result }} -Errors by service: -{{ steps.errors.output.result }} +Open incidents: +{{ steps.incidents.output.result }} ``` -When a reference sits inside other text, an object or a list renders as JSON, which the model reads well. So the prompt doesn't depend on the exact shape each tool returns, and the model does the arithmetic, such as each service's error rate. +Each tool has already worked out the numbers, such as each service's error rate and its latency in milliseconds, so the model only picks and summarizes them. The prompt asks the model to use only numbers from the data, and to say so when a section has none, which keeps it from filling a gap with a guess. Read the first few reports against the numbers in KloudMate before you rely on them. @@ -208,17 +195,17 @@ A Slack **Post Message** step posts `steps.write.output.text`. The `replace` fil ## Step 3: Test and publish -1. Test each of the four MCP Tool Call steps from its **Test** tab. The output shows what the tool returned for your workspace, and a connection problem shows up here first. +1. Test each of the three MCP Tool Call steps from its **Test** tab. The output shows what the tool returned for your workspace, and a connection problem shows up here first. 2. Click **Test run**. It writes a report from this week's data and posts it in the channel. 3. Compare the report with what KloudMate shows for the same week, and adjust the prompt if you want a different focus. 4. Click **Publish**. The first publish also switches the workflow on. ## Adapt the workflow -- **Report on other data.** Add another MCP Tool Call step to **Collect the week's data**, such as `search_alert_states` for the alerts firing now. Then add its result to the prompt, with a line that says what it is. The [KloudMate MCP server](/kloudmate-assistant/mcp-server/#available-tools) page lists the tools. -- **Report every day.** Change the trigger to **Daily**, and change `now-7d` to `now-1d` in the arguments. +- **Report on other data.** Add another MCP Tool Call step to **Collect the week's data**, such as `check_alerts` for the open alert groups. Then add its result to the prompt, with a line that says what it is. The [KloudMate MCP server](/kloudmate-assistant/mcp-server/#available-tools) page lists the tools. +- **Report every day.** Change the trigger to **Daily**, and change the `window` of **Traffic and errors by service** to `1d`. - **Email the report instead.** Replace **Post the report** with a **Send Email** step, and ask for Markdown instead of Slack formatting in the prompt, because Send Email reads Markdown. -- **Use another MCP server.** An MCP Tool Call step can call any server you connect as a **Custom MCP Server**, such as an internal service your team built. Use **API key in a header** under **Authentication** for a server that wants its key in a named header. +- **Use another MCP server.** An MCP Tool Call step can call any server you connect as a **Custom MCP Server**, such as an internal service your team built. Use **API key** under **Authentication** for a server that reads its key from a custom header. ## Related