Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 25 additions & 24 deletions autofix.mdx
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
---
title: "Autofix"
description: "Repair a malformed request and resend it once, before the fallback chain runs. Turn it on per harness."
description: "Use Manifest Autofix through the LLM Gateway to repair a malformed request and retry it once before fallback."
icon: "wrench"
keywords:
["auto-fix", "autofix", "self-healing", "LLM request repair", "malformed request", "400 bad request", "422 unprocessable", "model not found", "agent debugging", "harness debugging"]
canonical: "https://manifest.build/llm-gateway/docs/autofix"
---

<div className="deploy-mode-toggle" role="group" aria-label="Deployment mode">
Expand All @@ -13,7 +14,7 @@ keywords:

Some requests fail because of the request, not the model. A parameter the provider doesn't accept, a tool schema in the wrong shape, a model name that moved. Switching to a fallback model doesn't help, because the next model rejects the same body.

Autofix handles that case. When a request fails with a repairable error, Manifest sends the failed request and the provider's error to a healing service, gets a corrected request back, and sends it once more. It runs **before** the fallback chain, so fallback is still the safety net if the repair doesn't land.
Autofix handles that case. When a request fails with a repairable error, the gateway sends the failed request and the provider's error to a healing service, gets a corrected request back, and sends it once more. It runs **before** the fallback chain, so fallback is still the safety net if the repair doesn't land.

## When it runs

Expand All @@ -25,27 +26,27 @@ Autofix only touches request-side 4xx errors, where the provider is telling you
| `404` | Not found, usually a model name that moved |
| `422` | Unprocessable, usually a schema the API rejects |

It deliberately skips `401`, `403`, `429`, and every `5xx`. A bad key, a rate limit, or a provider outage isn't something a rewritten body can fix, so those go straight to [fallback](/llm-gateway#fallback).
It deliberately skips `401`, `403`, `429`, and every `5xx`. A bad key, a rate limit, or a provider outage isn't something a rewritten body can fix, so those go straight to [fallback](/llm-gateway/docs/llm-gateway#fallback).

Renamed models are covered through the provider itself. When a request names a model Manifest hasn't catalogued but the provider and credentials are unambiguous, Manifest forwards it anyway. The provider answers with its real error, typically a `404` for a model that no longer exists, and that error goes through the normal Autofix path. A patch that corrects the model name serves the request.
Renamed models are covered through the provider itself. When a request names a model the gateway hasn't catalogued but the provider and credentials are unambiguous, the gateway forwards it anyway. The provider answers with its real error, typically a `404` for a model that no longer exists, and that error goes through the normal Autofix path. A patch that corrects the model name serves the request.

## How it works

<Steps>
<Step title="The request fails">
A provider returns a repairable 4xx. Manifest normalizes the error into a
A provider returns a repairable 4xx. The gateway normalizes the error into a
message, type, and parameter.
</Step>
<Step title="Manifest asks for a patch">
<Step title="The gateway asks for a patch">
The failed request and the normalized error go to the hosted healing
service, which returns a corrected request body.
</Step>
<Step title="One retry">
Manifest sends the patched request. There is no retry budget: it tries
The gateway sends the patched request. There is no retry budget: it tries
exactly once.
</Step>
<Step title="Fallback if it still fails">
If the patched request fails too, Manifest reports the outcome and hands
If the patched request fails too, the gateway reports the outcome and hands
off to your fallback chain as normal.
</Step>
</Steps>
Expand All @@ -58,7 +59,7 @@ Autofix is a per-harness toggle, and you meet it at two moments.

<div data-deploy="selfhosted">

Nothing to deploy or configure: your install talks to Manifest's hosted healing service directly.
Nothing to deploy or configure: your install talks to the hosted Manifest Autofix service directly.

</div>

Expand All @@ -72,11 +73,11 @@ The Connect Harness modal shows an Autofix toggle under the name field, pre-set

### Harnesses you already have

A harness that existed before Autofix keeps its state: Manifest never flips the toggle for you.
A harness that existed before Autofix keeps its state: the gateway never flips the toggle for you.

<div data-deploy="cloud">

On Manifest Cloud, those harnesses are already on. You can switch any of them off on its **Settings** page.
On LLM Gateway Cloud, those harnesses are already on. You can switch any of them off on its **Settings** page.

</div>

Expand Down Expand Up @@ -108,11 +109,11 @@ Here is exactly what Autofix sends to the healing service, and when nothing is s

Self-hosting is often a privacy choice, so here is exactly what leaves your machine, and when nothing does.

The first time you enable Autofix, your install records a one-time consent: enabling it means you agree that failing requests are sent to the hosted healing service, as set out in the [terms](https://manifest.build/terms/). Any enable path records it. Your install identifies itself with an anonymous install id, the same id [telemetry](/reference/telemetry) uses.
The first time you enable Autofix, your install records a one-time consent: enabling it means you agree that failing requests are sent to the hosted healing service, as set out in the [terms](https://manifest.build/terms/). Any enable path records it. Your install identifies itself with an anonymous install id, the same id [telemetry](/llm-gateway/docs/reference/telemetry) uses.

</div>

When Autofix repairs a request, Manifest sends the failed request, including its message content, plus the provider's error response to the healing service.
When Autofix repairs a request, the gateway sends the failed request, including its message content, plus the provider's error response to the healing service.

Provider API keys and OAuth tokens are never sent: credentials travel in request headers, which are not part of the payload sent to the healing service. The provider's error response is scrubbed for secrets before it is sent.

Expand All @@ -131,9 +132,9 @@ See the [privacy policy](https://manifest.build/privacy/) and the [terms](https:
Autofix is built to stay out of the way when it isn't working:

- **One attempt.** A patched request is sent once. It never loops.
- **Failures degrade quietly.** If the healing service errors or times out, your agent gets the original provider error. Autofix never turns a provider's 400 into a Manifest 500.
- **A slow healer stops being asked.** After three consecutive transport failures, Manifest stops calling it for 30 seconds so a down service doesn't add latency to every failing request. One successful call clears the streak.
- **Startup never waits.** In production, Manifest pings the healing service once at boot, fire-and-forget, and logs a warning if it's unreachable. It never blocks startup, and it's skipped entirely when Autofix is disabled globally.
- **Failures degrade quietly.** If the healing service errors or times out, your agent gets the original provider error. Autofix never turns a provider's 400 into a gateway 500.
- **A slow healer stops being asked.** After three consecutive transport failures, the gateway stops calling it for 30 seconds so a down service doesn't add latency to every failing request. One successful call clears the streak.
- **Startup never waits.** In production, the gateway pings the healing service once at boot, fire-and-forget, and logs a warning if it's unreachable. It never blocks startup, and it's skipped entirely when Autofix is disabled globally.
- **Fallback still runs.** Autofix sits in front of the chain, it doesn't replace it.

## Seeing what it did
Expand All @@ -144,7 +145,7 @@ Open any request in the dashboard's request log. Requests that went through Auto
<img src="/images/autofix-request-panel.png" alt="The request drawer on a repaired request, with the failed and patched attempts listed and the Autofix panel showing what changed" />
</Frame>

The failed original and its patched retry are separate [provider attempts](/observability#requests-and-provider-attempts), so with [request logs](/request-logs) on you can read both bodies side by side and see exactly which field was rewritten.
The failed original and its patched retry are separate [provider attempts](/llm-gateway/docs/observability#requests-and-provider-attempts), so with [request logs](/llm-gateway/docs/request-logs) on you can read both bodies side by side and see exactly which field was rewritten.

The **Overview** page aggregates the same data across a time range:

Expand All @@ -161,7 +162,7 @@ Both **Recovered** cards link through to the matching filter on the request log.
## Environment variables

<Note>
Self-hosted only. On [Manifest Cloud](https://app.manifest.build) there is no
Self-hosted only. On [LLM Gateway Cloud](https://app.manifest.build) there is no
server to configure: everything Autofix related lives in the dashboard.
</Note>

Expand All @@ -172,16 +173,16 @@ Autofix needs no setup. Every variable below is optional and adjusts behavior:
| `AUTOFIX_GLOBAL_ENABLED` | `true` | Set `false` to turn Autofix off for the whole deployment. No call reaches the healing service, including the boot health check |
| `AUTOFIX_TIMEOUT_MS` | `10000` | Timeout per healing call, in milliseconds |
| `AUTOFIX_REPAIRABLE_STATUSES` | `400,404,422` | Which provider statuses are eligible for a repair |
| `AUTOFIX_HEALING_API_KEY` | unset | Sent as `x-api-key` to the healing service. Only relevant if Manifest issues you one |
| `AUTOFIX_HEALING_API_KEY` | unset | Sent as `x-api-key` to the healing service. Only relevant when provided by the hosted healing service |
| `AUTOFIX_REPORT_ALL_4XX` | `false` | Opt-in. Also report request-side 4xx errors (except `401`/`402`/`403`/`429`) from harnesses with Autofix on, as diagnostic evidence with the secret-scrubbed request body. Bodies over 256 KB are dropped, not truncated. No fix comes back from it |

You don't need `AUTOFIX_HEALING_API_KEY`. With no key set, the install announces its anonymous install id instead. The full list of variables lives in [Environment variables](/reference/environment-variables).
You don't need `AUTOFIX_HEALING_API_KEY`. With no key set, the install announces its anonymous install id instead. The full list of variables lives in [Environment variables](/llm-gateway/docs/reference/environment-variables).

</div>

## Related

- [LLM Gateway](/llm-gateway): where Autofix sits in the request flow
- [Observability](/observability): finding failing requests in the log
- [Request logs](/request-logs): reading the original and patched bodies
- [Error codes](/errors)
- [LLM Gateway](/llm-gateway/docs/llm-gateway): where Autofix sits in the request flow
- [Observability](/llm-gateway/docs/observability): finding failing requests in the log
- [Request logs](/llm-gateway/docs/request-logs): reading the original and patched bodies
- [Error codes](/llm-gateway/docs/errors)
39 changes: 20 additions & 19 deletions deploy.mdx
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
---
title: "Self-hosting Manifest"
title: "Deploy the LLM Gateway"
sidebarTitle: "Overview"
description: "Every way to run your own Manifest instance: Docker on your machine, a managed platform like Railway, your own AWS or GCP account, or a server panel like Coolify."
description: "Run the Manifest LLM Gateway with Docker, on a managed platform, or in your cloud account."
icon: "rocket"
keywords:
[
"self-hosted Manifest",
"self-hosted gateway",
"Manifest deployment",
"Docker",
"Railway",
Expand All @@ -16,13 +16,14 @@ keywords:
"Easypanel",
"Coolify",
]
canonical: "https://manifest.build/llm-gateway/docs/deploy"
---

Everything on this page runs the same open-source Manifest image on infrastructure you control, with your own PostgreSQL database. Pick the path that matches where you want it to live. If you'd rather not run anything yourself, [Manifest Cloud](https://app.manifest.build) is the hosted alternative.
Everything on this page runs the same open-source gateway image on infrastructure you control, with your own PostgreSQL database. Pick the path that matches where you want it to live. If you'd rather not run anything yourself, [LLM Gateway Cloud](https://app.manifest.build) is the hosted alternative.

## Your own machine

<Card title="Docker" icon="arrow-up-right" href="/self-hosted">
<Card title="Docker" icon="arrow-up-right" href="/llm-gateway/docs/self-hosted">
The most direct path. One install command, or docker-compose if you want to
edit the config first. Also covers upgrades, backups, and bringing your own
PostgreSQL.
Expand All @@ -33,27 +34,27 @@ Everything on this page runs the same open-source Manifest image on infrastructu
Someone else runs the infrastructure. Usually a one-click template or deploy button.

<CardGroup cols={2}>
<Card title="Railway" icon="arrow-up-right" href="/deploy/railway">
Best one-click path for most users. The template provisions Manifest and
<Card title="Railway" icon="arrow-up-right" href="/llm-gateway/docs/deploy/railway">
Best one-click path for most users. The template provisions the gateway and
PostgreSQL.
</Card>
<Card title="Render" icon="arrow-up-right" href="/deploy/render">
<Card title="Render" icon="arrow-up-right" href="/llm-gateway/docs/deploy/render">
Blueprint-based deploy using the public Docker image, generated secrets, and
Render PostgreSQL.
</Card>
<Card title="DigitalOcean" icon="arrow-up-right" href="/deploy/digitalocean">
App Platform deploy from the public repository with Manifest and a Dev
<Card title="DigitalOcean" icon="arrow-up-right" href="/llm-gateway/docs/deploy/digitalocean">
App Platform deploy from the public repository with the gateway and a Dev
PostgreSQL database.
</Card>
<Card title="Fly.io" icon="arrow-up-right" href="/deploy/fly">
<Card title="Fly.io" icon="arrow-up-right" href="/llm-gateway/docs/deploy/fly">
CLI-based deploy using the public Docker image, Fly Postgres, and generated
Manifest secrets.
gateway secrets.
</Card>
<Card title="Heroku" icon="arrow-up-right" href="/deploy/heroku">
<Card title="Heroku" icon="arrow-up-right" href="/llm-gateway/docs/deploy/heroku">
Deploy-button guide with Heroku Postgres, generated secrets, and one
required BETTER_AUTH_URL value.
</Card>
<Card title="Koyeb" icon="arrow-up-right" href="/deploy/koyeb">
<Card title="Koyeb" icon="arrow-up-right" href="/llm-gateway/docs/deploy/koyeb">
Deploy-button guide for Koyeb using the public Docker image, manual Postgres
URL, and runtime secrets.
</Card>
Expand All @@ -64,11 +65,11 @@ Someone else runs the infrastructure. Usually a one-click template or deploy but
Your own cloud account, provisioned with the provider's own infrastructure tooling.

<CardGroup cols={2}>
<Card title="AWS" icon="arrow-up-right" href="/deploy/aws">
<Card title="AWS" icon="arrow-up-right" href="/llm-gateway/docs/deploy/aws">
CloudFormation quick-create for ECS Fargate, RDS PostgreSQL, Secrets
Manager, and ALB.
</Card>
<Card title="GCP" icon="arrow-up-right" href="/deploy/gcp">
<Card title="GCP" icon="arrow-up-right" href="/llm-gateway/docs/deploy/gcp">
Cloud Shell guided deploy for Cloud Run, Cloud SQL PostgreSQL, and Secret
Manager.
</Card>
Expand All @@ -79,11 +80,11 @@ Your own cloud account, provisioned with the provider's own infrastructure tooli
A PaaS you run on your own server, so you get deploy buttons on hardware you own.

<CardGroup cols={2}>
<Card title="Coolify" icon="arrow-up-right" href="/deploy/coolify">
<Card title="Coolify" icon="arrow-up-right" href="/llm-gateway/docs/deploy/coolify">
Docker Compose stack using Coolify magic variables and a private PostgreSQL
container.
</Card>
<Card title="Easypanel" icon="arrow-up-right" href="/deploy/easypanel">
<Card title="Easypanel" icon="arrow-up-right" href="/llm-gateway/docs/deploy/easypanel">
Easypanel app service using PostgreSQL and the Manifest Docker image.
</Card>
</CardGroup>
Expand All @@ -94,4 +95,4 @@ Most hosted deployments create paid resources. Review each provider's pricing be

For public deployments, make sure `BETTER_AUTH_URL` matches the exact browser URL. If it does not, Better Auth can reject login with an invalid-origin error.

Whichever path you pick, [Environment variables](/reference/environment-variables) is the full configuration list, and [Data and telemetry](/reference/telemetry) covers what your instance stores and what it reports back.
Whichever path you pick, [Environment variables](/llm-gateway/docs/reference/environment-variables) is the full configuration list, and [Data and telemetry](/llm-gateway/docs/reference/telemetry) covers what your instance stores and what it reports back.
17 changes: 9 additions & 8 deletions deploy/aws.mdx
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
---
title: "Deploy Manifest on AWS"
title: "Deploy the LLM Gateway on AWS"
sidebarTitle: "AWS"
description: "Deploy Manifest on AWS with CloudFormation, ECS Fargate, RDS PostgreSQL, Secrets Manager, and an Application Load Balancer."
description: "Deploy the Manifest LLM Gateway with CloudFormation, ECS Fargate, RDS PostgreSQL, Secrets Manager, and an Application Load Balancer."
icon: "/icons/deploy/aws.svg"
keywords:
["Manifest deployment", "AWS", "self-hosted Manifest", "PostgreSQL", "Docker"]
["Manifest deployment", "AWS", "self-hosted gateway", "PostgreSQL", "Docker"]
canonical: "https://manifest.build/llm-gateway/docs/deploy/aws"
---

<Card
Expand All @@ -18,7 +19,7 @@ keywords:

## Overview

The AWS template deploys Manifest with ECS Fargate, RDS PostgreSQL, Secrets Manager, CloudWatch Logs, and an Application Load Balancer. It is a good fit when you want Manifest in your own AWS account.
The AWS template deploys the gateway with ECS Fargate, RDS PostgreSQL, Secrets Manager, CloudWatch Logs, and an Application Load Balancer. It is a good fit when you want the gateway in your own AWS account.

## Prerequisites

Expand Down Expand Up @@ -64,11 +65,11 @@ The CloudFormation template generates:
- `BETTER_AUTH_SECRET`.
- `MANIFEST_ENCRYPTION_KEY`.

Manifest runs with `PORT=2099`, `BIND_ADDRESS=0.0.0.0`, and `MANIFEST_MODE=selfhosted`.
The gateway runs with `PORT=2099`, `BIND_ADDRESS=0.0.0.0`, and `MANIFEST_MODE=selfhosted`.

The template also creates an S3 bucket for [request logs](/request-logs). It is retained by design, so deleting the CloudFormation stack leaves the bucket in place. Delete it by hand when you tear the deployment down.
The template also creates an S3 bucket for [request logs](/llm-gateway/docs/request-logs). It is retained by design, so deleting the CloudFormation stack leaves the bucket in place. Delete it by hand when you tear the deployment down.

## Open Manifest
## Open the gateway

After deployment, the script prints `ServiceUrl` and `HealthCheckUrl`.

Expand All @@ -82,7 +83,7 @@ Open `ServiceUrl` and create the first account. Fresh installs redirect to `/set

## Production notes

- The default template exposes Manifest over HTTP on the generated load balancer DNS name.
- The default template exposes the gateway over HTTP on the generated load balancer DNS name.
- Configure TLS with your own domain and ACM certificate before using the deployment for production authentication traffic.
- Set database deletion protection if you want to avoid accidental RDS deletion.

Expand Down
Loading