|
1 | 1 | --- |
2 | 2 | title: OpenAI Compatible |
3 | | -description: Connect to any OpenAI-compatible API endpoint including custom deployments, self-hosted models, and alternative AI services. |
| 3 | +description: Connect to any OpenAI-compatible API endpoint including custom deployments and self-hosted models. |
4 | 4 | --- |
5 | 5 |
|
6 | | -The OpenAI Compatible provider allows you to connect to any service that implements the OpenAI API specification, including custom deployments, alternative providers, and self-hosted models that maintain API compatibility. |
| 6 | +Connect to any service that implements the OpenAI API specification. |
7 | 7 |
|
8 | | -## Overview |
| 8 | +## Configuration |
9 | 9 |
|
10 | | -This flexible provider enables integration with any OpenAI-compatible API, making it easy to use custom AI deployments, alternative hosting services, or self-hosted models that follow the OpenAI API standard. |
| 10 | +Set these environment variables: |
| 11 | +- `OPENAI_LIKE_API_BASE_URL` - Your API endpoint URL |
| 12 | +- `OPENAI_LIKE_API_KEY` - Authentication token |
| 13 | +- `OPENAI_LIKE_API_MODELS` (optional) - Manual model list in format: `model1:limit;model2:limit` |
11 | 14 |
|
12 | | -<CardGroup cols={3}> |
13 | | - <Card title="API Compatibility" icon="plug"> |
14 | | - Works with any OpenAI-compatible API |
15 | | - </Card> |
16 | | - <Card title="Custom Deployments" icon="server"> |
17 | | - Connect to self-hosted or custom AI services |
18 | | - </Card> |
19 | | - <Card title="Flexible Configuration" icon="settings"> |
20 | | - Highly customizable connection settings |
21 | | - </Card> |
22 | | -</CardGroup> |
| 15 | +## Setup |
23 | 16 |
|
24 | | -## How It Works |
25 | | - |
26 | | -<AccordionGroup> |
27 | | - <Accordion title="API Compatibility" icon="check-circle"> |
28 | | - ### OpenAI Standard |
29 | | - Connects to services that implement the OpenAI API specification. |
30 | | - |
31 | | - - **Standard Endpoints**: Uses familiar `/chat/completions` and `/models` endpoints |
32 | | - - **Compatible Formats**: Supports standard OpenAI request/response formats |
33 | | - - **Authentication**: Uses Bearer token authentication like OpenAI |
34 | | - - **Streaming Support**: Compatible with streaming responses |
35 | | - |
36 | | - </Accordion> |
37 | | - |
38 | | - <Accordion title="Configuration Options" icon="cog"> |
39 | | - ### Setup Flexibility |
40 | | - Highly configurable to work with different API providers. |
41 | | - |
42 | | - - **Custom Base URL**: Specify any API endpoint URL |
43 | | - - **API Key**: Configure authentication tokens |
44 | | - - **Model List**: Define available models manually or auto-discover |
45 | | - - **Environment Variables**: Support for different deployment environments |
46 | | - |
47 | | - </Accordion> |
48 | | - |
49 | | - <Accordion title="Model Discovery" icon="search"> |
50 | | - ### Dynamic Model Loading |
51 | | - Automatically discovers available models from compatible APIs. |
52 | | - |
53 | | - - **API Query**: Fetches model list from `/models` endpoint |
54 | | - - **Fallback Configuration**: Manual model specification if API discovery fails |
55 | | - - **Model Parsing**: Intelligent model name and capability detection |
56 | | - - **Real-time Updates**: Reflects current API capabilities |
57 | | - |
58 | | - </Accordion> |
59 | | -</AccordionGroup> |
60 | | - |
61 | | -## Setup Instructions |
62 | | - |
63 | | -<Steps> |
64 | | - <Step title="Identify API Endpoint">Determine the base URL of your OpenAI-compatible API service</Step> |
65 | | - <Step title="Obtain API Key">Get the authentication token or API key for the service</Step> |
66 | | - <Step title="Configure Environment">Set the required environment variables in your deployment</Step> |
67 | | - <Step title="Test Connection">Verify the API endpoint and authentication work correctly</Step> |
68 | | - <Step title="Configure Models">Set up model list either through API discovery or manual configuration</Step> |
69 | | -</Steps> |
70 | | - |
71 | | -## Configuration Options |
72 | | - |
73 | | -<AccordionGroup> |
74 | | - <Accordion title="Environment Variables" icon="variable"> |
75 | | - ### Required Settings |
76 | | - Environment variables needed for OpenAI-compatible provider setup. |
77 | | - |
78 | | - - **OPENAI_LIKE_API_BASE_URL**: The base URL of your API service |
79 | | - - **OPENAI_LIKE_API_KEY**: Authentication token for API access |
80 | | - - **OPENAI_LIKE_API_MODELS** (optional): Manual model specification |
81 | | - |
82 | | - </Accordion> |
83 | | - |
84 | | - <Accordion title="Manual Model Configuration" icon="list"> |
85 | | - ### Model Specification |
86 | | - How to manually specify models when API discovery is not available. |
87 | | - |
88 | | - - **Format**: `model1:limit;model2:limit;model3:limit` |
89 | | - - **Example**: `gpt-4:8000;claude-3:4000;llama-2:2000` |
90 | | - - **Token Limits**: Specify context window limits per model |
91 | | - - **Naming**: Use clear, descriptive model names |
92 | | - |
93 | | - </Accordion> |
94 | | - |
95 | | - <Accordion title="Docker Considerations" icon="container"> |
96 | | - ### Container Deployment |
97 | | - Special considerations for Docker and containerized deployments. |
98 | | - |
99 | | - - **Network Access**: Ensure API endpoints are accessible from containers |
100 | | - - **Environment Variables**: Pass configuration through Docker environment |
101 | | - - **Volume Mounting**: Mount configuration files if needed |
102 | | - - **Service Discovery**: Use container networking for service communication |
103 | | - |
104 | | - </Accordion> |
105 | | -</AccordionGroup> |
106 | | - |
107 | | -## Use Cases |
108 | | - |
109 | | -<AccordionGroup> |
110 | | - <Accordion title="Self-Hosted Models" icon="home"> |
111 | | - ### Local AI Deployment |
112 | | - Connect to locally hosted AI models and services. |
113 | | - |
114 | | - - Local LLM deployments (Ollama, LM Studio, etc.) |
115 | | - - Custom model servers |
116 | | - - Private AI infrastructure |
117 | | - - Development environments |
118 | | - |
119 | | - </Accordion> |
120 | | - |
121 | | - <Accordion title="Alternative Providers" icon="shuffle"> |
122 | | - ### Third-Party Services |
123 | | - Integrate with alternative AI providers using OpenAI compatibility. |
124 | | - |
125 | | - - Alternative hosting services |
126 | | - - Specialized AI providers |
127 | | - - Regional AI services |
128 | | - - Custom AI platforms |
129 | | - |
130 | | - </Accordion> |
131 | | - |
132 | | - <Accordion title="Enterprise Deployments" icon="building"> |
133 | | - ### Corporate AI |
134 | | - Connect to enterprise AI deployments and private clouds. |
135 | | - |
136 | | - - Corporate AI infrastructure |
137 | | - - Private cloud deployments |
138 | | - - On-premises AI services |
139 | | - - Hybrid cloud setups |
140 | | - |
141 | | - </Accordion> |
142 | | - |
143 | | - <Accordion title="Development Testing" icon="flask"> |
144 | | - ### Development and Testing |
145 | | - Useful for development, testing, and prototyping scenarios. |
146 | | - |
147 | | - - Local development servers |
148 | | - - Staging environment testing |
149 | | - - API compatibility testing |
150 | | - - Mock AI services for development |
151 | | - |
152 | | - </Accordion> |
153 | | -</AccordionGroup> |
| 17 | +1. Identify your OpenAI-compatible API endpoint |
| 18 | +2. Obtain the API key or authentication token |
| 19 | +3. Set environment variables in your deployment |
| 20 | +4. Test the connection |
| 21 | +5. Configure available models |
154 | 22 |
|
155 | 23 | ## Compatible Services |
156 | 24 |
|
157 | | -<AccordionGroup> |
158 | | - <Accordion title="Local AI Tools" icon="computer"> |
159 | | - ### Desktop Applications |
160 | | - Popular local AI tools that provide OpenAI-compatible APIs. |
161 | | - |
162 | | - - **LM Studio**: Local model server with web UI |
163 | | - - **Ollama**: Command-line tool for running models locally |
164 | | - - **LocalAI**: Self-hosted OpenAI-compatible API |
165 | | - - **Text Generation WebUI**: Local web interface for models |
166 | | - |
167 | | - </Accordion> |
168 | | - |
169 | | - <Accordion title="Cloud Alternatives" icon="cloud"> |
170 | | - ### Alternative Cloud Providers |
171 | | - Cloud services that offer OpenAI-compatible APIs. |
172 | | - |
173 | | - - **Together AI**: Open-source model hosting |
174 | | - - **Replicate**: Model deployment platform |
175 | | - - **Modal**: Serverless model inference |
176 | | - - **Anthropic-compatible services**: Alternative Claude hosting |
177 | | - |
178 | | - </Accordion> |
179 | | - |
180 | | - <Accordion title="Custom Deployments" icon="wrench"> |
181 | | - ### Custom AI Services |
182 | | - Self-hosted or custom AI service deployments. |
183 | | - |
184 | | - - **vLLM**: High-performance LLM serving |
185 | | - - **TGI (Text Generation Inference)**: Optimized text generation |
186 | | - - **FastChat**: Open-source chat platform |
187 | | - - **Custom model servers**: Your own AI service implementations |
188 | | - |
189 | | - </Accordion> |
190 | | -</AccordionGroup> |
191 | | - |
192 | | -## Troubleshooting |
193 | | - |
194 | | -<AccordionGroup> |
195 | | - <Accordion title="Connection Issues" icon="wifi-off"> |
196 | | - ### API Connectivity |
197 | | - Common connection and authentication problems. |
198 | | - |
199 | | - - **Network Access**: Verify API endpoint is reachable |
200 | | - - **Authentication**: Check API key validity and format |
201 | | - - **CORS Issues**: Ensure proper cross-origin headers |
202 | | - - **SSL/TLS**: Verify certificate validity for HTTPS endpoints |
| 25 | +**Local AI Tools:** |
| 26 | +- LM Studio |
| 27 | +- Ollama |
| 28 | +- LocalAI |
| 29 | +- Text Generation WebUI |
203 | 30 |
|
204 | | - </Accordion> |
| 31 | +**Cloud Alternatives:** |
| 32 | +- Together AI |
| 33 | +- Replicate |
| 34 | +- Modal |
| 35 | +- Custom deployments |
205 | 36 |
|
206 | | - <Accordion title="Model Discovery" icon="search-off"> |
207 | | - ### Model Loading Issues |
208 | | - Problems with model list retrieval and configuration. |
| 37 | +**Self-Hosted:** |
| 38 | +- vLLM |
| 39 | +- TGI (Text Generation Inference) |
| 40 | +- FastChat |
| 41 | +- Custom model servers |
209 | 42 |
|
210 | | - - **API Endpoint**: Verify `/models` endpoint exists and works |
211 | | - - **Authentication**: Ensure proper API key for model discovery |
212 | | - - **Manual Configuration**: Use environment variable fallback |
213 | | - - **Model Format**: Check model ID format and naming conventions |
214 | | - |
215 | | - </Accordion> |
216 | | - |
217 | | - <Accordion title="Performance Issues" icon="gauge"> |
218 | | - ### Speed and Reliability |
219 | | - Addressing performance and reliability concerns. |
220 | | - |
221 | | - - **Response Times**: Check network latency to API endpoint |
222 | | - - **Rate Limits**: Monitor API rate limiting and quotas |
223 | | - - **Model Size**: Consider model size vs. available resources |
224 | | - - **Caching**: Implement response caching for repeated queries |
225 | | - |
226 | | - </Accordion> |
227 | | -</AccordionGroup> |
228 | | - |
229 | | -<Callout type="info"> |
230 | | - **Compatibility Check**: Always verify that your target service implements the OpenAI API specification correctly, |
231 | | - including proper request/response formats and authentication. |
232 | | -</Callout> |
233 | | - |
234 | | -<Callout type="tip"> |
235 | | - **Testing Strategy**: Start with simple requests to verify connectivity, then test model discovery, and finally test |
236 | | - actual model inference before full deployment. |
237 | | -</Callout> |
238 | | - |
239 | | -<Callout type="warning"> |
240 | | - **Security Considerations**: Ensure your API keys are properly secured and that the API endpoint uses HTTPS for secure |
241 | | - communication. |
242 | | -</Callout> |
243 | | - |
244 | | -## Advanced Configuration |
245 | | - |
246 | | -<AccordionGroup> |
247 | | - <Accordion title="Custom Headers" icon="header"> |
248 | | - ### Additional Headers |
249 | | - Configure custom headers for special API requirements. |
250 | | - |
251 | | - - **Authorization Variants**: Different authentication header formats |
252 | | - - **API Version Headers**: Specify API version requirements |
253 | | - - **Custom Metadata**: Service-specific header requirements |
254 | | - - **Rate Limiting**: Custom rate limit headers |
255 | | - |
256 | | - </Accordion> |
257 | | - |
258 | | - <Accordion title="Proxy Configuration" icon="route"> |
259 | | - ### Network Proxies |
260 | | - Configure proxy settings for restricted network environments. |
261 | | - |
262 | | - - **HTTP Proxies**: Route API calls through proxy servers |
263 | | - - **Corporate Networks**: Work within enterprise network restrictions |
264 | | - - **VPN Requirements**: Handle VPN-dependent API access |
265 | | - - **Load Balancing**: Distribute requests across multiple endpoints |
266 | | - |
267 | | - </Accordion> |
| 43 | +## Use Cases |
268 | 44 |
|
269 | | - <Accordion title="Monitoring Integration" icon="activity"> |
270 | | - ### Observability |
271 | | - Integrate with monitoring and logging systems. |
| 45 | +- Self-hosted models and services |
| 46 | +- Alternative AI providers |
| 47 | +- Enterprise private deployments |
| 48 | +- Development and testing environments |
272 | 49 |
|
273 | | - - **Request Logging**: Track API usage and performance |
274 | | - - **Error Monitoring**: Capture and analyze API errors |
275 | | - - **Usage Analytics**: Monitor token consumption and costs |
276 | | - - **Health Checks**: Implement API endpoint health monitoring |
| 50 | +## Notes |
277 | 51 |
|
278 | | - </Accordion> |
279 | | -</AccordionGroup> |
| 52 | +- Verify API implements OpenAI specification correctly |
| 53 | +- Ensure HTTPS for secure communication |
| 54 | +- Test with simple requests first |
| 55 | +- Use manual model configuration if auto-discovery fails |
0 commit comments