Skip to content

Commit 5a87ac0

Browse files
Update providers/openai-like.mdx
Co-Authored-By: mintlify[bot] <109931778+mintlify[bot]@users.noreply.github.com>
1 parent 9019220 commit 5a87ac0

1 file changed

Lines changed: 38 additions & 262 deletions

File tree

providers/openai-like.mdx

Lines changed: 38 additions & 262 deletions
Original file line numberDiff line numberDiff line change
@@ -1,279 +1,55 @@
11
---
22
title: OpenAI Compatible
3-
description: Connect to any OpenAI-compatible API endpoint including custom deployments, self-hosted models, and alternative AI services.
3+
description: Connect to any OpenAI-compatible API endpoint including custom deployments and self-hosted models.
44
---
55

6-
The OpenAI Compatible provider allows you to connect to any service that implements the OpenAI API specification, including custom deployments, alternative providers, and self-hosted models that maintain API compatibility.
6+
Connect to any service that implements the OpenAI API specification.
77

8-
## Overview
8+
## Configuration
99

10-
This flexible provider enables integration with any OpenAI-compatible API, making it easy to use custom AI deployments, alternative hosting services, or self-hosted models that follow the OpenAI API standard.
10+
Set these environment variables:
11+
- `OPENAI_LIKE_API_BASE_URL` - Your API endpoint URL
12+
- `OPENAI_LIKE_API_KEY` - Authentication token
13+
- `OPENAI_LIKE_API_MODELS` (optional) - Manual model list in format: `model1:limit;model2:limit`
1114

12-
<CardGroup cols={3}>
13-
<Card title="API Compatibility" icon="plug">
14-
Works with any OpenAI-compatible API
15-
</Card>
16-
<Card title="Custom Deployments" icon="server">
17-
Connect to self-hosted or custom AI services
18-
</Card>
19-
<Card title="Flexible Configuration" icon="settings">
20-
Highly customizable connection settings
21-
</Card>
22-
</CardGroup>
15+
## Setup
2316

24-
## How It Works
25-
26-
<AccordionGroup>
27-
<Accordion title="API Compatibility" icon="check-circle">
28-
### OpenAI Standard
29-
Connects to services that implement the OpenAI API specification.
30-
31-
- **Standard Endpoints**: Uses familiar `/chat/completions` and `/models` endpoints
32-
- **Compatible Formats**: Supports standard OpenAI request/response formats
33-
- **Authentication**: Uses Bearer token authentication like OpenAI
34-
- **Streaming Support**: Compatible with streaming responses
35-
36-
</Accordion>
37-
38-
<Accordion title="Configuration Options" icon="cog">
39-
### Setup Flexibility
40-
Highly configurable to work with different API providers.
41-
42-
- **Custom Base URL**: Specify any API endpoint URL
43-
- **API Key**: Configure authentication tokens
44-
- **Model List**: Define available models manually or auto-discover
45-
- **Environment Variables**: Support for different deployment environments
46-
47-
</Accordion>
48-
49-
<Accordion title="Model Discovery" icon="search">
50-
### Dynamic Model Loading
51-
Automatically discovers available models from compatible APIs.
52-
53-
- **API Query**: Fetches model list from `/models` endpoint
54-
- **Fallback Configuration**: Manual model specification if API discovery fails
55-
- **Model Parsing**: Intelligent model name and capability detection
56-
- **Real-time Updates**: Reflects current API capabilities
57-
58-
</Accordion>
59-
</AccordionGroup>
60-
61-
## Setup Instructions
62-
63-
<Steps>
64-
<Step title="Identify API Endpoint">Determine the base URL of your OpenAI-compatible API service</Step>
65-
<Step title="Obtain API Key">Get the authentication token or API key for the service</Step>
66-
<Step title="Configure Environment">Set the required environment variables in your deployment</Step>
67-
<Step title="Test Connection">Verify the API endpoint and authentication work correctly</Step>
68-
<Step title="Configure Models">Set up model list either through API discovery or manual configuration</Step>
69-
</Steps>
70-
71-
## Configuration Options
72-
73-
<AccordionGroup>
74-
<Accordion title="Environment Variables" icon="variable">
75-
### Required Settings
76-
Environment variables needed for OpenAI-compatible provider setup.
77-
78-
- **OPENAI_LIKE_API_BASE_URL**: The base URL of your API service
79-
- **OPENAI_LIKE_API_KEY**: Authentication token for API access
80-
- **OPENAI_LIKE_API_MODELS** (optional): Manual model specification
81-
82-
</Accordion>
83-
84-
<Accordion title="Manual Model Configuration" icon="list">
85-
### Model Specification
86-
How to manually specify models when API discovery is not available.
87-
88-
- **Format**: `model1:limit;model2:limit;model3:limit`
89-
- **Example**: `gpt-4:8000;claude-3:4000;llama-2:2000`
90-
- **Token Limits**: Specify context window limits per model
91-
- **Naming**: Use clear, descriptive model names
92-
93-
</Accordion>
94-
95-
<Accordion title="Docker Considerations" icon="container">
96-
### Container Deployment
97-
Special considerations for Docker and containerized deployments.
98-
99-
- **Network Access**: Ensure API endpoints are accessible from containers
100-
- **Environment Variables**: Pass configuration through Docker environment
101-
- **Volume Mounting**: Mount configuration files if needed
102-
- **Service Discovery**: Use container networking for service communication
103-
104-
</Accordion>
105-
</AccordionGroup>
106-
107-
## Use Cases
108-
109-
<AccordionGroup>
110-
<Accordion title="Self-Hosted Models" icon="home">
111-
### Local AI Deployment
112-
Connect to locally hosted AI models and services.
113-
114-
- Local LLM deployments (Ollama, LM Studio, etc.)
115-
- Custom model servers
116-
- Private AI infrastructure
117-
- Development environments
118-
119-
</Accordion>
120-
121-
<Accordion title="Alternative Providers" icon="shuffle">
122-
### Third-Party Services
123-
Integrate with alternative AI providers using OpenAI compatibility.
124-
125-
- Alternative hosting services
126-
- Specialized AI providers
127-
- Regional AI services
128-
- Custom AI platforms
129-
130-
</Accordion>
131-
132-
<Accordion title="Enterprise Deployments" icon="building">
133-
### Corporate AI
134-
Connect to enterprise AI deployments and private clouds.
135-
136-
- Corporate AI infrastructure
137-
- Private cloud deployments
138-
- On-premises AI services
139-
- Hybrid cloud setups
140-
141-
</Accordion>
142-
143-
<Accordion title="Development Testing" icon="flask">
144-
### Development and Testing
145-
Useful for development, testing, and prototyping scenarios.
146-
147-
- Local development servers
148-
- Staging environment testing
149-
- API compatibility testing
150-
- Mock AI services for development
151-
152-
</Accordion>
153-
</AccordionGroup>
17+
1. Identify your OpenAI-compatible API endpoint
18+
2. Obtain the API key or authentication token
19+
3. Set environment variables in your deployment
20+
4. Test the connection
21+
5. Configure available models
15422

15523
## Compatible Services
15624

157-
<AccordionGroup>
158-
<Accordion title="Local AI Tools" icon="computer">
159-
### Desktop Applications
160-
Popular local AI tools that provide OpenAI-compatible APIs.
161-
162-
- **LM Studio**: Local model server with web UI
163-
- **Ollama**: Command-line tool for running models locally
164-
- **LocalAI**: Self-hosted OpenAI-compatible API
165-
- **Text Generation WebUI**: Local web interface for models
166-
167-
</Accordion>
168-
169-
<Accordion title="Cloud Alternatives" icon="cloud">
170-
### Alternative Cloud Providers
171-
Cloud services that offer OpenAI-compatible APIs.
172-
173-
- **Together AI**: Open-source model hosting
174-
- **Replicate**: Model deployment platform
175-
- **Modal**: Serverless model inference
176-
- **Anthropic-compatible services**: Alternative Claude hosting
177-
178-
</Accordion>
179-
180-
<Accordion title="Custom Deployments" icon="wrench">
181-
### Custom AI Services
182-
Self-hosted or custom AI service deployments.
183-
184-
- **vLLM**: High-performance LLM serving
185-
- **TGI (Text Generation Inference)**: Optimized text generation
186-
- **FastChat**: Open-source chat platform
187-
- **Custom model servers**: Your own AI service implementations
188-
189-
</Accordion>
190-
</AccordionGroup>
191-
192-
## Troubleshooting
193-
194-
<AccordionGroup>
195-
<Accordion title="Connection Issues" icon="wifi-off">
196-
### API Connectivity
197-
Common connection and authentication problems.
198-
199-
- **Network Access**: Verify API endpoint is reachable
200-
- **Authentication**: Check API key validity and format
201-
- **CORS Issues**: Ensure proper cross-origin headers
202-
- **SSL/TLS**: Verify certificate validity for HTTPS endpoints
25+
**Local AI Tools:**
26+
- LM Studio
27+
- Ollama
28+
- LocalAI
29+
- Text Generation WebUI
20330

204-
</Accordion>
31+
**Cloud Alternatives:**
32+
- Together AI
33+
- Replicate
34+
- Modal
35+
- Custom deployments
20536

206-
<Accordion title="Model Discovery" icon="search-off">
207-
### Model Loading Issues
208-
Problems with model list retrieval and configuration.
37+
**Self-Hosted:**
38+
- vLLM
39+
- TGI (Text Generation Inference)
40+
- FastChat
41+
- Custom model servers
20942

210-
- **API Endpoint**: Verify `/models` endpoint exists and works
211-
- **Authentication**: Ensure proper API key for model discovery
212-
- **Manual Configuration**: Use environment variable fallback
213-
- **Model Format**: Check model ID format and naming conventions
214-
215-
</Accordion>
216-
217-
<Accordion title="Performance Issues" icon="gauge">
218-
### Speed and Reliability
219-
Addressing performance and reliability concerns.
220-
221-
- **Response Times**: Check network latency to API endpoint
222-
- **Rate Limits**: Monitor API rate limiting and quotas
223-
- **Model Size**: Consider model size vs. available resources
224-
- **Caching**: Implement response caching for repeated queries
225-
226-
</Accordion>
227-
</AccordionGroup>
228-
229-
<Callout type="info">
230-
**Compatibility Check**: Always verify that your target service implements the OpenAI API specification correctly,
231-
including proper request/response formats and authentication.
232-
</Callout>
233-
234-
<Callout type="tip">
235-
**Testing Strategy**: Start with simple requests to verify connectivity, then test model discovery, and finally test
236-
actual model inference before full deployment.
237-
</Callout>
238-
239-
<Callout type="warning">
240-
**Security Considerations**: Ensure your API keys are properly secured and that the API endpoint uses HTTPS for secure
241-
communication.
242-
</Callout>
243-
244-
## Advanced Configuration
245-
246-
<AccordionGroup>
247-
<Accordion title="Custom Headers" icon="header">
248-
### Additional Headers
249-
Configure custom headers for special API requirements.
250-
251-
- **Authorization Variants**: Different authentication header formats
252-
- **API Version Headers**: Specify API version requirements
253-
- **Custom Metadata**: Service-specific header requirements
254-
- **Rate Limiting**: Custom rate limit headers
255-
256-
</Accordion>
257-
258-
<Accordion title="Proxy Configuration" icon="route">
259-
### Network Proxies
260-
Configure proxy settings for restricted network environments.
261-
262-
- **HTTP Proxies**: Route API calls through proxy servers
263-
- **Corporate Networks**: Work within enterprise network restrictions
264-
- **VPN Requirements**: Handle VPN-dependent API access
265-
- **Load Balancing**: Distribute requests across multiple endpoints
266-
267-
</Accordion>
43+
## Use Cases
26844

269-
<Accordion title="Monitoring Integration" icon="activity">
270-
### Observability
271-
Integrate with monitoring and logging systems.
45+
- Self-hosted models and services
46+
- Alternative AI providers
47+
- Enterprise private deployments
48+
- Development and testing environments
27249

273-
- **Request Logging**: Track API usage and performance
274-
- **Error Monitoring**: Capture and analyze API errors
275-
- **Usage Analytics**: Monitor token consumption and costs
276-
- **Health Checks**: Implement API endpoint health monitoring
50+
## Notes
27751

278-
</Accordion>
279-
</AccordionGroup>
52+
- Verify API implements OpenAI specification correctly
53+
- Ensure HTTPS for secure communication
54+
- Test with simple requests first
55+
- Use manual model configuration if auto-discovery fails

0 commit comments

Comments
 (0)