gns3-server/docs/gns3-copilot/implemented/llm-model-configs.md

LLM Model Configurations API

Overview

This API provides LLM model configuration management for users and user groups with inheritance support.

Key Features

• User-level configurations: Each user can have their own LLM model configurations
• Group-level configurations: User groups can share LLM model configurations
• Inheritance: Users automatically inherit configurations from their groups (when they have no own configs)
• Default configuration: Both users and groups can set a default configuration
• API Key Encryption: API keys are automatically encrypted in the database
• Optimistic Locking: Prevents concurrent modification conflicts using version tracking

Inheritance Logic

User requests configs:
  ├─ Always return user's own configs (if any)
  └─ Always return inherited group configs (if any)

Note: Users can see both their own configurations AND configurations inherited from their groups. The source field in the response indicates the origin of each configuration.

Configuration Priority

User's own config > User's group config

---

Database Schema

Table: llm_model_configs

Column	Type	Description
config_id	UUID	Primary key
name	VARCHAR(100)	Configuration name (table-level for indexing)
model_type	VARCHAR(50)	Model type (table-level for filtering)
config	JSONB	Configuration data (provider, base_url, model, temperature, api_key, etc.)
user_id	UUID (nullable)	Foreign key to users table
group_id	UUID (nullable)	Foreign key to user_groups table
is_default	BOOLEAN	Default configuration flag
version	INTEGER	Optimistic locking version (starts at 0, increments on each update)
reserved_jsonb_1	JSONB (nullable)	Reserved field for future use
reserved_jsonb_2	JSONB (nullable)	Reserved field for future use
reserved_jsonb_3	JSONB (nullable)	Reserved field for future use
created_at	TIMESTAMP	Creation timestamp
updated_at	TIMESTAMP	Last update timestamp

Model Types

The model_type field accepts the following values:

• text - Text generation models
• vision - Vision/image understanding models
• stt - Speech-to-Text models
• tts - Text-to-Speech models
• multimodal - Multimodal models supporting multiple input types
• embedding - Text embedding models
• reranking - Reranking models
• other - Other model types

Constraints

• Each config belongs to either a user or a group (not both)
• Each user can have at most one default configuration
• Each group can have at most one default configuration
• version field is automatically incremented on each update

---

API Endpoints

User Configuration Endpoints

Method	Path	Description	Privilege
GET	/v3/access/users/{user_id}/llm-model-configs	Get user's effective configs (own + inherited)	User.Audit
GET	/v3/access/users/{user_id}/llm-model-configs/own	Get user's own configs only	User.Audit
GET	/v3/access/users/{user_id}/llm-model-configs/default	Get user's default configuration	User.Audit
POST	/v3/access/users/{user_id}/llm-model-configs	Create a new configuration	User.Modify
PUT	/v3/access/users/{user_id}/llm-model-configs/{config_id}	Update a configuration	User.Modify
DELETE	/v3/access/users/{user_id}/llm-model-configs/{config_id}	Delete a configuration	User.Modify
PUT	/v3/access/users/{user_id}/llm-model-configs/default/{config_id}	Set default configuration	User.Modify

Group Configuration Endpoints

Method	Path	Description	Privilege
GET	/v3/access/groups/{group_id}/llm-model-configs	Get all group configurations	Group.Audit
GET	/v3/access/groups/{group_id}/llm-model-configs/default	Get group's default configuration	Group.Audit
POST	/v3/access/groups/{group_id}/llm-model-configs	Create a new configuration	Group.Modify
PUT	/v3/access/groups/{group_id}/llm-model-configs/{config_id}	Update a configuration	Group.Modify
DELETE	/v3/access/groups/{group_id}/llm-model-configs/{config_id}	Delete a configuration	Group.Modify
PUT	/v3/access/groups/{group_id}/llm-model-configs/default/{config_id}	Set default configuration	Group.Modify

Note: The GET endpoints for groups return the same structure as user endpoints: configs, default_config, and total.

---

Request/Response Schemas

LLMModelConfigCreate

Required Fields:

Field	Type	Description
name	string	Configuration name (1-100 chars)
model_type	string	Model type (text, vision, stt, tts, multimodal, embedding, reranking, other)
provider	string	LLM provider (e.g., "openai", "anthropic", "ollama")
base_url	string	API base URL
model	string	Model name
temperature	float	Temperature (0.0-2.0, default: 0.7)
context_limit	integer	Model context window limit in K tokens (e.g., 128 = 128K = 128,000 tokens)

Optional Fields:

Field	Type	Description
api_key	string	API key (auto-encrypted)
max_tokens	integer	⚠️ Reserved for future use, not currently implemented
context_strategy	string	Context trimming strategy: "conservative" (60%), "balanced" (75%), "aggressive" (85%). Default: "balanced"
copilot_mode	string	GNS3-Copilot mode: "teaching_assistant" (diagnostics only, default) or "lab_automation_assistant" (full configuration access)
is_default	boolean	Set as default (default: false)

Important Notes:

• context_limit is required: You must specify the model's context window limit. Refer to the model provider's official documentation for the current value.
• Unit is K tokens: The value is in thousands of tokens (1 K = 1,000 tokens). For example:
  • GPT-4o: 128,000 tokens → configure as "context_limit": 128
  • Claude 3.5 Sonnet: 200,000 tokens → configure as "context_limit": 200
  • Gemini 1.5 Pro: 2,800,000 tokens → configure as "context_limit": 2800

Extra Fields: Any custom fields are supported for future extensibility.

LLMModelConfigUpdate

Field	Type	Description
name	string (optional)	Configuration name
model_type	string (optional)	Model type
provider	string (optional)	LLM provider
base_url	string (optional)	API base URL
model	string (optional)	Model name
temperature	float (optional)	Temperature
api_key	string (optional)	API key
max_tokens	integer or string (optional)	⚠️ Reserved for future use, not currently implemented. Accepts integers, null, or the string "null" (which will be converted to null)
context_limit	integer (optional)	Model context window limit in K tokens
context_strategy	string (optional)	Context trimming strategy
is_default	boolean (optional)	Default flag
expected_version	integer (optional)	Optimistic locking version

Note: When using expected_version, the API will verify the version hasn't changed since you read the data. If it has, you'll receive a 409 Conflict error.

Reserved Fields:

• max_tokens: Reserved for future implementation. Currently not used by the system. The maximum output tokens are controlled automatically by the LLM provider based on the model and input size.

LLMModelConfigResponse

Field	Type	Description
config_id	UUID	Configuration ID
name	string	Configuration name
model_type	string	Model type
config	LLMModelConfigData	Configuration data (provider, base_url, model, temperature, etc.)
user_id	UUID (nullable)	Owner user ID
group_id	UUID (nullable)	Owner group ID
is_default	boolean	Default flag
version	integer	Current version number (for optimistic locking)
created_at	TIMESTAMP	Creation time
updated_at	TIMESTAMP	Last update time

LLMModelConfigInheritedResponse

Field	Type	Description
configs	list[LLMModelConfigWithSource]	Effective configurations
default_config	LLMModelConfigWithSource (nullable)	Default configuration (never null if configs list is not empty)
total	integer	Total count

Default Configuration Selection Logic:

1. User's config marked with is_default: true (highest priority)
2. Group's config marked with is_default: true
3. First config in the list (user configs come before group configs)

LLMModelConfigListResponse

Field	Type	Description
configs	list[LLMModelConfigResponse]	Configuration list
default_config	LLMModelConfigResponse (nullable)	Default configuration (never null if configs list is not empty)
total	integer	Total count

Default Configuration Selection Logic:

1. Config marked with is_default: true
2. First config in the list (fallback if no default is marked)

Usage: This schema is used for group configuration endpoints (e.g., GET /groups/{group_id}/llm-model-configs).

LLMModelConfigWithSource

Field	Type	Description
config_id	UUID	Configuration ID
name	string	Configuration name
model_type	string	Model type
config	LLMModelConfigData	Configuration data (provider, base_url, model, temperature, etc.)
user_id	UUID (nullable)	Owner user ID
group_id	UUID (nullable)	Owner group ID
is_default	boolean	Default flag
version	integer	Optimistic locking version
created_at	TIMESTAMP	Creation time
updated_at	TIMESTAMP	Last update time
source	string	Source: "user" or "group"
group_name	string (nullable)	Group name if source is "group"

---

Usage Examples

1. Create a user configuration

curl -X POST http://localhost:3080/v3/access/users/{user_id}/llm-model-configs \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "GPT-4o",
    "model_type": "text",
    "provider": "openai",
    "base_url": "https://api.openai.com/v1",
    "model": "gpt-4o",
    "temperature": 0.7,
    "context_limit": 128,
    "context_strategy": "balanced",
    "api_key": "sk-xxx",
    "copilot_mode": "teaching_assistant",
    "is_default": true
  }'

Important:

• context_limit is required and specified in K tokens (e.g., 128 = 128K = 128,000 tokens)
• Refer to the model provider's official documentation for the current context window size

Response:

{
  "config_id": "uuid-1",
  "name": "GPT-4o",
  "model_type": "text",
  "config": {
    "provider": "openai",
    "base_url": "https://api.openai.com/v1",
    "model": "gpt-4o",
    "temperature": 0.7,
    "context_limit": 128,
    "context_strategy": "balanced",
    "api_key": null,
    "copilot_mode": null
  },
  "user_id": "uuid-user",
  "group_id": null,
  "is_default": true,
  "version": 0,
  "created_at": "2026-03-03T12:00:00Z",
  "updated_at": "2026-03-03T12:00:00Z"
}

Note: The api_key field is always null in responses for security. The API key is encrypted and stored in the database, but never returned via the API.

2. Create a group configuration

curl -X POST http://localhost:3080/v3/access/groups/{group_id}/llm-model-configs \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Claude-3.5 Sonnet",
    "model_type": "text",
    "provider": "anthropic",
    "base_url": "https://api.anthropic.com",
    "model": "claude-3-5-sonnet-20241022",
    "temperature": 0.7,
    "context_limit": 200,
    "context_strategy": "balanced",
    "api_key": "sk-ant-xxx",
    "copilot_mode": "lab_automation_assistant",
    "is_default": true
  }'

3. Get user's effective configurations (with inheritance)

curl -X GET http://localhost:3080/v3/access/users/{user_id}/llm-model-configs \
  -H "Authorization: Bearer <token>"

Response (user has both own configs and inherited group configs):

{
  "configs": [
    {
      "config_id": "uuid-1",
      "name": "GPT-4o",
      "model_type": "text",
      "config": {
        "provider": "openai",
        "base_url": "https://api.openai.com/v1",
        "model": "gpt-4o",
        "temperature": 0.7,
        "context_limit": 128,
        "context_strategy": "balanced",
        "api_key": null,
        "copilot_mode": "lab_automation_assistant"
      },
      "user_id": "uuid-user",
      "group_id": null,
      "is_default": true,
      "version": 0,
      "created_at": "2026-03-03T14:32:48.158880Z",
      "updated_at": "2026-03-03T14:32:48.158880Z",
      "source": "user",
      "group_name": null
    },
    {
      "config_id": "uuid-2",
      "name": "Claude-3.5 Sonnet",
      "model_type": "text",
      "config": {
        "provider": "anthropic",
        "base_url": "https://api.anthropic.com",
        "model": "claude-3-5-sonnet-20241022",
        "temperature": 0.7,
        "context_limit": 200,
        "context_strategy": "balanced",
        "api_key": null,
        "copilot_mode": null
      },
      "user_id": null,
      "group_id": "uuid-group",
      "is_default": true,
      "version": 0,
      "created_at": "2026-03-03T14:32:48.158880Z",
      "updated_at": "2026-03-03T14:32:48.158880Z",
      "source": "group",
      "group_name": "Developers"
    }
  ],
  "default_config": {
    "config_id": "uuid-1",
    "name": "GPT-4o",
    "model_type": "text",
    "config": {
      "provider": "openai",
      "base_url": "https://api.openai.com/v1",
      "model": "gpt-4o",
      "temperature": 0.7,
      "context_limit": 128,
      "context_strategy": "balanced",
      "api_key": null,
      "copilot_mode": "lab_automation_assistant"
    },
    "user_id": "uuid-user",
    "group_id": null,
    "is_default": true,
    "version": 0,
    ...
  },
  "total": 2
}

Note:

• All configs show config.api_key: null (always hidden for security)
• source: "user" indicates the config belongs to the user
• source: "group" indicates the config is inherited from a group
• Configuration fields are nested in the config object (same structure as group endpoints)
• context_limit is in K tokens (128 = 128K = 128,000 tokens)

4. Get group configurations

curl -X GET http://localhost:3080/v3/access/groups/{group_id}/llm-model-configs \
  -H "Authorization: Bearer <token>"

Response:

{
  "configs": [
    {
      "config_id": "uuid-1",
      "name": "Claude-3",
      "model_type": "text",
      "config": {
        "provider": "anthropic",
        "base_url": "https://api.anthropic.com",
        "model": "claude-3-opus-20240229",
        "temperature": 0.7,
        "context_limit": 200,
        "context_strategy": "balanced",
        "api_key": null,
        "copilot_mode": null
      },
      "user_id": null,
      "group_id": "uuid-group",
      "is_default": true,
      "version": 0,
      "created_at": "2026-03-03T12:00:00Z",
      "updated_at": "2026-03-03T12:00:00Z"
    },
    {
      "config_id": "uuid-2",
      "name": "GPT-4",
      "model_type": "text",
      "config": {
        "provider": "openai",
        "base_url": "https://api.openai.com/v1",
        "model": "gpt-4",
        "temperature": 0.7,
        "context_limit": 128,
        "context_strategy": "balanced",
        "api_key": null,
        "copilot_mode": null
      },
      "user_id": null,
      "group_id": "uuid-group",
      "is_default": false,
      "version": 0,
      "created_at": "2026-03-03T12:00:00Z",
      "updated_at": "2026-03-03T12:00:00Z"
    }
  ],
  "default_config": {
    "config_id": "uuid-1",
    "name": "Claude-3",
    "model_type": "text",
    "config": {
      "provider": "anthropic",
      ...
    },
    "user_id": null,
    "group_id": "uuid-group",
    "is_default": true,
    "version": 0,
    ...
  },
  "total": 2
}

Note: The response structure is the same as user endpoints, with configs, default_config, and total fields. All api_key values are null for security.

5. Update a configuration (without optimistic locking)

curl -X PUT http://localhost:3080/v3/access/users/{user_id}/llm-model-configs/{config_id} \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "temperature": 0.9,
    "max_tokens": 4000,
    "context_strategy": "aggressive"
  }'

6. Update a configuration (WITH optimistic locking)

Best practice for avoiding concurrent modification conflicts:

# Step 1: Read the config (get the current version)
curl -X GET http://localhost:3080/v3/access/users/{user_id}/llm-model-configs/own \
  -H "Authorization: Bearer <token>"

# Response includes "version": 5

# Step 2: Update with expected_version
curl -X PUT http://localhost:3080/v3/access/users/{user_id}/llm-model-configs/{config_id} \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "temperature": 0.9,
    "max_tokens": 4000,
    "expected_version": 5
  }'

# Response includes incremented "version": 6

If someone else modified the config before you:

HTTP 409 Conflict
{
  "detail": "Concurrent modification detected. Expected version 5, but current version is 6. Please retry."
}

Client retry flow:

1. Receive 409 Conflict error
2. Re-fetch the config to get the latest version
3. Apply your changes on top of the latest data
4. Retry the update with the new expected_version

7. Set default configuration

curl -X PUT http://localhost:3080/v3/access/users/{user_id}/llm-model-configs/default/{config_id} \
  -H "Authorization: Bearer <token>"

8. Get default configuration

Get the user's default configuration:

curl -X GET http://localhost:3080/v3/access/users/{user_id}/llm-model-configs/default \
  -H "Authorization: Bearer <token>"

Note: This endpoint only returns configurations explicitly marked with is_default: true. If no configuration is marked as default, it returns 404.

Response:

{
  "config_id": "uuid-1",
  "name": "GPT-4o",
  "model_type": "text",
  "config": {
    "provider": "openai",
    "base_url": "https://api.openai.com/v1",
    "model": "gpt-4o",
    "temperature": 0.7,
    "context_limit": 128,
    "context_strategy": "balanced",
    "api_key": null,
    "copilot_mode": null
  },
  "user_id": "uuid-user",
  "group_id": null,
  "is_default": true,
  "version": 0,
  "created_at": "2026-03-03T18:15:00Z",
  "updated_at": "2026-03-03T18:15:00Z"
}

Note: The api_key field is always null in responses for security.

If no default configuration is set:

HTTP 404 Not Found
{
  "detail": "No default LLM model configuration found for user '{user_id}'"
}

Get the group's default configuration:

curl -X GET http://localhost:3080/v3/access/groups/{group_id}/llm-model-configs/default \
  -H "Authorization: Bearer <token>"

The response format is the same as for users.

---

Important Note: This dedicated /default endpoint is different from the default_config field in the list response:

• /default endpoint: Requires explicit is_default: true flag, returns 404 if not found
• default_config field in list: Falls back to first config if no explicit default is marked

9. Delete a configuration

curl -X DELETE http://localhost:3080/v3/access/users/{user_id}/llm-model-configs/{config_id} \
  -H "Authorization: Bearer <token>"

---

Error Codes

Status	Description
200	Success
201	Created
204	Deleted (no content)
400	Bad request
401	Unauthorized
404	Not found
409	Conflict (optimistic lock violation)
500	Server error

409 Conflict Response

{
  "detail": "Concurrent modification detected. Expected version 5, but current version is 6. Please retry."
}

---

Concurrency Control

Optimistic Locking

This API uses optimistic locking to prevent concurrent modification conflicts:

1. Version Tracking: Each configuration has a version field that starts at 0 and increments on each update
2. Read-Modify-Write: When updating, clients should include the expected_version from their last read
3. Conflict Detection: If the provided version doesn't match the current version, the update is rejected with HTTP 409

When to Use Optimistic Locking

Use expected_version when:

• Multiple users/admins might modify the same configuration
• You want to prevent accidental overwrites of concurrent changes
• Building interactive UIs that display and edit configurations

Skip expected_version when:

• You're sure no one else is modifying the config
• Performance is more important than data integrity (not recommended)

---

Security Notes

API Key Encryption

All API keys are encrypted using Fernet symmetric encryption (AES-128-CBC). Encryption keys are stored in {secrets_dir}/gns3_encryption_key with 0600 permissions.

Access Control

All endpoints require appropriate privileges:

• User.Audit: View user configurations
• User.Modify: Create, update, delete user configurations
• Group.Audit: View group configurations
• Group.Modify: Create, update, delete group configurations

API Key Visibility

The API implements strict API key visibility controls to protect sensitive credentials:

Security Policy: API keys are NEVER returned via API endpoints

Scenario	User Configs	Group Configs
User viewing own configs	Hidden (null)	N/A
User viewing inherited group configs	N/A	Hidden (null)
Admin viewing other users' configs	Hidden (null)	N/A
Viewing group configs directly (with Group.Audit)	N/A	Hidden (null)

Rules:

1. All API responses: api_key field is always set to null in all API responses
2. Database storage: API keys are encrypted using Fernet symmetric encryption before storage
3. Internal use only: API keys are only decrypted internally by the system (e.g., when the Copilot Agent makes LLM API calls)
4. Super admins: Can access the database directly and decrypt any API key using the encryption key - this is by design as super admins have system-level access

Important Notes:

• API keys are stored in the database in encrypted format using Fernet symmetric encryption
• No API endpoint returns API keys - the api_key field is always null in responses
• API keys can still be created and updated via POST/PUT endpoints
• The system internally decrypts API keys when needed (e.g., for making LLM API calls)
• Super admins with database access can retrieve & decrypt any API key - this is intentional and reflects their system-level privileges

⚠️ Security First Design

API keys are never exposed through the application API.

• All GET endpoints return api_key: null
• POST/PUT endpoints accept API keys for storage, but responses still return api_key: null
• This prevents API keys from being leaked through logs, browser devtools, or network monitoring
• API keys are only decrypted internally by the system when making actual LLM API calls

Design Rationale: This defense-in-depth approach ensures that even if someone gains access to API logs or responses, they will never find valid API keys.

Example:

// User viewing their own configs
{
  "configs": [
    {
      "config_id": "uuid-1",
      "name": "GPT-4",
      "source": "user",
      "config": {
        "api_key": null  // Always hidden, even for own configs
      }
    },
    {
      "config_id": "uuid-2",
      "name": "Claude-3",
      "source": "group",
      "config": {
        "api_key": null  // Always hidden
      }
    }
  ]
}

// Admin viewing another user's configs
{
  "configs": [
    {
      "config_id": "uuid-1",
      "name": "GPT-4",
      "source": "user",
      "config": {
        "api_key": null  // Always hidden
      }
    }
  ]
}

// User with Group.Audit viewing group configs directly
{
  "configs": [
    {
      "config_id": "uuid-3",
      "name": "Gemini Pro",
      "config": {
        "api_key": null  // Always hidden
      }
    }
  ]
}

// Creating a new config (request includes api_key)
POST /users/{user_id}/llm-model-configs
{
  "name": "GPT-4",
  "api_key": "sk-xxxxx"  // Sent in request
}

// Response (api_key is filtered)
{
  "config_id": "uuid-1",
  "name": "GPT-4",
  "config": {
    "api_key": null  // Always null in responses
  }
}

---

Migration from Old User Settings API

The old user settings API (/v3/access/users/{user_id}/profiles) stored configurations in the users.model_configs JSON column. This new API uses a dedicated table with better inheritance support and optimistic locking.

Migration strategy:

1. Run the database migration to create the llm_model_configs table
2. Optionally migrate existing data from users.model_configs to the new table
3. Update clients to use the new API endpoints
4. Update clients to handle version field and 409 Conflict errors
5. Deprecate the old /profiles endpoints

Key differences:

• Inheritance: Users without configs inherit from groups (automatic fallback)
• Optimistic locking: New version field and expected_version parameter
• Dedicated table: Better query performance and data integrity
• Transparent encryption: API keys auto-encrypted/decrypted by the API
• Model type support: New model_type field for categorizing models (text, vision, stt, tts, multimodal, embedding, reranking, other)
• Table-level indexing: name and model_type stored as table columns for efficient filtering and querying

---

Model Type Filtering

The model_type table column enables efficient filtering and querying of configurations by model type:

Common Use Cases

1. Filter by model type: Retrieve only text generation models for chat features
2. Multi-model applications: Select appropriate model based on task type (text vs vision vs embedding)
3. Model type analytics: Query and analyze usage patterns by model type
4. Type-specific defaults: Set different default models for different model types

Example: Filter text models (client-side)

# After fetching configs, filter by model_type
configs = get_user_configs(user_id)
text_models = [c for c in configs if c["model_type"] == "text"]
vision_models = [c for c in configs if c["model_type"] == "vision"]

Database Index

The model_type column is indexed for efficient queries:

CREATE INDEX idx_llm_model_configs_model_type ON llm_model_configs(model_type);

This enables fast lookups when filtering by model type, even with large datasets.

---

Reserved Fields

The llm_model_configs table includes three reserved JSONB fields for future extensibility:

Field	Type	Description
reserved_jsonb_1	JSONB (nullable)	Reserved for future use
reserved_jsonb_2	JSONB (nullable)	Reserved for future use
reserved_jsonb_3	JSONB (nullable)	Reserved for future use

Purpose: These fields are reserved for future feature development without requiring schema changes. They are currently unused in the API code but are available in the database layer for future enhancements.

Use Cases: Future features might use these fields for:

• Advanced configuration options
• Metadata storage
• Feature flags
• Extension data
• Caching computed values

Note: These fields are not exposed in the current API schemas and are reserved for internal use.

---

Context Limit Configuration

What is context_limit?

The context_limit field specifies the maximum context window size for an LLM model. This is a required field for all model configurations.

Why is it required?

Model providers frequently update their models and change context window sizes:

• OpenAI GPT-4o: 128K tokens (may change)
• Anthropic Claude 3.5: 200K tokens (may change)
• Google Gemini 1.5: 2.8M tokens (may change)

Hardcoding these values in the system would quickly become outdated. Requiring users to configure this field ensures that the system always uses the correct, up-to-date values.

Unit: K tokens

The context_limit value is specified in K tokens (thousands of tokens) to make it easier to read and write:

Official Documentation	API Configuration
128,000 tokens	"context_limit": 128
200,000 tokens	"context_limit": 200
2,800,000 tokens	"context_limit": 2800

How to find the correct value

1. Check the official documentation for your model:

   • OpenAI: https://platform.openai.com/docs/models
   • Anthropic: https://docs.anthropic.com/claude/docs/models-overview
   • Google: https://ai.google.dev/gemini-api/docs/models
   • DeepSeek: https://platform.deepseek.com/api-docs/

2. Convert from tokens to K:

   context_limit = official_value_in_tokens / 1000
   
   Example:
   GPT-4o: 128,000 tokens / 1000 = 128
   

Example: Common Models

Model	Official Value	Configuration
GPT-4o	128,000	"context_limit": 128
GPT-3.5 Turbo	16,385	"context_limit": 17
Claude 3.5 Sonnet	200,000	"context_limit": 200
Gemini 1.5 Pro	2,800,000	"context_limit": 2800
DeepSeek Chat	128,000	"context_limit": 128

Context Strategy

The optional context_strategy field controls how aggressively the system uses the available context window:

Strategy	Usage	Best For
conservative	60% of limit	Long outputs, complex tasks, uncertain output size
balanced (default)	75% of limit	Most conversations, general use
aggressive	85% of limit	Short outputs, analysis tasks, predictable output size

Error Handling

If context_limit is missing or invalid, the API will return:

HTTP 400 Bad Request
{
  "detail": "context_limit is required (unit: K tokens, e.g., 128 = 128K = 128,000 tokens). Please check your model provider's documentation for the current context window size and specify it in the configuration."
}