NgTech-LTD/gns3-server
2
0
Fork 0
mirror of https://github.com/GNS3/gns3-server.git synced 2026-08-27 12:30:13 +03:00
Code Issues Packages Projects Releases Wiki Activity
gns3-server/docs/gns3-copilot/llm-model-configs-api.md
YueGuobin 13a032ea2c chore: update author name and copyright headers 
Updated the author name and copyright statements across the
gns3_copilot module. The name has been standardized from
"Guobin Yue" to "Yue Guobin (岳国宾)" to reflect the correct
author attribution including Chinese characters.
2026-03-09 11:46:28 +08:00

31 KiB
Raw Blame History

LLM Model Configurations API

Overview

This API provides LLM model configuration management for users and user groups with inheritance support.

Key Features

  • User-level configurations: Each user can have their own LLM model configurations
  • Group-level configurations: User groups can share LLM model configurations
  • Inheritance: Users automatically inherit configurations from their groups (when they have no own configs)
  • Default configuration: Both users and groups can set a default configuration
  • API Key Encryption: API keys are automatically encrypted in the database
  • Optimistic Locking: Prevents concurrent modification conflicts using version tracking

Inheritance Logic

User requests configs:
  ├─ Always return user's own configs (if any)
  └─ Always return inherited group configs (if any)

Note: Users can see both their own configurations AND configurations inherited from their groups. The source field in the response indicates the origin of each configuration.

Configuration Priority

User's own config > User's group config

Database Schema

Table: llm_model_configs

Column Type Description
config_id UUID Primary key
name VARCHAR(100) Configuration name (table-level for indexing)
model_type VARCHAR(50) Model type (table-level for filtering)
config JSONB Configuration data (provider, base_url, model, temperature, api_key, etc.)
user_id UUID (nullable) Foreign key to users table
group_id UUID (nullable) Foreign key to user_groups table
is_default BOOLEAN Default configuration flag
version INTEGER Optimistic locking version (starts at 0, increments on each update)
reserved_jsonb_1 JSONB (nullable) Reserved field for future use
reserved_jsonb_2 JSONB (nullable) Reserved field for future use
reserved_jsonb_3 JSONB (nullable) Reserved field for future use
created_at TIMESTAMP Creation timestamp
updated_at TIMESTAMP Last update timestamp

Model Types

The model_type field accepts the following values:

  • text - Text generation models
  • vision - Vision/image understanding models
  • stt - Speech-to-Text models
  • tts - Text-to-Speech models
  • multimodal - Multimodal models supporting multiple input types
  • embedding - Text embedding models
  • reranking - Reranking models
  • other - Other model types

Constraints

  • Each config belongs to either a user or a group (not both)
  • Each user can have at most one default configuration
  • Each group can have at most one default configuration
  • version field is automatically incremented on each update

API Endpoints

User Configuration Endpoints

Method Path Description Privilege
GET /v3/access/users/{user_id}/llm-model-configs Get user's effective configs (own + inherited) User.Audit
GET /v3/access/users/{user_id}/llm-model-configs/own Get user's own configs only User.Audit
GET /v3/access/users/{user_id}/llm-model-configs/default Get user's default configuration User.Audit
POST /v3/access/users/{user_id}/llm-model-configs Create a new configuration User.Modify
PUT /v3/access/users/{user_id}/llm-model-configs/{config_id} Update a configuration User.Modify
DELETE /v3/access/users/{user_id}/llm-model-configs/{config_id} Delete a configuration User.Modify
PUT /v3/access/users/{user_id}/llm-model-configs/default/{config_id} Set default configuration User.Modify

Group Configuration Endpoints

Method Path Description Privilege
GET /v3/access/groups/{group_id}/llm-model-configs Get all group configurations Group.Audit
GET /v3/access/groups/{group_id}/llm-model-configs/default Get group's default configuration Group.Audit
POST /v3/access/groups/{group_id}/llm-model-configs Create a new configuration Group.Modify
PUT /v3/access/groups/{group_id}/llm-model-configs/{config_id} Update a configuration Group.Modify
DELETE /v3/access/groups/{group_id}/llm-model-configs/{config_id} Delete a configuration Group.Modify
PUT /v3/access/groups/{group_id}/llm-model-configs/default/{config_id} Set default configuration Group.Modify

Note: The GET endpoints for groups return the same structure as user endpoints: configs, default_config, and total.

Request/Response Schemas

LLMModelConfigCreate

Required Fields:

Field Type Description
name string Configuration name (1-100 chars)
model_type string Model type (text, vision, stt, tts, multimodal, embedding, reranking, other)
provider string LLM provider (e.g., "openai", "anthropic", "ollama")
base_url string API base URL
model string Model name
temperature float Temperature (0.0-2.0, default: 0.7)
context_limit integer Model context window limit in K tokens (e.g., 128 = 128K = 128,000 tokens)

Optional Fields:

Field Type Description
api_key string API key (auto-encrypted)
max_tokens integer Max tokens for generation
context_strategy string Context trimming strategy: "conservative" (60%), "balanced" (75%), "aggressive" (85%). Default: "balanced"
copilot_mode string GNS3-Copilot mode: "teaching_assistant" (diagnostics only, default) or "lab_automation_assistant" (full configuration access)
is_default boolean Set as default (default: false)

Important Notes:

  • context_limit is required: You must specify the model's context window limit. Refer to the model provider's official documentation for the current value.
  • Unit is K tokens: The value is in thousands of tokens (1 K = 1,000 tokens). For example:
    • GPT-4o: 128,000 tokens → configure as "context_limit": 128
    • Claude 3.5 Sonnet: 200,000 tokens → configure as "context_limit": 200
    • Gemini 1.5 Pro: 2,800,000 tokens → configure as "context_limit": 2800

Extra Fields: Any custom fields are supported for future extensibility.

LLMModelConfigUpdate

Field Type Description
name string (optional) Configuration name
model_type string (optional) Model type
provider string (optional) LLM provider
base_url string (optional) API base URL
model string (optional) Model name
temperature float (optional) Temperature
api_key string (optional) API key
max_tokens integer (optional) Max tokens
context_limit integer (optional) Model context window limit in K tokens
context_strategy string (optional) Context trimming strategy
is_default boolean (optional) Default flag
expected_version integer (optional) Optimistic locking version

Note: When using expected_version, the API will verify the version hasn't changed since you read the data. If it has, you'll receive a 409 Conflict error.

LLMModelConfigResponse

Field Type Description
config_id UUID Configuration ID
name string Configuration name
model_type string Model type
config LLMModelConfigData Configuration data (provider, base_url, model, temperature, etc.)
user_id UUID (nullable) Owner user ID
group_id UUID (nullable) Owner group ID
is_default boolean Default flag
version integer Current version number (for optimistic locking)
created_at TIMESTAMP Creation time
updated_at TIMESTAMP Last update time

LLMModelConfigInheritedResponse

Field Type Description
configs list[LLMModelConfigWithSource] Effective configurations
default_config LLMModelConfigWithSource (nullable) Default configuration (never null if configs list is not empty)
total integer Total count

Default Configuration Selection Logic:

  1. User's config marked with is_default: true (highest priority)
  2. Group's config marked with is_default: true
  3. First config in the list (user configs come before group configs)

LLMModelConfigListResponse

Field Type Description
configs list[LLMModelConfigResponse] Configuration list
default_config LLMModelConfigResponse (nullable) Default configuration (never null if configs list is not empty)
total integer Total count

Default Configuration Selection Logic:

  1. Config marked with is_default: true
  2. First config in the list (fallback if no default is marked)

Usage: This schema is used for group configuration endpoints (e.g., GET /groups/{group_id}/llm-model-configs).

LLMModelConfigWithSource

Field Type Description
config_id UUID Configuration ID
name string Configuration name
model_type string Model type
config LLMModelConfigData Configuration data (provider, base_url, model, temperature, etc.)
user_id UUID (nullable) Owner user ID
group_id UUID (nullable) Owner group ID
is_default boolean Default flag
version integer Optimistic locking version
created_at TIMESTAMP Creation time
updated_at TIMESTAMP Last update time
source string Source: "user" or "group"
group_name string (nullable) Group name if source is "group"

Usage Examples

1. Create a user configuration

curl -X POST http://localhost:3080/v3/access/users/{user_id}/llm-model-configs \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "GPT-4o",
    "model_type": "text",
    "provider": "openai",
    "base_url": "https://api.openai.com/v1",
    "model": "gpt-4o",
    "temperature": 0.7,
    "context_limit": 128,
    "context_strategy": "balanced",
    "api_key": "sk-xxx",
    "copilot_mode": "teaching_assistant",
    "is_default": true
  }'

Important:

  • context_limit is required and specified in K tokens (e.g., 128 = 128K = 128,000 tokens)
  • Refer to the model provider's official documentation for the current context window size

Response:

{
  "config_id": "uuid-1",
  "name": "GPT-4o",
  "model_type": "text",
  "config": {
    "provider": "openai",
    "base_url": "https://api.openai.com/v1",
    "model": "gpt-4o",
    "temperature": 0.7,
    "context_limit": 128,
    "context_strategy": "balanced",
    "api_key": "sk-xxx",
    "copilot_mode": null
  },
  "user_id": "uuid-user",
  "group_id": null,
  "is_default": true,
  "version": 0,
  "created_at": "2026-03-03T12:00:00Z",
  "updated_at": "2026-03-03T12:00:00Z"
}

2. Create a group configuration

curl -X POST http://localhost:3080/v3/access/groups/{group_id}/llm-model-configs \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Claude-3.5 Sonnet",
    "model_type": "text",
    "provider": "anthropic",
    "base_url": "https://api.anthropic.com",
    "model": "claude-3-5-sonnet-20241022",
    "temperature": 0.7,
    "context_limit": 200,
    "context_strategy": "balanced",
    "api_key": "sk-ant-xxx",
    "copilot_mode": "lab_automation_assistant",
    "is_default": true
  }'

3. Get user's effective configurations (with inheritance)

curl -X GET http://localhost:3080/v3/access/users/{user_id}/llm-model-configs \
  -H "Authorization: Bearer <token>"

Response (user has both own configs and inherited group configs):

{
  "configs": [
    {
      "config_id": "uuid-1",
      "name": "GPT-4o",
      "model_type": "text",
      "config": {
        "provider": "openai",
        "base_url": "https://api.openai.com/v1",
        "model": "gpt-4o",
        "temperature": 0.7,
        "context_limit": 128,
        "context_strategy": "balanced",
        "api_key": "sk-xxx",
        "copilot_mode": "lab_automation_assistant"
      },
      "user_id": "uuid-user",
      "group_id": null,
      "is_default": true,
      "version": 0,
      "created_at": "2026-03-03T14:32:48.158880Z",
      "updated_at": "2026-03-03T14:32:48.158880Z",
      "source": "user",
      "group_name": null
    },
    {
      "config_id": "uuid-2",
      "name": "Claude-3.5 Sonnet",
      "model_type": "text",
      "config": {
        "provider": "anthropic",
        "base_url": "https://api.anthropic.com",
        "model": "claude-3-5-sonnet-20241022",
        "temperature": 0.7,
        "context_limit": 200,
        "context_strategy": "balanced",
        "api_key": null,
        "copilot_mode": null
      },
      "user_id": null,
      "group_id": "uuid-group",
      "is_default": true,
      "version": 0,
      "created_at": "2026-03-03T14:32:48.158880Z",
      "updated_at": "2026-03-03T14:32:48.158880Z",
      "source": "group",
      "group_name": "Developers"
    }
  ],
  "default_config": {
    "config_id": "uuid-1",
    "name": "GPT-4o",
    "model_type": "text",
    "config": {
      "provider": "openai",
      "base_url": "https://api.openai.com/v1",
      "model": "gpt-4o",
      "temperature": 0.7,
      "context_limit": 128,
      "context_strategy": "balanced",
      "api_key": "sk-xxx",
      "copilot_mode": "lab_automation_assistant"
    },
    "user_id": "uuid-user",
    "group_id": null,
    "is_default": true,
    "version": 0,
    ...
  },
  "total": 2
}

Note:

  • User's own config shows config.api_key: "sk-xxx" (visible to owner)
  • Inherited group config shows config.api_key: null (hidden from users)
  • source: "user" indicates the config belongs to the user
  • source: "group" indicates the config is inherited from a group
  • Configuration fields are nested in the config object (same structure as group endpoints)
  • context_limit is in K tokens (128 = 128K = 128,000 tokens)

4. Get group configurations

curl -X GET http://localhost:3080/v3/access/groups/{group_id}/llm-model-configs \
  -H "Authorization: Bearer <token>"

Response:

{
  "configs": [
    {
      "config_id": "uuid-1",
      "name": "Claude-3",
      "model_type": "text",
      "config": {
        "provider": "anthropic",
        "base_url": "https://api.anthropic.com",
        "model": "claude-3-opus-20240229",
        "temperature": 0.7,
        "context_limit": 200,
        "context_strategy": "balanced",
        "api_key": "sk-ant-xxx",
        "copilot_mode": null
      },
      "user_id": null,
      "group_id": "uuid-group",
      "is_default": true,
      "version": 0,
      "created_at": "2026-03-03T12:00:00Z",
      "updated_at": "2026-03-03T12:00:00Z"
    },
    {
      "config_id": "uuid-2",
      "name": "GPT-4",
      "model_type": "text",
      "config": {
        "provider": "openai",
        "base_url": "https://api.openai.com/v1",
        "model": "gpt-4",
        "temperature": 0.7,
        "context_limit": 128,
        "context_strategy": "balanced",
        "api_key": "sk-xxx",
        "copilot_mode": null
      },
      "user_id": null,
      "group_id": "uuid-group",
      "is_default": false,
      "version": 0,
      "created_at": "2026-03-03T12:00:00Z",
      "updated_at": "2026-03-03T12:00:00Z"
    }
  ],
  "default_config": {
    "config_id": "uuid-1",
    "name": "Claude-3",
    "model_type": "text",
    "config": {
      "provider": "anthropic",
      ...
    },
    "user_id": null,
    "group_id": "uuid-group",
    "is_default": true,
    "version": 0,
    ...
  },
  "total": 2
}

Note: The response structure is the same as user endpoints, with configs, default_config, and total fields.

5. Update a configuration (without optimistic locking)

curl -X PUT http://localhost:3080/v3/access/users/{user_id}/llm-model-configs/{config_id} \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "temperature": 0.9,
    "max_tokens": 4000,
    "context_strategy": "aggressive"
  }'

6. Update a configuration (WITH optimistic locking)

Best practice for avoiding concurrent modification conflicts:

# Step 1: Read the config (get the current version)
curl -X GET http://localhost:3080/v3/access/users/{user_id}/llm-model-configs/own \
  -H "Authorization: Bearer <token>"

# Response includes "version": 5

# Step 2: Update with expected_version
curl -X PUT http://localhost:3080/v3/access/users/{user_id}/llm-model-configs/{config_id} \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "temperature": 0.9,
    "max_tokens": 4000,
    "expected_version": 5
  }'

# Response includes incremented "version": 6

If someone else modified the config before you:

HTTP 409 Conflict
{
  "detail": "Concurrent modification detected. Expected version 5, but current version is 6. Please retry."
}

Client retry flow:

  1. Receive 409 Conflict error
  2. Re-fetch the config to get the latest version
  3. Apply your changes on top of the latest data
  4. Retry the update with the new expected_version

7. Set default configuration

curl -X PUT http://localhost:3080/v3/access/users/{user_id}/llm-model-configs/default/{config_id} \
  -H "Authorization: Bearer <token>"

8. Get default configuration

Get the user's default configuration:

curl -X GET http://localhost:3080/v3/access/users/{user_id}/llm-model-configs/default \
  -H "Authorization: Bearer <token>"

Note: This endpoint only returns configurations explicitly marked with is_default: true. If no configuration is marked as default, it returns 404.

Response:

{
  "config_id": "uuid-1",
  "name": "GPT-4o",
  "model_type": "text",
  "config": {
    "provider": "openai",
    "base_url": "https://api.openai.com/v1",
    "model": "gpt-4o",
    "temperature": 0.7,
    "context_limit": 128,
    "context_strategy": "balanced",
    "api_key": "sk-xxx",
    "copilot_mode": null
  },
  "user_id": "uuid-user",
  "group_id": null,
  "is_default": true,
  "version": 0,
  "created_at": "2026-03-03T18:15:00Z",
  "updated_at": "2026-03-03T18:15:00Z"
}

If no default configuration is set:

HTTP 404 Not Found
{
  "detail": "No default LLM model configuration found for user '{user_id}'"
}

Get the group's default configuration:

curl -X GET http://localhost:3080/v3/access/groups/{group_id}/llm-model-configs/default \
  -H "Authorization: Bearer <token>"

The response format is the same as for users.

Important Note: This dedicated /default endpoint is different from the default_config field in the list response:

  • /default endpoint: Requires explicit is_default: true flag, returns 404 if not found
  • default_config field in list: Falls back to first config if no explicit default is marked

9. Delete a configuration

curl -X DELETE http://localhost:3080/v3/access/users/{user_id}/llm-model-configs/{config_id} \
  -H "Authorization: Bearer <token>"

Error Codes

Status Description
200 Success
201 Created
204 Deleted (no content)
400 Bad request
401 Unauthorized
404 Not found
409 Conflict (optimistic lock violation)
500 Server error

409 Conflict Response

{
  "detail": "Concurrent modification detected. Expected version 5, but current version is 6. Please retry."
}

Concurrency Control

Optimistic Locking

This API uses optimistic locking to prevent concurrent modification conflicts:

  1. Version Tracking: Each configuration has a version field that starts at 0 and increments on each update
  2. Read-Modify-Write: When updating, clients should include the expected_version from their last read
  3. Conflict Detection: If the provided version doesn't match the current version, the update is rejected with HTTP 409

When to Use Optimistic Locking

Use expected_version when:

  • Multiple users/admins might modify the same configuration
  • You want to prevent accidental overwrites of concurrent changes
  • Building interactive UIs that display and edit configurations

Skip expected_version when:

  • You're sure no one else is modifying the config
  • Performance is more important than data integrity (not recommended)

Security Notes

API Key Encryption

All API keys are encrypted using Fernet symmetric encryption (AES-128-CBC). Encryption keys are stored in {secrets_dir}/gns3_encryption_key with 0600 permissions.

Access Control

All endpoints require appropriate privileges:

  • User.Audit: View user configurations
  • User.Modify: Create, update, delete user configurations
  • Group.Audit: View group configurations
  • Group.Modify: Create, update, delete group configurations

API Key Visibility

The API implements strict API key visibility controls to protect sensitive credentials:

Scenario User Configs Group Configs
User viewing own configs Visible (Plaintext) N/A
User viewing inherited group configs N/A Hidden (null)
Admin viewing other users' configs Hidden (null) N/A
Viewing group configs directly (with Group.Audit) N/A Visible (Encrypted)

Rules:

  1. Users viewing their own configs: Can see decrypted (plaintext) API keys in their own configurations
  2. Users viewing inherited group configs: API keys are hidden (set to null) in the inherited configs
  3. Admins viewing other users' configs: Cannot see API keys in any user configurations (set to null) - user privacy protection
  4. Viewing group configs directly: Users with Group.Audit privilege can see encrypted API keys in group configurations (not decrypted)
  5. Super admins: While application-layer restrictions apply, super admins can access the database directly and decrypt any API key using the encryption key. This is by design as super admins have system-level access.

Important Notes:

  • API keys are stored in the database in encrypted format using Fernet symmetric encryption
  • User configs are decrypted on-the-fly when retrieved by the owner
  • Group configs return the encrypted value as stored in the database (no automatic decryption)
  • Super admins have database access and can retrieve & decrypt any API key - this is intentional and reflects their system-level privileges

⚠️ Key Point: No Plaintext Group API Keys via API

Nobody (including super admins) can see plaintext API keys for group configurations through the application API.

  • Group configs always return the encrypted API key value (e.g., gAAAAABl1a2b3c4d5e6f7...)
  • There is no API endpoint that decrypts and returns group config API keys as plaintext
  • Even super admins with Group.Audit privilege receive encrypted values via the API
  • This is an application-layer restriction that applies to all users

Design Rationale: Group configs are intended to be inherited automatically, not viewed/copied manually. The encrypted values protect API keys while still allowing the inheritance mechanism to function (the system decrypts them internally when needed).

Access Paths:

  • ✅ Inheritance: Users inherit group configs → Agent uses them with internal decryption
  • ✅ Database Direct Access: Super admins can query DB and decrypt using the encryption key
  • ❌ API Viewing: No endpoint returns plaintext group config API keys

Example:

// User viewing their own configs
{
  "configs": [
    {
      "config_id": "uuid-1",
      "name": "GPT-4",
      "source": "user",
      "config": {
        "api_key": "sk-ant-xxxxx"  // Decrypted (own config)
      }
    },
    {
      "config_id": "uuid-2",
      "name": "Claude-3",
      "source": "group",
      "config": {
        "api_key": null  // Hidden (inherited from group)
      }
    }
  ]
}

// Admin viewing another user's configs
{
  "configs": [
    {
      "config_id": "uuid-1",
      "name": "GPT-4",
      "source": "user",
      "config": {
        "api_key": null  // Hidden (another user's config)
      }
    }
  ]
}

// User with Group.Audit viewing group configs directly
{
  "configs": [
    {
      "config_id": "uuid-3",
      "name": "Gemini Pro",
      "source": "group",
      "config": {
        "api_key": "gAAAAABl1a2b3c4d5e6f7..."  // Encrypted (as stored in DB)
      }
    }
  ]
}

Migration from Old User Settings API

The old user settings API (/v3/access/users/{user_id}/profiles) stored configurations in the users.model_configs JSON column. This new API uses a dedicated table with better inheritance support and optimistic locking.

Migration strategy:

  1. Run the database migration to create the llm_model_configs table
  2. Optionally migrate existing data from users.model_configs to the new table
  3. Update clients to use the new API endpoints
  4. Update clients to handle version field and 409 Conflict errors
  5. Deprecate the old /profiles endpoints

Key differences:

  • Inheritance: Users without configs inherit from groups (automatic fallback)
  • Optimistic locking: New version field and expected_version parameter
  • Dedicated table: Better query performance and data integrity
  • Transparent encryption: API keys auto-encrypted/decrypted by the API
  • Model type support: New model_type field for categorizing models (text, vision, stt, tts, multimodal, embedding, reranking, other)
  • Table-level indexing: name and model_type stored as table columns for efficient filtering and querying

Model Type Filtering

The model_type table column enables efficient filtering and querying of configurations by model type:

Common Use Cases

  1. Filter by model type: Retrieve only text generation models for chat features
  2. Multi-model applications: Select appropriate model based on task type (text vs vision vs embedding)
  3. Model type analytics: Query and analyze usage patterns by model type
  4. Type-specific defaults: Set different default models for different model types

Example: Filter text models (client-side)

# After fetching configs, filter by model_type
configs = get_user_configs(user_id)
text_models = [c for c in configs if c["model_type"] == "text"]
vision_models = [c for c in configs if c["model_type"] == "vision"]

Database Index

The model_type column is indexed for efficient queries:

CREATE INDEX idx_llm_model_configs_model_type ON llm_model_configs(model_type);

This enables fast lookups when filtering by model type, even with large datasets.

Reserved Fields

The llm_model_configs table includes three reserved JSONB fields for future extensibility:

Field Type Description
reserved_jsonb_1 JSONB (nullable) Reserved for future use
reserved_jsonb_2 JSONB (nullable) Reserved for future use
reserved_jsonb_3 JSONB (nullable) Reserved for future use

Purpose: These fields are reserved for future feature development without requiring schema changes. They are currently unused in the API code but are available in the database layer for future enhancements.

Use Cases: Future features might use these fields for:

  • Advanced configuration options
  • Metadata storage
  • Feature flags
  • Extension data
  • Caching computed values

Note: These fields are not exposed in the current API schemas and are reserved for internal use.

Context Limit Configuration

What is context_limit?

The context_limit field specifies the maximum context window size for an LLM model. This is a required field for all model configurations.

Why is it required?

Model providers frequently update their models and change context window sizes:

  • OpenAI GPT-4o: 128K tokens (may change)
  • Anthropic Claude 3.5: 200K tokens (may change)
  • Google Gemini 1.5: 2.8M tokens (may change)

Hardcoding these values in the system would quickly become outdated. Requiring users to configure this field ensures that the system always uses the correct, up-to-date values.

Unit: K tokens

The context_limit value is specified in K tokens (thousands of tokens) to make it easier to read and write:

Official Documentation API Configuration
128,000 tokens "context_limit": 128
200,000 tokens "context_limit": 200
2,800,000 tokens "context_limit": 2800

How to find the correct value

  1. Check the official documentation for your model:

  2. Convert from tokens to K:

    context_limit = official_value_in_tokens / 1000

Example:
GPT-4o: 128,000 tokens / 1000 = 128

Example: Common Models

Model Official Value Configuration
GPT-4o 128,000 "context_limit": 128
GPT-3.5 Turbo 16,385 "context_limit": 17
Claude 3.5 Sonnet 200,000 "context_limit": 200
Gemini 1.5 Pro 2,800,000 "context_limit": 2800
DeepSeek Chat 128,000 "context_limit": 128

Context Strategy

The optional context_strategy field controls how aggressively the system uses the available context window:

Strategy Usage Best For
conservative 60% of limit Long outputs, complex tasks, uncertain output size
balanced (default) 75% of limit Most conversations, general use
aggressive 85% of limit Short outputs, analysis tasks, predictable output size

Error Handling

If context_limit is missing or invalid, the API will return:

HTTP 400 Bad Request
{
  "detail": "context_limit is required (unit: K tokens, e.g., 128 = 128K = 128,000 tokens). Please check your model provider's documentation for the current context window size and specify it in the configuration."
}

License

Copyright © 2025 Yue Guobin (岳国宾)

This work is licensed under the Creative Commons Attribution-ShareAlike 4.0 International License (CC BY-SA 4.0).

CC BY-SA 4.0

Summary

You are free to:

  • Share — Copy and redistribute the material in any medium or format
  • Adapt — Remix, transform, and build upon the material for any purpose

Under the following terms:

  • Attribution — You must give appropriate credit to Yue Guobin (岳国宾), provide a link to the license, and indicate if changes were made.
  • ShareAlike — If you remix, transform, or build upon the material, you must distribute your contributions under the same license (CC BY-SA 4.0).

Full license text: DESIGN_DOCS_LICENSE