docs: reorganize AI Copilot documentation structure

- Add README.md with documentation overview and structure guide
- Move implemented designs to docs/gns3-copilot/implemented/:
  - chat-api.md (from ai-chat-api-design.md)
  - llm-model-configs.md (from llm-model-configs-api.md)
  - command-security.md
  - context-window-management.md
- Add Jinja2 configuration template system design documents:
  - jinja2-config-templates-system.md
  - config-templates-implementation-guide.md
  - ai-prompting-for-config-templates.md
- Remove obsolete documents (acl-web-ui, Chinese RBAC doc)

This reorganization makes it clearer which features are implemented
vs planned, following the established documentation structure.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
YueGuobinandClaude Sonnet 4.5 committed 2026-03-10 23:20:24 +08:00
1 parent a6c33061c4
commit e452f75ac3
10 files changed
+218 -1806

No files matched your search

File diff suppressed because it is too large. Load diff
@@ -0,0 +1,462 @@
# Command Security Configuration
## Overview
GNS3-Copilot includes a command filtering system to prevent execution of commands that may cause timeout or console availability issues in the lab environment. This helps:
- **Prevent tool timeouts**: Commands like `traceroute` may run longer than the tool timeout
- **Maintain console availability**: Long-running commands can leave the device console unavailable for subsequent commands
- **Ensure reliable execution**: Filtering problematic commands ensures the remaining commands can execute properly
## Implementation Status
**Status**: ✅ **Implemented and Verified**
The command filtering system is fully implemented and has been tested in a live GNS3 environment with actual network devices. Key features:
- ✅ Simple text-based configuration file
- ✅ Substring matching (case-insensitive)
- ✅ Non-blocking filtering (allowed commands execute normally)
- ✅ Detailed blocking feedback in tool results
- ✅ Multi-device support
- ✅ Verified with real Cisco IOS devices
See the [Implementation Verification](#implementation-verification) section for actual test results.
## Problem Context
### Why Filter Commands?
When GNS3-Copilot tools execute commands on network devices using Nornir/Netmiko, there is a timeout limit (typically 30-60 seconds). If a command exceeds this timeout:
1. The tool stops waiting and returns a timeout error
2. The device console may still be executing the command
3. Subsequent commands sent to the device fail or produce incorrect results
4. The user may need to manually interrupt the command on the device console
### Example Scenario
```
Time Agent Action Device Console Status
t0 Execute: traceroute 8.8.8.8 [Command starts]
t1 ...waiting... [Tracing...]
t2 ...waiting... [Tracing...]
t30 Timeout! Proceed to next tool [Still tracing!]
t31 Execute: show ip route [Ignored or corrupted]
t32 ❌ Command fails [Console still busy]
```
## Current Implementation
### Forbidden Commands List
Commands are listed in a simple text file at:
```
gns3server/agent/gns3_copilot/config/forbidden_commands.txt
```
**Format:**
- One command pattern per line
- Simple substring matching (case-insensitive)
- Empty lines and lines starting with `#` are ignored
- Match is performed on the beginning of each command
**Example:**
```
# Network diagnostic commands that may timeout
traceroute
tracepath
tracert
# Debug commands that may destabilize devices
debug
# Test commands that may affect device stability
test
```
### Filter Behavior
1. **Input Commands**: `["show version", "traceroute 8.8.8.8", "show ip int brief"]`
2. **Filtering**: `traceroute 8.8.8.8` is removed (matches `traceroute`)
3. **Executed**: `["show version", "show ip int brief"]`
4. **Result**: Returns successful output with blocked command information
### Result Format
When commands are filtered, the result includes additional fields:
```json
{
"device_name": "R-1",
"status": "partial_success",
"output": "R-1#show version\nCisco IOS Software...\nR-1#show ip int brief\nInterface...",
"diagnostic_commands": ["show version", "show ip int brief"],
"blocked_commands": ["traceroute 8.8.8.8"],
"blocked_info": {
"traceroute 8.8.8.8": "Command 'traceroute 8.8.8.8' is not allowed because it matches the forbidden pattern 'traceroute'. This command may run longer than the tool timeout or leave the device console unavailable for subsequent commands."
}
}
```
**Status values:**
- `"success"`: All commands executed successfully
- `"partial_success"`: Some commands were blocked, but remaining commands executed successfully
- `"failed"`: Command execution failed (device not found, connection error, etc.)
## Module Structure
### Command Filter Module
**File:** `gns3server/agent/gns3_copilot/utils/command_filter.py`
**Functions:**
- `filter_forbidden_commands(commands: list[str]) -> tuple[list[str], dict[str, str]]`
- Returns allowed commands and blocked command information
- `is_command_forbidden(command: str) -> bool`
- Check if a single command is forbidden
- `get_forbidden_commands() -> list[str]`
- Get the current list of forbidden patterns
- `reload_forbidden_commands() -> None`
- Reload the forbidden commands list (useful after editing the file)
### Integration Points
The filter is integrated into:
- **Display Tools** (`display_tools_nornir.py`): `ExecuteMultipleDeviceCommands`
- **Configuration Tools** (`config_tools_nornir.py`): `ExecuteMultipleDeviceConfigCommands`
Both tools use the same filtering logic and return format.
## Configuration
### Default Forbidden Commands
If the configuration file is not found, these defaults are used:
- `traceroute`
- `tracepath`
- `tracert`
- `ping -f`
- `debug`
- `test`
### Customizing the List
To add or remove forbidden commands:
1. Edit the configuration file:
```bash
nano gns3server/agent/gns3_copilot/config/forbidden_commands.txt
```
2. Add your command patterns (one per line):
```
# My custom blocked commands
my_dangerous_command
another_pattern
```
3. Restart GNS3 server to apply changes
### Reloading Without Restart
To reload the forbidden commands list without restarting the server:
```python
from gns3server.agent.gns3_copilot.utils.command_filter import reload_forbidden_commands
reload_forbidden_commands()
```
## Usage Examples
### Example 1: All Commands Allowed
**Input:**
```json
{
"project_id": "abc-123-def",
"device_configs": [
{
"device_name": "R-1",
"commands": ["show version", "show ip route"]
}
]
}
```
**Output:**
```json
{
"device_name": "R-1",
"status": "success",
"output": "...",
"diagnostic_commands": ["show version", "show ip route"]
}
```
### Example 2: Some Commands Blocked
**Input:**
```json
{
"project_id": "abc-123-def",
"device_configs": [
{
"device_name": "R-1",
"commands": ["show version", "traceroute 8.8.8.8", "show ip route"]
}
]
}
```
**Output:**
```json
{
"device_name": "R-1",
"status": "partial_success",
"output": "...",
"diagnostic_commands": ["show version", "show ip route"],
"blocked_commands": ["traceroute 8.8.8.8"],
"blocked_info": {
"traceroute 8.8.8.8": "Command 'traceroute 8.8.8.8' is not allowed because it matches the forbidden pattern 'traceroute'. This command may run longer than the tool timeout or leave the device console unavailable for subsequent commands."
}
}
```
### Example 3: All Commands Blocked
**Input:**
```json
{
"project_id": "abc-123-def",
"device_configs": [
{
"device_name": "R-1",
"commands": ["traceroute 8.8.8.8", "debug ip routing"]
}
]
}
```
**Output:**
```json
{
"device_name": "R-1",
"status": "success",
"output": "",
"diagnostic_commands": [],
"blocked_commands": ["traceroute 8.8.8.8", "debug ip routing"],
"blocked_info": {
"traceroute 8.8.8.8": "Command 'traceroute 8.8.8.8' is not allowed because it matches the forbidden pattern 'traceroute'. This command may run longer than the tool timeout or leave the device console unavailable for subsequent commands.",
"debug ip routing": "Command 'debug ip routing' is not allowed because it matches the forbidden pattern 'debug'. This command may run longer than the tool timeout or leave the device console unavailable for subsequent commands."
}
}
```
## Implementation Verification
### Real-World Test Results
The command filtering system has been tested in a live GNS3 environment with actual network devices. Below are actual execution results:
**Test Scenario:**
- Devices: IOU-L2-1, IOU-L2-2 (Cisco IOS Layer 3 switches)
- Commands: Mixed allowed and forbidden commands
- Forbidden pattern: `traceroute`
**Actual Output:**
```json
{
"device_name": "IOU-L2-1",
"status": "partial_success",
"diagnostic_commands": [
"show ip route",
"show ip interface brief",
"ping 10.0.0.1",
"ping 10.0.0.2",
"ping 10.0.0.4"
],
"blocked_commands": ["traceroute 10.0.0.2"],
"blocked_info": {
"traceroute 10.0.0.2": "Command 'traceroute 10.0.0.2' is not allowed because it matches the forbidden pattern 'traceroute'. This command may run longer than the tool timeout or leave the device console unavailable for subsequent commands."
}
}
```
**Key Observations:**
1. ✅ `traceroute` command was successfully filtered
2. ✅ All other commands (`show`, `ping`) executed normally
3. ✅ Status correctly set to `partial_success`
4. ✅ Both `diagnostic_commands` (executed) and `blocked_commands` (filtered) are clearly listed
5. ✅ Detailed blocking reason provided in `blocked_info`
6. ✅ Tool execution continued without timeout or console lockup issues
### Functionality Verification Matrix
| Feature | Status | Notes |
|---------|--------|-------|
| Command filtering (substring match) | ✅ Verified | `traceroute` correctly matched and blocked |
| Partial execution | ✅ Verified | Other commands executed successfully |
| Return format consistency | ✅ Verified | Contains all expected fields |
| Multi-device support | ✅ Verified | Each device filtered independently |
| Error messages | ✅ Verified | Clear, informative blocking reasons |
| Status field accuracy | ✅ Verified | `partial_success` set correctly |
| Non-blocking behavior | ✅ Verified | No tool timeouts or console issues |
### Benefits Confirmed
1. **Timeout Prevention**: The `traceroute` command that could have taken 30+ seconds was filtered, preventing tool timeout
2. **Console Availability**: Since `traceroute` was not executed, the device console remained available for subsequent commands
3. **Clear Feedback**: The LLM receives clear information about which commands were blocked and why
4. **Partial Execution**: Useful commands (`show`, `ping`) still executed, providing valuable diagnostic information
## Future Enhancements (TODO)
### Planned Improvements
1. **Regex Support**: Allow more sophisticated pattern matching
```python
# Current: simple substring match
"traceroute"
# Future: regex patterns
"^traceroute\\s+"
"ping\\s+.*\\s+-f"
```
2. **User Override File**: Allow per-project or user-specific overrides
```
/etc/gns3-server/forbidden_commands_override.txt
<project_dir>/forbidden_commands_override.txt
```
3. **Web UI Configuration**: Manage forbidden commands through GNS3 web interface
4. **Audit Logging**: Log blocked commands for security analysis
5. **Per-Command Timeouts**: Configure timeouts for specific commands instead of blocking
```python
"command_timeouts": {
"traceroute.*": 120,
"debug.*": 5
}
```
6. **Interrupt Mechanism**: Send Ctrl+C to interrupt long-running commands instead of blocking
```python
def execute_with_timeout(cmd, timeout=30):
try:
return device.execute(cmd, timeout=timeout)
except Timeout:
device.send_break() # Ctrl+C
return f"Command interrupted after {timeout}s"
```
7. **Command State Tracking**: Track device console state to ensure availability
```python
device_state = {
"console_available": True,
"current_command": None,
"last_prompt_seen": timestamp
}
```
### Advanced Features (Long-term)
- **Per-Device Filtering**: Different rules for different device types
- **Time-Based Restrictions**: Block certain commands during specific hours
- **Severity Levels**: Classify commands by severity (warn, block, allow)
- **ML-Based Detection**: Learn which commands cause problems and auto-block them
## Troubleshooting
### Commands Are Being Blocked Unexpectedly
**Problem:** A command you want to use is being blocked.
**Solution:**
1. Check the blocked command list in the result output
2. Identify which pattern is matching your command
3. Edit `forbidden_commands.txt` to remove or modify the pattern
4. Restart GNS3 server
### Forbidden Commands File Not Found
**Problem:** The system logs "Forbidden commands file not found. Using default list."
**Solution:**
1. Verify the file exists at the expected location
2. Check file permissions (should be readable by the GNS3 server process)
3. Ensure the file is not empty
### Changes Not Taking Effect
**Problem:** You edited the file but commands are still being blocked.
**Solution:**
1. Restart the GNS3 server (required to reload the configuration)
2. Or use the `reload_forbidden_commands()` function if available in your context
## Security Considerations
### Why These Commands Are Blocked
| Command | Reason |
|---------|--------|
| `traceroute` | Can run for 30+ seconds, exceeds typical tool timeout |
| `tracepath` | Similar to traceroute, long execution time |
| `tracert` | Windows traceroute, same timeout issues |
| `ping -f` | Flood ping can overwhelm lab devices |
| `debug` | Debug commands can produce overwhelming output and destabilize devices |
| `test` | Test commands may affect device stability |
### Best Practices
1. **Education Environment**: Use the default filtering for safety
2. **Personal Lab**: Consider which commands you actually need
3. **Production-like Environment**: Keep restrictions enabled
4. **Always Understand**: Before allowing a command, understand why it was blocked
## Related Documentation
- [Tool Implementation](../gns3-copilot/tools_v2/README.md)
- [GNS3-Copilot Documentation](../README.md)
- [Contributing Guide](../../CONTRIBUTING.md)
## Feedback and Issues
If you:
- Find commands that should be blocked by default
- Need to allow commands for legitimate use cases
- Have suggestions for improving the filtering system
Please submit an issue: https://github.com/yueguobin/gns3-copilot/issues
---
## License
**Copyright © 2025 Yue Guobin (岳国宾)**
This work is licensed under the [Creative Commons Attribution-ShareAlike 4.0
International License (CC BY-SA 4.0)](https://creativecommons.org/licenses/by-sa/4.0/).
![CC BY-SA 4.0](https://i.creativecommons.org/l/by-sa/4.0/88x31.png)
### Summary
You are free to:
- **Share** — Copy and redistribute the material in any medium or format
- **Adapt** — Remix, transform, and build upon the material for any purpose
Under the following terms:
- **Attribution** — You must give appropriate credit to **Yue Guobin (岳国宾)**, provide
a link to the license, and indicate if changes were made.
- **ShareAlike** — If you remix, transform, or build upon the material, you must
distribute your contributions under the **same license** (CC BY-SA 4.0).
Full license text: [DESIGN_DOCS_LICENSE](../DESIGN_DOCS_LICENSE.md)
@@ -0,0 +1,355 @@
# LLM Context Window Management Implementation Document
## Overview
This document explains the context window management implementation mechanism for GNS3 Copilot, including message trimming, token counting, and configuration validation.
## Implementation Architecture
### 1. Core Modules
**File Location**: `gns3server/agent/gns3_copilot/agent/context_manager.py`
#### Token Counting Strategy
The system uses **tiktoken** for token counting (context_manager.py:60):
```python
_tiktoken_encoding = tiktoken.get_encoding("cl100k_base")
```
**Required Dependency**:
```bash
pip install tiktoken>=0.8.0
```
If tiktoken is not installed, the system will throw a `ModuleNotFoundError` at startup.
#### Key Functions
**`count_tokens(text: str) -> int`** (context_manager.py:84-100)
- Uses tiktoken to accurately count tokens in text
- Uses `cl100k_base` encoding
- Returns the exact token count
**`estimate_tool_tokens(tools: list) -> int`** (context_manager.py:103-169)
- Serializes tool schema to JSON
- Uses tiktoken to count token consumption of tool definitions
- Supports Pydantic v1/v2 compatibility
- Falls back to 1000 tokens on failure
**`create_pre_model_hook(...)`** (context_manager.py:195-402)
- Creates a preprocessing function (pre_model_hook)
- Automatically executes before each LLM call:
1. Injects topology information into system prompt
2. Estimates token consumption of tool definitions
3. Trims message history to fit context limits
- Returns a callable function for preparing messages
### 2. Detailed Trimming Logic
#### 2.1 Token Budget Allocation
When calling the LLM, the content sent consists of two parts:
```
Complete request sent to LLM:
┌─────────────────────────────────────────────────────────────┐
│ 1. Messages (managed by us) │
│ ├─ SystemMessage: system prompt + topology (template injection) │
│ └─ HumanMessage/AIMessage: user messages / history messages │
├─────────────────────────────────────────────────────────────┤
│ 2. Tool Definitions (LangChain adds automatically, not in messages) │
│ ├─ Tool 1 schema (name, description, parameters) │
│ ├─ Tool 2 schema │
│ └─ ... (about 500-1500 tokens per tool) │
└─────────────────────────────────────────────────────────────┘
```
**System Message Structure**:
- Uses template variable `{{topology_info}}` to dynamically inject topology
- System prompt contains placeholder: `"### CURRENT TOPOLOGY\n{{topology_info}}"`
- If topology exists, replaces with actual content
- If no topology, replaces with `"(No topology information available)"`
#### 2.2 Trimming Process
```
Step 1: Calculate Input Budget
┌─────────────────────────────────────────────────────────────┐
│ context_limit: 128,000 tokens (128K) │
│ strategy: balanced (75%) │
│ │
│ Input budget = 128 × 1000 × 0.75 = 96,000 tokens │
└─────────────────────────────────────────────────────────────┘
↓
Step 2: Subtract Tool Definitions
┌─────────────────────────────────────────────────────────────┐
│ Input budget: 96,000 tokens │
│ Tool definitions: 1,725 tokens │
│ │
│ Available for messages = 96,000 - 1,725 = 94,275 tokens │
└─────────────────────────────────────────────────────────────┘
↓
Step 3: trim_messages Processing
┌─────────────────────────────────────────────────────────────┐
│ Call LangChain's trim_messages: │
│ - max_tokens = 94,275 (includes system message) │
│ - strategy = "last" (keep latest messages) │
│ - token_counter = tiktoken counting function │
│ - include_system = True (always keep system) │
│ │
│ trim_messages will: │
│ 1. Keep SystemMessage (system + topology) │
│ 2. Starting from latest messages, keep as much history │
│ 3. When exceeding limit, discard oldest messages │
└─────────────────────────────────────────────────────────────┘
```
#### 2.3 Trimming Priority
The system preserves content in the following priority order:
| Priority | Content | Description |
|----------|---------|-------------|
| 1️⃣ | System Message (system prompt + topology) | Never removed |
| 2️⃣ | Latest user message | Keep at least the last 1 |
| 3️⃣ | Old conversation history | Discarded in chronological order |
**Note**: System prompt and topology info are merged into one SystemMessage via template variable and cannot be separated.
#### 2.4 Edge Case Handling
| Scenario | Handling |
|----------|----------|
| System (including topology) > budget | Keep complete SystemMessage (cannot separate system and topology) |
| Tools > budget | ERROR log, suggest increasing context_limit or reducing tool count |
| All history trimmed | Keep last 1 user message |
**Important Notes**:
- When system + topology exceed available budget, **both are preserved**
- Cannot discard only topology while keeping system prompt (already merged)
### 3. Integration with GNS3 Copilot
**File Location**: `gns3server/agent/gns3_copilot/agent/gns3_copilot.py`
#### Implementation Method
**Key Point**: The system uses a **custom StateGraph**, not LangGraph's pre-built agent.
Therefore, `pre_model_hook` cannot be passed via `model.invoke(config={"configurable": {"pre_model_hook": ...}})`.
**Correct Usage**: **Directly call** the `pre_hook` function to prepare messages.
```python
def llm_call(state: dict, config: RunnableConfig | None = None):
"""LLM decides whether to call a tool or not."""
# 1. Get topology information
project_id = config["configurable"].get("project_id")
topology_info = None
if project_id:
topology_tool = GNS3TopologyTool()
topology = topology_tool._run(project_id=project_id)
if topology and "error" not in topology:
topology_info = topology
# 2. Create pre_model_hook
system_prompt = load_system_prompt()
pre_hook = create_pre_model_hook(
system_prompt=system_prompt,
get_topology_func=lambda s: s.get("topology_info"),
get_llm_config_func=get_current_llm_config,
get_tools_func=lambda: tools,
)
# 3. Create model with tools
model_with_tools = create_base_model_with_tools(tools, llm_config=llm_config)
# 4. ⭐ Key: directly call pre_hook to prepare messages
logger.info("Calling pre_hook to prepare %d messages", len(messages))
prepared_state = pre_hook({"messages": messages, "topology_info": topology_info})
prepared_messages = prepared_state["messages"]
# 5. Use prepared messages to call LLM
response = model_with_tools.invoke(prepared_messages)
return {"messages": [response], ...}
```
#### Why Not Pass via Config?
LangGraph's `pre_model_hook` parameter only applies to **pre-built agents**, not custom StateGraphs.
| Agent Type | pre_model_hook Support |
|------------|------------------------|
| `create_react_agent` | ✅ Via `pre_model_hook` parameter |
| `chat_agent_executor` | ✅ Via `pre_model_hook` parameter |
| **Custom StateGraph** | ❌ **Not supported**, need to call directly |
Our implementation uses a custom StateGraph (`agent_builder = StateGraph(MessagesState)`), so we must call `pre_hook` directly.
### 4. Execution Flow
```
User sends message
↓
llm_call node is called
↓
Get project_id (from config["configurable"])
↓
Call GNS3TopologyTool._run(project_id) to get topology
↓
Store topology_info to state
↓
Create pre_model_hook (via create_pre_model_hook())
↓
[Key] Directly call pre_hook({"messages": messages, "topology_info": topology_info})
├─ 1. Inject topology into system prompt
├─ 2. Estimate tool definitions tokens
├─ 3. Call trim_messages() to trim messages
└─ 4. Return prepared message list
↓
Call model.invoke() with prepared messages
↓
Return LLM response
```
---
## Strategy Implementation
### Context Strategy Ratios
**Definition** (context_manager.py:68-72):
```python
CONTEXT_STRATEGY_RATIOS = {
"conservative": 0.60,
"balanced": 0.75,
"aggressive": 0.85,
}
```
**Default Value** (context_manager.py:74):
```python
DEFAULT_CONTEXT_STRATEGY = "balanced"
```
### Strategy Comparison
| Strategy | Input Ratio | Output Reserved | Calculation Formula |
|----------|-------------|-----------------|---------------------|
| Conservative | 60% | 40% | `context_limit × 1000 × 0.60` |
| Balanced | 75% | 25% | `context_limit × 1000 × 0.75` |
| Aggressive | 85% | 15% | `context_limit × 1000 × 0.85` |
---
## Log Output
### Normal Case (topology successfully injected)
```
INFO: Calling pre_hook to prepare 1 messages
INFO: ✓ Topology injected: 7722 chars, nodes: ['netshoot-1', 'R1', 'R2', 'IOU-L3-1', 'IOU-L3-2']
INFO: Context ready: 2 msgs, ~3815 tokens + 1725 tools = 5540 / 128K (4.3%), strategy=conservative
INFO: Messages prepared: 1 → 2
INFO: LLM call completed: tool_calls=0
```
### When Trimming Occurs
```
INFO: Calling pre_hook to prepare 50 messages
INFO: ✓ Topology injected: 8500 chars, nodes: ['R1', 'R2', ...]
INFO: Messages trimmed: 50 → 25 msgs. Total: ~82000 tokens + 1725 tools = 83725 / 128K (65.4%), strategy=balanced
INFO: Messages prepared: 50 → 25
```
### When topology is None
```
INFO: Calling pre_hook to prepare 1 messages
WARNING: ✗ Topology data is None, injecting placeholder
INFO: Context ready: 2 msgs, ~800 tokens + 1725 tools = 2525 / 128K (2.0%), strategy=balanced
```
---
## Error Handling
### tiktoken Not Installed
If tiktoken is not installed, the system will throw an error at startup:
```python
ModuleNotFoundError: No module named 'tiktoken'
```
**Solution**:
```bash
pip install tiktoken>=0.8.0
```
### context_limit Missing or Invalid
If there is no `context_limit` in the LLM configuration or the value is invalid (context_manager.py:285-295):
```python
if "context_limit" not in llm_config:
raise ValueError("context_limit is required in LLM config")
limit = llm_config["context_limit"]
if not isinstance(limit, int) or limit <= 0:
raise ValueError(f"Invalid context_limit: {limit}")
```
### Trimming Failure
```python
try:
trimmed = trim_messages(...)
except Exception as e:
logger.error("Failed to trim messages: %s", e)
logger.warning("Returning original messages due to trimming error")
return {"messages": messages_with_system}
```
---
## Related Source Files
- `gns3server/agent/gns3_copilot/agent/context_manager.py` - Context management core logic
- `gns3server/agent/gns3_copilot/agent/gns3_copilot.py` - LLM call node (StateGraph)
- `gns3server/agent/gns3_copilot/agent/model_factory.py` - Model creation and tool binding
---
## License
**Copyright © 2025 Yue Guobin (岳国宾)**
This work is licensed under the [Creative Commons Attribution-ShareAlike 4.0
International License (CC BY-SA 4.0)](https://creativecommons.org/licenses/by-sa/4.0/).
![CC BY-SA 4.0](https://i.creativecommons.org/l/by-sa/4.0/88x31.png)
### Summary
You are free to:
- **Share** — Copy and redistribute the material in any medium or format
- **Adapt** — Remix, transform, and build upon the material for any purpose
Under the following terms:
- **Attribution** — You must give appropriate credit to **Yue Guobin (岳国宾)**, provide
a link to the license, and indicate if changes were made.
- **ShareAlike** — If you remix, transform, or build upon the material, you must
distribute your contributions under the **same license** (CC BY-SA 4.0).
Full license text: [DESIGN_DOCS_LICENSE](../DESIGN_DOCS_LICENSE.md)
@@ -0,0 +1,915 @@
# LLM Model Configurations API
## Overview
This API provides LLM model configuration management for users and user groups with inheritance support.
### Key Features
- **User-level configurations**: Each user can have their own LLM model configurations
- **Group-level configurations**: User groups can share LLM model configurations
- **Inheritance**: Users automatically inherit configurations from their groups (when they have no own configs)
- **Default configuration**: Both users and groups can set a default configuration
- **API Key Encryption**: API keys are automatically encrypted in the database
- **Optimistic Locking**: Prevents concurrent modification conflicts using version tracking
### Inheritance Logic
```
User requests configs:
├─ Always return user's own configs (if any)
└─ Always return inherited group configs (if any)
```
**Note:** Users can see both their own configurations AND configurations inherited from their groups. The `source` field in the response indicates the origin of each configuration.
### Configuration Priority
```
User's own config > User's group config
```
---
## Database Schema
### Table: `llm_model_configs`
| Column | Type | Description |
|--------|------|-------------|
| `config_id` | UUID | Primary key |
| `name` | VARCHAR(100) | Configuration name (table-level for indexing) |
| `model_type` | VARCHAR(50) | Model type (table-level for filtering) |
| `config` | JSONB | Configuration data (provider, base_url, model, temperature, api_key, etc.) |
| `user_id` | UUID (nullable) | Foreign key to users table |
| `group_id` | UUID (nullable) | Foreign key to user_groups table |
| `is_default` | BOOLEAN | Default configuration flag |
| `version` | INTEGER | Optimistic locking version (starts at 0, increments on each update) |
| `reserved_jsonb_1` | JSONB (nullable) | Reserved field for future use |
| `reserved_jsonb_2` | JSONB (nullable) | Reserved field for future use |
| `reserved_jsonb_3` | JSONB (nullable) | Reserved field for future use |
| `created_at` | TIMESTAMP | Creation timestamp |
| `updated_at` | TIMESTAMP | Last update timestamp |
### Model Types
The `model_type` field accepts the following values:
- `text` - Text generation models
- `vision` - Vision/image understanding models
- `stt` - Speech-to-Text models
- `tts` - Text-to-Speech models
- `multimodal` - Multimodal models supporting multiple input types
- `embedding` - Text embedding models
- `reranking` - Reranking models
- `other` - Other model types
### Constraints
- Each config belongs to **either** a user **or** a group (not both)
- Each user can have **at most one** default configuration
- Each group can have **at most one** default configuration
- `version` field is automatically incremented on each update
---
## API Endpoints
### User Configuration Endpoints
| Method | Path | Description | Privilege |
|--------|------|-------------|-----------|
| GET | `/v3/access/users/{user_id}/llm-model-configs` | Get user's effective configs (own + inherited) | User.Audit |
| GET | `/v3/access/users/{user_id}/llm-model-configs/own` | Get user's own configs only | User.Audit |
| GET | `/v3/access/users/{user_id}/llm-model-configs/default` | Get user's default configuration | User.Audit |
| POST | `/v3/access/users/{user_id}/llm-model-configs` | Create a new configuration | User.Modify |
| PUT | `/v3/access/users/{user_id}/llm-model-configs/{config_id}` | Update a configuration | User.Modify |
| DELETE | `/v3/access/users/{user_id}/llm-model-configs/{config_id}` | Delete a configuration | User.Modify |
| PUT | `/v3/access/users/{user_id}/llm-model-configs/default/{config_id}` | Set default configuration | User.Modify |
### Group Configuration Endpoints
| Method | Path | Description | Privilege |
|--------|------|-------------|-----------|
| GET | `/v3/access/groups/{group_id}/llm-model-configs` | Get all group configurations | Group.Audit |
| GET | `/v3/access/groups/{group_id}/llm-model-configs/default` | Get group's default configuration | Group.Audit |
| POST | `/v3/access/groups/{group_id}/llm-model-configs` | Create a new configuration | Group.Modify |
| PUT | `/v3/access/groups/{group_id}/llm-model-configs/{config_id}` | Update a configuration | Group.Modify |
| DELETE | `/v3/access/groups/{group_id}/llm-model-configs/{config_id}` | Delete a configuration | Group.Modify |
| PUT | `/v3/access/groups/{group_id}/llm-model-configs/default/{config_id}` | Set default configuration | Group.Modify |
**Note:** The GET endpoints for groups return the same structure as user endpoints: `configs`, `default_config`, and `total`.
---
## Request/Response Schemas
### LLMModelConfigCreate
**Required Fields:**
| Field | Type | Description |
|-------|------|-------------|
| `name` | string | Configuration name (1-100 chars) |
| `model_type` | string | Model type (text, vision, stt, tts, multimodal, embedding, reranking, other) |
| `provider` | string | LLM provider (e.g., "openai", "anthropic", "ollama") |
| `base_url` | string | API base URL |
| `model` | string | Model name |
| `temperature` | float | Temperature (0.0-2.0, default: 0.7) |
| `context_limit` | integer | **Model context window limit in K tokens** (e.g., 128 = 128K = 128,000 tokens) |
**Optional Fields:**
| Field | Type | Description |
|-------|------|-------------|
| `api_key` | string | API key (auto-encrypted) |
| `max_tokens` | integer | Max tokens for generation |
| `context_strategy` | string | Context trimming strategy: "conservative" (60%), "balanced" (75%), "aggressive" (85%). Default: "balanced" |
| `copilot_mode` | string | GNS3-Copilot mode: "teaching_assistant" (diagnostics only, default) or "lab_automation_assistant" (full configuration access) |
| `is_default` | boolean | Set as default (default: false) |
**Important Notes:**
- **`context_limit` is required**: You must specify the model's context window limit. Refer to the model provider's official documentation for the current value.
- **Unit is K tokens**: The value is in thousands of tokens (1 K = 1,000 tokens). For example:
- GPT-4o: 128,000 tokens → configure as `"context_limit": 128`
- Claude 3.5 Sonnet: 200,000 tokens → configure as `"context_limit": 200`
- Gemini 1.5 Pro: 2,800,000 tokens → configure as `"context_limit": 2800`
**Extra Fields:** Any custom fields are supported for future extensibility.
### LLMModelConfigUpdate
| Field | Type | Description |
|-------|------|-------------|
| `name` | string (optional) | Configuration name |
| `model_type` | string (optional) | Model type |
| `provider` | string (optional) | LLM provider |
| `base_url` | string (optional) | API base URL |
| `model` | string (optional) | Model name |
| `temperature` | float (optional) | Temperature |
| `api_key` | string (optional) | API key |
| `max_tokens` | integer (optional) | Max tokens |
| `context_limit` | integer (optional) | Model context window limit in K tokens |
| `context_strategy` | string (optional) | Context trimming strategy |
| `is_default` | boolean (optional) | Default flag |
| `expected_version` | integer (optional) | **Optimistic locking version** |
**Note:** When using `expected_version`, the API will verify the version hasn't changed since you read the data. If it has, you'll receive a 409 Conflict error.
### LLMModelConfigResponse
| Field | Type | Description |
|-------|------|-------------|
| `config_id` | UUID | Configuration ID |
| `name` | string | Configuration name |
| `model_type` | string | Model type |
| `config` | LLMModelConfigData | Configuration data (provider, base_url, model, temperature, etc.) |
| `user_id` | UUID (nullable) | Owner user ID |
| `group_id` | UUID (nullable) | Owner group ID |
| `is_default` | boolean | Default flag |
| `version` | integer | **Current version number** (for optimistic locking) |
| `created_at` | TIMESTAMP | Creation time |
| `updated_at` | TIMESTAMP | Last update time |
### LLMModelConfigInheritedResponse
| Field | Type | Description |
|-------|------|-------------|
| `configs` | list[LLMModelConfigWithSource] | Effective configurations |
| `default_config` | LLMModelConfigWithSource (nullable) | Default configuration (never null if configs list is not empty) |
| `total` | integer | Total count |
**Default Configuration Selection Logic:**
1. User's config marked with `is_default: true` (highest priority)
2. Group's config marked with `is_default: true`
3. First config in the list (user configs come before group configs)
### LLMModelConfigListResponse
| Field | Type | Description |
|-------|------|-------------|
| `configs` | list[LLMModelConfigResponse] | Configuration list |
| `default_config` | LLMModelConfigResponse (nullable) | Default configuration (never null if configs list is not empty) |
| `total` | integer | Total count |
**Default Configuration Selection Logic:**
1. Config marked with `is_default: true`
2. First config in the list (fallback if no default is marked)
**Usage:** This schema is used for group configuration endpoints (e.g., `GET /groups/{group_id}/llm-model-configs`).
### LLMModelConfigWithSource
| Field | Type | Description |
|-------|------|-------------|
| `config_id` | UUID | Configuration ID |
| `name` | string | Configuration name |
| `model_type` | string | Model type |
| `config` | LLMModelConfigData | Configuration data (provider, base_url, model, temperature, etc.) |
| `user_id` | UUID (nullable) | Owner user ID |
| `group_id` | UUID (nullable) | Owner group ID |
| `is_default` | boolean | Default flag |
| `version` | integer | Optimistic locking version |
| `created_at` | TIMESTAMP | Creation time |
| `updated_at` | TIMESTAMP | Last update time |
| `source` | string | Source: "user" or "group" |
| `group_name` | string (nullable) | Group name if source is "group" |
---
## Usage Examples
### 1. Create a user configuration
```bash
curl -X POST http://localhost:3080/v3/access/users/{user_id}/llm-model-configs \
-H "Authorization: Bearer <token>" \
-H "Content-Type: application/json" \
-d '{
"name": "GPT-4o",
"model_type": "text",
"provider": "openai",
"base_url": "https://api.openai.com/v1",
"model": "gpt-4o",
"temperature": 0.7,
"context_limit": 128,
"context_strategy": "balanced",
"api_key": "sk-xxx",
"copilot_mode": "teaching_assistant",
"is_default": true
}'
```
**Important:**
- `context_limit` is **required** and specified in K tokens (e.g., 128 = 128K = 128,000 tokens)
- Refer to the model provider's official documentation for the current context window size
**Response:**
```json
{
"config_id": "uuid-1",
"name": "GPT-4o",
"model_type": "text",
"config": {
"provider": "openai",
"base_url": "https://api.openai.com/v1",
"model": "gpt-4o",
"temperature": 0.7,
"context_limit": 128,
"context_strategy": "balanced",
"api_key": "sk-xxx",
"copilot_mode": null
},
"user_id": "uuid-user",
"group_id": null,
"is_default": true,
"version": 0,
"created_at": "2026-03-03T12:00:00Z",
"updated_at": "2026-03-03T12:00:00Z"
}
```
### 2. Create a group configuration
```bash
curl -X POST http://localhost:3080/v3/access/groups/{group_id}/llm-model-configs \
-H "Authorization: Bearer <token>" \
-H "Content-Type: application/json" \
-d '{
"name": "Claude-3.5 Sonnet",
"model_type": "text",
"provider": "anthropic",
"base_url": "https://api.anthropic.com",
"model": "claude-3-5-sonnet-20241022",
"temperature": 0.7,
"context_limit": 200,
"context_strategy": "balanced",
"api_key": "sk-ant-xxx",
"copilot_mode": "lab_automation_assistant",
"is_default": true
}'
```
### 3. Get user's effective configurations (with inheritance)
```bash
curl -X GET http://localhost:3080/v3/access/users/{user_id}/llm-model-configs \
-H "Authorization: Bearer <token>"
```
**Response (user has both own configs and inherited group configs):**
```json
{
"configs": [
{
"config_id": "uuid-1",
"name": "GPT-4o",
"model_type": "text",
"config": {
"provider": "openai",
"base_url": "https://api.openai.com/v1",
"model": "gpt-4o",
"temperature": 0.7,
"context_limit": 128,
"context_strategy": "balanced",
"api_key": "sk-xxx",
"copilot_mode": "lab_automation_assistant"
},
"user_id": "uuid-user",
"group_id": null,
"is_default": true,
"version": 0,
"created_at": "2026-03-03T14:32:48.158880Z",
"updated_at": "2026-03-03T14:32:48.158880Z",
"source": "user",
"group_name": null
},
{
"config_id": "uuid-2",
"name": "Claude-3.5 Sonnet",
"model_type": "text",
"config": {
"provider": "anthropic",
"base_url": "https://api.anthropic.com",
"model": "claude-3-5-sonnet-20241022",
"temperature": 0.7,
"context_limit": 200,
"context_strategy": "balanced",
"api_key": null,
"copilot_mode": null
},
"user_id": null,
"group_id": "uuid-group",
"is_default": true,
"version": 0,
"created_at": "2026-03-03T14:32:48.158880Z",
"updated_at": "2026-03-03T14:32:48.158880Z",
"source": "group",
"group_name": "Developers"
}
],
"default_config": {
"config_id": "uuid-1",
"name": "GPT-4o",
"model_type": "text",
"config": {
"provider": "openai",
"base_url": "https://api.openai.com/v1",
"model": "gpt-4o",
"temperature": 0.7,
"context_limit": 128,
"context_strategy": "balanced",
"api_key": "sk-xxx",
"copilot_mode": "lab_automation_assistant"
},
"user_id": "uuid-user",
"group_id": null,
"is_default": true,
"version": 0,
...
},
"total": 2
}
```
**Note:**
- User's own config shows `config.api_key: "sk-xxx"` (visible to owner)
- Inherited group config shows `config.api_key: null` (hidden from users)
- `source: "user"` indicates the config belongs to the user
- `source: "group"` indicates the config is inherited from a group
- Configuration fields are nested in the `config` object (same structure as group endpoints)
- `context_limit` is in K tokens (128 = 128K = 128,000 tokens)
### 4. Get group configurations
```bash
curl -X GET http://localhost:3080/v3/access/groups/{group_id}/llm-model-configs \
-H "Authorization: Bearer <token>"
```
**Response:**
```json
{
"configs": [
{
"config_id": "uuid-1",
"name": "Claude-3",
"model_type": "text",
"config": {
"provider": "anthropic",
"base_url": "https://api.anthropic.com",
"model": "claude-3-opus-20240229",
"temperature": 0.7,
"context_limit": 200,
"context_strategy": "balanced",
"api_key": "sk-ant-xxx",
"copilot_mode": null
},
"user_id": null,
"group_id": "uuid-group",
"is_default": true,
"version": 0,
"created_at": "2026-03-03T12:00:00Z",
"updated_at": "2026-03-03T12:00:00Z"
},
{
"config_id": "uuid-2",
"name": "GPT-4",
"model_type": "text",
"config": {
"provider": "openai",
"base_url": "https://api.openai.com/v1",
"model": "gpt-4",
"temperature": 0.7,
"context_limit": 128,
"context_strategy": "balanced",
"api_key": "sk-xxx",
"copilot_mode": null
},
"user_id": null,
"group_id": "uuid-group",
"is_default": false,
"version": 0,
"created_at": "2026-03-03T12:00:00Z",
"updated_at": "2026-03-03T12:00:00Z"
}
],
"default_config": {
"config_id": "uuid-1",
"name": "Claude-3",
"model_type": "text",
"config": {
"provider": "anthropic",
...
},
"user_id": null,
"group_id": "uuid-group",
"is_default": true,
"version": 0,
...
},
"total": 2
}
```
**Note:** The response structure is the same as user endpoints, with `configs`, `default_config`, and `total` fields.
### 5. Update a configuration (without optimistic locking)
```bash
curl -X PUT http://localhost:3080/v3/access/users/{user_id}/llm-model-configs/{config_id} \
-H "Authorization: Bearer <token>" \
-H "Content-Type: application/json" \
-d '{
"temperature": 0.9,
"max_tokens": 4000,
"context_strategy": "aggressive"
}'
```
### 6. Update a configuration (WITH optimistic locking)
**Best practice for avoiding concurrent modification conflicts:**
```bash
# Step 1: Read the config (get the current version)
curl -X GET http://localhost:3080/v3/access/users/{user_id}/llm-model-configs/own \
-H "Authorization: Bearer <token>"
# Response includes "version": 5
# Step 2: Update with expected_version
curl -X PUT http://localhost:3080/v3/access/users/{user_id}/llm-model-configs/{config_id} \
-H "Authorization: Bearer <token>" \
-H "Content-Type: application/json" \
-d '{
"temperature": 0.9,
"max_tokens": 4000,
"expected_version": 5
}'
# Response includes incremented "version": 6
```
**If someone else modified the config before you:**
```json
HTTP 409 Conflict
{
"detail": "Concurrent modification detected. Expected version 5, but current version is 6. Please retry."
}
```
**Client retry flow:**
1. Receive 409 Conflict error
2. Re-fetch the config to get the latest version
3. Apply your changes on top of the latest data
4. Retry the update with the new `expected_version`
### 7. Set default configuration
```bash
curl -X PUT http://localhost:3080/v3/access/users/{user_id}/llm-model-configs/default/{config_id} \
-H "Authorization: Bearer <token>"
```
### 8. Get default configuration
Get the user's default configuration:
```bash
curl -X GET http://localhost:3080/v3/access/users/{user_id}/llm-model-configs/default \
-H "Authorization: Bearer <token>"
```
**Note:** This endpoint only returns configurations explicitly marked with `is_default: true`. If no configuration is marked as default, it returns 404.
**Response:**
```json
{
"config_id": "uuid-1",
"name": "GPT-4o",
"model_type": "text",
"config": {
"provider": "openai",
"base_url": "https://api.openai.com/v1",
"model": "gpt-4o",
"temperature": 0.7,
"context_limit": 128,
"context_strategy": "balanced",
"api_key": "sk-xxx",
"copilot_mode": null
},
"user_id": "uuid-user",
"group_id": null,
"is_default": true,
"version": 0,
"created_at": "2026-03-03T18:15:00Z",
"updated_at": "2026-03-03T18:15:00Z"
}
```
**If no default configuration is set:**
```json
HTTP 404 Not Found
{
"detail": "No default LLM model configuration found for user '{user_id}'"
}
```
Get the group's default configuration:
```bash
curl -X GET http://localhost:3080/v3/access/groups/{group_id}/llm-model-configs/default \
-H "Authorization: Bearer <token>"
```
The response format is the same as for users.
---
**Important Note:** This dedicated `/default` endpoint is different from the `default_config` field in the list response:
- `/default` endpoint: Requires explicit `is_default: true` flag, returns 404 if not found
- `default_config` field in list: Falls back to first config if no explicit default is marked
### 9. Delete a configuration
```bash
curl -X DELETE http://localhost:3080/v3/access/users/{user_id}/llm-model-configs/{config_id} \
-H "Authorization: Bearer <token>"
```
---
## Error Codes
| Status | Description |
|--------|-------------|
| 200 | Success |
| 201 | Created |
| 204 | Deleted (no content) |
| 400 | Bad request |
| 401 | Unauthorized |
| 404 | Not found |
| **409** | **Conflict (optimistic lock violation)** |
| 500 | Server error |
### 409 Conflict Response
```json
{
"detail": "Concurrent modification detected. Expected version 5, but current version is 6. Please retry."
}
```
---
## Concurrency Control
### Optimistic Locking
This API uses **optimistic locking** to prevent concurrent modification conflicts:
1. **Version Tracking**: Each configuration has a `version` field that starts at 0 and increments on each update
2. **Read-Modify-Write**: When updating, clients should include the `expected_version` from their last read
3. **Conflict Detection**: If the provided version doesn't match the current version, the update is rejected with HTTP 409
### When to Use Optimistic Locking
**Use `expected_version` when:**
- Multiple users/admins might modify the same configuration
- You want to prevent accidental overwrites of concurrent changes
- Building interactive UIs that display and edit configurations
**Skip `expected_version` when:**
- You're sure no one else is modifying the config
- Performance is more important than data integrity (not recommended)
---
## Security Notes
### API Key Encryption
All API keys are encrypted using Fernet symmetric encryption (AES-128-CBC). Encryption keys are stored in `{secrets_dir}/gns3_encryption_key` with 0600 permissions.
### Access Control
All endpoints require appropriate privileges:
- **User.Audit**: View user configurations
- **User.Modify**: Create, update, delete user configurations
- **Group.Audit**: View group configurations
- **Group.Modify**: Create, update, delete group configurations
### API Key Visibility
The API implements strict API key visibility controls to protect sensitive credentials:
| Scenario | User Configs | Group Configs |
|----------|-------------|---------------|
| User viewing own configs | **Visible (Plaintext)** | N/A |
| User viewing inherited group configs | N/A | **Hidden (null)** |
| Admin viewing other users' configs | **Hidden (null)** | N/A |
| Viewing group configs directly (with `Group.Audit`) | N/A | **Visible (Encrypted)** |
**Rules:**
1. **Users viewing their own configs**: Can see **decrypted (plaintext)** API keys in their own configurations
2. **Users viewing inherited group configs**: API keys are **hidden** (set to `null`) in the inherited configs
3. **Admins viewing other users' configs**: Cannot see API keys in any user configurations (set to `null`) - user privacy protection
4. **Viewing group configs directly**: Users with `Group.Audit` privilege can see **encrypted** API keys in group configurations (not decrypted)
5. **Super admins**: While application-layer restrictions apply, super admins can access the database directly and decrypt any API key using the encryption key. This is by design as super admins have system-level access.
**Important Notes:**
- API keys are stored in the database in **encrypted** format using Fernet symmetric encryption
- User configs are **decrypted on-the-fly** when retrieved by the owner
- Group configs return the **encrypted value** as stored in the database (no automatic decryption)
- Super admins have database access and can retrieve & decrypt any API key - this is intentional and reflects their system-level privileges
**⚠️ Key Point: No Plaintext Group API Keys via API**
> **Nobody (including super admins) can see plaintext API keys for group configurations through the application API.**
>
> - Group configs always return the **encrypted** API key value (e.g., `gAAAAABl1a2b3c4d5e6f7...`)
> - There is **no API endpoint** that decrypts and returns group config API keys as plaintext
> - Even super admins with `Group.Audit` privilege receive encrypted values via the API
> - This is an application-layer restriction that applies to all users
>
> **Design Rationale**: Group configs are intended to be **inherited** automatically, not viewed/copied manually. The encrypted values protect API keys while still allowing the inheritance mechanism to function (the system decrypts them internally when needed).
>
> **Access Paths**:
> - ✅ **Inheritance**: Users inherit group configs → Agent uses them with internal decryption
> - ✅ **Database Direct Access**: Super admins can query DB and decrypt using the encryption key
> - ❌ **API Viewing**: No endpoint returns plaintext group config API keys
**Example:**
```json
// User viewing their own configs
{
"configs": [
{
"config_id": "uuid-1",
"name": "GPT-4",
"source": "user",
"config": {
"api_key": "sk-ant-xxxxx" // Decrypted (own config)
}
},
{
"config_id": "uuid-2",
"name": "Claude-3",
"source": "group",
"config": {
"api_key": null // Hidden (inherited from group)
}
}
]
}
// Admin viewing another user's configs
{
"configs": [
{
"config_id": "uuid-1",
"name": "GPT-4",
"source": "user",
"config": {
"api_key": null // Hidden (another user's config)
}
}
]
}
// User with Group.Audit viewing group configs directly
{
"configs": [
{
"config_id": "uuid-3",
"name": "Gemini Pro",
"source": "group",
"config": {
"api_key": "gAAAAABl1a2b3c4d5e6f7..." // Encrypted (as stored in DB)
}
}
]
}
```
---
## Migration from Old User Settings API
The old user settings API (`/v3/access/users/{user_id}/profiles`) stored configurations in the `users.model_configs` JSON column. This new API uses a dedicated table with better inheritance support and optimistic locking.
**Migration strategy:**
1. Run the database migration to create the `llm_model_configs` table
2. Optionally migrate existing data from `users.model_configs` to the new table
3. Update clients to use the new API endpoints
4. Update clients to handle `version` field and 409 Conflict errors
5. Deprecate the old `/profiles` endpoints
**Key differences:**
- **Inheritance**: Users without configs inherit from groups (automatic fallback)
- **Optimistic locking**: New `version` field and `expected_version` parameter
- **Dedicated table**: Better query performance and data integrity
- **Transparent encryption**: API keys auto-encrypted/decrypted by the API
- **Model type support**: New `model_type` field for categorizing models (text, vision, stt, tts, multimodal, embedding, reranking, other)
- **Table-level indexing**: `name` and `model_type` stored as table columns for efficient filtering and querying
---
## Model Type Filtering
The `model_type` table column enables efficient filtering and querying of configurations by model type:
### Common Use Cases
1. **Filter by model type**: Retrieve only text generation models for chat features
2. **Multi-model applications**: Select appropriate model based on task type (text vs vision vs embedding)
3. **Model type analytics**: Query and analyze usage patterns by model type
4. **Type-specific defaults**: Set different default models for different model types
### Example: Filter text models (client-side)
```python
# After fetching configs, filter by model_type
configs = get_user_configs(user_id)
text_models = [c for c in configs if c["model_type"] == "text"]
vision_models = [c for c in configs if c["model_type"] == "vision"]
```
### Database Index
The `model_type` column is indexed for efficient queries:
```sql
CREATE INDEX idx_llm_model_configs_model_type ON llm_model_configs(model_type);
```
This enables fast lookups when filtering by model type, even with large datasets.
---
## Reserved Fields
The `llm_model_configs` table includes three reserved JSONB fields for future extensibility:
| Field | Type | Description |
|-------|------|-------------|
| `reserved_jsonb_1` | JSONB (nullable) | Reserved for future use |
| `reserved_jsonb_2` | JSONB (nullable) | Reserved for future use |
| `reserved_jsonb_3` | JSONB (nullable) | Reserved for future use |
**Purpose:** These fields are reserved for future feature development without requiring schema changes. They are currently unused in the API code but are available in the database layer for future enhancements.
**Use Cases:** Future features might use these fields for:
- Advanced configuration options
- Metadata storage
- Feature flags
- Extension data
- Caching computed values
**Note:** These fields are not exposed in the current API schemas and are reserved for internal use.
---
## Context Limit Configuration
### What is `context_limit`?
The `context_limit` field specifies the maximum context window size for an LLM model. This is a **required field** for all model configurations.
### Why is it required?
Model providers frequently update their models and change context window sizes:
- OpenAI GPT-4o: 128K tokens (may change)
- Anthropic Claude 3.5: 200K tokens (may change)
- Google Gemini 1.5: 2.8M tokens (may change)
Hardcoding these values in the system would quickly become outdated. Requiring users to configure this field ensures that the system always uses the correct, up-to-date values.
### Unit: K tokens
The `context_limit` value is specified in **K tokens** (thousands of tokens) to make it easier to read and write:
| Official Documentation | API Configuration |
|---------------------|-------------------|
| 128,000 tokens | `"context_limit": 128` |
| 200,000 tokens | `"context_limit": 200` |
| 2,800,000 tokens | `"context_limit": 2800` |
### How to find the correct value
1. **Check the official documentation** for your model:
- OpenAI: https://platform.openai.com/docs/models
- Anthropic: https://docs.anthropic.com/claude/docs/models-overview
- Google: https://ai.google.dev/gemini-api/docs/models
- DeepSeek: https://platform.deepseek.com/api-docs/
2. **Convert from tokens to K**:
```
context_limit = official_value_in_tokens / 1000
Example:
GPT-4o: 128,000 tokens / 1000 = 128
```
### Example: Common Models
| Model | Official Value | Configuration |
|-------|---------------|--------------|
| GPT-4o | 128,000 | `"context_limit": 128` |
| GPT-3.5 Turbo | 16,385 | `"context_limit": 17` |
| Claude 3.5 Sonnet | 200,000 | `"context_limit": 200` |
| Gemini 1.5 Pro | 2,800,000 | `"context_limit": 2800` |
| DeepSeek Chat | 128,000 | `"context_limit": 128` |
### Context Strategy
The optional `context_strategy` field controls how aggressively the system uses the available context window:
| Strategy | Usage | Best For |
|----------|-------|----------|
| `conservative` | 60% of limit | Long outputs, complex tasks, uncertain output size |
| `balanced` (default) | 75% of limit | Most conversations, general use |
| `aggressive` | 85% of limit | Short outputs, analysis tasks, predictable output size |
### Error Handling
If `context_limit` is missing or invalid, the API will return:
```json
HTTP 400 Bad Request
{
"detail": "context_limit is required (unit: K tokens, e.g., 128 = 128K = 128,000 tokens). Please check your model provider's documentation for the current context window size and specify it in the configuration."
}
```
---
## License
**Copyright © 2025 Yue Guobin (岳国宾)**
This work is licensed under the [Creative Commons Attribution-ShareAlike 4.0
International License (CC BY-SA 4.0)](https://creativecommons.org/licenses/by-sa/4.0/).
![CC BY-SA 4.0](https://i.creativecommons.org/l/by-sa/4.0/88x31.png)
### Summary
You are free to:
- **Share** — Copy and redistribute the material in any medium or format
- **Adapt** — Remix, transform, and build upon the material for any purpose
Under the following terms:
- **Attribution** — You must give appropriate credit to **Yue Guobin (岳国宾)**, provide
a link to the license, and indicate if changes were made.
- **ShareAlike** — If you remix, transform, or build upon the material, you must
distribute your contributions under the **same license** (CC BY-SA 4.0).
Full license text: [DESIGN_DOCS_LICENSE](../DESIGN_DOCS_LICENSE.md)