mirror of
https://github.com/GNS3/gns3-server.git
synced 2026-11-04 22:41:12 +02:00
docs: add context limit and strategy to LLM model configs API
- Add `context_limit` as required field for LLM model configurations - Add `context_strategy` as optional field with three trimming strategies - Update API documentation with detailed examples for GPT-4o and Claude 3.5 Sonnet - Clarify that context limit is specified in K tokens (thousands of tokens) - Update example payloads to reflect current model versions and new fields
This commit is contained in:
YueGuobin committed
1 parent
6a9f745c13
commit
7368ac098a
7 files changed
+1246
-25
No files matched your search
@@ -0,0 +1,540 @@
|
||||
# LLM 上下文窗口管理实现文档
|
||||
|
||||
## 概述
|
||||
|
||||
本文档说明了 GNS3 Copilot 如何处理不同 LLM 模型的上下文窗口限制，以及如何实现自动消息裁剪功能。
|
||||
|
||||
## ⚠️ 重要：context_limit 必须手动配置
|
||||
|
||||
**由于模型供应商频繁更新上下文窗口大小，系统不再提供内置默认值。**
|
||||
|
||||
用户在创建 LLM 模型配置时**必须提供 `context_limit`**。请从模型供应商的官方文档获取最新的上下文窗口大小。
|
||||
|
||||
### ⚡ 单位说明
|
||||
|
||||
**`context_limit` 的单位是 K tokens（千 tokens）**
|
||||
|
||||
- `1` = 1K = 1,000 tokens
|
||||
- `128` = 128K = 128,000 tokens
|
||||
- `200` = 200K = 200,000 tokens
|
||||
- `2800` = 2800K = 2,800,000 tokens
|
||||
|
||||
### 为什么使用 K tokens？
|
||||
|
||||
1. **更简洁** - `128` 比 `128000` 更易读易写
|
||||
2. **减少错误** - 避免少写或多写 0
|
||||
3. **符合习惯** - 与业界常用的 "128K", "200K" 表示法一致
|
||||
|
||||
### 为什么必须手动配置？
|
||||
|
||||
1. **模型更新频繁** - OpenAI、Anthropic、Google 等供应商经常发布模型更新
|
||||
2. **上下文大小变化** - 新版本往往增加上下文窗口，旧版本可能被废弃
|
||||
3. **维护成本高** - 内置默认值很快过时，可能导致配置错误
|
||||
4. **责任明确** - 用户根据实际使用的模型配置，避免混淆
|
||||
|
||||
---
|
||||
|
||||
## 数据库配置方式
|
||||
|
||||
### 配置要求
|
||||
|
||||
`context_limit` 是**必填字段**，单位为 **K tokens**。创建或更新 LLM 模型配置时必须提供。
|
||||
|
||||
| 字段 | 类型 | 必填 | 单位 | 说明 |
|
||||
|------|------|------|------|------|
|
||||
| `context_limit` | `int` | ✅ **是** | K tokens | 模型上下文窗口限制（128 = 128K tokens） |
|
||||
| `context_strategy` | `"conservative" \| "balanced" \| "aggressive"` | 否 | - | 裁剪策略，默认 "balanced" |
|
||||
|
||||
### 获取上下文窗口大小
|
||||
|
||||
#### 官方文档链接
|
||||
|
||||
| 供应商 | 文档链接 | 示例值（tokens） | 配置值（K） |
|
||||
|--------|---------|------------------|------------|
|
||||
| OpenAI | https://platform.openai.com/docs/models | 128,000 | `128` |
|
||||
| Anthropic | https://docs.anthropic.com/claude/docs/models-overview | 200,000 | `200` |
|
||||
| Google | https://ai.google.dev/gemini-api/docs/models | 2,800,000 | `2800` |
|
||||
| DeepSeek | https://platform.deepseek.com/api-docs/ | 128,000 | `128` |
|
||||
| xAI | https://docs.x.ai/ | 128,000 | `128` |
|
||||
|
||||
**转换示例**：
|
||||
- 官方文档显示 `128,000 tokens` → 配置为 `128`
|
||||
- 官方文档显示 `200,000 tokens` → 配置为 `200`
|
||||
- 官方文档显示 `2,800,000 tokens` → 配置为 `2800`
|
||||
|
||||
#### 参考工具
|
||||
|
||||
运行参考脚本查看常见模型的上下文限制（显示为 K tokens）：
|
||||
|
||||
```bash
|
||||
python scripts/show_model_context_limits.py
|
||||
```
|
||||
|
||||
**注意**：此脚本仅提供参考值，请以官方文档为准。
|
||||
|
||||
---
|
||||
|
||||
## API 使用示例
|
||||
|
||||
### 创建配置（必须提供 context_limit）
|
||||
|
||||
```bash
|
||||
POST /v3/users/{user_id}/llm-model-configs
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"name": "GPT-4o Configuration",
|
||||
"model_type": "text",
|
||||
"is_default": true,
|
||||
"provider": "openai",
|
||||
"base_url": "https://api.openai.com/v1",
|
||||
"model": "gpt-4o",
|
||||
"temperature": 0.7,
|
||||
"api_key": "sk-...",
|
||||
"context_limit": 128, # ← 必填：单位是 K tokens (128 = 128K = 128,000 tokens)
|
||||
"context_strategy": "balanced" # 可选，默认 "balanced"
|
||||
}
|
||||
```
|
||||
|
||||
### 配置示例对比
|
||||
|
||||
#### 官方文档 → API 配置
|
||||
|
||||
| 模型 | 官方文档 | API 配置 |
|
||||
|------|---------|---------|
|
||||
| GPT-4o | 128,000 tokens | `"context_limit": 128` |
|
||||
| GPT-3.5 Turbo | 16,385 tokens | `"context_limit": 17` (向上取整) |
|
||||
| Claude 3.5 Sonnet | 200,000 tokens | `"context_limit": 200` |
|
||||
| Gemini 1.5 Pro | 2,800,000 tokens | `"context_limit": 2800` |
|
||||
|
||||
### 错误示例（缺少 context_limit）
|
||||
|
||||
```bash
|
||||
POST /v3/users/{user_id}/llm-model-configs
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"name": "Invalid Config",
|
||||
"provider": "openai",
|
||||
"model": "gpt-4o",
|
||||
# ❌ 缺少 context_limit
|
||||
}
|
||||
```
|
||||
|
||||
**响应**：
|
||||
```json
|
||||
{
|
||||
"detail": "context_limit is required (unit: K tokens, e.g., 128 = 128K = 128,000 tokens). Please check your model provider's documentation for the current context window size and specify it in the configuration."
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 常见模型配置参考
|
||||
|
||||
以下是一些常见模型的 `context_limit` 配置值（**请以官方文档为准**）：
|
||||
|
||||
### OpenAI
|
||||
| 模型 | 官方文档 | 配置值 |
|
||||
|------|---------|-------|
|
||||
| GPT-4o | 128,000 tokens | `"context_limit": 128` |
|
||||
| GPT-4o-mini | 128,000 tokens | `"context_limit": 128` |
|
||||
| GPT-4 Turbo | 128,000 tokens | `"context_limit": 128` |
|
||||
| GPT-3.5 Turbo | 16,385 tokens | `"context_limit": 17` |
|
||||
|
||||
### Anthropic
|
||||
| 模型 | 官方文档 | 配置值 |
|
||||
|------|---------|-------|
|
||||
| Claude 3.5 Sonnet | 200,000 tokens | `"context_limit": 200` |
|
||||
| Claude 3 Opus | 200,000 tokens | `"context_limit": 200` |
|
||||
| Claude 3 Haiku | 200,000 tokens | `"context_limit": 200` |
|
||||
|
||||
### Google
|
||||
| 模型 | 官方文档 | 配置值 |
|
||||
|------|---------|-------|
|
||||
| Gemini 2.0 Flash | 1,000,000 tokens | `"context_limit": 1000` |
|
||||
| Gemini 1.5 Pro | 2,800,000 tokens | `"context_limit": 2800` |
|
||||
| Gemini 1.5 Flash | 2,800,000 tokens | `"context_limit": 2800` |
|
||||
|
||||
### DeepSeek
|
||||
| 模型 | 官方文档 | 配置值 |
|
||||
|------|---------|-------|
|
||||
| DeepSeek Chat | 128,000 tokens | `"context_limit": 128` |
|
||||
| DeepSeek Coder | 128,000 tokens | `"context_limit": 128` |
|
||||
|
||||
### xAI
|
||||
| 模型 | 官方文档 | 配置值 |
|
||||
|------|---------|-------|
|
||||
| Grok Beta | 128,000 tokens | `"context_limit": 128` |
|
||||
|
||||
---
|
||||
|
||||
## 问题背景
|
||||
|
||||
### 原始问题
|
||||
- 消息历史无限制累积，使用 `operator.add` 追加所有消息
|
||||
- 不同模型的上下文窗口限制不同
|
||||
- 当消息历史超过模型限制时，LLM 调用会失败
|
||||
- 没有优雅的降级或裁剪机制
|
||||
|
||||
### 解决方案
|
||||
使用 LangChain/LangGraph 的内置功能实现智能上下文管理：
|
||||
- **`trim_messages`**: LangChain 的消息裁剪工具
|
||||
- **`count_tokens_approximately`**: Token 计数功能
|
||||
- **用户配置**: 必须提供 context_limit（单位：K tokens），无默认值
|
||||
|
||||
## 实现架构
|
||||
|
||||
### 1. 核心模块
|
||||
|
||||
**文件位置**: `gns3server/agent/gns3_copilot/agent/context_manager.py`
|
||||
|
||||
#### 关键函数
|
||||
|
||||
**`get_model_context_limit(model_name: str, llm_config: dict) -> int`**
|
||||
- 从数据库配置获取模型的上下文窗口大小
|
||||
- **`llm_config` 必须包含 `context_limit` 字段（单位：K tokens）**
|
||||
- 如果未提供或无效，抛出 `ValueError`
|
||||
- 返回值单位为实际 tokens（K tokens × 1000）
|
||||
|
||||
**`calculate_max_tokens(model_limit, strategy) -> int`**
|
||||
- 计算可用 token 数量（预留输出空间）
|
||||
- 三种策略：
|
||||
- `conservative`: 使用 60% 限制（更安全）
|
||||
- `balanced`: 使用 75% 限制（默认）
|
||||
- `aggressive`: 使用 85% 限制（最大化输入）
|
||||
|
||||
**`trim_messages_for_context(messages, model_name, strategy) -> list`**
|
||||
- 使用 LangChain 的 `trim_messages` 裁剪消息
|
||||
- 保留最近的消息
|
||||
- 始终保留系统消息
|
||||
|
||||
**`prepare_context_messages(state_messages, system_prompt, topology_context, model_name, trim_strategy) -> list`**
|
||||
- GNS3 Copilot 的主要入口点
|
||||
- 构建完整上下文（系统提示 + 拓扑信息 + 消息历史）
|
||||
- 自动裁剪以适应模型限制
|
||||
|
||||
### 2. 集成到 GNS3 Copilot
|
||||
|
||||
**修改文件**: `gns3server/agent/gns3_copilot/agent/gns3_copilot.py`
|
||||
|
||||
#### 改动内容
|
||||
|
||||
**导入新模块**:
|
||||
```python
|
||||
from gns3server.agent.gns3_copilot.agent.context_manager import (
|
||||
prepare_context_messages,
|
||||
)
|
||||
```
|
||||
|
||||
**替换消息构建逻辑**:
|
||||
```python
|
||||
# 旧代码（手动构建）
|
||||
full_messages = (
|
||||
[SystemMessage(content=current_prompt)]
|
||||
+ context_messages
|
||||
+ state["messages"]
|
||||
)
|
||||
|
||||
# 新代码（自动裁剪）
|
||||
full_messages = prepare_context_messages(
|
||||
state_messages=state["messages"],
|
||||
system_prompt=current_prompt,
|
||||
topology_context=topology_context,
|
||||
model_name=llm_config.get("model", "default"),
|
||||
llm_config=llm_config, # ← 包含 context_limit 和 context_strategy
|
||||
)
|
||||
```
|
||||
|
||||
### 3. LangChain/LangGraph 的内置功能
|
||||
|
||||
#### `trim_messages` 参数说明
|
||||
|
||||
```python
|
||||
from langchain_core.messages.utils import trim_messages
|
||||
|
||||
trimmed = trim_messages(
|
||||
messages,
|
||||
strategy="last", # 保留最近的 N 条消息
|
||||
max_tokens=10000, # 最大 token 数
|
||||
token_counter=count_tokens_approximately, # Token 计数器
|
||||
preserve_system=True, # 保留系统消息
|
||||
start_on="human", # 确保从人类消息开始
|
||||
end_on=("human", "tool", "ai"), # 在特定类型消息结束
|
||||
)
|
||||
```
|
||||
|
||||
#### `count_tokens_approximately`
|
||||
|
||||
- 快速估算消息的 token 数量
|
||||
- 不需要调用 LLM API
|
||||
- 用于判断是否需要裁剪
|
||||
|
||||
---
|
||||
|
||||
## 策略详细说明
|
||||
|
||||
### Conservative（保守策略）
|
||||
|
||||
**参数配置**：
|
||||
```json
|
||||
{
|
||||
"context_limit": 128,
|
||||
"context_strategy": "conservative"
|
||||
}
|
||||
```
|
||||
|
||||
**实际效果**：
|
||||
- **使用比例**：60% (context_limit × 0.6)
|
||||
- **计算公式**：`max_tokens = context_limit × 1000 × 0.6`
|
||||
- **示例**：对于 128K 上下文限制
|
||||
- 保留空间：`128 × 1000 × 0.6 = 76,800` tokens 用于输入
|
||||
- 预留空间：`128,000 - 76,800 = 51,200` tokens 用于输出
|
||||
|
||||
**适用场景**：
|
||||
1. **长输出任务**
|
||||
- 代码生成（可能生成数百行代码）
|
||||
- 文章写作（需要完整文章内容）
|
||||
- 详细报告生成（包含多个章节）
|
||||
|
||||
2. **复杂任务**
|
||||
- 多步骤推理任务
|
||||
- 需要深度分析的请求
|
||||
- 综合性问题的解决方案
|
||||
|
||||
3. **不确定输出大小时**
|
||||
- 不确定 LLM 会返回多长的内容
|
||||
- 首次尝试某种类型的任务
|
||||
- 需要额外安全边界的场景
|
||||
|
||||
**优缺点**：
|
||||
- ✅ **优点**：输出更不容易被截断，安全性高
|
||||
- ❌ **缺点**：输入上下文较少，可能遗漏早期信息
|
||||
|
||||
**日志示例**：
|
||||
```
|
||||
INFO: Context prepared: 20 msgs, ~72800 tokens / 128K limit (56.9%), strategy=conservative
|
||||
INFO: Available for output: ~51200 tokens
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Balanced（平衡策略，推荐）
|
||||
|
||||
**参数配置**：
|
||||
```json
|
||||
{
|
||||
"context_limit": 128,
|
||||
"context_strategy": "balanced"
|
||||
}
|
||||
```
|
||||
|
||||
**实际效果**：
|
||||
- **使用比例**：75% (context_limit × 0.75)
|
||||
- **计算公式**：`max_tokens = context_limit × 1000 × 0.75`
|
||||
- **示例**：对于 128K 上下文限制
|
||||
- 保留空间：`128 × 1000 × 0.75 = 96,000` tokens 用于输入
|
||||
- 预留空间：`128,000 - 96,000 = 32,000` tokens 用于输出
|
||||
|
||||
**适用场景**：
|
||||
1. **一般对话**
|
||||
- 日常聊天交互
|
||||
- 问答式对话
|
||||
- 技术支持和咨询
|
||||
|
||||
2. **大多数场景**
|
||||
- 网络配置命令生成
|
||||
- 故障排查建议
|
||||
- 知识问答
|
||||
|
||||
3. **平衡输入和输出**
|
||||
- 需要较多上下文，但输出也较长的场景
|
||||
- 中等复杂度的任务
|
||||
|
||||
**优缺点**：
|
||||
- ✅ **优点**：在输入上下文和输出空间之间取得良好平衡
|
||||
- ✅ **优点**：适用于大多数使用场景
|
||||
- ⚠️ **注意**：对于特别长的输出可能被截断
|
||||
|
||||
**日志示例**：
|
||||
```
|
||||
INFO: Context prepared: 30 msgs, ~85600 tokens / 128K limit (66.9%), strategy=balanced
|
||||
INFO: Available for output: ~32000 tokens
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Aggressive（激进策略）
|
||||
|
||||
**参数配置**：
|
||||
```json
|
||||
{
|
||||
"context_limit": 128,
|
||||
"context_strategy": "aggressive"
|
||||
}
|
||||
```
|
||||
|
||||
**实际效果**：
|
||||
- **使用比例**：85% (context_limit × 0.85)
|
||||
- **计算公式**：`max_tokens = context_limit × 1000 × 0.85`
|
||||
- **示例**：对于 128K 上下文限制
|
||||
- 保留空间：`128 × 1000 × 0.85 = 108,800` tokens 用于输入
|
||||
- 预留空间：`128,000 - 108,800 = 19,200` tokens 用于输出
|
||||
|
||||
**适用场景**：
|
||||
1. **简短输出任务**
|
||||
- 是/否判断
|
||||
- 简短回答确认
|
||||
- 状态查询类请求
|
||||
|
||||
2. **分析类任务**
|
||||
- 日志分析（输出分析结果，但不需要很长）
|
||||
- 数据解读（输出简洁的结论）
|
||||
- 配置检查（返回 OK 或简短说明）
|
||||
|
||||
3. **输出比较确定的场景**
|
||||
- 明确知道输出会很短
|
||||
- 只需要简单确认或状态
|
||||
- 不需要长篇解释的任务
|
||||
|
||||
**优缺点**：
|
||||
- ✅ **优点**：最大化输入上下文，保留更多历史信息
|
||||
- ❌ **缺点**：输出容易被截断，不适用于长输出任务
|
||||
|
||||
**日志示例**：
|
||||
```
|
||||
INFO: Context prepared: 50 msgs, ~105200 tokens / 128K limit (82.2%), strategy=aggressive
|
||||
INFO: Available for output: ~19200 tokens
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 策略对比总结
|
||||
|
||||
| 特性 | Conservative | Balanced | Aggressive |
|
||||
|------|-------------|----------|------------|
|
||||
| **输入比例** | 60% | 75% | 85% |
|
||||
| **输出预留** | 40% | 25% | 15% |
|
||||
| **上下文数量** | 最少 | 中等 | 最多 |
|
||||
| **输出空间** | 最大 | 中等 | 最小 |
|
||||
| **适用性** | 长输出 | 一般使用 | 短输出 |
|
||||
| **风险** | 上下文不足 | 平衡 | 输出截断 |
|
||||
|
||||
**选择建议流程**：
|
||||
```
|
||||
1. 任务类型是什么？
|
||||
├─ 代码生成/长文档 → Conservative
|
||||
├─ 日常对话/一般任务 → Balanced (推荐)
|
||||
└─ 简短确认/分析 → Aggressive
|
||||
|
||||
2. 输出长度预估？
|
||||
├─ 不确定/可能很长 → Conservative
|
||||
├─ 中等长度 → Balanced
|
||||
└─ 很短/简洁 → Aggressive
|
||||
|
||||
3. 上下文重要性？
|
||||
├─ 早期历史不太重要 → Aggressive (更多上下文)
|
||||
├─ 需要平衡 → Balanced
|
||||
└─ 重点是输出完整性 → Conservative (更多输出空间)
|
||||
```
|
||||
|
||||
## 工作流程
|
||||
|
||||
```
|
||||
用户发送消息
|
||||
↓
|
||||
获取消息历史 state["messages"]
|
||||
↓
|
||||
构建系统提示 + 拓扑信息
|
||||
↓
|
||||
调用 prepare_context_messages()
|
||||
↓
|
||||
├─ 估算 token 数量
|
||||
├─ 获取模型上下文限制
|
||||
├─ 判断是否需要裁剪
|
||||
└─ 如果需要 → 调用 trim_messages()
|
||||
↓
|
||||
调用 LLM（带裁剪后的消息）
|
||||
↓
|
||||
返回响应
|
||||
```
|
||||
|
||||
## 日志输出示例
|
||||
|
||||
### 正常情况
|
||||
```
|
||||
INFO: Using database config context limit: 128000 tokens for model 'gpt-4o'
|
||||
INFO: Context prepared: 15 msgs, ~8432 tokens / 128000 limit (6.6%), strategy=balanced
|
||||
INFO: LLM call completed: tool_calls=2
|
||||
```
|
||||
|
||||
### 发生裁剪时
|
||||
```
|
||||
INFO: Using database config context limit: 128000 tokens for model 'gpt-4o'
|
||||
INFO: Trimming messages: 18500 → 9600 tokens (model: gpt-4o)
|
||||
INFO: Trimmed 50 → 25 messages
|
||||
INFO: Context prepared: 27 msgs, ~9432 tokens / 128000 limit (7.4%), strategy=balanced
|
||||
```
|
||||
|
||||
### 配置错误时
|
||||
```
|
||||
ERROR: context_limit is required but not provided for model 'gpt-4o'.
|
||||
Please configure context_limit in your LLM model configuration.
|
||||
Refer to the model provider's documentation for the current context window size.
|
||||
```
|
||||
|
||||
## 错误处理
|
||||
|
||||
### Token 计数失败
|
||||
```python
|
||||
try:
|
||||
tokens = count_tokens_approximately(messages)
|
||||
except Exception as e:
|
||||
logger.warning("Failed to count tokens: %s", e)
|
||||
# 降级到简单的消息数量限制
|
||||
return messages[-50:]
|
||||
```
|
||||
|
||||
### 裁剪失败
|
||||
```python
|
||||
try:
|
||||
trimmed = trim_messages(...)
|
||||
except Exception as e:
|
||||
logger.error("Failed to trim: %s", e)
|
||||
# 回退到简单的切片操作
|
||||
return system_msgs + other_msgs[-N:]
|
||||
```
|
||||
|
||||
## 参考资源
|
||||
|
||||
- [LangChain Messages Utils](https://python.langchain.com/docs/messages/)
|
||||
- [LangGraph Memory Management](https://langchain-ai.github.io/langgraph/concepts/agentic_concepts/#memory)
|
||||
- [OpenAI Models Context Limits](https://platform.openai.com/docs/models)
|
||||
- [Anthropic Models Context Limits](https://docs.anthropic.com/claude/docs/models-overview)
|
||||
|
||||
## 相关源文件
|
||||
|
||||
- `gns3server/agent/gns3_copilot/agent/context_manager.py` - 上下文管理核心逻辑
|
||||
- `gns3server/agent/gns3_copilot/agent/gns3_copilot.py` - LLM 调用节点
|
||||
- `gns3server/agent/gns3_copilot/agent_service.py` - Agent 服务接口
|
||||
|
||||
## 总结
|
||||
|
||||
通过使用 LangChain/LangGraph 的内置功能，我们实现了：
|
||||
1. ✅ 智能裁剪消息历史，避免超限
|
||||
2. ✅ **必须手动配置 context_limit**，确保使用正确的上下文窗口大小
|
||||
3. ✅ 可配置的裁剪策略（保守/平衡/激进）
|
||||
4. ✅ 始终保留系统消息和拓扑信息
|
||||
5. ✅ 详细的日志输出，便于调试
|
||||
6. ✅ 优雅的错误处理和明确的错误提示
|
||||
7. ✅ 提供参考工具，帮助查找常见模型的上下文限制
|
||||
|
||||
### 关键优势
|
||||
|
||||
- **准确性**：用户从官方文档获取最新的上下文限制，避免使用过时数据
|
||||
- **灵活性**：每个配置独立设置，支持不同用户使用不同限制
|
||||
- **明确性**：缺少配置时立即报错，避免静默失败
|
||||
- **可维护性**：无需维护内置默认值，减少代码维护负担
|
||||
- **可观测性**：详细日志显示上下文使用情况和裁剪决策
|
||||
|
||||
这个实现确保了即使在进行长对话时，系统也不会因为上下文溢出而失败。
|
||||
@@ -115,6 +115,7 @@ The `model_type` field accepts the following values:
|
||||
| `base_url` | string | API base URL |
|
||||
| `model` | string | Model name |
|
||||
| `temperature` | float | Temperature (0.0-2.0, default: 0.7) |
|
||||
| `context_limit` | integer | **Model context window limit in K tokens** (e.g., 128 = 128K = 128,000 tokens) |
|
||||
|
||||
**Optional Fields:**
|
||||
|
||||
@@ -122,8 +123,17 @@ The `model_type` field accepts the following values:
|
||||
|-------|------|-------------|
|
||||
| `api_key` | string | API key (auto-encrypted) |
|
||||
| `max_tokens` | integer | Max tokens for generation |
|
||||
| `context_strategy` | string | Context trimming strategy: "conservative" (60%), "balanced" (75%), "aggressive" (85%). Default: "balanced" |
|
||||
| `is_default` | boolean | Set as default (default: false) |
|
||||
|
||||
**Important Notes:**
|
||||
|
||||
- **`context_limit` is required**: You must specify the model's context window limit. Refer to the model provider's official documentation for the current value.
|
||||
- **Unit is K tokens**: The value is in thousands of tokens (1 K = 1,000 tokens). For example:
|
||||
- GPT-4o: 128,000 tokens → configure as `"context_limit": 128`
|
||||
- Claude 3.5 Sonnet: 200,000 tokens → configure as `"context_limit": 200`
|
||||
- Gemini 1.5 Pro: 2,800,000 tokens → configure as `"context_limit": 2800`
|
||||
|
||||
**Extra Fields:** Any custom fields are supported for future extensibility.
|
||||
|
||||
### LLMModelConfigUpdate
|
||||
@@ -138,6 +148,8 @@ The `model_type` field accepts the following values:
|
||||
| `temperature` | float (optional) | Temperature |
|
||||
| `api_key` | string (optional) | API key |
|
||||
| `max_tokens` | integer (optional) | Max tokens |
|
||||
| `context_limit` | integer (optional) | Model context window limit in K tokens |
|
||||
| `context_strategy` | string (optional) | Context trimming strategy |
|
||||
| `is_default` | boolean (optional) | Default flag |
|
||||
| `expected_version` | integer (optional) | **Optimistic locking version** |
|
||||
|
||||
@@ -213,28 +225,36 @@ curl -X POST http://localhost:3080/v3/access/users/{user_id}/llm-model-configs \
|
||||
-H "Authorization: Bearer <token>" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"name": "GPT-4",
|
||||
"name": "GPT-4o",
|
||||
"model_type": "text",
|
||||
"provider": "openai",
|
||||
"base_url": "https://api.openai.com/v1",
|
||||
"model": "gpt-4",
|
||||
"model": "gpt-4o",
|
||||
"temperature": 0.7,
|
||||
"context_limit": 128,
|
||||
"context_strategy": "balanced",
|
||||
"api_key": "sk-xxx",
|
||||
"is_default": true
|
||||
}'
|
||||
```
|
||||
|
||||
**Important:**
|
||||
- `context_limit` is **required** and specified in K tokens (e.g., 128 = 128K = 128,000 tokens)
|
||||
- Refer to the model provider's official documentation for the current context window size
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"config_id": "uuid-1",
|
||||
"name": "GPT-4",
|
||||
"name": "GPT-4o",
|
||||
"model_type": "text",
|
||||
"config": {
|
||||
"provider": "openai",
|
||||
"base_url": "https://api.openai.com/v1",
|
||||
"model": "gpt-4",
|
||||
"model": "gpt-4o",
|
||||
"temperature": 0.7,
|
||||
"context_limit": 128,
|
||||
"context_strategy": "balanced",
|
||||
"api_key": "sk-xxx"
|
||||
},
|
||||
"user_id": "uuid-user",
|
||||
@@ -253,12 +273,14 @@ curl -X POST http://localhost:3080/v3/access/groups/{group_id}/llm-model-configs
|
||||
-H "Authorization: Bearer <token>" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"name": "Claude-3",
|
||||
"name": "Claude-3.5 Sonnet",
|
||||
"model_type": "text",
|
||||
"provider": "anthropic",
|
||||
"base_url": "https://api.anthropic.com",
|
||||
"model": "claude-3-opus-20240229",
|
||||
"model": "claude-3-5-sonnet-20241022",
|
||||
"temperature": 0.7,
|
||||
"context_limit": 200,
|
||||
"context_strategy": "balanced",
|
||||
"api_key": "sk-ant-xxx",
|
||||
"is_default": true
|
||||
}'
|
||||
@@ -277,13 +299,15 @@ curl -X GET http://localhost:3080/v3/access/users/{user_id}/llm-model-configs \
|
||||
"configs": [
|
||||
{
|
||||
"config_id": "uuid-1",
|
||||
"name": "GPT-4",
|
||||
"name": "GPT-4o",
|
||||
"model_type": "text",
|
||||
"config": {
|
||||
"provider": "openai",
|
||||
"base_url": "https://api.openai.com/v1",
|
||||
"model": "gpt-4",
|
||||
"model": "gpt-4o",
|
||||
"temperature": 0.7,
|
||||
"context_limit": 128,
|
||||
"context_strategy": "balanced",
|
||||
"api_key": "sk-xxx"
|
||||
},
|
||||
"user_id": "uuid-user",
|
||||
@@ -297,13 +321,15 @@ curl -X GET http://localhost:3080/v3/access/users/{user_id}/llm-model-configs \
|
||||
},
|
||||
{
|
||||
"config_id": "uuid-2",
|
||||
"name": "Claude-3",
|
||||
"name": "Claude-3.5 Sonnet",
|
||||
"model_type": "text",
|
||||
"config": {
|
||||
"provider": "anthropic",
|
||||
"base_url": "https://api.anthropic.com",
|
||||
"model": "claude-3-opus-20240229",
|
||||
"model": "claude-3-5-sonnet-20241022",
|
||||
"temperature": 0.7,
|
||||
"context_limit": 200,
|
||||
"context_strategy": "balanced",
|
||||
"api_key": null
|
||||
},
|
||||
"user_id": null,
|
||||
@@ -318,11 +344,16 @@ curl -X GET http://localhost:3080/v3/access/users/{user_id}/llm-model-configs \
|
||||
],
|
||||
"default_config": {
|
||||
"config_id": "uuid-1",
|
||||
"name": "GPT-4",
|
||||
"name": "GPT-4o",
|
||||
"model_type": "text",
|
||||
"config": {
|
||||
"provider": "openai",
|
||||
...
|
||||
"base_url": "https://api.openai.com/v1",
|
||||
"model": "gpt-4o",
|
||||
"temperature": 0.7,
|
||||
"context_limit": 128,
|
||||
"context_strategy": "balanced",
|
||||
"api_key": "sk-xxx"
|
||||
},
|
||||
"user_id": "uuid-user",
|
||||
"group_id": null,
|
||||
@@ -340,6 +371,7 @@ curl -X GET http://localhost:3080/v3/access/users/{user_id}/llm-model-configs \
|
||||
- `source: "user"` indicates the config belongs to the user
|
||||
- `source: "group"` indicates the config is inherited from a group
|
||||
- Configuration fields are nested in the `config` object (same structure as group endpoints)
|
||||
- `context_limit` is in K tokens (128 = 128K = 128,000 tokens)
|
||||
|
||||
### 4. Get group configurations
|
||||
|
||||
@@ -417,7 +449,8 @@ curl -X PUT http://localhost:3080/v3/access/users/{user_id}/llm-model-configs/{c
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"temperature": 0.9,
|
||||
"max_tokens": 4000
|
||||
"max_tokens": 4000,
|
||||
"context_strategy": "aggressive"
|
||||
}'
|
||||
```
|
||||
|
||||
@@ -482,13 +515,15 @@ curl -X GET http://localhost:3080/v3/access/users/{user_id}/llm-model-configs/de
|
||||
```json
|
||||
{
|
||||
"config_id": "uuid-1",
|
||||
"name": "GPT-4",
|
||||
"name": "GPT-4o",
|
||||
"model_type": "text",
|
||||
"config": {
|
||||
"provider": "openai",
|
||||
"base_url": "https://api.openai.com/v1",
|
||||
"model": "gpt-4",
|
||||
"model": "gpt-4o",
|
||||
"temperature": 0.7,
|
||||
"context_limit": 128,
|
||||
"context_strategy": "balanced",
|
||||
"api_key": "sk-xxx"
|
||||
},
|
||||
"user_id": "uuid-user",
|
||||
@@ -723,3 +758,82 @@ The `llm_model_configs` table includes three reserved JSONB fields for future ex
|
||||
- Caching computed values
|
||||
|
||||
**Note:** These fields are not exposed in the current API schemas and are reserved for internal use.
|
||||
|
||||
---
|
||||
|
||||
## Context Limit Configuration
|
||||
|
||||
### What is `context_limit`?
|
||||
|
||||
The `context_limit` field specifies the maximum context window size for an LLM model. This is a **required field** for all model configurations.
|
||||
|
||||
### Why is it required?
|
||||
|
||||
Model providers frequently update their models and change context window sizes:
|
||||
- OpenAI GPT-4o: 128K tokens (may change)
|
||||
- Anthropic Claude 3.5: 200K tokens (may change)
|
||||
- Google Gemini 1.5: 2.8M tokens (may change)
|
||||
|
||||
Hardcoding these values in the system would quickly become outdated. Requiring users to configure this field ensures that the system always uses the correct, up-to-date values.
|
||||
|
||||
### Unit: K tokens
|
||||
|
||||
The `context_limit` value is specified in **K tokens** (thousands of tokens) to make it easier to read and write:
|
||||
|
||||
| Official Documentation | API Configuration |
|
||||
|---------------------|-------------------|
|
||||
| 128,000 tokens | `"context_limit": 128` |
|
||||
| 200,000 tokens | `"context_limit": 200` |
|
||||
| 2,800,000 tokens | `"context_limit": 2800` |
|
||||
|
||||
### How to find the correct value
|
||||
|
||||
1. **Check the official documentation** for your model:
|
||||
- OpenAI: https://platform.openai.com/docs/models
|
||||
- Anthropic: https://docs.anthropic.com/claude/docs/models-overview
|
||||
- Google: https://ai.google.dev/gemini-api/docs/models
|
||||
- DeepSeek: https://platform.deepseek.com/api-docs/
|
||||
|
||||
2. **Convert from tokens to K**:
|
||||
```
|
||||
context_limit = official_value_in_tokens / 1000
|
||||
|
||||
Example:
|
||||
GPT-4o: 128,000 tokens / 1000 = 128
|
||||
```
|
||||
|
||||
3. **Use the reference tool**:
|
||||
```bash
|
||||
python scripts/show_model_context_limits.py
|
||||
```
|
||||
|
||||
### Example: Common Models
|
||||
|
||||
| Model | Official Value | Configuration |
|
||||
|-------|---------------|--------------|
|
||||
| GPT-4o | 128,000 | `"context_limit": 128` |
|
||||
| GPT-3.5 Turbo | 16,385 | `"context_limit": 17` |
|
||||
| Claude 3.5 Sonnet | 200,000 | `"context_limit": 200` |
|
||||
| Gemini 1.5 Pro | 2,800,000 | `"context_limit": 2800` |
|
||||
| DeepSeek Chat | 128,000 | `"context_limit": 128` |
|
||||
|
||||
### Context Strategy
|
||||
|
||||
The optional `context_strategy` field controls how aggressively the system uses the available context window:
|
||||
|
||||
| Strategy | Usage | Best For |
|
||||
|----------|-------|----------|
|
||||
| `conservative` | 60% of limit | Long outputs, complex tasks, uncertain output size |
|
||||
| `balanced` (default) | 75% of limit | Most conversations, general use |
|
||||
| `aggressive` | 85% of limit | Short outputs, analysis tasks, predictable output size |
|
||||
|
||||
### Error Handling
|
||||
|
||||
If `context_limit` is missing or invalid, the API will return:
|
||||
|
||||
```json
|
||||
HTTP 400 Bad Request
|
||||
{
|
||||
"detail": "context_limit is required (unit: K tokens, e.g., 128 = 128K = 128,000 tokens). Please check your model provider's documentation for the current context window size and specify it in the configuration."
|
||||
}
|
||||
```
|
||||
Reference in new issue
Block a user