diff --git a/docs/gns3-copilot/context-window-management.md b/docs/gns3-copilot/context-window-management.md new file mode 100644 index 000000000..1e4032a8b --- /dev/null +++ b/docs/gns3-copilot/context-window-management.md @@ -0,0 +1,540 @@ +# LLM 上下文窗口管理实现文档 + +## 概述 + +本文档说明了 GNS3 Copilot 如何处理不同 LLM 模型的上下文窗口限制,以及如何实现自动消息裁剪功能。 + +## ⚠️ 重要:context_limit 必须手动配置 + +**由于模型供应商频繁更新上下文窗口大小,系统不再提供内置默认值。** + +用户在创建 LLM 模型配置时**必须提供 `context_limit`**。请从模型供应商的官方文档获取最新的上下文窗口大小。 + +### ⚡ 单位说明 + +**`context_limit` 的单位是 K tokens(千 tokens)** + +- `1` = 1K = 1,000 tokens +- `128` = 128K = 128,000 tokens +- `200` = 200K = 200,000 tokens +- `2800` = 2800K = 2,800,000 tokens + +### 为什么使用 K tokens? + +1. **更简洁** - `128` 比 `128000` 更易读易写 +2. **减少错误** - 避免少写或多写 0 +3. **符合习惯** - 与业界常用的 "128K", "200K" 表示法一致 + +### 为什么必须手动配置? + +1. **模型更新频繁** - OpenAI、Anthropic、Google 等供应商经常发布模型更新 +2. **上下文大小变化** - 新版本往往增加上下文窗口,旧版本可能被废弃 +3. **维护成本高** - 内置默认值很快过时,可能导致配置错误 +4. **责任明确** - 用户根据实际使用的模型配置,避免混淆 + +--- + +## 数据库配置方式 + +### 配置要求 + +`context_limit` 是**必填字段**,单位为 **K tokens**。创建或更新 LLM 模型配置时必须提供。 + +| 字段 | 类型 | 必填 | 单位 | 说明 | +|------|------|------|------|------| +| `context_limit` | `int` | ✅ **是** | K tokens | 模型上下文窗口限制(128 = 128K tokens) | +| `context_strategy` | `"conservative" \| "balanced" \| "aggressive"` | 否 | - | 裁剪策略,默认 "balanced" | + +### 获取上下文窗口大小 + +#### 官方文档链接 + +| 供应商 | 文档链接 | 示例值(tokens) | 配置值(K) | +|--------|---------|------------------|------------| +| OpenAI | https://platform.openai.com/docs/models | 128,000 | `128` | +| Anthropic | https://docs.anthropic.com/claude/docs/models-overview | 200,000 | `200` | +| Google | https://ai.google.dev/gemini-api/docs/models | 2,800,000 | `2800` | +| DeepSeek | https://platform.deepseek.com/api-docs/ | 128,000 | `128` | +| xAI | https://docs.x.ai/ | 128,000 | `128` | + +**转换示例**: +- 官方文档显示 `128,000 tokens` → 配置为 `128` +- 官方文档显示 `200,000 tokens` → 配置为 `200` +- 官方文档显示 `2,800,000 tokens` → 配置为 `2800` + +#### 参考工具 + +运行参考脚本查看常见模型的上下文限制(显示为 K tokens): + +```bash +python scripts/show_model_context_limits.py +``` + +**注意**:此脚本仅提供参考值,请以官方文档为准。 + +--- + +## API 使用示例 + +### 创建配置(必须提供 context_limit) + +```bash +POST /v3/users/{user_id}/llm-model-configs +Content-Type: application/json + +{ + "name": "GPT-4o Configuration", + "model_type": "text", + "is_default": true, + "provider": "openai", + "base_url": "https://api.openai.com/v1", + "model": "gpt-4o", + "temperature": 0.7, + "api_key": "sk-...", + "context_limit": 128, # ← 必填:单位是 K tokens (128 = 128K = 128,000 tokens) + "context_strategy": "balanced" # 可选,默认 "balanced" +} +``` + +### 配置示例对比 + +#### 官方文档 → API 配置 + +| 模型 | 官方文档 | API 配置 | +|------|---------|---------| +| GPT-4o | 128,000 tokens | `"context_limit": 128` | +| GPT-3.5 Turbo | 16,385 tokens | `"context_limit": 17` (向上取整) | +| Claude 3.5 Sonnet | 200,000 tokens | `"context_limit": 200` | +| Gemini 1.5 Pro | 2,800,000 tokens | `"context_limit": 2800` | + +### 错误示例(缺少 context_limit) + +```bash +POST /v3/users/{user_id}/llm-model-configs +Content-Type: application/json + +{ + "name": "Invalid Config", + "provider": "openai", + "model": "gpt-4o", + # ❌ 缺少 context_limit +} +``` + +**响应**: +```json +{ + "detail": "context_limit is required (unit: K tokens, e.g., 128 = 128K = 128,000 tokens). Please check your model provider's documentation for the current context window size and specify it in the configuration." +} +``` + +--- + +## 常见模型配置参考 + +以下是一些常见模型的 `context_limit` 配置值(**请以官方文档为准**): + +### OpenAI +| 模型 | 官方文档 | 配置值 | +|------|---------|-------| +| GPT-4o | 128,000 tokens | `"context_limit": 128` | +| GPT-4o-mini | 128,000 tokens | `"context_limit": 128` | +| GPT-4 Turbo | 128,000 tokens | `"context_limit": 128` | +| GPT-3.5 Turbo | 16,385 tokens | `"context_limit": 17` | + +### Anthropic +| 模型 | 官方文档 | 配置值 | +|------|---------|-------| +| Claude 3.5 Sonnet | 200,000 tokens | `"context_limit": 200` | +| Claude 3 Opus | 200,000 tokens | `"context_limit": 200` | +| Claude 3 Haiku | 200,000 tokens | `"context_limit": 200` | + +### Google +| 模型 | 官方文档 | 配置值 | +|------|---------|-------| +| Gemini 2.0 Flash | 1,000,000 tokens | `"context_limit": 1000` | +| Gemini 1.5 Pro | 2,800,000 tokens | `"context_limit": 2800` | +| Gemini 1.5 Flash | 2,800,000 tokens | `"context_limit": 2800` | + +### DeepSeek +| 模型 | 官方文档 | 配置值 | +|------|---------|-------| +| DeepSeek Chat | 128,000 tokens | `"context_limit": 128` | +| DeepSeek Coder | 128,000 tokens | `"context_limit": 128` | + +### xAI +| 模型 | 官方文档 | 配置值 | +|------|---------|-------| +| Grok Beta | 128,000 tokens | `"context_limit": 128` | + +--- + +## 问题背景 + +### 原始问题 +- 消息历史无限制累积,使用 `operator.add` 追加所有消息 +- 不同模型的上下文窗口限制不同 +- 当消息历史超过模型限制时,LLM 调用会失败 +- 没有优雅的降级或裁剪机制 + +### 解决方案 +使用 LangChain/LangGraph 的内置功能实现智能上下文管理: +- **`trim_messages`**: LangChain 的消息裁剪工具 +- **`count_tokens_approximately`**: Token 计数功能 +- **用户配置**: 必须提供 context_limit(单位:K tokens),无默认值 + +## 实现架构 + +### 1. 核心模块 + +**文件位置**: `gns3server/agent/gns3_copilot/agent/context_manager.py` + +#### 关键函数 + +**`get_model_context_limit(model_name: str, llm_config: dict) -> int`** +- 从数据库配置获取模型的上下文窗口大小 +- **`llm_config` 必须包含 `context_limit` 字段(单位:K tokens)** +- 如果未提供或无效,抛出 `ValueError` +- 返回值单位为实际 tokens(K tokens × 1000) + +**`calculate_max_tokens(model_limit, strategy) -> int`** +- 计算可用 token 数量(预留输出空间) +- 三种策略: + - `conservative`: 使用 60% 限制(更安全) + - `balanced`: 使用 75% 限制(默认) + - `aggressive`: 使用 85% 限制(最大化输入) + +**`trim_messages_for_context(messages, model_name, strategy) -> list`** +- 使用 LangChain 的 `trim_messages` 裁剪消息 +- 保留最近的消息 +- 始终保留系统消息 + +**`prepare_context_messages(state_messages, system_prompt, topology_context, model_name, trim_strategy) -> list`** +- GNS3 Copilot 的主要入口点 +- 构建完整上下文(系统提示 + 拓扑信息 + 消息历史) +- 自动裁剪以适应模型限制 + +### 2. 集成到 GNS3 Copilot + +**修改文件**: `gns3server/agent/gns3_copilot/agent/gns3_copilot.py` + +#### 改动内容 + +**导入新模块**: +```python +from gns3server.agent.gns3_copilot.agent.context_manager import ( + prepare_context_messages, +) +``` + +**替换消息构建逻辑**: +```python +# 旧代码(手动构建) +full_messages = ( + [SystemMessage(content=current_prompt)] + + context_messages + + state["messages"] +) + +# 新代码(自动裁剪) +full_messages = prepare_context_messages( + state_messages=state["messages"], + system_prompt=current_prompt, + topology_context=topology_context, + model_name=llm_config.get("model", "default"), + llm_config=llm_config, # ← 包含 context_limit 和 context_strategy +) +``` + +### 3. LangChain/LangGraph 的内置功能 + +#### `trim_messages` 参数说明 + +```python +from langchain_core.messages.utils import trim_messages + +trimmed = trim_messages( + messages, + strategy="last", # 保留最近的 N 条消息 + max_tokens=10000, # 最大 token 数 + token_counter=count_tokens_approximately, # Token 计数器 + preserve_system=True, # 保留系统消息 + start_on="human", # 确保从人类消息开始 + end_on=("human", "tool", "ai"), # 在特定类型消息结束 +) +``` + +#### `count_tokens_approximately` + +- 快速估算消息的 token 数量 +- 不需要调用 LLM API +- 用于判断是否需要裁剪 + +--- + +## 策略详细说明 + +### Conservative(保守策略) + +**参数配置**: +```json +{ + "context_limit": 128, + "context_strategy": "conservative" +} +``` + +**实际效果**: +- **使用比例**:60% (context_limit × 0.6) +- **计算公式**:`max_tokens = context_limit × 1000 × 0.6` +- **示例**:对于 128K 上下文限制 + - 保留空间:`128 × 1000 × 0.6 = 76,800` tokens 用于输入 + - 预留空间:`128,000 - 76,800 = 51,200` tokens 用于输出 + +**适用场景**: +1. **长输出任务** + - 代码生成(可能生成数百行代码) + - 文章写作(需要完整文章内容) + - 详细报告生成(包含多个章节) + +2. **复杂任务** + - 多步骤推理任务 + - 需要深度分析的请求 + - 综合性问题的解决方案 + +3. **不确定输出大小时** + - 不确定 LLM 会返回多长的内容 + - 首次尝试某种类型的任务 + - 需要额外安全边界的场景 + +**优缺点**: +- ✅ **优点**:输出更不容易被截断,安全性高 +- ❌ **缺点**:输入上下文较少,可能遗漏早期信息 + +**日志示例**: +``` +INFO: Context prepared: 20 msgs, ~72800 tokens / 128K limit (56.9%), strategy=conservative +INFO: Available for output: ~51200 tokens +``` + +--- + +### Balanced(平衡策略,推荐) + +**参数配置**: +```json +{ + "context_limit": 128, + "context_strategy": "balanced" +} +``` + +**实际效果**: +- **使用比例**:75% (context_limit × 0.75) +- **计算公式**:`max_tokens = context_limit × 1000 × 0.75` +- **示例**:对于 128K 上下文限制 + - 保留空间:`128 × 1000 × 0.75 = 96,000` tokens 用于输入 + - 预留空间:`128,000 - 96,000 = 32,000` tokens 用于输出 + +**适用场景**: +1. **一般对话** + - 日常聊天交互 + - 问答式对话 + - 技术支持和咨询 + +2. **大多数场景** + - 网络配置命令生成 + - 故障排查建议 + - 知识问答 + +3. **平衡输入和输出** + - 需要较多上下文,但输出也较长的场景 + - 中等复杂度的任务 + +**优缺点**: +- ✅ **优点**:在输入上下文和输出空间之间取得良好平衡 +- ✅ **优点**:适用于大多数使用场景 +- ⚠️ **注意**:对于特别长的输出可能被截断 + +**日志示例**: +``` +INFO: Context prepared: 30 msgs, ~85600 tokens / 128K limit (66.9%), strategy=balanced +INFO: Available for output: ~32000 tokens +``` + +--- + +### Aggressive(激进策略) + +**参数配置**: +```json +{ + "context_limit": 128, + "context_strategy": "aggressive" +} +``` + +**实际效果**: +- **使用比例**:85% (context_limit × 0.85) +- **计算公式**:`max_tokens = context_limit × 1000 × 0.85` +- **示例**:对于 128K 上下文限制 + - 保留空间:`128 × 1000 × 0.85 = 108,800` tokens 用于输入 + - 预留空间:`128,000 - 108,800 = 19,200` tokens 用于输出 + +**适用场景**: +1. **简短输出任务** + - 是/否判断 + - 简短回答确认 + - 状态查询类请求 + +2. **分析类任务** + - 日志分析(输出分析结果,但不需要很长) + - 数据解读(输出简洁的结论) + - 配置检查(返回 OK 或简短说明) + +3. **输出比较确定的场景** + - 明确知道输出会很短 + - 只需要简单确认或状态 + - 不需要长篇解释的任务 + +**优缺点**: +- ✅ **优点**:最大化输入上下文,保留更多历史信息 +- ❌ **缺点**:输出容易被截断,不适用于长输出任务 + +**日志示例**: +``` +INFO: Context prepared: 50 msgs, ~105200 tokens / 128K limit (82.2%), strategy=aggressive +INFO: Available for output: ~19200 tokens +``` + +--- + +## 策略对比总结 + +| 特性 | Conservative | Balanced | Aggressive | +|------|-------------|----------|------------| +| **输入比例** | 60% | 75% | 85% | +| **输出预留** | 40% | 25% | 15% | +| **上下文数量** | 最少 | 中等 | 最多 | +| **输出空间** | 最大 | 中等 | 最小 | +| **适用性** | 长输出 | 一般使用 | 短输出 | +| **风险** | 上下文不足 | 平衡 | 输出截断 | + +**选择建议流程**: +``` +1. 任务类型是什么? + ├─ 代码生成/长文档 → Conservative + ├─ 日常对话/一般任务 → Balanced (推荐) + └─ 简短确认/分析 → Aggressive + +2. 输出长度预估? + ├─ 不确定/可能很长 → Conservative + ├─ 中等长度 → Balanced + └─ 很短/简洁 → Aggressive + +3. 上下文重要性? + ├─ 早期历史不太重要 → Aggressive (更多上下文) + ├─ 需要平衡 → Balanced + └─ 重点是输出完整性 → Conservative (更多输出空间) +``` + +## 工作流程 + +``` +用户发送消息 + ↓ +获取消息历史 state["messages"] + ↓ +构建系统提示 + 拓扑信息 + ↓ +调用 prepare_context_messages() + ↓ + ├─ 估算 token 数量 + ├─ 获取模型上下文限制 + ├─ 判断是否需要裁剪 + └─ 如果需要 → 调用 trim_messages() + ↓ +调用 LLM(带裁剪后的消息) + ↓ +返回响应 +``` + +## 日志输出示例 + +### 正常情况 +``` +INFO: Using database config context limit: 128000 tokens for model 'gpt-4o' +INFO: Context prepared: 15 msgs, ~8432 tokens / 128000 limit (6.6%), strategy=balanced +INFO: LLM call completed: tool_calls=2 +``` + +### 发生裁剪时 +``` +INFO: Using database config context limit: 128000 tokens for model 'gpt-4o' +INFO: Trimming messages: 18500 → 9600 tokens (model: gpt-4o) +INFO: Trimmed 50 → 25 messages +INFO: Context prepared: 27 msgs, ~9432 tokens / 128000 limit (7.4%), strategy=balanced +``` + +### 配置错误时 +``` +ERROR: context_limit is required but not provided for model 'gpt-4o'. + Please configure context_limit in your LLM model configuration. + Refer to the model provider's documentation for the current context window size. +``` + +## 错误处理 + +### Token 计数失败 +```python +try: + tokens = count_tokens_approximately(messages) +except Exception as e: + logger.warning("Failed to count tokens: %s", e) + # 降级到简单的消息数量限制 + return messages[-50:] +``` + +### 裁剪失败 +```python +try: + trimmed = trim_messages(...) +except Exception as e: + logger.error("Failed to trim: %s", e) + # 回退到简单的切片操作 + return system_msgs + other_msgs[-N:] +``` + +## 参考资源 + +- [LangChain Messages Utils](https://python.langchain.com/docs/messages/) +- [LangGraph Memory Management](https://langchain-ai.github.io/langgraph/concepts/agentic_concepts/#memory) +- [OpenAI Models Context Limits](https://platform.openai.com/docs/models) +- [Anthropic Models Context Limits](https://docs.anthropic.com/claude/docs/models-overview) + +## 相关源文件 + +- `gns3server/agent/gns3_copilot/agent/context_manager.py` - 上下文管理核心逻辑 +- `gns3server/agent/gns3_copilot/agent/gns3_copilot.py` - LLM 调用节点 +- `gns3server/agent/gns3_copilot/agent_service.py` - Agent 服务接口 + +## 总结 + +通过使用 LangChain/LangGraph 的内置功能,我们实现了: +1. ✅ 智能裁剪消息历史,避免超限 +2. ✅ **必须手动配置 context_limit**,确保使用正确的上下文窗口大小 +3. ✅ 可配置的裁剪策略(保守/平衡/激进) +4. ✅ 始终保留系统消息和拓扑信息 +5. ✅ 详细的日志输出,便于调试 +6. ✅ 优雅的错误处理和明确的错误提示 +7. ✅ 提供参考工具,帮助查找常见模型的上下文限制 + +### 关键优势 + +- **准确性**:用户从官方文档获取最新的上下文限制,避免使用过时数据 +- **灵活性**:每个配置独立设置,支持不同用户使用不同限制 +- **明确性**:缺少配置时立即报错,避免静默失败 +- **可维护性**:无需维护内置默认值,减少代码维护负担 +- **可观测性**:详细日志显示上下文使用情况和裁剪决策 + +这个实现确保了即使在进行长对话时,系统也不会因为上下文溢出而失败。 diff --git a/docs/gns3-copilot/llm-model-configs-api.md b/docs/gns3-copilot/llm-model-configs-api.md index 5a867e4b2..a94b513f4 100644 --- a/docs/gns3-copilot/llm-model-configs-api.md +++ b/docs/gns3-copilot/llm-model-configs-api.md @@ -115,6 +115,7 @@ The `model_type` field accepts the following values: | `base_url` | string | API base URL | | `model` | string | Model name | | `temperature` | float | Temperature (0.0-2.0, default: 0.7) | +| `context_limit` | integer | **Model context window limit in K tokens** (e.g., 128 = 128K = 128,000 tokens) | **Optional Fields:** @@ -122,8 +123,17 @@ The `model_type` field accepts the following values: |-------|------|-------------| | `api_key` | string | API key (auto-encrypted) | | `max_tokens` | integer | Max tokens for generation | +| `context_strategy` | string | Context trimming strategy: "conservative" (60%), "balanced" (75%), "aggressive" (85%). Default: "balanced" | | `is_default` | boolean | Set as default (default: false) | +**Important Notes:** + +- **`context_limit` is required**: You must specify the model's context window limit. Refer to the model provider's official documentation for the current value. +- **Unit is K tokens**: The value is in thousands of tokens (1 K = 1,000 tokens). For example: + - GPT-4o: 128,000 tokens → configure as `"context_limit": 128` + - Claude 3.5 Sonnet: 200,000 tokens → configure as `"context_limit": 200` + - Gemini 1.5 Pro: 2,800,000 tokens → configure as `"context_limit": 2800` + **Extra Fields:** Any custom fields are supported for future extensibility. ### LLMModelConfigUpdate @@ -138,6 +148,8 @@ The `model_type` field accepts the following values: | `temperature` | float (optional) | Temperature | | `api_key` | string (optional) | API key | | `max_tokens` | integer (optional) | Max tokens | +| `context_limit` | integer (optional) | Model context window limit in K tokens | +| `context_strategy` | string (optional) | Context trimming strategy | | `is_default` | boolean (optional) | Default flag | | `expected_version` | integer (optional) | **Optimistic locking version** | @@ -213,28 +225,36 @@ curl -X POST http://localhost:3080/v3/access/users/{user_id}/llm-model-configs \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{ - "name": "GPT-4", + "name": "GPT-4o", "model_type": "text", "provider": "openai", "base_url": "https://api.openai.com/v1", - "model": "gpt-4", + "model": "gpt-4o", "temperature": 0.7, + "context_limit": 128, + "context_strategy": "balanced", "api_key": "sk-xxx", "is_default": true }' ``` +**Important:** +- `context_limit` is **required** and specified in K tokens (e.g., 128 = 128K = 128,000 tokens) +- Refer to the model provider's official documentation for the current context window size + **Response:** ```json { "config_id": "uuid-1", - "name": "GPT-4", + "name": "GPT-4o", "model_type": "text", "config": { "provider": "openai", "base_url": "https://api.openai.com/v1", - "model": "gpt-4", + "model": "gpt-4o", "temperature": 0.7, + "context_limit": 128, + "context_strategy": "balanced", "api_key": "sk-xxx" }, "user_id": "uuid-user", @@ -253,12 +273,14 @@ curl -X POST http://localhost:3080/v3/access/groups/{group_id}/llm-model-configs -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{ - "name": "Claude-3", + "name": "Claude-3.5 Sonnet", "model_type": "text", "provider": "anthropic", "base_url": "https://api.anthropic.com", - "model": "claude-3-opus-20240229", + "model": "claude-3-5-sonnet-20241022", "temperature": 0.7, + "context_limit": 200, + "context_strategy": "balanced", "api_key": "sk-ant-xxx", "is_default": true }' @@ -277,13 +299,15 @@ curl -X GET http://localhost:3080/v3/access/users/{user_id}/llm-model-configs \ "configs": [ { "config_id": "uuid-1", - "name": "GPT-4", + "name": "GPT-4o", "model_type": "text", "config": { "provider": "openai", "base_url": "https://api.openai.com/v1", - "model": "gpt-4", + "model": "gpt-4o", "temperature": 0.7, + "context_limit": 128, + "context_strategy": "balanced", "api_key": "sk-xxx" }, "user_id": "uuid-user", @@ -297,13 +321,15 @@ curl -X GET http://localhost:3080/v3/access/users/{user_id}/llm-model-configs \ }, { "config_id": "uuid-2", - "name": "Claude-3", + "name": "Claude-3.5 Sonnet", "model_type": "text", "config": { "provider": "anthropic", "base_url": "https://api.anthropic.com", - "model": "claude-3-opus-20240229", + "model": "claude-3-5-sonnet-20241022", "temperature": 0.7, + "context_limit": 200, + "context_strategy": "balanced", "api_key": null }, "user_id": null, @@ -318,11 +344,16 @@ curl -X GET http://localhost:3080/v3/access/users/{user_id}/llm-model-configs \ ], "default_config": { "config_id": "uuid-1", - "name": "GPT-4", + "name": "GPT-4o", "model_type": "text", "config": { "provider": "openai", - ... + "base_url": "https://api.openai.com/v1", + "model": "gpt-4o", + "temperature": 0.7, + "context_limit": 128, + "context_strategy": "balanced", + "api_key": "sk-xxx" }, "user_id": "uuid-user", "group_id": null, @@ -340,6 +371,7 @@ curl -X GET http://localhost:3080/v3/access/users/{user_id}/llm-model-configs \ - `source: "user"` indicates the config belongs to the user - `source: "group"` indicates the config is inherited from a group - Configuration fields are nested in the `config` object (same structure as group endpoints) +- `context_limit` is in K tokens (128 = 128K = 128,000 tokens) ### 4. Get group configurations @@ -417,7 +449,8 @@ curl -X PUT http://localhost:3080/v3/access/users/{user_id}/llm-model-configs/{c -H "Content-Type: application/json" \ -d '{ "temperature": 0.9, - "max_tokens": 4000 + "max_tokens": 4000, + "context_strategy": "aggressive" }' ``` @@ -482,13 +515,15 @@ curl -X GET http://localhost:3080/v3/access/users/{user_id}/llm-model-configs/de ```json { "config_id": "uuid-1", - "name": "GPT-4", + "name": "GPT-4o", "model_type": "text", "config": { "provider": "openai", "base_url": "https://api.openai.com/v1", - "model": "gpt-4", + "model": "gpt-4o", "temperature": 0.7, + "context_limit": 128, + "context_strategy": "balanced", "api_key": "sk-xxx" }, "user_id": "uuid-user", @@ -723,3 +758,82 @@ The `llm_model_configs` table includes three reserved JSONB fields for future ex - Caching computed values **Note:** These fields are not exposed in the current API schemas and are reserved for internal use. + +--- + +## Context Limit Configuration + +### What is `context_limit`? + +The `context_limit` field specifies the maximum context window size for an LLM model. This is a **required field** for all model configurations. + +### Why is it required? + +Model providers frequently update their models and change context window sizes: +- OpenAI GPT-4o: 128K tokens (may change) +- Anthropic Claude 3.5: 200K tokens (may change) +- Google Gemini 1.5: 2.8M tokens (may change) + +Hardcoding these values in the system would quickly become outdated. Requiring users to configure this field ensures that the system always uses the correct, up-to-date values. + +### Unit: K tokens + +The `context_limit` value is specified in **K tokens** (thousands of tokens) to make it easier to read and write: + +| Official Documentation | API Configuration | +|---------------------|-------------------| +| 128,000 tokens | `"context_limit": 128` | +| 200,000 tokens | `"context_limit": 200` | +| 2,800,000 tokens | `"context_limit": 2800` | + +### How to find the correct value + +1. **Check the official documentation** for your model: + - OpenAI: https://platform.openai.com/docs/models + - Anthropic: https://docs.anthropic.com/claude/docs/models-overview + - Google: https://ai.google.dev/gemini-api/docs/models + - DeepSeek: https://platform.deepseek.com/api-docs/ + +2. **Convert from tokens to K**: + ``` + context_limit = official_value_in_tokens / 1000 + + Example: + GPT-4o: 128,000 tokens / 1000 = 128 + ``` + +3. **Use the reference tool**: + ```bash + python scripts/show_model_context_limits.py + ``` + +### Example: Common Models + +| Model | Official Value | Configuration | +|-------|---------------|--------------| +| GPT-4o | 128,000 | `"context_limit": 128` | +| GPT-3.5 Turbo | 16,385 | `"context_limit": 17` | +| Claude 3.5 Sonnet | 200,000 | `"context_limit": 200` | +| Gemini 1.5 Pro | 2,800,000 | `"context_limit": 2800` | +| DeepSeek Chat | 128,000 | `"context_limit": 128` | + +### Context Strategy + +The optional `context_strategy` field controls how aggressively the system uses the available context window: + +| Strategy | Usage | Best For | +|----------|-------|----------| +| `conservative` | 60% of limit | Long outputs, complex tasks, uncertain output size | +| `balanced` (default) | 75% of limit | Most conversations, general use | +| `aggressive` | 85% of limit | Short outputs, analysis tasks, predictable output size | + +### Error Handling + +If `context_limit` is missing or invalid, the API will return: + +```json +HTTP 400 Bad Request +{ + "detail": "context_limit is required (unit: K tokens, e.g., 128 = 128K = 128,000 tokens). Please check your model provider's documentation for the current context window size and specify it in the configuration." +} +``` diff --git a/gns3server/agent/gns3_copilot/agent/context_manager.py b/gns3server/agent/gns3_copilot/agent/context_manager.py new file mode 100644 index 000000000..cd9f6f10a --- /dev/null +++ b/gns3server/agent/gns3_copilot/agent/context_manager.py @@ -0,0 +1,364 @@ +# SPDX-License-Identifier: AGPL-3.0-or-later +# +# Context Window Manager for GNS3-Copilot +# +# This module manages context window limits for different LLM models, +# implementing intelligent message trimming and token counting strategies. + +""" +Context Window Manager for GNS3-Copilot + +This module provides context window management for different LLM models, +including: +- Model-specific context window limits +- Token counting for messages +- Message trimming strategies +- System message preservation +""" + +import logging +from typing import Any, Literal + +from langchain_core.messages import ( + AIMessage, + HumanMessage, + SystemMessage, + ToolMessage, +) +from langchain_core.messages.utils import trim_messages + +logger = logging.getLogger(__name__) + + +# NOTE: Built-in model context limits have been removed. +# Model providers frequently update context limits, and maintaining this list is not sustainable. +# Users MUST configure context_limit explicitly in their LLM model configurations. +# +# For reference, common model context limits as of 2025: +# - OpenAI GPT-4o: 128K tokens +# - OpenAI GPT-4 Turbo: 128K tokens +# - OpenAI GPT-3.5 Turbo: 16K tokens +# - Anthropic Claude 3.5 Sonnet: 200K tokens +# - Google Gemini 1.5 Pro: 2.8M tokens +# - DeepSeek Chat: 128K tokens +# +# Always verify current limits from official provider documentation. + + +def get_model_context_limit( + model_name: str, + llm_config: dict[str, Any] | None = None +) -> int: + """ + Get the context window limit for a given model. + + IMPORTANT: context_limit MUST be provided in llm_config. + Model providers frequently update context limits, so built-in defaults are NOT used. + Users must configure this value explicitly. + + NOTE: context_limit unit is K tokens (1 K = 1000 tokens). + Example: 128 means 128K = 128,000 tokens. + + Args: + model_name: Name of the model (e.g., "gpt-4o", "deepseek-chat") + llm_config: LLM config dict from database (must contain context_limit in K tokens) + + Returns: + int: Maximum context window size in tokens (actual number, not K) + + Raises: + ValueError: If context_limit is not provided or invalid + """ + # Check database config for context_limit + if llm_config and "context_limit" in llm_config: + db_limit_k = llm_config["context_limit"] + if isinstance(db_limit_k, int) and db_limit_k > 0: + # Convert K tokens to actual tokens + actual_tokens = db_limit_k * 1000 + logger.debug( + "Using database config context limit: %dK tokens (%d tokens) for model '%s'", + db_limit_k, actual_tokens, model_name + ) + return actual_tokens + else: + raise ValueError( + f"Invalid context_limit in database config: {db_limit_k} " + f"(type={type(db_limit_k).__name__}, expected positive integer in K tokens)" + ) + + # No context_limit provided - this is a configuration error + raise ValueError( + f"context_limit is required but not provided for model '{model_name}'. " + f"Please configure context_limit in your LLM model configuration (unit: K tokens). " + f"Example: 128 means 128K = 128,000 tokens. " + f"Refer to the model provider's documentation for the current context window size." + ) + + +def calculate_max_tokens( + model_limit: int, + strategy: Literal["conservative", "balanced", "aggressive"] = "balanced" +) -> int: + """ + Calculate the maximum tokens to use, reserving space for output. + + Args: + model_limit: Model's context window limit + strategy: How aggressively to use the context window + - "conservative": Use 60% of limit (safer, more reserved for output) + - "balanced": Use 75% of limit (default) + - "aggressive": Use 85% of limit (maximize input, minimal output reserve) + + Returns: + int: Maximum tokens for input messages + """ + ratios = { + "conservative": 0.60, + "balanced": 0.75, + "aggressive": 0.85, + } + + ratio = ratios.get(strategy, 0.75) + max_tokens = int(model_limit * ratio) + + logger.debug( + "Context limit: model=%d, strategy=%s, usable=%d tokens", + model_limit, strategy, max_tokens + ) + + return max_tokens + + +def trim_messages_for_context( + messages: list[Any], + model_name: str, + llm_config: dict[str, Any] | None = None, + strategy: Literal["conservative", "balanced", "aggressive"] = "balanced", + preserve_system: bool = True, +) -> list[Any]: + """ + Trim messages to fit within model's context window. + + This function uses LangChain's trim_messages utility to intelligently + reduce message history while preserving conversation flow. + + Args: + messages: List of LangChain messages (HumanMessage, AIMessage, etc.) + model_name: Name of the LLM model being used + llm_config: Optional LLM config dict from database (may contain context_limit) + strategy: How aggressively to use the context window + preserve_system: Whether to always preserve system messages + + Returns: + list: Trimmed list of messages that fit within context limit + + Examples: + >>> messages = [HumanMessage("Hello"), AIMessage("Hi there!")] + >>> trimmed = trim_messages_for_context(messages, "gpt-4o") + >>> len(trimmed) <= len(messages) + True + """ + if not messages: + return messages + + # Get model's context limit (from database or built-in defaults) + model_limit = get_model_context_limit(model_name, llm_config) + + # Calculate usable tokens (reserve space for output) + max_tokens = calculate_max_tokens(model_limit, strategy) + + # Check if trimming is needed + try: + from langchain_core.messages.utils import count_tokens_approximately + + # Estimate current token count + current_tokens = count_tokens_approximately(messages) + + if current_tokens <= max_tokens: + logger.debug( + "Messages fit in context: %d / %d tokens", + current_tokens, max_tokens + ) + return messages + + logger.info( + "Trimming messages: %d → %d tokens (model: %s)", + current_tokens, max_tokens, model_name + ) + + except Exception as e: + logger.warning("Failed to count tokens: %s, proceeding with trim", e) + + # Trim messages using LangChain's utility + try: + trimmed = trim_messages( + messages, + strategy="last", # Keep most recent messages + max_tokens=max_tokens, + preserve_system=preserve_system, + start_on="human", # Ensure we start with a human message + end_on=("human", "tool", "ai"), # End on human/tool/ai messages + ) + + logger.info( + "Trimmed %d → %d messages", + len(messages), len(trimmed) + ) + + return trimmed + + except Exception as e: + logger.error("Failed to trim messages: %s", e, exc_info=True) + + # Fallback: simple slicing (keep last N messages) + # Estimate average tokens per message (~100 tokens) + fallback_msg_count = max(1, max_tokens // 100) + + logger.warning( + "Using fallback trimming: keeping last %d messages", + fallback_msg_count + ) + + # Always preserve system messages + system_msgs = [m for m in messages if isinstance(m, SystemMessage)] + other_msgs = [m for m in messages if not isinstance(m, SystemMessage)] + + return system_msgs + other_msgs[-fallback_msg_count:] + + +def get_token_usage_summary( + messages: list[Any], + model_name: str, + llm_config: dict[str, Any] | None = None +) -> dict[str, Any]: + """ + Get a summary of token usage for the given messages. + + Args: + messages: List of LangChain messages + model_name: Name of the LLM model + llm_config: Optional LLM config dict from database (may contain context_limit in K tokens) + + Returns: + dict: Token usage summary including: + - estimated_tokens: Estimated total tokens + - model_limit_k: Model's context window limit in K tokens + - model_limit_tokens: Model's context window limit in actual tokens + - usage_percentage: Percentage of context used + - message_count: Number of messages + - needs_trimming: Whether messages exceed 80% of limit + """ + try: + from langchain_core.messages.utils import count_tokens_approximately + + estimated_tokens = count_tokens_approximately(messages) + except Exception as e: + logger.warning("Failed to count tokens: %s", e) + estimated_tokens = 0 + + model_limit_tokens = get_model_context_limit(model_name, llm_config) + model_limit_k = model_limit_tokens // 1000 + usage_percentage = (estimated_tokens / model_limit_tokens * 100) if model_limit_tokens > 0 else 0 + + return { + "estimated_tokens": estimated_tokens, + "model_limit_k": model_limit_k, + "model_limit_tokens": model_limit_tokens, + "usage_percentage": round(usage_percentage, 2), + "message_count": len(messages), + "needs_trimming": usage_percentage > 80, + } + + +# Convenience function for GNS3-Copilot integration +def prepare_context_messages( + state_messages: list[Any], + system_prompt: str, + topology_context: str | None, + model_name: str, + llm_config: dict[str, Any] | None = None, +) -> list[Any]: + """ + Prepare full context messages for LLM call with automatic trimming. + + This is the main entry point for GNS3-Copilot to prepare messages + before calling the LLM. + + Args: + state_messages: Message history from conversation state + system_prompt: System prompt text + topology_context: Optional topology information string + model_name: Name of the LLM model + llm_config: Optional LLM config dict from database (may contain context_limit and context_strategy) + + Returns: + list: Prepared messages ready for LLM invocation + + Examples: + >>> messages = prepare_context_messages( + ... state_messages=[HumanMessage("Help me")], + ... system_prompt="You are a helpful assistant", + ... topology_context=None, + ... model_name="gpt-4o" + ... ) + >>> len(messages) + 2 # System message + Human message + """ + # Get trimming strategy from config (default: "balanced") + trim_strategy = "balanced" + if llm_config and "context_strategy" in llm_config: + strategy = llm_config["context_strategy"] + if strategy in ["conservative", "balanced", "aggressive"]: + trim_strategy = strategy + logger.debug("Using context_strategy from config: %s", trim_strategy) + else: + logger.warning("Invalid context_strategy '%s', using 'balanced'", strategy) + + # Build base context (system + topology) + context_messages = [SystemMessage(content=system_prompt)] + + if topology_context: + context_messages.append( + SystemMessage(content=f"Current Topology:\n{topology_context}") + ) + + # Combine with conversation history + full_messages = context_messages + state_messages + + # Trim if needed + trimmed_messages = trim_messages_for_context( + full_messages, + model_name=model_name, + llm_config=llm_config, + strategy=trim_strategy, + preserve_system=True, # Always keep system prompts + ) + + # Log summary + summary = get_token_usage_summary(trimmed_messages, model_name, llm_config) + logger.info( + "Context prepared: %d msgs, ~%d tokens / %dK limit (%.1f%%), strategy=%s", + summary["message_count"], + summary["estimated_tokens"], + summary["model_limit_k"], + summary["usage_percentage"], + trim_strategy + ) + + return trimmed_messages + + +if __name__ == "__main__": + # Simple test + test_messages = [ + HumanMessage(f"Message {i}") for i in range(100) + ] + + result = trim_messages_for_context( + test_messages, + model_name="gpt-4o", + strategy="balanced" + ) + + print(f"Original: {len(test_messages)} messages") + print(f"Trimmed: {len(result)} messages") diff --git a/gns3server/agent/gns3_copilot/agent/gns3_copilot.py b/gns3server/agent/gns3_copilot/agent/gns3_copilot.py index 69fd58f04..1e821122a 100644 --- a/gns3server/agent/gns3_copilot/agent/gns3_copilot.py +++ b/gns3server/agent/gns3_copilot/agent/gns3_copilot.py @@ -48,6 +48,9 @@ from gns3server.agent.gns3_copilot.agent.model_factory import ( create_base_model_with_tools, create_title_model, ) +from gns3server.agent.gns3_copilot.agent.context_manager import ( + prepare_context_messages, +) from gns3server.agent.gns3_copilot.gns3_client import GNS3TopologyTool from gns3server.agent.gns3_copilot.prompts import TITLE_PROMPT, load_system_prompt import sys @@ -164,8 +167,8 @@ def llm_call(state: dict, config: RunnableConfig | None = None): if config and config.get("configurable"): project_id = config["configurable"].get("project_id") - # Construct context messages - context_messages = [] + # Retrieve topology information if available + topology_context = None topology_info = None if project_id: @@ -180,12 +183,8 @@ def llm_call(state: dict, config: RunnableConfig | None = None): "Successfully retrieved topology for project_id: %s, name: %s", project_id, topology.get("name") ) - # Convert topology dict to string for LLM consumption topology_context = str(topology) - context_messages.append( - SystemMessage(content=f"Current Topology:\n{topology_context}") - ) else: logger.warning( "Failed to retrieve topology for project_id %s: %s", @@ -194,11 +193,18 @@ def llm_call(state: dict, config: RunnableConfig | None = None): except Exception as e: logger.warning("Error retrieving topology for project_id %s: %s", project_id, e) - # Merge message lists - full_messages = ( - [SystemMessage(content=current_prompt)] + context_messages + state["messages"] + # Prepare context messages with automatic trimming based on model's context window + # This ensures we don't exceed the model's context limit + # llm_config may contain: + # - context_limit: Override built-in context window limit + # - context_strategy: Trimming strategy (conservative/balanced/aggressive) + full_messages = prepare_context_messages( + state_messages=state["messages"], + system_prompt=current_prompt, + topology_context=topology_context, + model_name=llm_config.get("model", "default"), + llm_config=llm_config, ) - # print(full_messages) # Create fresh model with tools for each LLM call logger.debug("Creating model with tools: provider=%s, model=%s", diff --git a/gns3server/api/routes/controller/llm_model_configs.py b/gns3server/api/routes/controller/llm_model_configs.py index 0e4dbcbd4..da36de67b 100644 --- a/gns3server/api/routes/controller/llm_model_configs.py +++ b/gns3server/api/routes/controller/llm_model_configs.py @@ -182,6 +182,9 @@ async def create_user_llm_model_config( Create a new LLM model configuration for a user. Required privilege: User.Modify + + IMPORTANT: context_limit is REQUIRED. Unit is K tokens (e.g., 128 = 128K = 128,000 tokens). + Please check your model provider's documentation for the current context window size. """ # Verify user exists @@ -189,6 +192,14 @@ async def create_user_llm_model_config( if not user: raise ControllerNotFoundError(f"User '{user_id}' not found") + # Validate context_limit is provided + if not hasattr(config_create, 'context_limit') or config_create.context_limit is None: + raise ControllerBadRequestError( + "context_limit is required (unit: K tokens, e.g., 128 = 128K = 128,000 tokens). " + "Please check your model provider's documentation for the current context window size " + "and specify it in the configuration." + ) + try: # Extract config fields (excluding table-level fields) config_fields = config_create.model_dump(exclude={"name", "model_type", "is_default"}) @@ -493,6 +504,9 @@ async def create_group_llm_model_config( Create a new LLM model configuration for a user group. Required privilege: Group.Modify + + IMPORTANT: context_limit is REQUIRED. Unit is K tokens (e.g., 128 = 128K = 128,000 tokens). + Please check your model provider's documentation for the current context window size. """ # Verify group exists @@ -500,6 +514,14 @@ async def create_group_llm_model_config( if not group: raise ControllerNotFoundError(f"User group '{group_id}' not found") + # Validate context_limit is provided + if not hasattr(config_create, 'context_limit') or config_create.context_limit is None: + raise ControllerBadRequestError( + "context_limit is required (unit: K tokens, e.g., 128 = 128K = 128,000 tokens). " + "Please check your model provider's documentation for the current context window size " + "and specify it in the configuration." + ) + try: # Extract config fields (excluding table-level fields) config_fields = config_create.model_dump(exclude={"name", "model_type", "is_default"}) diff --git a/gns3server/schemas/controller/llm_model_configs.py b/gns3server/schemas/controller/llm_model_configs.py index 9b70f3ec6..7e4961e8b 100644 --- a/gns3server/schemas/controller/llm_model_configs.py +++ b/gns3server/schemas/controller/llm_model_configs.py @@ -30,6 +30,12 @@ class LLMModelConfigData(BaseModel): """ LLM model configuration data. Stored in the config JSONB column (provider, base_url, model, etc.). + + IMPORTANT: context_limit is REQUIRED to ensure proper context window management. + Model providers frequently update context limits, so users must configure this value. + + NOTE: context_limit unit is K tokens (1 K = 1000 tokens). + Example: 128 means 128K tokens (128,000 tokens). """ provider: str = Field(..., description="LLM provider (e.g., 'openai', 'anthropic', 'ollama')") @@ -38,6 +44,10 @@ class LLMModelConfigData(BaseModel): temperature: float = Field(default=0.7, ge=0.0, le=2.0, description="Temperature parameter") api_key: Optional[str] = Field(None, description="API key (will be encrypted)") max_tokens: Optional[int] = Field(None, gt=0, description="Max tokens for generation") + context_limit: int = Field(..., gt=0, description="Model context window limit in K tokens (REQUIRED, e.g., 128 = 128K = 128,000 tokens)") + context_strategy: Literal["conservative", "balanced", "aggressive"] = Field( + "balanced", description="Context trimming strategy: conservative (60%), balanced (75%), aggressive (85%)" + ) # Allow extra fields for extensibility model_config = ConfigDict(extra="allow") @@ -57,6 +67,10 @@ class LLMModelConfigCreate(BaseModel): temperature: float = Field(default=0.7, ge=0.0, le=2.0) api_key: Optional[str] = None max_tokens: Optional[int] = Field(None, gt=0) + context_limit: int = Field(..., gt=0, description="Model context window limit in K tokens (REQUIRED, e.g., 128 = 128K tokens)") + context_strategy: Literal["conservative", "balanced", "aggressive"] = Field( + "balanced", description="Context trimming strategy" + ) # Allow extra config fields model_config = ConfigDict(extra="allow") @@ -78,6 +92,10 @@ class LLMModelConfigUpdate(BaseModel): temperature: Optional[float] = Field(None, ge=0.0, le=2.0) api_key: Optional[str] = None max_tokens: Optional[int] = Field(None, gt=0) + context_limit: Optional[int] = Field(None, gt=0, description="Model context window limit in K tokens (e.g., 128 = 128K tokens)") + context_strategy: Optional[Literal["conservative", "balanced", "aggressive"]] = Field( + None, description="Context trimming strategy" + ) # Allow extra config fields model_config = ConfigDict(extra="allow") diff --git a/scripts/show_model_context_limits.py b/scripts/show_model_context_limits.py new file mode 100644 index 000000000..f07475e4a --- /dev/null +++ b/scripts/show_model_context_limits.py @@ -0,0 +1,157 @@ +#!/usr/bin/env python3 +""" +Reference tool for LLM model context limits. + +This script provides reference context limit values for common LLM models. +It helps users find the correct context_limit value when creating LLM model configurations. + +IMPORTANT: +- context_limit unit is K tokens (1 K = 1000 tokens) +- This tool only displays reference values. You MUST manually configure + context_limit when creating or updating LLM model configurations via API. + +Usage: + python scripts/show_model_context_limits.py + +For official documentation, always check: +- OpenAI: https://platform.openai.com/docs/models +- Anthropic: https://docs.anthropic.com/claude/docs/models-overview +- Google: https://ai.google.dev/gemini-api/docs/models +- DeepSeek: https://platform.deepseek.com/api-docs/ +""" + +import sys +from pathlib import Path + +# Add parent directory to path +sys.path.insert(0, str(Path(__file__).parent.parent)) + + +# Reference context limits (as of 2025) +# Displayed in K tokens for easier configuration +# Users should verify from official provider documentation +MODEL_CONTEXT_LIMITS_K = { + # OpenAI Models + "gpt-4o": 128, + "gpt-4o-mini": 128, + "gpt-4-turbo": 128, + "gpt-4": 8, + "gpt-4-32k": 33, + "gpt-3.5-turbo": 17, + "gpt-3.5-turbo-16k": 17, + + # Anthropic Models + "claude-3-5-sonnet-20241022": 200, + "claude-3-5-sonnet-20240620": 200, + "claude-3-opus-20240229": 200, + "claude-3-sonnet-20240229": 200, + "claude-3-haiku-20240307": 200, + + # Google Models + "gemini-2.0-flash-exp": 1000, + "gemini-1.5-pro": 2800, + "gemini-1.5-flash": 2800, + "gemini-pro": 92, + + # DeepSeek Models + "deepseek-chat": 128, + "deepseek-coder": 128, + + # xAI Models + "grok-beta": 128, +} + + +def find_context_limit_for_model(model_name: str) -> int | None: + """Find the context limit for a given model name (in K tokens).""" + model_lower = model_name.lower().strip() + + # Try exact match + if model_lower in MODEL_CONTEXT_LIMITS_K: + return MODEL_CONTEXT_LIMITS_K[model_lower] + + # Try prefix match + for key, limit in MODEL_CONTEXT_LIMITS_K.items(): + if model_lower.startswith(key.lower()): + return limit + + return None + + +def main(): + print("=" * 70) + print("LLM Model Context Limits Reference Tool") + print("=" * 70) + print() + print("This tool displays reference context limit values for common LLM models.") + print("Please verify from official provider documentation before configuring.") + print() + print("IMPORTANT: context_limit unit is K tokens (1 K = 1,000 tokens)") + print() + print("Official Documentation:") + print(" - OpenAI: https://platform.openai.com/docs/models") + print(" - Anthropic: https://docs.anthropic.com/claude/docs/models-overview") + print(" - Google: https://ai.google.dev/gemini-api/docs/models") + print(" - DeepSeek: https://platform.deepseek.com/api-docs/") + print() + + # Display all reference values + print("=" * 70) + print("Reference Context Limits (in K tokens)") + print("=" * 70) + print() + + # Group by provider + providers = { + "OpenAI": ["gpt-4o", "gpt-4o-mini", "gpt-4-turbo", "gpt-4", "gpt-4-32k", "gpt-3.5-turbo", "gpt-3.5-turbo-16k"], + "Anthropic": ["claude-3-5-sonnet-20241022", "claude-3-5-sonnet-20240620", "claude-3-opus-20240229", "claude-3-sonnet-20240229", "claude-3-haiku-20240307"], + "Google": ["gemini-2.0-flash-exp", "gemini-1.5-pro", "gemini-1.5-flash", "gemini-pro"], + "DeepSeek": ["deepseek-chat", "deepseek-coder"], + "xAI": ["grok-beta"], + } + + for provider, models in providers.items(): + print(f"\n{provider}:") + for model in models: + if model in MODEL_CONTEXT_LIMITS_K: + limit_k = MODEL_CONTEXT_LIMITS_K[model] + limit_actual = limit_k * 1000 + print(f" {model:40s} → {limit_k:4d}K (= {limit_actual:,} tokens)") + + print() + print("=" * 70) + print() + print("Conversion Examples:") + print() + print(" Official Documentation: 128,000 tokens") + print(" ↓") + print(" API Configuration: \"context_limit\": 128") + print() + print(" Official Documentation: 200,000 tokens") + print(" ↓") + print(" API Configuration: \"context_limit\": 200") + print() + print(" Official Documentation: 2,800,000 tokens") + print(" ↓") + print(" API Configuration: \"context_limit\": 2800") + print() + print("=" * 70) + print() + print("Usage Example:") + print() + print("When creating a model configuration, specify context_limit in K:") + print() + print(' POST /v3/users/{user_id}/llm-model-configs') + print(' {') + print(' "name": "GPT-4o Configuration",') + print(' "provider": "openai",') + print(' "model": "gpt-4o",') + print(' "context_limit": 128, // ← Required: 128K = 128,000 tokens') + print(' "context_strategy": "balanced"') + print(' }') + print() + print("=" * 70) + + +if __name__ == "__main__": + main()