预测输出

属性	详情
描述	当 LLM 的大部分输出已知时使用此功能。例如，如果您要求模型重写一些只有微小改动的文本或代码，您可以通过使用预测输出（传递现有内容作为您的预测）显著降低延迟。
支持的提供商	openai
OpenAI 关于预测输出的文档链接	预测输出 ↗
LiteLLM 支持版本	v1.51.4

使用预测输出

LiteLLM Python SDK

在此示例中，我们希望重构一段 C# 代码，并将 Username 属性改为 Email

import litellm
os.environ["OPENAI_API_KEY"] = "your-api-key"
code = """
/// <summary>
/// Represents a user with a first name, last name, and username.
/// </summary>
public class User
{
    /// <summary>
    /// Gets or sets the user's first name.
    /// </summary>
    public string FirstName { get; set; }

    /// <summary>
    /// Gets or sets the user's last name.
    /// </summary>
    public string LastName { get; set; }

    /// <summary>
    /// Gets or sets the user's username.
    /// </summary>
    public string Username { get; set; }
}
"""

completion = litellm.completion(
    model="gpt-4o-mini",
    messages=[
        {
            "role": "user",
            "content": "Replace the Username property with an Email property. Respond only with code, and with no markdown formatting.",
        },
        {"role": "user", "content": code},
    ],
    prediction={"type": "content", "content": code},
)

print(completion)

LiteLLM 代理服务器

1. 在 config.yaml 中定义模型

model_list:
  - model_name: gpt-4o-mini # OpenAI gpt-4o-mini
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY 

2. 运行代理服务器

litellm --config config.yaml

3. 使用 OpenAI Python SDK 进行测试

from openai import OpenAI

client = OpenAI(
    api_key="LITELLM_PROXY_KEY", # sk-1234
    base_url="LITELLM_PROXY_BASE" # http://0.0.0.0:4000
)

completion = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {
            "role": "user",
            "content": "Replace the Username property with an Email property. Respond only with code, and with no markdown formatting.",
        },
        {"role": "user", "content": code},
    ],
    prediction={"type": "content", "content": code},
)

print(completion)