跳到主要内容

LiteLLM - 入门指南

https://github.com/BerriAI/litellm

使用 OpenAI 输入/输出格式调用 100+ 个 LLM​

  • 将输入转换为提供商的 completion、embedding 和 image_generation 端点格式
  • 输出一致，文本响应始终可在 ['choices'][0]['message']['content'] 获取
  • 跨多个部署（例如 Azure/OpenAI）的重试/回退逻辑 - 路由器
  • 跟踪每个项目的花费并设置预算 LiteLLM 代理服务器

如何使用 LiteLLM​

您可以通过以下方式使用 litellm：

  1. LiteLLM 代理服务器 - 服务器（LLM 网关），用于调用 100+ 个 LLM，实现负载均衡和跨项目成本跟踪
  2. LiteLLM python SDK - Python 客户端，用于调用 100+ 个 LLM，实现负载均衡和成本跟踪

何时使用 LiteLLM 代理服务器（LLM 网关）​

提示

如果您想要一个中央服务（LLM 网关）来访问多个 LLM，请使用 LiteLLM 代理服务器。

通常由 Gen AI 赋能 / ML 平台团队使用

  • LiteLLM 代理提供统一接口，用于访问多个 LLM (100+ 个 LLM)
  • 跟踪 LLM 使用情况并设置防护措施
  • 为每个项目自定义日志记录、防护措施和缓存

何时使用 LiteLLM Python SDK​

提示

如果您想在您的 python 代码中使用 LiteLLM，请使用 LiteLLM Python SDK。

通常由开发 LLM 项目的开发者使用

  • LiteLLM SDK 提供统一接口，用于访问多个 LLM (100+ 个 LLM)
  • 跨多个部署（例如 Azure/OpenAI）的重试/回退逻辑 - 路由器

LiteLLM Python SDK​

基本用法​

Open In Colab
pip install litellm
from litellm import completion
import os

## set ENV variables
os.environ["OPENAI_API_KEY"] = "your-api-key"

response = completion(
  model="openai/gpt-4o",
  messages=[{ "content": "Hello, how are you?","role": "user"}]
)

响应格式 (OpenAI 格式)​

{
    "id": "chatcmpl-565d891b-a42e-4c39-8d14-82a1f5208885",
    "created": 1734366691,
    "model": "claude-3-sonnet-20240229",
    "object": "chat.completion",
    "system_fingerprint": null,
    "choices": [
        {
            "finish_reason": "stop",
            "index": 0,
            "message": {
                "content": "Hello! As an AI language model, I don't have feelings, but I'm operating properly and ready to assist you with any questions or tasks you may have. How can I help you today?",
                "role": "assistant",
                "tool_calls": null,
                "function_call": null
            }
        }
    ],
    "usage": {
        "completion_tokens": 43,
        "prompt_tokens": 13,
        "total_tokens": 56,
        "completion_tokens_details": null,
        "prompt_tokens_details": {
            "audio_tokens": null,
            "cached_tokens": 0
        },
        "cache_creation_input_tokens": 0,
        "cache_read_input_tokens": 0
    }
}

流式传输​

在 completion 参数中设置 stream=True。

from litellm import completion
import os

## set ENV variables
os.environ["OPENAI_API_KEY"] = "your-api-key"

response = completion(
  model="openai/gpt-4o",
  messages=[{ "content": "Hello, how are you?","role": "user"}],
  stream=True,
)

流式传输响应格式 (OpenAI 格式)​

{
    "id": "chatcmpl-2be06597-eb60-4c70-9ec5-8cd2ab1b4697",
    "created": 1734366925,
    "model": "claude-3-sonnet-20240229",
    "object": "chat.completion.chunk",
    "system_fingerprint": null,
    "choices": [
        {
            "finish_reason": null,
            "index": 0,
            "delta": {
                "content": "Hello",
                "role": "assistant",
                "function_call": null,
                "tool_calls": null,
                "audio": null
            },
            "logprobs": null
        }
    ]
}

异常处理​

LiteLLM 将所有支持的提供商的异常映射到 OpenAI 异常。我们所有的异常都继承自 OpenAI 的异常类型，因此您现有的任何错误处理机制都应该能直接与 LiteLLM 一起工作。

from openai.error import OpenAIError
from litellm import completion

os.environ["ANTHROPIC_API_KEY"] = "bad-key"
try:
    # some code
    completion(model="claude-instant-1", messages=[{"role": "user", "content": "Hey, how's it going?"}])
except OpenAIError as e:
    print(e)

日志记录和可观测性 - 记录 LLM 输入/输出 (文档)​

LiteLLM 提供预定义的回调函数，可将数据发送到 Lunary, MLflow, Langfuse, Helicone, Promptlayer, Traceloop, Slack

from litellm import completion

## set env variables for logging tools (API key set up is not required when using MLflow)
os.environ["LUNARY_PUBLIC_KEY"] = "your-lunary-public-key" # get your public key at https://app.lunary.ai/settings
os.environ["HELICONE_API_KEY"] = "your-helicone-key"
os.environ["LANGFUSE_PUBLIC_KEY"] = ""
os.environ["LANGFUSE_SECRET_KEY"] = ""

os.environ["OPENAI_API_KEY"]

# set callbacks
litellm.success_callback = ["lunary", "mlflow", "langfuse", "helicone"] # log input/output to lunary, mlflow, langfuse, helicone

#openai call
response = completion(model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Hi 👋 - i'm openai"}])

跟踪流式传输的成本、使用情况和延迟​

为此，请使用回调函数 - 有关自定义回调函数的更多信息：https://docs.litellm.com.cn/docs/observability/custom_callback

import litellm

# track_cost_callback
def track_cost_callback(
    kwargs,                 # kwargs to completion
    completion_response,    # response from completion
    start_time, end_time    # start/end time
):
    try:
      response_cost = kwargs.get("response_cost", 0)
      print("streaming response_cost", response_cost)
    except:
        pass
# set callback
litellm.success_callback = [track_cost_callback] # set custom callback function

# litellm.completion() call
response = completion(
    model="gpt-3.5-turbo",
    messages=[
        {
            "role": "user",
            "content": "Hi 👋 - i'm openai"
        }
    ],
    stream=True
)

LiteLLM 代理服务器 (LLM 网关)​

跟踪跨多个项目/人员的花费

ui_3

代理提供

  1. 用于认证的 Hook
  2. 用于日志记录的 Hook
  3. 成本跟踪
  4. 速率限制

📖 代理端点 - Swagger 文档​

点击此处获取包含密钥 + 速率限制的完整教程 - 此处

代理快速入门 - CLI​

pip install 'litellm[proxy]'

步骤 1: 启动 litellm 代理​

$ litellm --model huggingface/bigcode/starcoder

#INFO: Proxy running on http://0.0.0.0:4000

步骤 2: 向代理发送 ChatCompletions 请求​

import openai # openai v1.0.0+
client = openai.OpenAI(api_key="anything",base_url="http://0.0.0.0:4000") # set proxy to base_url
# request sent to model set on litellm proxy, `litellm --model`
response = client.chat.completions.create(model="gpt-3.5-turbo", messages = [
    {
        "role": "user",
        "content": "this is a test request, write a short poem"
    }
])

print(response)

更多详情​