跳到主要内容

AWS Bedrock

支持所有 Bedrock 模型 (Anthropic, Meta, Deepseek, Mistral, Amazon 等)

属性详情
描述Amazon Bedrock 是一项完全托管的服务，提供多种高性能基础模型 (FM)。
LiteLLM 上的提供商路由bedrock/, bedrock/converse/, bedrock/invoke/, bedrock/converse_like/, bedrock/llama/, bedrock/deepseek_r1/
提供商文档Amazon Bedrock ↗
支持的 OpenAI 端点/chat/completions, /completions, /embeddings, /images/generations
Rerank 端点/rerank
直通端点支持

LiteLLM 需要在您的系统上安装 boto3 来处理 Bedrock 请求

pip install boto3>=1.28.57
信息

对于 Amazon Nova 模型：升级到 v1.53.5+

信息

LiteLLM 使用 boto3 处理身份验证。支持所有这些选项 - https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#credentials。

用法​

Open In Colab
import os
from litellm import completion

os.environ["AWS_ACCESS_KEY_ID"] = ""
os.environ["AWS_SECRET_ACCESS_KEY"] = ""
os.environ["AWS_REGION_NAME"] = ""

response = completion(
  model="bedrock/anthropic.claude-3-sonnet-20240229-v1:0",
  messages=[{ "content": "Hello, how are you?","role": "user"}]
)

LiteLLM 代理用法​

以下是如何使用 LiteLLM 代理服务器调用 Bedrock

1. 设置 config.yaml​

model_list:
  - model_name: bedrock-claude-3-5-sonnet
    litellm_params:
      model: bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0
      aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
      aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
      aws_region_name: os.environ/AWS_REGION_NAME

所有可能的认证参数

aws_access_key_id: Optional[str],
aws_secret_access_key: Optional[str],
aws_session_token: Optional[str],
aws_region_name: Optional[str],
aws_session_name: Optional[str],
aws_profile_name: Optional[str],
aws_role_name: Optional[str],
aws_web_identity_token: Optional[str],
aws_bedrock_runtime_endpoint: Optional[str],

2. 启动代理​

litellm --config /path/to/config.yaml

3. 测试​

curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
--data ' {
      "model": "bedrock-claude-v1",
      "messages": [
        {
          "role": "user",
          "content": "what llm are you"
        }
      ]
    }
'

设置 temperature, top p 等​

import os
from litellm import completion

os.environ["AWS_ACCESS_KEY_ID"] = ""
os.environ["AWS_SECRET_ACCESS_KEY"] = ""
os.environ["AWS_REGION_NAME"] = ""

response = completion(
  model="bedrock/anthropic.claude-3-sonnet-20240229-v1:0",
  messages=[{ "content": "Hello, how are you?","role": "user"}],
  temperature=0.7,
  top_p=1
)

传递特定于提供商的参数​

如果您向 litellm 传递非 openai 参数，我们将假定它是特定于提供商的，并将其作为 kwarg 发送到请求体中。了解更多

import os
from litellm import completion

os.environ["AWS_ACCESS_KEY_ID"] = ""
os.environ["AWS_SECRET_ACCESS_KEY"] = ""
os.environ["AWS_REGION_NAME"] = ""

response = completion(
  model="bedrock/anthropic.claude-3-sonnet-20240229-v1:0",
  messages=[{ "content": "Hello, how are you?","role": "user"}],
  top_k=1 # 👈 PROVIDER-SPECIFIC PARAM
)

用法 - 函数调用 / 工具调用​

LiteLLM 通过 Bedrock 的 Converse 和 Invoke API 支持工具调用。

from litellm import completion

# set env
os.environ["AWS_ACCESS_KEY_ID"] = ""
os.environ["AWS_SECRET_ACCESS_KEY"] = ""
os.environ["AWS_REGION_NAME"] = ""

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_current_weather",
            "description": "Get the current weather in a given location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city and state, e.g. San Francisco, CA",
                    },
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                },
                "required": ["location"],
            },
        },
    }
]
messages = [{"role": "user", "content": "What's the weather like in Boston today?"}]

response = completion(
    model="bedrock/anthropic.claude-3-sonnet-20240229-v1:0",
    messages=messages,
    tools=tools,
    tool_choice="auto",
)
# Add any assertions, here to check response args
print(response)
assert isinstance(response.choices[0].message.tool_calls[0].function.name, str)
assert isinstance(
    response.choices[0].message.tool_calls[0].function.arguments, str
)

用法 - 视觉​

from litellm import completion

# set env
os.environ["AWS_ACCESS_KEY_ID"] = ""
os.environ["AWS_SECRET_ACCESS_KEY"] = ""
os.environ["AWS_REGION_NAME"] = ""


def encode_image(image_path):
    import base64

    with open(image_path, "rb") as image_file:
        return base64.b64encode(image_file.read()).decode("utf-8")


image_path = "../proxy/cached_logo.jpg"
# Getting the base64 string
base64_image = encode_image(image_path)
resp = litellm.completion(
    model="bedrock/anthropic.claude-3-sonnet-20240229-v1:0",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Whats in this image?"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "data:image/jpeg;base64," + base64_image
                    },
                },
            ],
        }
    ],
)
print(f"\nResponse: {resp}")

用法 - '思考' / '推理内容'​

目前仅支持 Anthropic 的 Claude 3.7 Sonnet + Deepseek R1。

适用于 v1.61.20+。

在 message 和 delta 对象中返回 2 个新字段

  • reasoning_content - 字符串 - 响应的推理内容
  • thinking_blocks - 对象列表 (仅 Anthropic) - 响应的思考块

每个对象包含以下字段

  • type - 字面量["thinking"]- 思考块的类型
  • thinking - 字符串 - 响应的思考内容。也包含在 reasoning_content 中返回
  • signature - 字符串 - Anthropic 返回的 base64 编码字符串。

如果在后续调用中传递了 'thinking' 内容，Anthropic 需要 signature（仅在使用 thinking 进行工具调用时需要）。了解更多

from litellm import completion

# set env
os.environ["AWS_ACCESS_KEY_ID"] = ""
os.environ["AWS_SECRET_ACCESS_KEY"] = ""
os.environ["AWS_REGION_NAME"] = ""


resp = completion(
    model="bedrock/us.anthropic.claude-3-7-sonnet-20250219-v1:0",
    messages=[{"role": "user", "content": "What is the capital of France?"}],
    reasoning_effort="low",
)

print(resp)

预期响应

与 Anthropic API 响应相同。

{
    "id": "chatcmpl-c661dfd7-7530-49c9-b0cc-d5018ba4727d",
    "created": 1740640366,
    "model": "us.anthropic.claude-3-7-sonnet-20250219-v1:0",
    "object": "chat.completion",
    "system_fingerprint": null,
    "choices": [
        {
            "finish_reason": "stop",
            "index": 0,
            "message": {
                "content": "The capital of France is Paris. It's not only the capital city but also the largest city in France, serving as the country's major cultural, economic, and political center.",
                "role": "assistant",
                "tool_calls": null,
                "function_call": null,
                "reasoning_content": "The capital of France is Paris. This is a straightforward factual question.",
                "thinking_blocks": [
                    {
                        "type": "thinking",
                        "thinking": "The capital of France is Paris. This is a straightforward factual question.",
                        "signature": "EqoBCkgIARABGAIiQL2UoU0b1OHYi+yCHpBY7U6FQW8/FcoLewocJQPa2HnmLM+NECy50y44F/kD4SULFXi57buI9fAvyBwtyjlOiO0SDE3+r3spdg6PLOo9PBoMma2ku5OTAoR46j9VIjDRlvNmBvff7YW4WI9oU8XagaOBSxLPxElrhyuxppEn7m6bfT40dqBSTDrfiw4FYB4qEPETTI6TA6wtjGAAqmFqKTo="
                    }
                ]
            }
        }
    ],
    "usage": {
        "completion_tokens": 64,
        "prompt_tokens": 42,
        "total_tokens": 106,
        "completion_tokens_details": null,
        "prompt_tokens_details": null
    }
}

将 thinking 传递给 Anthropic 模型​

与 Anthropic API 响应相同。

用法 - 结构化输出 / JSON 模式​

from litellm import completion
import os 
from pydantic import BaseModel

# set env
os.environ["AWS_ACCESS_KEY_ID"] = ""
os.environ["AWS_SECRET_ACCESS_KEY"] = ""
os.environ["AWS_REGION_NAME"] = ""

class CalendarEvent(BaseModel):
  name: str
  date: str
  participants: list[str]

class EventsList(BaseModel):
    events: list[CalendarEvent]

response = completion(
  model="bedrock/anthropic.claude-3-7-sonnet-20250219-v1:0", # specify invoke via `bedrock/invoke/anthropic.claude-3-7-sonnet-20250219-v1:0`
  response_format=EventsList,
  messages=[
    {"role": "system", "content": "You are a helpful assistant designed to output JSON."},
    {"role": "user", "content": "Who won the world series in 2020?"}
  ],
)
print(response.choices[0].message.content)

用法 - 延迟优化推理​

从 v1.65.1+ 版本开始有效

from litellm import completion

response = completion(
    model="bedrock/anthropic.claude-3-7-sonnet-20250219-v1:0",
    messages=[{"role": "user", "content": "What is the capital of France?"}],
    performanceConfig={"latency": "optimized"},
)

用法 - Bedrock Guardrails​

使用 Bedrock Guardrails 与 LiteLLM 的示例

from litellm import completion

# set env
os.environ["AWS_ACCESS_KEY_ID"] = ""
os.environ["AWS_SECRET_ACCESS_KEY"] = ""
os.environ["AWS_REGION_NAME"] = ""

response = completion(
    model="anthropic.claude-v2",
    messages=[
        {
            "content": "where do i buy coffee from? ",
            "role": "user",
        }
    ],
    max_tokens=10,
    guardrailConfig={
        "guardrailIdentifier": "ff6ujrregl1q", # The identifier (ID) for the guardrail.
        "guardrailVersion": "DRAFT",           # The version of the guardrail.
        "trace": "disabled",                   # The trace behavior for the guardrail. Can either be "disabled" or "enabled"
    },
)

用法 - “助手预填充”​

如果您在使用 Bedrock 上的 Anthropic Claude，您可以通过在 messages 数组的最后一个项中包含 assistant 角色消息来“替 Claude 说话”。

[!重要]返回的补全文本将不包含您的“预填充”文本，因为它本身是 prompt 的一部分。请确保在 Claude 的补全文本前加上您的预填充内容。

import os
from litellm import completion

os.environ["AWS_ACCESS_KEY_ID"] = ""
os.environ["AWS_SECRET_ACCESS_KEY"] = ""
os.environ["AWS_REGION_NAME"] = ""

messages = [
    {"role": "user", "content": "How do you say 'Hello' in German? Return your answer as a JSON object, like this:\n\n{ \"Hello\": \"Hallo\" }"},
    {"role": "assistant", "content": "{"},
]
response = completion(model="bedrock/anthropic.claude-v2", messages=messages)

发送给 Claude 的 Prompt 示例​


Human: How do you say 'Hello' in German? Return your answer as a JSON object, like this:

{ "Hello": "Hallo" }

Assistant: {

用法 - “系统”消息​

如果您在使用 Bedrock 上的 Anthropic Claude 2.1，system 角色消息将为您正确格式化。

import os
from litellm import completion

os.environ["AWS_ACCESS_KEY_ID"] = ""
os.environ["AWS_SECRET_ACCESS_KEY"] = ""
os.environ["AWS_REGION_NAME"] = ""

messages = [
    {"role": "system", "content": "You are a snarky assistant."},
    {"role": "user", "content": "How do I boil water?"},
]
response = completion(model="bedrock/anthropic.claude-v2:1", messages=messages)

发送给 Claude 的 Prompt 示例​

You are a snarky assistant.

Human: How do I boil water?

Assistant:

用法 - 流式传输​

import os
from litellm import completion

os.environ["AWS_ACCESS_KEY_ID"] = ""
os.environ["AWS_SECRET_ACCESS_KEY"] = ""
os.environ["AWS_REGION_NAME"] = ""

response = completion(
  model="bedrock/anthropic.claude-instant-v1",
  messages=[{ "content": "Hello, how are you?","role": "user"}],
  stream=True
)
for chunk in response:
  print(chunk)

流式输出块示例​

{
  "choices": [
    {
      "finish_reason": null,
      "index": 0,
      "delta": {
        "content": "ase can appeal the case to a higher federal court. If a higher federal court rules in a way that conflicts with a ruling from a lower federal court or conflicts with a ruling from a higher state court, the parties involved in the case can appeal the case to the Supreme Court. In order to appeal a case to the Sup"
      }
    }
  ],
  "created": null,
  "model": "anthropic.claude-instant-v1",
  "usage": {
    "prompt_tokens": null,
    "completion_tokens": null,
    "total_tokens": null
  }
}

跨区域推理​

LiteLLM 支持 Bedrock 跨区域推理，覆盖所有 支持的 Bedrock 模型。

from litellm import completion 
import os 


os.environ["AWS_ACCESS_KEY_ID"] = ""
os.environ["AWS_SECRET_ACCESS_KEY"] = ""
os.environ["AWS_REGION_NAME"] = ""


litellm.set_verbose = True #  👈 SEE RAW REQUEST 

response = completion(
    model="bedrock/us.anthropic.claude-3-haiku-20240307-v1:0",
    messages=messages,
    max_tokens=10,
    temperature=0.1,
)

print("Final Response: {}".format(response))

设置 'converse' / 'invoke' 路由​

信息

从 LiteLLM 版本 v1.53.5 开始支持

LiteLLM 默认使用 invoke 路由。对于支持 converse 路由的 Bedrock 模型，LiteLLM 使用 converse 路由。

要明确设置路由，请使用 bedrock/converse/<model> 或 bedrock/invoke/<model>。

例如：

from litellm import completion

completion(model="bedrock/converse/us.amazon.nova-pro-v1:0")

交替的 user/assistant 消息​

使用 user_continue_message 添加默认的 user 消息，适用于客户端可能不遵循以 user 消息开始和结束的交替 user/assistant 消息的情况（例如 Autogen）。

model_list:
  - model_name: "bedrock-claude"
    litellm_params:
      model: "bedrock/anthropic.claude-instant-v1"
      user_continue_message: {"role": "user", "content": "Please continue"}

或者

只需设置 litellm.modify_params=True，LiteLLM 将自动使用默认的 user_continue_message 处理此事。

model_list:
  - model_name: "bedrock-claude"
    litellm_params:
      model: "bedrock/anthropic.claude-instant-v1"

litellm_settings:
   modify_params: true

测试！

curl -X POST 'http://0.0.0.0:4000/chat/completions' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer sk-1234' \
-d '{
    "model": "bedrock-claude",
    "messages": [{"role": "assistant", "content": "Hey, how's it going?"}]
}'

用法 - PDF / 文档理解​

LiteLLM 支持 Bedrock 模型的文档理解 - AWS Bedrock 文档。

信息

LiteLLM 支持所有 Bedrock 文档类型 -

例如：
"pdf", "csv", "doc", "docx", "xls", "xlsx", "html", "txt", "md"

您也可以将其作为 image_url 或 base64 传递

url​

from litellm.utils import supports_pdf_input, completion

# set aws credentials
os.environ["AWS_ACCESS_KEY_ID"] = ""
os.environ["AWS_SECRET_ACCESS_KEY"] = ""
os.environ["AWS_REGION_NAME"] = ""


# pdf url
image_url = "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf"

# Download the file
response = requests.get(url)
file_data = response.content

encoded_file = base64.b64encode(file_data).decode("utf-8")

# model
model = "bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0"

image_content = [
    {"type": "text", "text": "What's this file about?"},
    {
        "type": "file",
        "file": {
            "file_data": f"data:application/pdf;base64,{encoded_file}", # 👈 PDF
        }
    },
]


if not supports_pdf_input(model, None):
    print("Model does not support image input")

response = completion(
    model=model,
    messages=[{"role": "user", "content": image_content}],
)
assert response is not None

base64​

from litellm.utils import supports_pdf_input, completion

# set aws credentials
os.environ["AWS_ACCESS_KEY_ID"] = ""
os.environ["AWS_SECRET_ACCESS_KEY"] = ""
os.environ["AWS_REGION_NAME"] = ""


# pdf url
image_url = "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf"
response = requests.get(url)
file_data = response.content

encoded_file = base64.b64encode(file_data).decode("utf-8")
base64_url = f"data:application/pdf;base64,{encoded_file}"

# model
model = "bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0"

image_content = [
    {"type": "text", "text": "What's this file about?"},
    {
        "type": "image_url",
        "image_url": base64_url, # OR {"url": base64_url}
    },
]


if not supports_pdf_input(model, None):
    print("Model does not support image input")

response = completion(
    model=model,
    messages=[{"role": "user", "content": image_content}],
)
assert response is not None

Bedrock 导入模型 (Deepseek, Deepseek R1)​

Deepseek R1​

这是另一个独立的路由，因为聊天模板不同。

属性详情
提供商路由bedrock/deepseek_r1/{model_arn}
提供商文档Bedrock 导入模型, Deepseek Bedrock 导入模型
from litellm import completion
import os

response = completion(
    model="bedrock/deepseek_r1/arn:aws:bedrock:us-east-1:086734376398:imported-model/r4c4kewx2s0n",  # bedrock/deepseek_r1/{your-model-arn}
    messages=[{"role": "user", "content": "Tell me a joke"}],
)

Deepseek (非 R1)​

属性详情
提供商路由bedrock/llama/{model_arn}
提供商文档Bedrock 导入模型, Deepseek Bedrock 导入模型

使用此路由调用遵循 llama Invoke 请求/响应规范的 Bedrock 导入模型

from litellm import completion
import os

response = completion(
    model="bedrock/llama/arn:aws:bedrock:us-east-1:086734376398:imported-model/r4c4kewx2s0n",  # bedrock/llama/{your-model-arn}
    messages=[{"role": "user", "content": "Tell me a joke"}],
)

预置吞吐量模型​

要使用预置吞吐量 Bedrock 模型，请传递

  • model=bedrock/<base-model>，例如 model=bedrock/anthropic.claude-v2。将 model 设置为任何 支持的 AWS 模型
  • model_id=provisioned-model-arn

补全

import litellm
response = litellm.completion(
    model="bedrock/anthropic.claude-instant-v1",
    model_id="provisioned-model-arn",
    messages=[{"content": "Hello, how are you?", "role": "user"}]
)

嵌入

import litellm
response = litellm.embedding(
    model="bedrock/amazon.titan-embed-text-v1",
    model_id="provisioned-model-arn",
    input=["hi"],
)

支持的 AWS Bedrock 模型​

LiteLLM 支持所有 Bedrock 模型。

以下是使用 LiteLLM 调用 Bedrock 模型的示例。完整列表请参考 模型成本图

模型名称命令
Deepseek R1completion(model='bedrock/us.deepseek.r1-v1:0', messages=messages)
Anthropic Claude-V3.5 Sonnetcompletion(model='bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0', messages=messages)
Anthropic Claude-V3 sonnetcompletion(model='bedrock/anthropic.claude-3-sonnet-20240229-v1:0', messages=messages)
Anthropic Claude-V3 Haikucompletion(model='bedrock/anthropic.claude-3-haiku-20240307-v1:0', messages=messages)
Anthropic Claude-V3 Opuscompletion(model='bedrock/anthropic.claude-3-opus-20240229-v1:0', messages=messages)
Anthropic Claude-V2.1completion(model='bedrock/anthropic.claude-v2:1', messages=messages)
Anthropic Claude-V2completion(model='bedrock/anthropic.claude-v2', messages=messages)
Anthropic Claude-Instant V1completion(model='bedrock/anthropic.claude-instant-v1', messages=messages)
Meta llama3-1-405bcompletion(model='bedrock/meta.llama3-1-405b-instruct-v1:0', messages=messages)
Meta llama3-1-70bcompletion(model='bedrock/meta.llama3-1-70b-instruct-v1:0', messages=messages)
Meta llama3-1-8bcompletion(model='bedrock/meta.llama3-1-8b-instruct-v1:0', messages=messages)
Meta llama3-70bcompletion(model='bedrock/meta.llama3-70b-instruct-v1:0', messages=messages)
Meta llama3-8bcompletion(model='bedrock/meta.llama3-8b-instruct-v1:0', messages=messages)
Amazon Titan Litecompletion(model='bedrock/amazon.titan-text-lite-v1', messages=messages)
Amazon Titan Expresscompletion(model='bedrock/amazon.titan-text-express-v1', messages=messages)
Cohere Commandcompletion(model='bedrock/cohere.command-text-v14', messages=messages)
AI21 J2-Midcompletion(model='bedrock/ai21.j2-mid-v1', messages=messages)
AI21 J2-Ultracompletion(model='bedrock/ai21.j2-ultra-v1', messages=messages)
AI21 Jamba-Instructcompletion(model='bedrock/ai21.jamba-instruct-v1:0', messages=messages)
Meta Llama 2 Chat 13bcompletion(model='bedrock/meta.llama2-13b-chat-v1', messages=messages)
Meta Llama 2 Chat 70bcompletion(model='bedrock/meta.llama2-70b-chat-v1', messages=messages)
Mistral 7B Instructcompletion(model='bedrock/mistral.mistral-7b-instruct-v0:2', messages=messages)
Mixtral 8x7B Instructcompletion(model='bedrock/mistral.mixtral-8x7b-instruct-v0:1', messages=messages)

Bedrock Embedding​

API 密钥​

这可以设置为环境变量或作为参数传递给 litellm.embedding()

import os
os.environ["AWS_ACCESS_KEY_ID"] = ""        # Access key
os.environ["AWS_SECRET_ACCESS_KEY"] = ""    # Secret access key
os.environ["AWS_REGION_NAME"] = ""           # us-east-1, us-east-2, us-west-1, us-west-2

用法​

from litellm import embedding
response = embedding(
    model="bedrock/amazon.titan-embed-text-v1",
    input=["good morning from litellm"],
)
print(response)

支持的 AWS Bedrock Embedding 模型​

模型名称用法支持的额外 OpenAI 参数
Titan Embeddings V2embedding(model="bedrock/amazon.titan-embed-text-v2:0", input=input)此处
Titan Embeddings - V1embedding(model="bedrock/amazon.titan-embed-text-v1", input=input)此处
Titan 多模态 Embeddingsembedding(model="bedrock/amazon.titan-embed-image-v1", input=input)此处
Cohere Embeddings - 英语embedding(model="bedrock/cohere.embed-english-v3", input=input)此处
Cohere Embeddings - 多语言embedding(model="bedrock/cohere.embed-multilingual-v3", input=input)此处

高级 - 丢弃不支持的参数​

高级 - 传递模型/提供商特定参数​

图像生成​

将其用于 stable diffusion 和 bedrock 上的 amazon nova canvas

用法​

import os
from litellm import image_generation

os.environ["AWS_ACCESS_KEY_ID"] = ""
os.environ["AWS_SECRET_ACCESS_KEY"] = ""
os.environ["AWS_REGION_NAME"] = ""

response = image_generation(
            prompt="A cute baby sea otter",
            model="bedrock/stability.stable-diffusion-xl-v0",
        )
print(f"response: {response}")

设置可选参数

import os
from litellm import image_generation

os.environ["AWS_ACCESS_KEY_ID"] = ""
os.environ["AWS_SECRET_ACCESS_KEY"] = ""
os.environ["AWS_REGION_NAME"] = ""

response = image_generation(
            prompt="A cute baby sea otter",
            model="bedrock/stability.stable-diffusion-xl-v0",
            ### OPENAI-COMPATIBLE ###
            size="128x512", # width=128, height=512
            ### PROVIDER-SPECIFIC ### see `AmazonStabilityConfig` in bedrock.py for all params
            seed=30
        )
print(f"response: {response}")

支持的 AWS Bedrock 图像生成模型​

模型名称函数调用
Stable Diffusion 3 - v0embedding(model="bedrock/stability.stability.sd3-large-v1:0", prompt=prompt)
Stable Diffusion - v0embedding(model="bedrock/stability.stable-diffusion-xl-v0", prompt=prompt)
Stable Diffusion - v0embedding(model="bedrock/stability.stable-diffusion-xl-v1", prompt=prompt)

Rerank API​

使用 Cohere /rerank 格式调用 Bedrock 的 Rerank API。

支持的 Cohere Rerank 参数

  • model - 基础模型的 ARN
  • query - 用于 rerank 的查询
  • documents - 用于 rerank 的文档列表
  • top_n - 返回结果的数量
from litellm import rerank
import os 

os.environ["AWS_ACCESS_KEY_ID"] = ""
os.environ["AWS_SECRET_ACCESS_KEY"] = ""
os.environ["AWS_REGION_NAME"] = ""

response = rerank(
    model="bedrock/arn:aws:bedrock:us-west-2::foundation-model/amazon.rerank-v1:0", # provide the model ARN - get this here https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/bedrock/client/list_foundation_models.html
    query="hello",
    documents=["hello", "world"],
    top_n=2,
)

print(response)

Bedrock 应用推理配置文件​

使用 Bedrock 应用推理配置文件跟踪 AWS 项目的成本。

您可以在模型名称中传递它 - model="bedrock/arn:... 或作为单独的 model_id="arn:.. 参数。

通过 model_id 设置​

from litellm import completion
import os 

os.environ["AWS_ACCESS_KEY_ID"] = ""
os.environ["AWS_SECRET_ACCESS_KEY"] = ""
os.environ["AWS_REGION_NAME"] = ""

response = completion(
    model="bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0",
    messages=[{"role": "user", "content": "Hello, how are you?"}],
    model_id="arn:aws:bedrock:eu-central-1:000000000000:application-inference-profile/a0a0a0a0a0a0",
)

print(response)

Boto3 - 身份验证​

将凭证作为参数传递 - Completion()​

将 AWS 凭证作为参数传递给 litellm.completion

import os
from litellm import completion

response = completion(
            model="bedrock/anthropic.claude-instant-v1",
            messages=[{ "content": "Hello, how are you?","role": "user"}],
            aws_access_key_id="",
            aws_secret_access_key="",
            aws_region_name="",
)

传递额外的请求头 + 自定义 API 端点​

这可用于调用自定义 API 端点时覆盖现有请求头（例如 Authorization）

import os
import litellm
from litellm import completion

litellm.set_verbose = True # 👈 SEE RAW REQUEST

response = completion(
            model="bedrock/anthropic.claude-instant-v1",
            messages=[{ "content": "Hello, how are you?","role": "user"}],
            aws_access_key_id="",
            aws_secret_access_key="",
            aws_region_name="",
            aws_bedrock_runtime_endpoint="https://my-fake-endpoint.com",
            extra_headers={"key": "value"}
)

SSO 登录 (AWS Profile)​

  • 设置 AWS_PROFILE 环境变量
  • 调用 Bedrock 补全
import os
from litellm import completion

response = completion(
            model="bedrock/anthropic.claude-instant-v1",
            messages=[{ "content": "Hello, how are you?","role": "user"}]
)

或传递 aws_profile_name

import os
from litellm import completion

response = completion(
            model="bedrock/anthropic.claude-instant-v1",
            messages=[{ "content": "Hello, how are you?","role": "user"}],
            aws_profile_name="dev-profile",
)

STS (基于角色的认证)​

  • 设置 aws_role_name 和 aws_session_name
LiteLLM 参数Boto3 参数描述Boto3 文档
aws_access_key_idaws_access_key_id与 IAM 用户或角色关联的 AWS 访问密钥凭证
aws_secret_access_keyaws_secret_access_key与访问密钥关联的 AWS 秘密密钥凭证
aws_role_nameRoleArn要假定的角色的亚马逊资源名称 (ARN)AssumeRole API
aws_session_nameRoleSessionName假定角色会话的标识符AssumeRole API

进行 Bedrock 补全调用

from litellm import completion

response = completion(
            model="bedrock/anthropic.claude-instant-v1",
            messages=messages,
            max_tokens=10,
            temperature=0.1,
            aws_role_name=aws_role_name,
            aws_session_name="my-test-session",
        )

如果您还需要动态设置访问角色的 aws 用户，请在 completion()/embedding() 函数中添加额外的参数

from litellm import completion

response = completion(
            model="bedrock/anthropic.claude-instant-v1",
            messages=messages,
            max_tokens=10,
            temperature=0.1,
            aws_region_name=aws_region_name,
            aws_access_key_id=aws_access_key_id,
            aws_secret_access_key=aws_secret_access_key,
            aws_role_name=aws_role_name,
            aws_session_name="my-test-session",
        )

文本到图像

curl -L -X POST 'http://0.0.0.0:4000/v1/images/generations' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer $LITELLM_VIRTUAL_KEY' \
-d '{
    "model": "amazon.nova-canvas-v1:0",
    "prompt": "A cute baby sea otter"
}'

颜色引导生成

curl -L -X POST 'http://0.0.0.0:4000/v1/images/generations' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer $LITELLM_VIRTUAL_KEY' \
-d '{
    "model": "amazon.nova-canvas-v1:0",
    "prompt": "A cute baby sea otter",
    "taskType": "COLOR_GUIDED_GENERATION",
    "colorGuidedGenerationParams":{"colors":["#FFFFFF"]}
}'
模型名称函数调用
Stable Diffusion 3 - v0image_generation(model="bedrock/stability.stability.sd3-large-v1:0", prompt=prompt)
Stable Diffusion - v0image_generation(model="bedrock/stability.stable-diffusion-xl-v0", prompt=prompt)
Stable Diffusion - v1image_generation(model="bedrock/stability.stable-diffusion-xl-v1", prompt=prompt)
Amazon Nova Canvas - v0image_generation(model="bedrock/amazon.nova-canvas-v1:0", prompt=prompt)

将外部 BedrockRuntime.Client 作为参数传递 - Completion()​

这是一个已弃用的流程。Boto3 不是异步的。并且 boto3.client 不允许我们通过 httpx 进行 http 调用。请通过上方的方法👆传递您的 aws 参数。查看认证代码 添加新的认证流程

危险

实验性 - 2024年6月23日：aws_access_key_id, aws_secret_access_key 和 aws_session_token 将从 boto3.client 中提取并传递给 httpx client

将外部 BedrockRuntime.Client 对象作为参数传递给 litellm.completion。在使用 AWS 凭证配置文件、SSO 会话、假定角色会话或环境变量不可用于认证时非常有用。

从会话凭证创建客户端

import boto3
from litellm import completion

bedrock = boto3.client(
            service_name="bedrock-runtime",
            region_name="us-east-1",
            aws_access_key_id="",
            aws_secret_access_key="",
            aws_session_token="",
)

response = completion(
            model="bedrock/anthropic.claude-instant-v1",
            messages=[{ "content": "Hello, how are you?","role": "user"}],
            aws_bedrock_client=bedrock,
)

从 ~/.aws/config 中的 AWS Profile 创建客户端

import boto3
from litellm import completion

dev_session = boto3.Session(profile_name="dev-profile")
bedrock = dev_session.client(
            service_name="bedrock-runtime",
            region_name="us-east-1",
)

response = completion(
            model="bedrock/anthropic.claude-instant-v1",
            messages=[{ "content": "Hello, how are you?","role": "user"}],
            aws_bedrock_client=bedrock,
)

通过内部代理调用 (与 bedrock url 不兼容)​

使用 bedrock/converse_like/model 端点通过内部代理调用 bedrock converse 模型。

from litellm import completion

response = completion(
    model="bedrock/converse_like/some-model",
    messages=[{"role": "user", "content": "What's AWS?"}],
    api_key="sk-1234",
    api_base="https://some-api-url/models",
    extra_headers={"test": "hello world"},
)

预期输出 URL

https://some-api-url/models