Together AI

LiteLLM 支持 Together AI 上的所有模型。

API 密钥

import os 
os.environ["TOGETHERAI_API_KEY"] = "your-api-key"

示例用法

from litellm import completion 

os.environ["TOGETHERAI_API_KEY"] = "your-api-key"

messages = [{"role": "user", "content": "Write me a poem about the blue sky"}]

completion(model="together_ai/togethercomputer/Llama-2-7B-32K-Instruct", messages=messages)

Together AI 模型

liteLLM 支持 非流式 和 流式 请求到 https://api.together.xyz/ 上的所有模型

TogetherAI 用法示例 - 注意：liteLLM 支持部署在 TogetherAI 上的所有模型

Llama LLM - 对话

模型名称	函数调用	必需的操作系统变量
togethercomputer/llama-2-70b-chat	`completion('together_ai/togethercomputer/llama-2-70b-chat', messages)`	`os.environ['TOGETHERAI_API_KEY']`

Llama LLM - 语言 / 指令

模型名称	函数调用	必需的操作系统变量
togethercomputer/llama-2-70b	`completion('together_ai/togethercomputer/llama-2-70b', messages)`	`os.environ['TOGETHERAI_API_KEY']`
togethercomputer/LLaMA-2-7B-32K	`completion('together_ai/togethercomputer/LLaMA-2-7B-32K', messages)`	`os.environ['TOGETHERAI_API_KEY']`
togethercomputer/Llama-2-7B-32K-Instruct	`completion('together_ai/togethercomputer/Llama-2-7B-32K-Instruct', messages)`	`os.environ['TOGETHERAI_API_KEY']`
togethercomputer/llama-2-7b	`completion('together_ai/togethercomputer/llama-2-7b', messages)`	`os.environ['TOGETHERAI_API_KEY']`

Falcon LLM

模型名称	函数调用	必需的操作系统变量
togethercomputer/falcon-40b-instruct	`completion('together_ai/togethercomputer/falcon-40b-instruct', messages)`	`os.environ['TOGETHERAI_API_KEY']`
togethercomputer/falcon-7b-instruct	`completion('together_ai/togethercomputer/falcon-7b-instruct', messages)`	`os.environ['TOGETHERAI_API_KEY']`

Alpaca LLM

模型名称	函数调用	必需的操作系统变量
togethercomputer/alpaca-7b	`completion('together_ai/togethercomputer/alpaca-7b', messages)`	`os.environ['TOGETHERAI_API_KEY']`

其他对话 LLM

模型名称	函数调用	必需的操作系统变量
HuggingFaceH4/starchat-alpha	`completion('together_ai/HuggingFaceH4/starchat-alpha', messages)`	`os.environ['TOGETHERAI_API_KEY']`

代码 LLM

模型名称	函数调用	必需的操作系统变量
togethercomputer/CodeLlama-34b	`completion('together_ai/togethercomputer/CodeLlama-34b', messages)`	`os.environ['TOGETHERAI_API_KEY']`
togethercomputer/CodeLlama-34b-Instruct	`completion('together_ai/togethercomputer/CodeLlama-34b-Instruct', messages)`	`os.environ['TOGETHERAI_API_KEY']`
togethercomputer/CodeLlama-34b-Python	`completion('together_ai/togethercomputer/CodeLlama-34b-Python', messages)`	`os.environ['TOGETHERAI_API_KEY']`
defog/sqlcoder	`completion('together_ai/defog/sqlcoder', messages)`	`os.environ['TOGETHERAI_API_KEY']`
NumbersStation/nsql-llama-2-7B	`completion('together_ai/NumbersStation/nsql-llama-2-7B', messages)`	`os.environ['TOGETHERAI_API_KEY']`
WizardLM/WizardCoder-15B-V1.0	`completion('together_ai/WizardLM/WizardCoder-15B-V1.0', messages)`	`os.environ['TOGETHERAI_API_KEY']`
WizardLM/WizardCoder-Python-34B-V1.0	`completion('together_ai/WizardLM/WizardCoder-Python-34B-V1.0', messages)`	`os.environ['TOGETHERAI_API_KEY']`

语言 LLM

模型名称	函数调用	必需的操作系统变量
NousResearch/Nous-Hermes-Llama2-13b	`completion('together_ai/NousResearch/Nous-Hermes-Llama2-13b', messages)`	`os.environ['TOGETHERAI_API_KEY']`
Austism/chronos-hermes-13b	`completion('together_ai/Austism/chronos-hermes-13b', messages)`	`os.environ['TOGETHERAI_API_KEY']`
upstage/SOLAR-0-70b-16bit	`completion('together_ai/upstage/SOLAR-0-70b-16bit', messages)`	`os.environ['TOGETHERAI_API_KEY']`
WizardLM/WizardLM-70B-V1.0	`completion('together_ai/WizardLM/WizardLM-70B-V1.0', messages)`	`os.environ['TOGETHERAI_API_KEY']`

提示模板

在 Together AI 上使用带有其自身提示格式的对话模型？

使用 Llama2 指令模型

如果你正在使用 Together AI 的 Llama2 变体（model=togethercomputer/llama-2..-instruct），LiteLLM 可以自动在 OpenAI 提示格式和 TogetherAI Llama2 格式（[INST]..[/INST]）之间进行转换。

from litellm import completion 

# set env variable 
os.environ["TOGETHERAI_API_KEY"] = ""

messages = [{"role": "user", "content": "Write me a poem about the blue sky"}]

completion(model="together_ai/togethercomputer/Llama-2-7B-32K-Instruct", messages=messages)

使用其他模型

你可以在 LiteLLM 上创建自定义提示模板（我们欢迎 PR 将它们添加到主仓库 🤗）

让我们为 OpenAssistant/llama2-70b-oasst-sft-v10 创建一个！

可接受的模板格式为：参考

"""
<|im_start|>system
{system_message}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
"""

让我们注册自定义提示模板：实现代码

import litellm 

litellm.register_prompt_template(
        model="OpenAssistant/llama2-70b-oasst-sft-v10",
        roles={
            "system": {
                "pre_message": "[<|im_start|>system",
                "post_message": "\n"
            },
            "user": {
                "pre_message": "<|im_start|>user",
                "post_message": "\n"
            }, 
            "assistant": {
                "pre_message": "<|im_start|>assistant",
                "post_message": "\n"
            }
        }
    )

让我们使用它！

from litellm import completion 

# set env variable 
os.environ["TOGETHERAI_API_KEY"] = ""

messages=[{"role":"user", "content": "Write me a poem about the blue sky"}]

completion(model="together_ai/OpenAssistant/llama2-70b-oasst-sft-v10", messages=messages)

完整代码

import litellm 
from litellm import completion

# set env variable 
os.environ["TOGETHERAI_API_KEY"] = ""

litellm.register_prompt_template(
        model="OpenAssistant/llama2-70b-oasst-sft-v10",
        roles={
            "system": {
                "pre_message": "[<|im_start|>system",
                "post_message": "\n"
            },
            "user": {
                "pre_message": "<|im_start|>user",
                "post_message": "\n"
            }, 
            "assistant": {
                "pre_message": "<|im_start|>assistant",
                "post_message": "\n"
            }
        }
    )

messages=[{"role":"user", "content": "Write me a poem about the blue sky"}]

response = completion(model="together_ai/OpenAssistant/llama2-70b-oasst-sft-v10", messages=messages)

print(response)

输出

{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": ".\n\nThe sky is a canvas of blue,\nWith clouds that drift and move,",
        "role": "assistant",
        "logprobs": null
      }
    }
  ],
  "created": 1693941410.482018,
  "model": "OpenAssistant/llama2-70b-oasst-sft-v10",
  "usage": {
    "prompt_tokens": 7,
    "completion_tokens": 16,
    "total_tokens": 23
  },
  "litellm_call_id": "f21315db-afd6-4c1e-b43a-0b5682de4b06"
}

Rerank

用法

LiteLLM SDK 用法
LiteLLM 代理用法

from litellm import rerank
import os

os.environ["TOGETHERAI_API_KEY"] = "sk-.."

query = "What is the capital of the United States?"
documents = [
    "Carson City is the capital city of the American state of Nevada.",
    "The Commonwealth of the Northern Mariana Islands is a group of islands in the Pacific Ocean. Its capital is Saipan.",
    "Washington, D.C. is the capital of the United States.",
    "Capital punishment has existed in the United States since before it was a country.",
]

response = rerank(
    model="together_ai/rerank-english-v3.0",
    query=query,
    documents=documents,
    top_n=3,
)
print(response)

LiteLLM 提供一个与 cohere api 兼容的 /rerank 端点用于 Rerank 调用。

设置

将此添加到你的 litellm 代理 config.yaml

model_list:
  - model_name: Salesforce/Llama-Rank-V1
    litellm_params:
      model: together_ai/Salesforce/Llama-Rank-V1
      api_key: os.environ/TOGETHERAI_API_KEY

启动 litellm

litellm --config /path/to/config.yaml

# RUNNING on http://0.0.0.0:4000

测试请求

curl http://0.0.0.0:4000/rerank \
  -H "Authorization: Bearer sk-1234" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Salesforce/Llama-Rank-V1",
    "query": "What is the capital of the United States?",
    "documents": [
        "Carson City is the capital city of the American state of Nevada.",
        "The Commonwealth of the Northern Mariana Islands is a group of islands in the Pacific Ocean. Its capital is Saipan.",
        "Washington, D.C. is the capital of the United States.",
        "Capital punishment has existed in the United States since before it was a country."
    ],
    "top_n": 3
  }'

Together AI

API 密钥​

示例用法​

Together AI 模型​

Llama LLM - 对话​

Llama LLM - 语言 / 指令​

Falcon LLM​

Alpaca LLM​

其他对话 LLM​

代码 LLM​

语言 LLM​

提示模板​

使用 Llama2 指令模型​

使用其他模型​

Rerank​

用法​