Predibase

LiteLLM 支持 Predibase 上的所有模型

用法

SDK

API 密钥​

import os 
os.environ["PREDIBASE_API_KEY"] = ""

示例调用

from litellm import completion
import os
## set ENV variables
os.environ["PREDIBASE_API_KEY"] = "predibase key"
os.environ["PREDIBASE_TENANT_ID"] = "predibase tenant id"

# predibase llama-3 call
response = completion(
    model="predibase/llama-3-8b-instruct", 
    messages = [{ "content": "Hello, how are you?","role": "user"}]
)

代理

1. 将模型添加到你的 config.yaml

   model_list:
     - model_name: llama-3
       litellm_params:
         model: predibase/llama-3-8b-instruct
         api_key: os.environ/PREDIBASE_API_KEY
         tenant_id: os.environ/PREDIBASE_TENANT_ID

2. 启动代理

   $ litellm --config /path/to/config.yaml --debug

3. 向 LiteLLM 代理服务器发送请求

   OpenAI Python v1.0.0+

   import openai
   client = openai.OpenAI(
       api_key="sk-1234",             # pass litellm proxy key, if you're using virtual keys
       base_url="http://0.0.0.0:4000" # litellm-proxy-base url
   )
   
   response = client.chat.completions.create(
       model="llama-3",
       messages = [
         {
             "role": "system",
             "content": "Be a good human!"
         },
         {
             "role": "user",
             "content": "What do you know about earth?"
         }
     ]
   )
   
   print(response)

   curl

   curl --location 'http://0.0.0.0:4000/chat/completions' \
       --header 'Authorization: Bearer sk-1234' \
       --header 'Content-Type: application/json' \
       --data '{
       "model": "llama-3",
       "messages": [
         {
             "role": "system",
             "content": "Be a good human!"
         },
         {
             "role": "user",
             "content": "What do you know about earth?"
         }
         ],
   }'

高级用法 - Prompt 格式化

LiteLLM 为所有 meta-llama llama3 指令模型提供了 prompt 模板映射。 查看代码

应用自定义 prompt 模板

SDK

import litellm

import os 
os.environ["PREDIBASE_API_KEY"] = ""

# Create your own custom prompt template 
litellm.register_prompt_template(
        model="togethercomputer/LLaMA-2-7B-32K",
        initial_prompt_value="You are a good assistant" # [OPTIONAL]
        roles={
            "system": {
                "pre_message": "[INST] <<SYS>>\n", # [OPTIONAL]
                "post_message": "\n<</SYS>>\n [/INST]\n" # [OPTIONAL]
            },
            "user": { 
                "pre_message": "[INST] ", # [OPTIONAL]
                "post_message": " [/INST]" # [OPTIONAL]
            }, 
            "assistant": {
                "pre_message": "\n" # [OPTIONAL]
                "post_message": "\n" # [OPTIONAL]
            }
        }
        final_prompt_value="Now answer as best you can:" # [OPTIONAL]
)

def predibase_custom_model():
    model = "predibase/togethercomputer/LLaMA-2-7B-32K"
    response = completion(model=model, messages=messages)
    print(response['choices'][0]['message']['content'])
    return response

predibase_custom_model()

代理

# Model-specific parameters
model_list:
  - model_name: mistral-7b # model alias
    litellm_params: # actual params for litellm.completion()
      model: "predibase/mistralai/Mistral-7B-Instruct-v0.1" 
      api_key: os.environ/PREDIBASE_API_KEY
      initial_prompt_value: "\n"
      roles: {"system":{"pre_message":"<|im_start|>system\n", "post_message":"<|im_end|>"}, "assistant":{"pre_message":"<|im_start|>assistant\n","post_message":"<|im_end|>"}, "user":{"pre_message":"<|im_start|>user\n","post_message":"<|im_end|>"}}
      final_prompt_value: "\n"
      bos_token: "<s>"
      eos_token: "</s>"
      max_tokens: 4096

传递附加参数 - max_tokens, temperature

此处查看所有 litellm.completion 支持的参数 here

# !pip install litellm
from litellm import completion
import os
## set ENV variables
os.environ["PREDIBASE_API_KEY"] = "predibase key"

# predibae llama-3 call
response = completion(
    model="predibase/llama3-8b-instruct", 
    messages = [{ "content": "Hello, how are you?","role": "user"}],
    max_tokens=20,
    temperature=0.5
)

代理

  model_list:
    - model_name: llama-3
      litellm_params:
        model: predibase/llama-3-8b-instruct
        api_key: os.environ/PREDIBASE_API_KEY
        max_tokens: 20
        temperature: 0.5

传递 Predibase 特定参数 - adapter_id, adapter_source,

发送 `litellm.completion()` 不支持但 Predibase 支持的参数，通过将它们传递给 litellm.completion

示例：adapter_id, adapter_source 是 Predibase 特定参数 - 查看列表

# !pip install litellm
from litellm import completion
import os
## set ENV variables
os.environ["PREDIBASE_API_KEY"] = "predibase key"

# predibase llama3 call
response = completion(
    model="predibase/llama-3-8b-instruct", 
    messages = [{ "content": "Hello, how are you?","role": "user"}],
    adapter_id="my_repo/3",
    adapter_source="pbase",
)

代理

  model_list:
    - model_name: llama-3
      litellm_params:
        model: predibase/llama-3-8b-instruct
        api_key: os.environ/PREDIBASE_API_KEY
        adapter_id: my_repo/3
        adapter_source: pbase