LiteRT-LM Python API

適用於 Linux、macOS 和 Windows 的 LiteRT-LM Python API。支援多模態、工具使用和 GPU 與 NPU 加速等功能。

簡介

以下是使用 Python API 建構的終端機即時通訊應用程式範例：

import litert_lm

litert_lm.set_min_log_severity(litert_lm.LogSeverity.ERROR) # Hide log for TUI app

with litert_lm.Engine("path/to/model.litertlm") as engine:
  with engine.create_conversation() as conversation:
    while True:
      user_input = input("\n>>> ")
      for chunk in conversation.send_message_async(user_input):
        print(chunk["content"][0]["text"], end="", flush=True)

開始使用

LiteRT-LM 以 Python 程式庫的形式提供，您可以從 PyPI 安裝套件：

# Using pip
pip install litert-lm-api

# Using uv
uv pip install litert-lm-api

初始化引擎

Engine 是 API 的進入點。負責處理模型載入和資源管理作業。將其做為內容管理員 (使用 with 陳述式) 可確保資源及時釋出。

注意：初始化引擎可能需要幾秒鐘才能載入模型。

import litert_lm

# Initialize with the model path and optionally specify the backend.
# backend can be Backend.CPU() (default), Backend.GPU() or Backend.NPU().
with litert_lm.Engine(
    "path/to/your/model.litertlm",
    backend=litert_lm.Backend.GPU(),
    # Optional: Pick a writable dir for caching compiled artifacts.
    # cache_dir="/tmp/litert-lm-cache"
) as engine:
    # ... Use the engine to create a conversation ...
    pass

建立對話

Conversation會管理您與模型互動的狀態和記錄。

# Optional: Configure system instruction and initial messages
messages = [litert_lm.Message.system("You are a helpful assistant.")]

# Create the conversation
with engine.create_conversation(messages=messages) as conversation:
    # ... Interact with the conversation ...
    pass

傳送訊息

您可以同步或非同步 (串流) 傳送訊息。

send_message 和 send_message_async 方法接受：

  • str (自動包裝為使用者訊息)。
  • litert_lm.Contents 物件 (適用於多模態輸入內容)。
  • litert_lm.Message 物件 (適用於完整訊息結構)。
  • 類似 JSON 的字典物件，做為提示範本輸入內容。

同步範例：

# Simple string input
response = conversation.send_message("What is the capital of France?")
print(response["content"][0]["text"])

# Or with a Message object
# response = conversation.send_message(litert_lm.Message.user("What is the capital of France?"))

非同步 (串流) 範例：

# sendMessageAsync returns an iterator of response chunks
stream = conversation.send_message_async("Tell me a long story.")
for chunk in stream:
    # Chunks are dictionaries containing pieces of the response
    for item in chunk.get("content", []):
      if item.get("type") == "text":
        print(item["text"], end="", flush=True)
print()

🔴 新功能：多權杖預測 (MTP)

多權杖預測 (MTP) 是一項效能最佳化功能，可大幅提升解碼速度。建議您一律使用 MTP，在 GPU 後端執行所有工作。

如要使用 MTP，請在初始化引擎時啟用推測解碼。

import litert_lm

# Enable MTP by setting enable_speculative_decoding=True
with litert_lm.Engine(
    "path/to/your/model.litertlm",
    backend=litert_lm.Backend.GPU(),
    enable_speculative_decoding=True,
) as engine:
    with engine.create_conversation() as conversation:
        response = conversation.send_message("What is the capital of France?")
        print(response["content"][0]["text"])

多模態

# Initialize with vision and/or audio backends if needed
with litert_lm.Engine(
    "path/to/multimodal_model.litertlm",
    audio_backend=litert_lm.Backend.CPU(),
    vision_backend=litert_lm.Backend.GPU(),
) as engine:
    with engine.create_conversation() as conversation:
        response = conversation.send_message(
            litert_lm.Contents.of(
                "Describe this audio.",
                litert_lm.Content.AudioFile(absolute_path="/path/to/audio.wav"),
            )
        )
        print(response["content"][0]["text"])

定義及使用工具

您可以將 Python 函式定義為工具，供模型自動呼叫。

def add_numbers(a: float, b: float) -> float:
    """Adds two numbers.

    Args:
        a: The first number.
        b: The second number.
    """
    return a + b

# Register the tool in the conversation
tools = [add_numbers]
with engine.create_conversation(tools=tools) as conversation:
    # The model will call add_numbers automatically if it needs to sum values
    response = conversation.send_message("What is 123 + 456?")
    print(response["content"][0]["text"])

LiteRT-LM 會使用函式的 docstring 和型別提示，為模型產生工具結構定義。