 在 ai.google.dev 上查看
|
 在 Google Colab 中執行
|
 在 Kaggle 中執行
|
 在 Vertex AI 中開啟
|
 在 GitHub 上查看來源
|
從 Gemma 3n 開始，您可以直接在提示和工作流程中使用音訊。音訊和口語是豐富的資料來源，可擷取使用者意圖、記錄周遭世界資訊，以及瞭解待解決的特定問題。
本指南將概略說明 Gemma 4 的音訊處理功能，包括自動語音辨識 (ASR)、翻譯和一般語音理解。
這個筆記本將在 T4 GPU 上執行。
安裝 Python 套件
安裝執行 Gemma 模型及發出要求時所需的 Hugging Face 程式庫。
# Install PyTorch & other librariespip install torch accelerate# Install the transformers librarypip install "transformers>=5.10.1"
載入模型
使用 transformers 程式庫，透過 AutoProcessor 和 AutoModelForImageTextToText 類別建立 processor 和 model 的執行個體，如以下程式碼範例所示：
MODEL_ID = "google/gemma-4-E2B-it" # @param ["google/gemma-4-E2B-it","google/gemma-4-E4B-it", "google/gemma-4-12B-it"]
from transformers import pipeline
pipe = pipeline(
task="any-to-any",
model=MODEL_ID,
device_map="auto",
dtype="auto"
)
config.json: 0%| | 0.00/4.95k [00:00<?, ?B/s] model.safetensors: 0%| | 0.00/10.2G [00:00<?, ?B/s] Loading weights: 0%| | 0/1951 [00:00<?, ?it/s] generation_config.json: 0%| | 0.00/208 [00:00<?, ?B/s] processor_config.json: 0%| | 0.00/1.69k [00:00<?, ?B/s] chat_template.jinja: 0%| | 0.00/17.3k [00:00<?, ?B/s] tokenizer_config.json: 0%| | 0.00/2.10k [00:00<?, ?B/s] tokenizer.json: 0%| | 0.00/32.2M [00:00<?, ?B/s]
音訊資料
數位音訊資料的格式和解析度可能各不相同。您可搭配 Gemma 使用的實際音訊格式 (例如 MP3 和 WAV 格式)，取決於您選擇將聲音資料轉換為張量的架構。以下是使用 Gemma 處理音訊資料時，需要注意的具體事項：
- 權杖費用：Gemma 4 每秒音訊為 25 個權杖。(Gemma 3n 為 6.25 個符記)。
- 剪輯片段長度：語音長度上限為 30 秒。
- 音訊聲道：音訊資料會以單一音訊聲道處理。如果使用多聲道音訊 (例如左右聲道)，請考慮移除聲道或將音訊資料合併為單一聲道，以減少資料量。
- 技術編碼：
- 取樣率：16 kHz
- 位元深度：32 位元浮點格式，樣本已在 [-1, 1] 範圍內正規化。
如果打算處理的音訊資料與輸入處理的資料有顯著差異，特別是在聲道、取樣率和位元深度方面，請考慮重新取樣或修剪音訊資料，使其符合模型處理的資料解析度。
音訊編碼
雖然高階程式庫 (例如 Hugging Face AutoProcessor) 通常會自動處理音訊前置處理作業，但有時您可能需要實作自訂編碼。
使用自己的程式碼實作項目編碼音訊資料，以搭配 Gemma 使用時，請按照建議的轉換程序操作。如果您要處理以特定格式編碼的音訊檔案，例如 MP3 或 WAV 編碼資料，請先使用 ffmpeg 等程式庫將這些檔案解碼為樣本。資料解碼後，請將音訊轉換為單聲道、16 kHz 的 float32 波形，範圍為 [-1, 1]。舉例來說，如果您要處理 44.1 kHz 的立體聲帶正負號 16 位元 PCM 整數 WAV 檔案，請按照下列步驟操作：
- 將音訊資料重新取樣為 16 kHz
- 將立體聲降混為單聲道，方法是計算 2 個聲道的平均值
- 從 int16 轉換為 float32，然後除以 32768.0，縮放至 [-1, 1] 範圍
Speech-to-Text
Gemma 4 E2B、E4B 和 12B Unified 經過多語言語音辨識訓練，可將各種語言的音訊輸入內容轉錄為文字。
請使用下列提示結構進行語音辨識 (ASR)。
Transcribe the following speech segment in {LANGUAGE} into {LANGUAGE} text.
Follow these specific instructions for formatting the answer:
* Only output the transcription, with no newlines.
* When transcribing numbers, write the digits, i.e. write 1.7 and not one point seven, and write 3 instead of three.
下列程式碼範例說明如何使用 Hugging Face Transformers，提示模型轉錄音訊檔案中的文字：
from transformers import GenerationConfig
config = GenerationConfig.from_pretrained(MODEL_ID)
config.max_new_tokens = 64
gen_kwargs = dict(generation_config=config)
RESOURCE_URL_PREFIX = "https://raw.githubusercontent.com/google-gemma/cookbook/refs/heads/main/apps/sample-data/"
messages = [
{
"role": "user",
"content": [
{"type": "text", "text": "Transcribe the following speech segment in its original language. Follow these specific instructions for formatting the answer:\n* Only output the transcription, with no newlines.\n* When transcribing numbers, write the digits, i.e. write 1.7 and not one point seven, and write 3 instead of three."},
#{"type": "text", "text": "Transcribe the following speech segment in English into English text. Follow these specific instructions for formatting the answer:\n* Only output the transcription, with no newlines.\n* When transcribing numbers, write the digits, i.e. write 1.7 and not one point seven, and write 3 instead of three."},
{"type": "audio", "audio": f"{RESOURCE_URL_PREFIX}journal1.wav"},
]
}
]
outputs = pipe(messages, return_full_text=False, generate_kwargs=gen_kwargs)
print(outputs[0]['generated_text'])
I woke up early today feeling really fresh the morning light was beautiful and I enjoyed a nice cup of coffee<turn|>
from transformers import GenerationConfig
config = GenerationConfig.from_pretrained(MODEL_ID)
config.max_new_tokens = 1024
gen_kwargs = dict(generation_config=config)
messages = [
{
"role": "user",
"content": [
{"type": "text", "text": "Give me a concise overview of these audio files."},
{"type": "text", "text": "journal1:"},
{"type": "audio", "audio": f"{RESOURCE_URL_PREFIX}journal1.wav"},
{"type": "text", "text": "journal2:"},
{"type": "audio", "audio": f"{RESOURCE_URL_PREFIX}journal2.wav"},
{"type": "text", "text": "journal3:"},
{"type": "audio", "audio": f"{RESOURCE_URL_PREFIX}journal3.wav"},
{"type": "text", "text": "journal4:"},
{"type": "audio", "audio": f"{RESOURCE_URL_PREFIX}journal4.wav"},
{"type": "text", "text": "journal5:"},
{"type": "audio", "audio": f"{RESOURCE_URL_PREFIX}journal5.wav"},
]
}
]
outputs = pipe(messages, return_full_text=False, generate_kwargs=gen_kwargs)
print(outputs[0]['generated_text'])
Here is a concise overview of each audio file: **journal1:** The speaker describes a fresh and peaceful day, enjoying a cup of coffee. **journal2:** The speaker had a perfect day at the park, including a walk and watching cherry blossoms. **journal3:** The speaker finished the day with a good book, feeling grateful for simple moments. **journal4:** The speaker returned from work and noted the beautiful night sky and a clear view from the train. **journal5:** The speaker had a great lunch with an old friend, which was a pleasant way to catch up and made their day. <turn|>
自動語音翻譯
Gemma 4 E2B、E4B 和 12B Unified 經過多語言語音翻譯工作訓練，可直接將語音音訊翻譯成其他語言。
請使用下列提示結構進行自動語音翻譯 (AST)。
Transcribe the following speech segment in {SOURCE_LANGUAGE}, then translate it into {TARGET_LANGUAGE}.
When formatting the answer, first output the transcription in {SOURCE_LANGUAGE}, then one newline, then output the string '{TARGET_LANGUAGE}: ', then the translation in {TARGET_LANGUAGE}.
下列程式碼範例說明如何使用 Hugging Face Transformers，提示模型將語音音訊翻譯成文字：
from transformers import GenerationConfig
config = GenerationConfig.from_pretrained(MODEL_ID)
config.max_new_tokens = 64
gen_kwargs = dict(generation_config=config)
messages = [
{
"role": "user",
"content": [
{"type": "text", "text": "Transcribe the following speech segment in English, then translate it into Korean. When formatting the answer, first output the transcription in English, then one newline, then output the string 'Korean: ', then the translation in Korean."},
{"type": "audio", "audio": "https://ai.google.dev/gemma/docs/audio/roses-are.wav"},
]
}
]
outputs = pipe(messages, return_full_text=False, generate_kwargs=gen_kwargs)
print(outputs[0]['generated_text'])
Roses are red, violets are blue. Korean: 장미는 빨갛고, 제비꽃은 파랗다.<turn|>
自動語音翻譯 / 自動語音辨識
親自試試
pip install ipywebrtc按下圓形按鈕並開始說話，說完後再次按下圓形按鈕。小工具會立即播放錄音內容。
from google.colab import output
output.enable_custom_widget_manager()
from ipywebrtc import AudioRecorder, CameraStream
camera = CameraStream(constraints={'audio': True,'video':False})
recorder = AudioRecorder(stream=camera)
recorder
AudioRecorder(audio=Audio(value=b'', format='webm'), stream=CameraStream(constraints={'audio': True, 'video': …
將 webm 檔案轉換為 PyTorch 可辨識的 wav 格式。
with open('/content/recording.webm', 'wb') as f:
f.write(recorder.audio.value)
!ffmpeg -i /content/recording.webm /content/recording.wav -y
ffmpeg version 4.4.2-0ubuntu0.22.04.1 Copyright (c) 2000-2021 the FFmpeg developers
built with gcc 11 (Ubuntu 11.2.0-19ubuntu1)
configuration: --prefix=/usr --extra-version=0ubuntu0.22.04.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-librsvg --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
libavutil 56. 70.100 / 56. 70.100
libavcodec 58.134.100 / 58.134.100
libavformat 58. 76.100 / 58. 76.100
libavdevice 58. 13.100 / 58. 13.100
libavfilter 7.110.100 / 7.110.100
libswscale 5. 9.100 / 5. 9.100
libswresample 3. 9.100 / 3. 9.100
libpostproc 55. 9.100 / 55. 9.100
Input #0, matroska,webm, from '/content/recording.webm':
Metadata:
encoder : Chrome
Duration: 00:00:03.00, start: 0.000000, bitrate: 132 kb/s
Stream #0:0(eng): Audio: opus, 48000 Hz, mono, fltp (default)
Stream mapping:
Stream #0:0 -> #0:0 (opus (native) -> pcm_s16le (native))
Press [q] to stop, [?] for help
Output #0, wav, to '/content/recording.wav':
Metadata:
ISFT : Lavf58.76.100
Stream #0:0(eng): Audio: pcm_s16le ([1][0][0][0] / 0x0001), 48000 Hz, mono, s16, 768 kb/s (default)
Metadata:
encoder : Lavc58.134.100 pcm_s16le
size= 287kB time=00:00:02.99 bitrate= 783.7kbits/s speed=79.4x
video:0kB audio:287kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.026552%
ASR
from transformers import GenerationConfig
config = GenerationConfig.from_pretrained(MODEL_ID)
config.max_new_tokens = 64
gen_kwargs = dict(generation_config=config)
messages = [{
"role": "user",
"content": [
{"type": "text", "text": "Transcribe the following speech segment in its original language. Follow these specific instructions for formatting the answer:\n* Only output the transcription, with no newlines.\n* When transcribing numbers, write the digits, i.e. write 1.7 and not one point seven, and write 3 instead of three."},
{"type": "audio", "audio": "/content/recording.wav"},
]
}]
outputs = pipe(messages, return_full_text=False, generate_kwargs=gen_kwargs)
print(outputs[0]['generated_text'])
How can I get to the station?<turn|>
AST
messages = [{
"role": "user",
"content": [
{"type": "text", "text": "Transcribe the following speech segment in English, then translate it into Korean. When formatting the answer, first output the transcription in English, then one newline, then output the string 'Korean: ', then the translation in Korean."},
{"type": "audio", "audio": "/content/recording.wav"},
]
}]
outputs = pipe(messages, return_full_text=False, generate_kwargs=gen_kwargs)
print(outputs[0]['generated_text'])
How can I get to the station? Korean: 역에 어떻게 가나요?<turn|>
摘要和後續步驟
在本指南中，您瞭解如何使用 Gemma 4 模型處理音訊。範例說明如何執行語音轉文字 (ASR) 來轉錄口語，以及如何執行自動語音翻譯 (AST) 將語音音訊直接翻譯成另一種語言。您也瞭解如何在筆記本環境中擷取麥克風的音訊以供處理。
如要瞭解詳情，請參閱下列說明文件。