 在 ai.google.dev 上查看
|
 在 Google Colab 中运行
|
 在 Kaggle 中运行
|
 在 Vertex AI 中打开
|
 在 GitHub 上查看源代码
|
从 Gemma 3n 开始，您可以直接在提示和工作流中使用音频。音频和口语是捕获用户意图、记录周围世界的信息以及了解要解决的特定问题的丰富数据来源。
本指南概述了 Gemma 4 的音频处理功能，包括自动语音识别 (ASR)、翻译和一般语音理解。
此笔记本将在 T4 GPU 上运行。
安装 Python 软件包
安装运行 Gemma 模型和发出请求所需的 Hugging Face 库。
# Install PyTorch & other librariespip install torch accelerate# Install the transformers librarypip install "transformers>=5.10.1"
加载模型
使用 transformers 库，通过 AutoProcessor 和 AutoModelForImageTextToText 类创建 processor 和 model 的实例，如以下代码示例所示：
MODEL_ID = "google/gemma-4-E2B-it" # @param ["google/gemma-4-E2B-it","google/gemma-4-E4B-it", "google/gemma-4-12B-it"]
from transformers import pipeline
pipe = pipeline(
task="any-to-any",
model=MODEL_ID,
device_map="auto",
dtype="auto"
)
config.json: 0%| | 0.00/4.95k [00:00<?, ?B/s] model.safetensors: 0%| | 0.00/10.2G [00:00<?, ?B/s] Loading weights: 0%| | 0/1951 [00:00<?, ?it/s] generation_config.json: 0%| | 0.00/208 [00:00<?, ?B/s] processor_config.json: 0%| | 0.00/1.69k [00:00<?, ?B/s] chat_template.jinja: 0%| | 0.00/17.3k [00:00<?, ?B/s] tokenizer_config.json: 0%| | 0.00/2.10k [00:00<?, ?B/s] tokenizer.json: 0%| | 0.00/32.2M [00:00<?, ?B/s]
音频数据
数字音频数据可以有多种格式和分辨率级别。您可以使用 Gemma 的实际音频格式（例如 MP3 和 WAV 格式）取决于您选择将声音数据转换为张量的框架。以下是一些针对准备音频数据以供 Gemma 处理的具体注意事项：
- token 费用： 对于 Gemma 4，每秒音频为 25 个 token。（对于 Gemma 3n，为 6.25 个 token）。
- 剪辑长度： 音频支持的最大长度为 30 秒。
- 音频声道： 音频数据作为单个音频声道进行处理。 如果您使用的是多声道音频（例如左声道和右声道），请考虑通过移除声道或将声音数据合并到单个声道来将数据减少到单个声道。
- 技术编码：
- 采样率： 16kHz
- 位深度： 32 位浮点格式，样本归一化在 [-1, 1] 范围内。
如果您计划处理的音频数据与输入处理（尤其是在声道、采样率和位深度方面）有显著不同，请考虑对音频数据进行重新采样或修剪，以匹配模型处理的数据分辨率。
音频编码
虽然高级库（例如 Hugging Face AutoProcessor）通常会自动处理音频预处理，但您有时可能需要实现自定义编码。
使用您自己的代码实现对音频数据进行编码以供 Gemma 使用时，您应遵循建议的转换流程。如果您处理的是以特定格式编码的音频文件（例如 MP3 或 WAV 编码的数据），则必须先使用 ffmpeg 等库将这些文件解码为样本。数据解码后，将音频转换为 [-1, 1] 范围内的单声道 16 kHz float32 波形。例如，如果您处理的是 44.1 kHz 的立体声带符号 16 位 PCM 整数 WAV 文件，请按以下步骤操作：
- 将音频数据重新采样为 16 kHz
- 通过对 2 个声道求平均值，从立体声缩混为单声道
- 从 int16 转换为 float32，然后除以 32768.0 以缩放至 [-1, 1] 范围
语音转文字
Gemma 4 E2B、E4B 和 12B Unified 经过多语言语音识别训练，可让您将各种语言的音频输入转录为文字。
对于音频语音识别 (ASR) ，请使用以下提示结构。
Transcribe the following speech segment in {LANGUAGE} into {LANGUAGE} text.
Follow these specific instructions for formatting the answer:
* Only output the transcription, with no newlines.
* When transcribing numbers, write the digits, i.e. write 1.7 and not one point seven, and write 3 instead of three.
以下代码示例展示了如何提示模型使用 Hugging Face Transformers 从音频文件转录文字：
from transformers import GenerationConfig
config = GenerationConfig.from_pretrained(MODEL_ID)
config.max_new_tokens = 64
gen_kwargs = dict(generation_config=config)
RESOURCE_URL_PREFIX = "https://raw.githubusercontent.com/google-gemma/cookbook/refs/heads/main/apps/sample-data/"
messages = [
{
"role": "user",
"content": [
{"type": "text", "text": "Transcribe the following speech segment in its original language. Follow these specific instructions for formatting the answer:\n* Only output the transcription, with no newlines.\n* When transcribing numbers, write the digits, i.e. write 1.7 and not one point seven, and write 3 instead of three."},
#{"type": "text", "text": "Transcribe the following speech segment in English into English text. Follow these specific instructions for formatting the answer:\n* Only output the transcription, with no newlines.\n* When transcribing numbers, write the digits, i.e. write 1.7 and not one point seven, and write 3 instead of three."},
{"type": "audio", "audio": f"{RESOURCE_URL_PREFIX}journal1.wav"},
]
}
]
outputs = pipe(messages, return_full_text=False, generate_kwargs=gen_kwargs)
print(outputs[0]['generated_text'])
I woke up early today feeling really fresh the morning light was beautiful and I enjoyed a nice cup of coffee<turn|>
from transformers import GenerationConfig
config = GenerationConfig.from_pretrained(MODEL_ID)
config.max_new_tokens = 1024
gen_kwargs = dict(generation_config=config)
messages = [
{
"role": "user",
"content": [
{"type": "text", "text": "Give me a concise overview of these audio files."},
{"type": "text", "text": "journal1:"},
{"type": "audio", "audio": f"{RESOURCE_URL_PREFIX}journal1.wav"},
{"type": "text", "text": "journal2:"},
{"type": "audio", "audio": f"{RESOURCE_URL_PREFIX}journal2.wav"},
{"type": "text", "text": "journal3:"},
{"type": "audio", "audio": f"{RESOURCE_URL_PREFIX}journal3.wav"},
{"type": "text", "text": "journal4:"},
{"type": "audio", "audio": f"{RESOURCE_URL_PREFIX}journal4.wav"},
{"type": "text", "text": "journal5:"},
{"type": "audio", "audio": f"{RESOURCE_URL_PREFIX}journal5.wav"},
]
}
]
outputs = pipe(messages, return_full_text=False, generate_kwargs=gen_kwargs)
print(outputs[0]['generated_text'])
Here is a concise overview of each audio file: **journal1:** The speaker describes a fresh and peaceful day, enjoying a cup of coffee. **journal2:** The speaker had a perfect day at the park, including a walk and watching cherry blossoms. **journal3:** The speaker finished the day with a good book, feeling grateful for simple moments. **journal4:** The speaker returned from work and noted the beautiful night sky and a clear view from the train. **journal5:** The speaker had a great lunch with an old friend, which was a pleasant way to catch up and made their day. <turn|>
自动语音翻译
Gemma 4 E2B、E4B 和 12B Unified 经过多语言语音翻译任务训练，可让您将口语音频直接翻译成另一种语言。
对于自动语音翻译 (AST) ，请使用以下提示结构。
Transcribe the following speech segment in {SOURCE_LANGUAGE}, then translate it into {TARGET_LANGUAGE}.
When formatting the answer, first output the transcription in {SOURCE_LANGUAGE}, then one newline, then output the string '{TARGET_LANGUAGE}: ', then the translation in {TARGET_LANGUAGE}.
以下代码示例展示了如何提示模型使用 Hugging Face Transformers 将口语音频翻译为文字：
from transformers import GenerationConfig
config = GenerationConfig.from_pretrained(MODEL_ID)
config.max_new_tokens = 64
gen_kwargs = dict(generation_config=config)
messages = [
{
"role": "user",
"content": [
{"type": "text", "text": "Transcribe the following speech segment in English, then translate it into Korean. When formatting the answer, first output the transcription in English, then one newline, then output the string 'Korean: ', then the translation in Korean."},
{"type": "audio", "audio": "https://ai.google.dev/gemma/docs/audio/roses-are.wav"},
]
}
]
outputs = pipe(messages, return_full_text=False, generate_kwargs=gen_kwargs)
print(outputs[0]['generated_text'])
Roses are red, violets are blue. Korean: 장미는 빨갛고, 제비꽃은 파랗다.<turn|>
自动语音翻译 / 自动语音识别
自行试用
pip install ipywebrtc按圆形按钮，然后开始说话。说完后，再次点击圆形按钮。该小部件会立即开始回放其捕获的内容。
from google.colab import output
output.enable_custom_widget_manager()
from ipywebrtc import AudioRecorder, CameraStream
camera = CameraStream(constraints={'audio': True,'video':False})
recorder = AudioRecorder(stream=camera)
recorder
AudioRecorder(audio=Audio(value=b'', format='webm'), stream=CameraStream(constraints={'audio': True, 'video': …
将 webm 文件转换为 PyTorch 可以理解的 wav 格式。
with open('/content/recording.webm', 'wb') as f:
f.write(recorder.audio.value)
!ffmpeg -i /content/recording.webm /content/recording.wav -y
ffmpeg version 4.4.2-0ubuntu0.22.04.1 Copyright (c) 2000-2021 the FFmpeg developers
built with gcc 11 (Ubuntu 11.2.0-19ubuntu1)
configuration: --prefix=/usr --extra-version=0ubuntu0.22.04.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-librsvg --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
libavutil 56. 70.100 / 56. 70.100
libavcodec 58.134.100 / 58.134.100
libavformat 58. 76.100 / 58. 76.100
libavdevice 58. 13.100 / 58. 13.100
libavfilter 7.110.100 / 7.110.100
libswscale 5. 9.100 / 5. 9.100
libswresample 3. 9.100 / 3. 9.100
libpostproc 55. 9.100 / 55. 9.100
Input #0, matroska,webm, from '/content/recording.webm':
Metadata:
encoder : Chrome
Duration: 00:00:03.00, start: 0.000000, bitrate: 132 kb/s
Stream #0:0(eng): Audio: opus, 48000 Hz, mono, fltp (default)
Stream mapping:
Stream #0:0 -> #0:0 (opus (native) -> pcm_s16le (native))
Press [q] to stop, [?] for help
Output #0, wav, to '/content/recording.wav':
Metadata:
ISFT : Lavf58.76.100
Stream #0:0(eng): Audio: pcm_s16le ([1][0][0][0] / 0x0001), 48000 Hz, mono, s16, 768 kb/s (default)
Metadata:
encoder : Lavc58.134.100 pcm_s16le
size= 287kB time=00:00:02.99 bitrate= 783.7kbits/s speed=79.4x
video:0kB audio:287kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.026552%
ASR
from transformers import GenerationConfig
config = GenerationConfig.from_pretrained(MODEL_ID)
config.max_new_tokens = 64
gen_kwargs = dict(generation_config=config)
messages = [{
"role": "user",
"content": [
{"type": "text", "text": "Transcribe the following speech segment in its original language. Follow these specific instructions for formatting the answer:\n* Only output the transcription, with no newlines.\n* When transcribing numbers, write the digits, i.e. write 1.7 and not one point seven, and write 3 instead of three."},
{"type": "audio", "audio": "/content/recording.wav"},
]
}]
outputs = pipe(messages, return_full_text=False, generate_kwargs=gen_kwargs)
print(outputs[0]['generated_text'])
How can I get to the station?<turn|>
AST
messages = [{
"role": "user",
"content": [
{"type": "text", "text": "Transcribe the following speech segment in English, then translate it into Korean. When formatting the answer, first output the transcription in English, then one newline, then output the string 'Korean: ', then the translation in Korean."},
{"type": "audio", "audio": "/content/recording.wav"},
]
}]
outputs = pipe(messages, return_full_text=False, generate_kwargs=gen_kwargs)
print(outputs[0]['generated_text'])
How can I get to the station? Korean: 역에 어떻게 가나요?<turn|>
总结与后续步骤
在本指南中，您学习了如何使用 Gemma 4 模型处理音频。这些示例展示了如何执行语音转文字 (ASR) 以转录口语，以及如何执行自动语音翻译 (AST) 以将口语音频直接翻译成另一种语言。您还了解了如何在笔记本环境中从麦克风捕获音频以进行处理。
如需进一步阅读，请查看以下文档。