The Gemini API supports content generation with images, audio, code, tools, and more. For details on each of these features, read on and check out the task-focused sample code, or read the comprehensive guides.
- Text generation
- Vision
- Audio
- Embeddings
- Long context
- Code execution
- JSON Mode
- Function calling
- System instructions
Method: models.generateContent
Generates a model response given an input GenerateContentRequest. Refer to the text generation guide for detailed usage information. Input capabilities differ between models, including tuned models. Refer to the model guide and tuning guide for details.
Endpoint
posthttps: / /generativelanguage.googleapis.com /v1beta /{model=models /*}:generateContent
Path parameters
modelstring
Required. The name of the Model to use for generating the completion.
Format: models/{model}. It takes the form models/{model}.
Request body
The request body contains data with the following structure:
contents[]object (Content)
Required. The content of the current conversation with the model.
For single-turn queries, this is a single instance. For multi-turn queries like chat, this is a repeated field that contains the conversation history and the latest request.
Optional. A list of Tools the Model may use to generate the next response.
A Tool is a piece of code that enables the system to interact with external systems to perform an action, or set of actions, outside of knowledge and scope of the Model. Supported Tools are Function and codeExecution. Refer to the Function calling and the Code execution guides to learn more.
Optional. Tool configuration for any Tool specified in the request. Refer to the Function calling guide for a usage example.
Optional. A list of unique SafetySetting instances for blocking unsafe content.
This will be enforced on the GenerateContentRequest.contents and GenerateContentResponse.candidates. There should not be more than one setting for each SafetyCategory type. The API will block any contents and responses that fail to meet the thresholds set by these settings. This list overrides the default settings for each SafetyCategory specified in the safetySettings. If there is no SafetySetting for a given SafetyCategory provided in the list, the API will use the default safety setting for that category. Harm categories HARM_CATEGORY_HATE_SPEECH, HARM_CATEGORY_SEXUALLY_EXPLICIT, HARM_CATEGORY_DANGEROUS_CONTENT, HARM_CATEGORY_HARASSMENT, HARM_CATEGORY_CIVIC_INTEGRITY, HARM_CATEGORY_JAILBREAK are supported. Refer to the guide for detailed information on available safety settings. Also refer to the Safety guidance to learn how to incorporate safety considerations in your AI applications.
systemInstructionobject (Content)
Optional. Developer set system instruction(s). Currently, text only.
Optional. Configuration options for model generation and outputs.
cachedContentstring
Optional. The name of the content cached to use as context to serve the prediction. Format: cachedContents/{cachedContent}
serviceTierenum (ServiceTier)
Optional. The service tier of the request.
storeboolean
Optional. Configures the logging behavior for a given request. If set, it takes precedence over the project-level logging config.
Example request
Text
Python
Node.js
Go
Shell
Java
Image
Python
Node.js
Go
Shell
Java
Audio
Python
Node.js
Go
Shell
Video
Python
Node.js
Go
Shell
Python
Go
Shell
Chat
Python
Node.js
Go
Shell
Java
Cache
Python
Node.js
Go
Tuned Model
Python
JSON Mode
Python
Node.js
Go
Shell
Java
Code execution
Python
Go
Java
Function Calling
Python
Go
Node.js
Shell
Java
Generation config
Python
Node.js
Go
Shell
Java
Safety Settings
Python
Node.js
Go
Shell
Java
System Instruction
Python
Node.js
Go
Shell
Java
Response body
If successful, the response body contains an instance of GenerateContentResponse.
Method: models.streamGenerateContent
Generates a streamed response from the model given an input GenerateContentRequest.
Endpoint
posthttps: / /generativelanguage.googleapis.com /v1beta /{model=models /*}:streamGenerateContent
Path parameters
modelstring
Required. The name of the Model to use for generating the completion.
Format: models/{model}. It takes the form models/{model}.
Request body
The request body contains data with the following structure:
contents[]object (Content)
Required. The content of the current conversation with the model.
For single-turn queries, this is a single instance. For multi-turn queries like chat, this is a repeated field that contains the conversation history and the latest request.
Optional. A list of Tools the Model may use to generate the next response.
A Tool is a piece of code that enables the system to interact with external systems to perform an action, or set of actions, outside of knowledge and scope of the Model. Supported Tools are Function and codeExecution. Refer to the Function calling and the Code execution guides to learn more.
Optional. Tool configuration for any Tool specified in the request. Refer to the Function calling guide for a usage example.
Optional. A list of unique SafetySetting instances for blocking unsafe content.
This will be enforced on the GenerateContentRequest.contents and GenerateContentResponse.candidates. There should not be more than one setting for each SafetyCategory type. The API will block any contents and responses that fail to meet the thresholds set by these settings. This list overrides the default settings for each SafetyCategory specified in the safetySettings. If there is no SafetySetting for a given SafetyCategory provided in the list, the API will use the default safety setting for that category. Harm categories HARM_CATEGORY_HATE_SPEECH, HARM_CATEGORY_SEXUALLY_EXPLICIT, HARM_CATEGORY_DANGEROUS_CONTENT, HARM_CATEGORY_HARASSMENT, HARM_CATEGORY_CIVIC_INTEGRITY, HARM_CATEGORY_JAILBREAK are supported. Refer to the guide for detailed information on available safety settings. Also refer to the Safety guidance to learn how to incorporate safety considerations in your AI applications.
systemInstructionobject (Content)
Optional. Developer set system instruction(s). Currently, text only.
Optional. Configuration options for model generation and outputs.
cachedContentstring
Optional. The name of the content cached to use as context to serve the prediction. Format: cachedContents/{cachedContent}
serviceTierenum (ServiceTier)
Optional. The service tier of the request.
storeboolean
Optional. Configures the logging behavior for a given request. If set, it takes precedence over the project-level logging config.
Example request
Text
Python
Node.js
Go
Shell
Java
Image
Python
Node.js
Go
Shell
Java
Audio
Python
Go
Shell
Video
Python
Node.js
Go
Shell
Python
Go
Shell
Chat
Python
Node.js
Go
Shell
Response body
If successful, the response body contains a stream of GenerateContentResponse instances.
GenerateContentResponse
Response from the model supporting multiple candidate responses.
Safety ratings and content filtering are reported for both prompt in GenerateContentResponse.prompt_feedback and for each candidate in finishReason and in safetyRatings. The API: - Returns either all requested candidates or none of them - Returns no candidates at all only if there was something wrong with the prompt (check promptFeedback) - Reports feedback on each candidate in finishReason and safetyRatings.
Candidate responses from the model.
Returns the prompt's feedback related to the content filters.
Output only. Metadata on the generation requests' token usage.
modelVersionstring
Output only. The model version used to generate the response.
responseIdstring
Output only. responseId is used to identify each response.
Output only. The current model status of this model.
| JSON representation |
|---|
{ "candidates": [ { object ( |
PromptFeedback
A set of the feedback metadata the prompt specified in GenerateContentRequest.content.
Optional. If set, the prompt was blocked and no candidates are returned. Rephrase the prompt.
Ratings for safety of the prompt. There is at most one rating per category.
| JSON representation |
|---|
{ "blockReason": enum ( |
BlockReason
Specifies the reason why the prompt was blocked.
| Enums | |
|---|---|
BLOCK_REASON_UNSPECIFIED |
Default value. This value is unused. |
SAFETY |
Prompt was blocked due to safety reasons. Inspect safetyRatings to understand which safety category blocked it. |
OTHER |
Prompt was blocked due to unknown reasons. |
BLOCKLIST |
Prompt was blocked due to the terms which are included from the terminology blocklist. |
PROHIBITED_CONTENT |
Prompt was blocked due to prohibited content. |
IMAGE_SAFETY |
Candidates blocked due to unsafe image generation content. |
UsageMetadata
Metadata on the generation request's token usage.
promptTokenCountinteger
Number of tokens in the prompt. When cachedContent is set, this is still the total effective prompt size meaning this includes the number of tokens in the cached content.
cachedContentTokenCountinteger
Number of tokens in the cached part of the prompt (the cached content)
candidatesTokenCountinteger
Total number of tokens across all the generated response candidates.
toolUsePromptTokenCountinteger
Output only. Number of tokens present in tool-use prompt(s).
thoughtsTokenCountinteger
Output only. Number of tokens of thoughts for thinking models.
totalTokenCountinteger
Total token count for the generation request (prompt + thoughts + response candidates).
Output only. List of modalities that were processed in the request input.
Output only. List of modalities of the cached content in the request input.
Output only. List of modalities that were returned in the response.
Output only. List of modalities that were processed for tool-use request inputs.
serviceTierenum (ServiceTier)
Output only. Service tier of the request.
| JSON representation |
|---|
{ "promptTokenCount": integer, "cachedContentTokenCount": integer, "candidatesTokenCount": integer, "toolUsePromptTokenCount": integer, "thoughtsTokenCount": integer, "totalTokenCount": integer, "promptTokensDetails": [ { object ( |
ModelStatus
The status of the underlying model. This is used to indicate the stage of the underlying model and the retirement time if applicable.
The stage of the underlying model.
The time at which the model will be retired.
Uses RFC 3339, where generated output will always be Z-normalized and use 0, 3, 6 or 9 fractional digits. Offsets other than "Z" are also accepted. Examples: "2014-10-02T15:01:23Z", "2014-10-02T15:01:23.045123456Z" or "2014-10-02T15:01:23+05:30".
messagestring
A message explaining the model status.
| JSON representation |
|---|
{
"modelStage": enum ( |
ModelStage
Defines the stage of the underlying model.
| Enums | |
|---|---|
MODEL_STAGE_UNSPECIFIED |
Unspecified model stage. |
UNSTABLE_EXPERIMENTAL |
The underlying model is subject to lots of tunings. |
EXPERIMENTAL |
Models in this stage are for experimental purposes only. |
PREVIEW |
Models in this stage are more mature than experimental models. |
STABLE |
Models in this stage are considered stable and ready for production use. |
LEGACY |
If the model is on this stage, it means that this model is on the path to deprecation in near future. Only existing customers can use this model. |
DEPRECATED |
Models in this stage are deprecated. These models cannot be used. |
RETIRED |
Models in this stage are retired. These models cannot be used. |
Candidate
- JSON representation
- FinishReason
- GroundingAttribution
- AttributionSourceId
- GroundingPassageId
- SemanticRetrieverChunk
- GroundingMetadata
- SearchEntryPoint
- GroundingChunk
- Web
- Image
- RetrievedContext
- CustomMetadata
- StringList
- Maps
- PlaceAnswerSources
- ReviewSnippet
- GroundingSupport
- Segment
- RetrievalMetadata
- LogprobsResult
- TopCandidates
- Candidate
- UrlContextMetadata
- UrlMetadata
- UrlRetrievalStatus
A response candidate generated from the model.
contentobject (Content)
Output only. Generated content returned from the model.
Optional. Output only. The reason why the model stopped generating tokens.
If empty, the model has not stopped generating tokens.
List of ratings for the safety of a response candidate.
There is at most one rating per category.
Output only. Citation information for model-generated candidate.
This field may be populated with recitation information for any text included in the content. These are passages that are "recited" from copyrighted material in the foundational LLM's training data.
tokenCountinteger
Output only. Token count for this candidate.
Output only. Attribution information for sources that contributed to a grounded answer.
This field is populated for GenerateAnswer calls.
Output only. Grounding metadata for the candidate.
This field is populated for GenerateContent calls.
avgLogprobsnumber
Output only. Average log probability score of the candidate.
Output only. Log-likelihood scores for the response tokens and top tokens
Output only. Metadata related to url context retrieval tool.
indexinteger
Output only. Index of the candidate in the list of response candidates.
finishMessagestring
Optional. Output only. Details the reason why the model stopped generating tokens. This is populated only when finishReason is set.
| JSON representation |
|---|
{ "content": { object ( |
FinishReason
Defines the reason why the model stopped generating tokens.
| Enums | |
|---|---|
FINISH_REASON_UNSPECIFIED |
Default value. This value is unused. |
STOP |
Natural stop point of the model or provided stop sequence. |
MAX_TOKENS |
The maximum number of tokens as specified in the request was reached. |
SAFETY |
The response candidate content was flagged for safety reasons. |
RECITATION |
The response candidate content was flagged for recitation reasons. |
LANGUAGE |
The response candidate content was flagged for using an unsupported language. |
OTHER |
Unknown reason. |
BLOCKLIST |
Token generation stopped because the content contains forbidden terms. |
PROHIBITED_CONTENT |
Token generation stopped for potentially containing prohibited content. |
SPII |
Token generation stopped because the content potentially contains Sensitive Personally Identifiable Information (SPII). |
MALFORMED_FUNCTION_CALL |
The function call generated by the model is invalid. |
IMAGE_SAFETY |
Token generation stopped because generated images contain safety violations. |
IMAGE_PROHIBITED_CONTENT |
Image generation stopped because generated images has other prohibited content. |
IMAGE_OTHER |
Image generation stopped because of other miscellaneous issue. |
NO_IMAGE |
The model was expected to generate an image, but none was generated. |
IMAGE_RECITATION |
Image generation stopped due to recitation. |
UNEXPECTED_TOOL_CALL |
Model generated a tool call but no tools were enabled in the request. |
TOO_MANY_TOOL_CALLS |
Model called too many tools consecutively, thus the system exited execution. |
MISSING_THOUGHT_SIGNATURE |
Request has at least one thought signature missing. |
MALFORMED_RESPONSE |
Finished due to malformed response. |
ESCALATION |
Request was filtered by an escalation rule. |
GroundingAttribution
Attribution for a source that contributed to an answer.
Output only. Identifier for the source contributing to this attribution.
contentobject (Content)
Grounding source content that makes up this attribution.
| JSON representation |
|---|
{ "sourceId": { object ( |
AttributionSourceId
Identifier for the source contributing to this attribution.
sourceUnion type
source can be only one of the following:Identifier for an inline passage.
Identifier for a Chunk fetched via Semantic Retriever.
| JSON representation |
|---|
{ // source "groundingPassage": { object ( |
GroundingPassageId
Identifier for a part within a GroundingPassage.
passageIdstring
Output only. ID of the passage matching the GenerateAnswerRequest's GroundingPassage.id.
partIndexinteger
Output only. Index of the part within the GenerateAnswerRequest's GroundingPassage.content.
| JSON representation |
|---|
{ "passageId": string, "partIndex": integer } |
SemanticRetrieverChunk
Identifier for a Chunk retrieved via Semantic Retriever specified in the GenerateAnswerRequest using SemanticRetrieverConfig.
sourcestring
Output only. Name of the source matching the request's SemanticRetrieverConfig.source. Example: corpora/123 or corpora/123/documents/abc
chunkstring
Output only. Name of the Chunk containing the attributed text. Example: corpora/123/documents/abc/chunks/xyz
| JSON representation |
|---|
{ "source": string, "chunk": string } |
GroundingMetadata
Metadata returned to client when grounding is enabled.
List of supporting references retrieved from specified grounding source. When streaming, this only contains the grounding chunks that have not been included in the grounding metadata of previous responses.
List of grounding support.
webSearchQueries[]string
Web search queries for the following-up web search.
imageSearchQueries[]string
Image search queries used for grounding.
Optional. Google search entry for the following-up web searches.
Metadata related to retrieval in the grounding flow.
googleMapsWidgetContextTokenstring
Optional. Resource name of the Google Maps widget context token that can be used with the PlacesContextElement widget in order to render contextual data. Only populated in the case that grounding with Google Maps is enabled.
| JSON representation |
|---|
{ "groundingChunks": [ { object ( |
SearchEntryPoint
Google search entry point.
renderedContentstring
Optional. Web content snippet that can be embedded in a web page or an app webview.
Optional. Base64 encoded JSON representing array of <search term, search url> tuple.
A base64-encoded string.
| JSON representation |
|---|
{ "renderedContent": string, "sdkBlob": string } |
GroundingChunk
A GroundingChunk represents a segment of supporting evidence that grounds the model's response. It can be a chunk from the web, a retrieved context from a file, or information from Google Maps.
chunk_typeUnion type
chunk_type can be only one of the following:Grounding chunk from the web.
Optional. Grounding chunk from image search.
Optional. Grounding chunk from context retrieved by the file search tool.
Optional. Grounding chunk from Google Maps.
| JSON representation |
|---|
{ // chunk_type "web": { object ( |
Web
Chunk from the web.
uristring
Output only. URI reference of the chunk.
titlestring
Output only. Title of the chunk.
| JSON representation |
|---|
{ "uri": string, "title": string } |
Image
Chunk from image search.
sourceUristring
The web page URI for attribution.
imageUristring
The image asset URL.
titlestring
The title of the web page that the image is from.
domainstring
The root domain of the web page that the image is from, e.g. "example.com".
| JSON representation |
|---|
{ "sourceUri": string, "imageUri": string, "title": string, "domain": string } |
RetrievedContext
Chunk from context retrieved by the file search tool.
Optional. User-provided metadata about the retrieved context.
uristring
Optional. URI reference of the semantic retrieval document.
titlestring
Optional. Title of the document.
textstring
Optional. Text of the chunk.
fileSearchStorestring
Optional. Name of the FileSearchStore containing the document. Example: fileSearchStores/123
pageNumberinteger
Optional. Page number of the retrieved context, if applicable.
mediaIdstring
Optional. The media blob resource name for multimodal file search results. Format: fileSearchStores/{file_search_store_id}/media/{blobId}
| JSON representation |
|---|
{
"customMetadata": [
{
object ( |
CustomMetadata
User provided metadata about the GroundingFact.
keystring
The key of the metadata.
valueUnion type
value can be only one of the following:stringValuestring
Optional. The string value of the metadata.
Optional. A list of string values for the metadata.
numericValuenumber
Optional. The numeric value of the metadata. The expected range for this value depends on the specific key used.
| JSON representation |
|---|
{
"key": string,
// value
"stringValue": string,
"stringListValue": {
object ( |
StringList
A list of string values.
values[]string
The string values of the list.
| JSON representation |
|---|
{ "values": [ string ] } |
Maps
A grounding chunk from Google Maps. A Maps chunk corresponds to a single place.
uristring
URI reference of the place.
titlestring
Title of the place.
textstring
Text description of the place answer.
placeIdstring
The ID of the place, in places/{placeId} format. A user can use this ID to look up that place.
Sources that provide answers about the features of a given place in Google Maps.
| JSON representation |
|---|
{
"uri": string,
"title": string,
"text": string,
"placeId": string,
"placeAnswerSources": {
object ( |
PlaceAnswerSources
Collection of sources that provide answers about the features of a given place in Google Maps. Each PlaceAnswerSources message corresponds to a specific place in Google Maps. The Google Maps tool used these sources in order to answer questions about features of the place (e.g: "does Bar Foo have Wifi" or "is Foo Bar wheelchair accessible?"). Currently we only support review snippets as sources.
Snippets of reviews that are used to generate answers about the features of a given place in Google Maps.
| JSON representation |
|---|
{
"reviewSnippets": [
{
object ( |
ReviewSnippet
Encapsulates a snippet of a user review that answers a question about the features of a specific place in Google Maps.
reviewIdstring
The ID of the review snippet.
googleMapsUristring
A link that corresponds to the user review on Google Maps.
titlestring
Title of the review.
| JSON representation |
|---|
{ "reviewId": string, "googleMapsUri": string, "title": string } |
GroundingSupport
Grounding support.
groundingChunkIndices[]integer
Optional. A list of indices (into 'grounding_chunk' in response.candidate.grounding_metadata) specifying the citations associated with the claim. For instance [1,3,4] means that grounding_chunk[1], grounding_chunk[3], grounding_chunk[4] are the retrieved content attributed to the claim. If the response is streaming, the groundingChunkIndices refer to the indices across all responses. It is the client's responsibility to accumulate the grounding chunks from all responses (while maintaining the same order).
confidenceScores[]number
Optional. Confidence score of the support references. Ranges from 0 to 1. 1 is the most confident. This list must have the same size as the groundingChunkIndices.
renderedParts[]integer
Output only. Indices into the parts field of the candidate's content. These indices specify which rendered parts are associated with this support source.
Segment of the content this support belongs to.
| JSON representation |
|---|
{
"groundingChunkIndices": [
integer
],
"confidenceScores": [
number
],
"renderedParts": [
integer
],
"segment": {
object ( |
Segment
Segment of the content.
partIndexinteger
The index of a Part object within its parent Content object.
startIndexinteger
Start index in the given Part, measured in bytes. Offset from the start of the Part, inclusive, starting at zero.
endIndexinteger
End index in the given Part, measured in bytes. Offset from the start of the Part, exclusive, starting at zero.
textstring
The text corresponding to the segment from the response.
| JSON representation |
|---|
{ "partIndex": integer, "startIndex": integer, "endIndex": integer, "text": string } |
RetrievalMetadata
Metadata related to retrieval in the grounding flow.
googleSearchDynamicRetrievalScorenumber
Optional. Score indicating how likely information from google search could help answer the prompt. The score is in the range [0, 1], where 0 is the least likely and 1 is the most likely. This score is only populated when google search grounding and dynamic retrieval is enabled. It will be compared to the threshold to determine whether to trigger google search.
| JSON representation |
|---|
{ "googleSearchDynamicRetrievalScore": number } |
LogprobsResult
Logprobs Result
Length = total number of decoding steps.
Length = total number of decoding steps. The chosen candidates may or may not be in topCandidates.
logProbabilitySumnumber
Sum of log probabilities for all tokens.
| JSON representation |
|---|
{ "topCandidates": [ { object ( |
TopCandidates
Candidates with top log probabilities at each decoding step.
Sorted by log probability in descending order.
| JSON representation |
|---|
{
"candidates": [
{
object ( |
Candidate
Candidate for the logprobs token and score.
tokenstring
The candidate’s token string value.
tokenIdinteger
The candidate’s token id value.
logProbabilitynumber
The candidate's log probability.
| JSON representation |
|---|
{ "token": string, "tokenId": integer, "logProbability": number } |
UrlContextMetadata
Metadata related to url context retrieval tool.
List of url context.
| JSON representation |
|---|
{
"urlMetadata": [
{
object ( |
UrlMetadata
Context of the a single url retrieval.
retrievedUrlstring
Retrieved url by the tool.
Status of the url retrieval.
| JSON representation |
|---|
{
"retrievedUrl": string,
"urlRetrievalStatus": enum ( |
UrlRetrievalStatus
Status of the url retrieval.
| Enums | |
|---|---|
URL_RETRIEVAL_STATUS_UNSPECIFIED |
Default value. This value is unused. |
URL_RETRIEVAL_STATUS_SUCCESS |
Url retrieval is successful. |
URL_RETRIEVAL_STATUS_ERROR |
Url retrieval is failed due to error. |
URL_RETRIEVAL_STATUS_PAYWALL |
Url retrieval is failed because the content is behind paywall. |
URL_RETRIEVAL_STATUS_UNSAFE |
Url retrieval is failed because the content is unsafe. |
CitationMetadata
A collection of source attributions for a piece of content.
Citations to sources for a specific response.
| JSON representation |
|---|
{
"citationSources": [
{
object ( |
CitationSource
A citation to a source for a portion of a specific response.
startIndexinteger
Optional. Start of segment of the response that is attributed to this source.
Index indicates the start of the segment, measured in bytes.
endIndexinteger
Optional. End of the attributed segment, exclusive.
uristring
Optional. URI that is attributed as a source for a portion of the text.
licensestring
Optional. License for the GitHub project that is attributed as a source for segment.
License info is required for code citations.
| JSON representation |
|---|
{ "startIndex": integer, "endIndex": integer, "uri": string, "license": string } |
HarmCategory
Harm categories that can be detected in user input and model responses.
| Enums | |
|---|---|
HARM_CATEGORY_UNSPECIFIED |
Default value. This value is unused. |
HARM_CATEGORY_HATE_SPEECH |
Content that promotes violence or incites hatred against individuals or groups based on certain attributes. |
HARM_CATEGORY_DANGEROUS_CONTENT |
Content that promotes, facilitates, or enables dangerous activities. |
HARM_CATEGORY_HARASSMENT |
Abusive, threatening, or content intended to bully, torment, or ridicule. |
HARM_CATEGORY_SEXUALLY_EXPLICIT |
Content that contains sexually explicit material. |
HARM_CATEGORY_CIVIC_INTEGRITY |
Deprecated: Election filter is not longer supported. The harm category is civic integrity. |
HARM_CATEGORY_IMAGE_HATE |
Images that contain hate speech. |
HARM_CATEGORY_IMAGE_DANGEROUS_CONTENT |
Images that contain dangerous content. |
HARM_CATEGORY_IMAGE_HARASSMENT |
Images that contain harassment. |
HARM_CATEGORY_IMAGE_SEXUALLY_EXPLICIT |
Images that contain sexually explicit content. |
HARM_CATEGORY_JAILBREAK |
Prompts designed to bypass safety filters. |
ModalityTokenCount
Represents token counting info for a single modality.
The modality associated with this token count.
tokenCountinteger
Number of tokens.
| JSON representation |
|---|
{
"modality": enum ( |
Modality
Content Part modality
| Enums | |
|---|---|
MODALITY_UNSPECIFIED |
Unspecified modality. |
TEXT |
Plain text. |
IMAGE |
Image. |
VIDEO |
Video. |
AUDIO |
Audio. |
DOCUMENT |
Document, e.g. PDF. |
SafetyRating
Safety rating for a piece of content.
The safety rating contains the category of harm and the harm probability level in that category for a piece of content. Content is classified for safety across a number of harm categories and the probability of the harm classification is included here.
categoryenum (HarmCategory)
Required. The category for this rating.
Required. The probability of harm for this content.
blockedboolean
Was this content blocked because of this rating?
| JSON representation |
|---|
{ "category": enum ( |
HarmProbability
The probability that a piece of content is harmful.
The classification system gives the probability of the content being unsafe. This does not indicate the severity of harm for a piece of content.
| Enums | |
|---|---|
HARM_PROBABILITY_UNSPECIFIED |
Probability is unspecified. |
NEGLIGIBLE |
Content has a negligible chance of being unsafe. |
LOW |
Content has a low chance of being unsafe. |
MEDIUM |
Content has a medium chance of being unsafe. |
HIGH |
Content has a high chance of being unsafe. |
SafetySetting
Safety setting, affecting the safety-blocking behavior.
Passing a safety setting for a category changes the allowed probability that content is blocked.
categoryenum (HarmCategory)
Required. The category for this setting.
Required. Controls the probability threshold at which harm is blocked.
| JSON representation |
|---|
{ "category": enum ( |
HarmBlockThreshold
Block at and beyond a specified harm probability.
| Enums | |
|---|---|
HARM_BLOCK_THRESHOLD_UNSPECIFIED |
Threshold is unspecified. |
BLOCK_LOW_AND_ABOVE |
Content with NEGLIGIBLE will be allowed. |
BLOCK_MEDIUM_AND_ABOVE |
Content with NEGLIGIBLE and LOW will be allowed. |
BLOCK_ONLY_HIGH |
Content with NEGLIGIBLE, LOW, and MEDIUM will be allowed. |
BLOCK_NONE |
All content will be allowed. |
OFF |
Turn off the safety filter. |
ServiceTier
The service tier of the interaction.
| Enums | |
|---|---|
SERVICE_TIER_UNSPECIFIED |
Default service tier, which is standard. |
SERVICE_TIER_FLEX |
Flex service tier. |
SERVICE_TIER_STANDARD |
Standard service tier. |
SERVICE_TIER_PRIORITY |
Priority service tier. |
AllowedTools
The configuration for allowed tools.
The mode of the tool choice.
tools[]string
The names of the allowed tools.
| JSON representation |
|---|
{
"mode": enum ( |
Annotation
Citation information for model-generated content.
startIndexinteger
Start of segment of the response that is attributed to this source.
Index indicates the start of the segment, measured in bytes.
endIndexinteger
End of the attributed segment, exclusive.
typeUnion type
type can be only one of the following:A URL citation annotation.
A file citation annotation.
A place citation annotation.
| JSON representation |
|---|
{ "startIndex": integer, "endIndex": integer, // type "urlCitation": { object ( |
UrlCitation
A URL citation annotation.
urlstring
The URL.
titlestring
The title of the URL.
| JSON representation |
|---|
{ "url": string, "title": string } |
FileCitation
A file citation annotation.
documentUristring
The URI of the file.
fileNamestring
The name of the file.
sourcestring
Source attributed for a portion of the text.
User provided metadata about the retrieved context.
pageNumberinteger
Page number of the cited document, if applicable.
mediaIdstring
Media ID in-case of image citations, if applicable.
| JSON representation |
|---|
{
"documentUri": string,
"fileName": string,
"source": string,
"customMetadata": {
object ( |
PlaceCitation
A place citation annotation.
placeIdstring
The ID of the place, in places/{placeId} format.
namestring
Title of the place.
urlstring
URI reference of the place.
Snippets of reviews that are used to generate answers about the features of a given place in Google Maps.
| JSON representation |
|---|
{
"placeId": string,
"name": string,
"url": string,
"reviewSnippets": [
{
object ( |
AspectRatio
Supported aspect ratios for image output.
| Enums | |
|---|---|
ASPECT_RATIO_UNSPECIFIED |
Default value. This value is unused. |
ASPECT_RATIO_ONE_BY_ONE |
1:1 aspect ratio. |
ASPECT_RATIO_TWO_BY_THREE |
2:3 aspect ratio. |
ASPECT_RATIO_THREE_BY_TWO |
3:2 aspect ratio. |
ASPECT_RATIO_THREE_BY_FOUR |
3:4 aspect ratio. |
ASPECT_RATIO_FOUR_BY_THREE |
4:3 aspect ratio. |
ASPECT_RATIO_FOUR_BY_FIVE |
4:5 aspect ratio. |
ASPECT_RATIO_FIVE_BY_FOUR |
5:4 aspect ratio. |
ASPECT_RATIO_NINE_BY_SIXTEEN |
9:16 aspect ratio. |
ASPECT_RATIO_SIXTEEN_BY_NINE |
16:9 aspect ratio. |
ASPECT_RATIO_TWENTY_ONE_BY_NINE |
21:9 aspect ratio. |
ASPECT_RATIO_ONE_BY_EIGHT |
1:8 aspect ratio. |
ASPECT_RATIO_EIGHT_BY_ONE |
8:1 aspect ratio. |
ASPECT_RATIO_ONE_BY_FOUR |
1:4 aspect ratio. |
ASPECT_RATIO_FOUR_BY_ONE |
4:1 aspect ratio. |
AudioResponseFormat
Configuration for audio output format.
mimeTypeenum (MimeType)
The MIME type of the audio output.
The delivery mode for the audio output.
sampleRateinteger
Sample rate in Hz.
bitRateinteger
Bit rate in bits per second (bps). Only applicable for compressed formats (MP3, Opus).
| JSON representation |
|---|
{ "mimeType": enum ( |
CodeExecutionCallStep
Code execution call step.
Required. The arguments to pass to the code execution.
| JSON representation |
|---|
{
"arguments": {
object ( |
CodeExecutionCallStepArguments
The arguments to pass to the code execution.
languageenum (Language)
Programming language of the code.
codestring
The code to be executed.
| JSON representation |
|---|
{
"language": enum ( |
CodeExecutionResultStep
Code execution result step.
resultstring
Required. The output of the code execution.
isErrorboolean
Whether the code execution resulted in an error.
| JSON representation |
|---|
{ "result": string, "isError": boolean } |
ComputerUse
A tool that can be used by the model to interact with the computer.
The environment being operated.
excludedPredefinedFunctions[]string
The list of predefined functions that are excluded from the model call.
enablePromptInjectionDetectionboolean
Whether enable the prompt injection detection check on computer-use request.
Optional. Disabled safety policies for computer use.
| JSON representation |
|---|
{ "environment": enum ( |
Content
- JSON representation
- TextContent
- ImageContent
- AudioContent
- DocumentContent
- VideoContent
- ThoughtContent
- ThoughtSummaryContent
- ToolCallContent
- FunctionCallContent
- CodeExecutionCallContent
- CodeExecutionCallArguments
- UrlContextCallContent
- UrlContextCallArguments
- McpServerToolCallContent
- GoogleSearchCallContent
- GoogleSearchCallArguments
- FileSearchCallContent
- GoogleMapsCallContent
- GoogleMapsCallArguments
- ToolResultContent
- FunctionResultContent
- FunctionResultSubcontentList
- FunctionResultSubcontent
- CodeExecutionResultContent
- UrlContextResultContent
- UrlContextResult
- GoogleSearchResultContent
- GoogleSearchResult
- McpServerToolResultContent
- FileSearchResultContent
- FileSearchResult
- GoogleMapsResultContent
- GoogleMapsResult
- Places
The content of the response.
typeUnion type
type can be only one of the following:| JSON representation |
|---|
{ // type "text": { object ( |
TextContent
A text content block.
textstring
Required. The text content.
Citation information for model-generated content.
| JSON representation |
|---|
{
"text": string,
"annotations": [
{
object ( |
ImageContent
An image content block.
mimeTypeenum (MimeType)
The mime type of the image.
The resolution of the media.
data_or_uriUnion type
data_or_uri can be only one of the following:The image content.
A base64-encoded string.
uristring
The URI of the image.
| JSON representation |
|---|
{ "mimeType": enum ( |
AudioContent
An audio content block.
The mime type of the audio.
channelsinteger
The number of audio channels.
sampleRateinteger
The sample rate of the audio.
data_or_uriUnion type
data_or_uri can be only one of the following:The audio content.
A base64-encoded string.
uristring
The URI of the audio.
| JSON representation |
|---|
{
"mimeType": enum ( |
DocumentContent
A document content block.
mimeTypeenum (MimeType)
The mime type of the document.
data_or_uriUnion type
data_or_uri can be only one of the following:The document content.
A base64-encoded string.
uristring
The URI of the document.
| JSON representation |
|---|
{
"mimeType": enum ( |
VideoContent
A video content block.
mimeTypeenum (MimeType)
The mime type of the video.
The resolution of the media.
data_or_uriUnion type
data_or_uri can be only one of the following:The video content.
A base64-encoded string.
uristring
The URI of the video.
| JSON representation |
|---|
{ "mimeType": enum ( |
ThoughtContent
A thought content block.
Signature to match the backend source to be part of the generation.
A base64-encoded string.
A summary of the thought.
| JSON representation |
|---|
{
"signature": string,
"summary": [
{
object ( |
ThoughtSummaryContent
typeUnion type
| JSON representation |
|---|
{ // type "text": { object ( |
ToolCallContent
Tool call content.
idstring
Required. A unique ID for this specific tool call.
A signature hash for backend validation.
A base64-encoded string.
typeUnion type
type can be only one of the following:| JSON representation |
|---|
{ "id": string, "signature": string, // type "functionCall": { object ( |
FunctionCallContent
A function tool call content block.
namestring
Required. The name of the tool to call.
Required. The arguments to pass to the function.
| JSON representation |
|---|
{
"name": string,
"arguments": {
object ( |
CodeExecutionCallContent
Code execution content.
Required. The arguments to pass to the code execution.
| JSON representation |
|---|
{
"arguments": {
object ( |
CodeExecutionCallArguments
The arguments to pass to the code execution.
Programming language of the code.
codestring
The code to be executed.
| JSON representation |
|---|
{
"language": enum ( |
UrlContextCallContent
URL context content.
Required. The arguments to pass to the URL context.
| JSON representation |
|---|
{
"arguments": {
object ( |
UrlContextCallArguments
The arguments to pass to the URL context.
urls[]string
The URLs to fetch.
| JSON representation |
|---|
{ "urls": [ string ] } |
McpServerToolCallContent
MCPServer tool call content.
namestring
Required. The name of the tool which was called.
serverNamestring
Required. The name of the used MCP server.
Required. The JSON object of arguments for the function.
| JSON representation |
|---|
{
"name": string,
"serverName": string,
"arguments": {
object ( |
GoogleSearchCallContent
Google Search content.
Required. The arguments to pass to Google Search.
The type of search grounding enabled.
| JSON representation |
|---|
{ "arguments": { object ( |
GoogleSearchCallArguments
The arguments to pass to Google Search.
queries[]string
Web search queries for the following-up web search.
| JSON representation |
|---|
{ "queries": [ string ] } |
FileSearchCallContent
This type has no fields.
File Search content.
GoogleMapsCallContent
Google Maps content.
The arguments to pass to the Google Maps tool.
| JSON representation |
|---|
{
"arguments": {
object ( |
GoogleMapsCallArguments
The arguments to pass to the Google Maps tool.
queries[]string
The queries to be executed.
| JSON representation |
|---|
{ "queries": [ string ] } |
ToolResultContent
Tool result content.
callIdstring
Required. ID to match the ID from the function call block.
A signature hash for backend validation.
A base64-encoded string.
typeUnion type
type can be only one of the following:| JSON representation |
|---|
{ "callId": string, "signature": string, // type "functionResult": { object ( |
FunctionResultContent
A function tool result content block.
namestring
The name of the tool that was called.
isErrorboolean
Whether the tool call resulted in an error.
resultUnion type
result can be only one of the following:stringResultstring
| JSON representation |
|---|
{ "name": string, "isError": boolean, // result "structResult": { object ( |
FunctionResultSubcontentList
| JSON representation |
|---|
{
"contents": [
{
object ( |
FunctionResultSubcontent
typeUnion type
| JSON representation |
|---|
{ // type "text": { object ( |
CodeExecutionResultContent
Code execution result content.
resultstring
Required. The output of the code execution.
isErrorboolean
Whether the code execution resulted in an error.
| JSON representation |
|---|
{ "result": string, "isError": boolean } |
UrlContextResultContent
URL context result content.
Required. The results of the URL context.
isErrorboolean
Whether the URL context resulted in an error.
| JSON representation |
|---|
{
"result": [
{
object ( |
UrlContextResult
The result of the URL context.
urlstring
The URL that was fetched.
statusenum (Status)
The status of the URL retrieval.
| JSON representation |
|---|
{
"url": string,
"status": enum ( |
GoogleSearchResultContent
Google Search result content.
Required. The results of the Google Search.
isErrorboolean
Whether the Google Search resulted in an error.
| JSON representation |
|---|
{
"result": [
{
object ( |
GoogleSearchResult
The result of the Google Search.
searchSuggestionsstring
Web content snippet that can be embedded in a web page or an app webview.
| JSON representation |
|---|
{ "searchSuggestions": string } |
McpServerToolResultContent
MCPServer tool result content.
namestring
Name of the tool which is called for this specific tool call.
serverNamestring
The name of the used MCP server.
resultUnion type
result can be only one of the following:stringResultstring
| JSON representation |
|---|
{ "name": string, "serverName": string, // result "structResult": { object ( |
FileSearchResultContent
File Search result content.
Optional. The results of the File Search.
| JSON representation |
|---|
{
"result": [
{
object ( |
FileSearchResult
This type has no fields.
The result of the File Search.
GoogleMapsResultContent
Google Maps result content.
Required. The results of the Google Maps.
| JSON representation |
|---|
{
"result": [
{
object ( |
GoogleMapsResult
The result of the Google Maps.
The places that were found.
widgetContextTokenstring
Resource name of the Google Maps widget context token.
| JSON representation |
|---|
{
"places": [
{
object ( |
Places
placeIdstring
The ID of the place, in places/{placeId} format.
namestring
Title of the place.
urlstring
URI reference of the place.
Snippets of reviews that are used to generate answers about the features of a given place in Google Maps.
| JSON representation |
|---|
{
"placeId": string,
"name": string,
"url": string,
"reviewSnippets": [
{
object ( |
CreateInteractionRequest
- JSON representation
- Interaction
- TurnList
- Turn
- StepList
- Step
- ThoughtStep
- ToolCallStep
- FunctionCallStep
- UrlContextCallStep
- UrlContextCallStepArguments
- McpServerToolCallStep
- GoogleSearchCallStep
- GoogleSearchCallStepArguments
- FileSearchCallStep
- GoogleMapsCallStep
- GoogleMapsCallStepArguments
- ToolResultStep
- FunctionResultStep
- UrlContextResultStep
- UrlContextResultItem
- GoogleSearchResultStep
- GoogleSearchResultItem
- McpServerToolResultStep
- FileSearchResultStep
- GoogleMapsResultStep
- GoogleMapsResultItem
- GoogleMapsResultPlaces
- UserInputStep
- ModelOutputStep
- ResponseFormatList
- ResponseFormat
- TextResponseFormat
- ImageResponseFormat
- VideoResponseFormat
- ModelInteraction
- GenerationConfig
- ToolChoiceConfig
- SpeechConfig
- ImageConfig
- VideoConfig
- EnvironmentConfig
- EnvironmentNetworkEgressAllowlist
- EgressRule
- Source
- LocalEnvironmentConfig
- Tool
- Function
- UrlContext
- McpServer
- GoogleSearch
- FileSearch
- GoogleMaps
- Usage
- ModalityTokens
- GroundingToolCount
- WebhookConfig
- SafetySetting
Configuration parameters for creating an interaction.
streamboolean
Input only. Whether the interaction will be streamed.
storeboolean
Input only. Whether to store the response and request for later retrieval.
The interaction to create.
backgroundboolean
Input only. Whether to run the model interaction in the background.
| JSON representation |
|---|
{
"stream": boolean,
"store": boolean,
"interaction": {
object ( |
Interaction
Response for InteractionService.CreateInteraction.
idstring
Required. Output only. A unique identifier for the interaction completion.
Required. Output only. The status of the interaction.
createdstring
Required. Output only. The time at which the response was created in ISO 8601 format (YYYY-MM-DDThh:mm:ssZ).
updatedstring
Required. Output only. The time at which the response was last updated in ISO 8601 format (YYYY-MM-DDThh:mm:ssZ).
role
(deprecated)string
Output only. The role of the interaction.
Output only. Responses from the model.
systemInstructionstring
System instruction for the interaction.
A list of tool declarations the model may call during interaction.
Output only. Statistics on the interaction request's token usage.
The requested modalities of the response (TEXT, IMAGE, AUDIO).
responseMimeType
(deprecated)string
The mime type of the response. This is required if responseFormat is set.
previousInteractionIdstring
The ID of the previous interaction, if any.
environmentIdstring
Output only. The environment ID for the interaction. Only populated if environment config is set in the request.
The service tier for the interaction.
Optional. Webhook configuration for receiving notifications when the interaction completes.
Required. Output only. The steps that make up the interaction.
inputUnion type
input can be only one of the following:The inputs for the interaction.
stringContentstring
A string input for the interaction, it will be processed as a single text input.
The turns for the interaction.
Input only. The steps for the interaction.
The content for the interaction.
response_format_configUnion type
response_format_config can be only one of the following:Enforces that the generated response is a JSON object that complies with the JSON schema specified in this field.
request_typeUnion type
request_type can be only one of the following:Interaction for generating the completion using models.
agentInteractionobject (AgentInteraction)
Interaction for generating the completion using agents.
environmentUnion type
environment can be only one of the following:envIdstring
The environment ID for the interaction. Can be 'remote' for default environment.
The agent's environment lives on the client connection: its built-in environment operations (filesystem ops and running commands) are yielded to the client to execute, instead of running in a server-managed sandbox. Mutually exclusive with remoteEnvironment. (Independent of any client-declared function tools, which are always executed on the client regardless of this field.)
| JSON representation |
|---|
{ "id": string, "status": enum ( |
TurnList
| JSON representation |
|---|
{
"turns": [
{
object ( |
Turn
rolestring
The originator of this turn. Must be user for input or model for model output.
contentUnion type
content can be only one of the following:The content of the turn. An array of Content objects.
contentStringstring
The content of the turn. A single string.
| JSON representation |
|---|
{
"role": string,
// content
"contentList": {
object ( |
StepList
A list of Steps.
The steps of the list.
| JSON representation |
|---|
{
"steps": [
{
object ( |
Step
A step in the interaction.
typeUnion type
type can be only one of the following:DO NOT USE -- These are for 3P JSON only
text
(deprecated)object (LegacyTextContent)
image
(deprecated)object (LegacyImageContent)
audio
(deprecated)object (LegacyAudioContent)
document
(deprecated)object (LegacyDocumentContent)
video
(deprecated)object (LegacyVideoContent)
| JSON representation |
|---|
{ // type "thought": { object ( |
ThoughtStep
A thought step.
A signature hash for backend validation.
A base64-encoded string.
A summary of the thought.
| JSON representation |
|---|
{
"signature": string,
"summary": [
{
object ( |
ToolCallStep
Tool call step.
idstring
Required. A unique ID for this specific tool call.
A signature hash for backend validation.
A base64-encoded string.
typeUnion type
type can be only one of the following:retrievalCallobject (RetrievalCallStep)
| JSON representation |
|---|
{ "id": string, "signature": string, // type "functionCall": { object ( |
FunctionCallStep
A function tool call step.
namestring
Required. The name of the tool to call.
Required. The arguments to pass to the function.
| JSON representation |
|---|
{
"name": string,
"arguments": {
object ( |
UrlContextCallStep
URL context call step.
Required. The arguments to pass to the URL context.
| JSON representation |
|---|
{
"arguments": {
object ( |
UrlContextCallStepArguments
The arguments to pass to the URL context.
urls[]string
The URLs to fetch.
| JSON representation |
|---|
{ "urls": [ string ] } |
McpServerToolCallStep
MCPServer tool call step.
namestring
Required. The name of the tool which was called.
serverNamestring
Required. The name of the used MCP server.
Required. The JSON object of arguments for the function.
| JSON representation |
|---|
{
"name": string,
"serverName": string,
"arguments": {
object ( |
GoogleSearchCallStep
Google Search call step.
Required. The arguments to pass to Google Search.
The type of search grounding enabled.
| JSON representation |
|---|
{ "arguments": { object ( |
GoogleSearchCallStepArguments
The arguments to pass to Google Search.
queries[]string
Web search queries for the following-up web search.
| JSON representation |
|---|
{ "queries": [ string ] } |
FileSearchCallStep
This type has no fields.
File Search call step.
GoogleMapsCallStep
Google Maps call step.
The arguments to pass to the Google Maps tool.
| JSON representation |
|---|
{
"arguments": {
object ( |
GoogleMapsCallStepArguments
The arguments to pass to the Google Maps tool.
queries[]string
The queries to be executed.
| JSON representation |
|---|
{ "queries": [ string ] } |
ToolResultStep
Tool result step.
callIdstring
Required. ID to match the ID from the function call block.
A signature hash for backend validation.
A base64-encoded string.
typeUnion type
type can be only one of the following:retrievalResultobject (RetrievalResultStep)
| JSON representation |
|---|
{ "callId": string, "signature": string, // type "functionResult": { object ( |
FunctionResultStep
Result of a function tool call.
namestring
The name of the tool that was called.
isErrorboolean
Whether the tool call resulted in an error.
Required. The result of the tool call.
| JSON representation |
|---|
{
"name": string,
"isError": boolean,
"result": {
object ( |
UrlContextResultStep
URL context result step.
Required. The results of the URL context.
isErrorboolean
Whether the URL context resulted in an error.
| JSON representation |
|---|
{
"result": [
{
object ( |
UrlContextResultItem
The result of the URL context.
urlstring
The URL that was fetched.
statusenum (Status)
The status of the URL retrieval.
| JSON representation |
|---|
{
"url": string,
"status": enum ( |
GoogleSearchResultStep
Google Search result step.
Required. The results of the Google Search.
isErrorboolean
Whether the Google Search resulted in an error.
| JSON representation |
|---|
{
"result": [
{
object ( |
GoogleSearchResultItem
The result of the Google Search.
searchSuggestionsstring
Web content snippet that can be embedded in a web page or an app webview.
| JSON representation |
|---|
{ "searchSuggestions": string } |
McpServerToolResultStep
MCPServer tool result step.
namestring
Name of the tool which is called for this specific tool call.
serverNamestring
The name of the used MCP server.
Required. The output from the MCP server call. Can be simple text or rich content.
| JSON representation |
|---|
{
"name": string,
"serverName": string,
"result": {
object ( |
FileSearchResultStep
This type has no fields.
File Search result step.
GoogleMapsResultStep
Google Maps result step.
| JSON representation |
|---|
{
"result": [
{
object ( |
GoogleMapsResultItem
The result of the Google Maps.
widgetContextTokenstring
| JSON representation |
|---|
{
"places": [
{
object ( |
GoogleMapsResultPlaces
placeIdstring
namestring
urlstring
| JSON representation |
|---|
{
"placeId": string,
"name": string,
"url": string,
"reviewSnippets": [
{
object ( |
UserInputStep
Input provided by the user.
contentUnion type
content can be only one of the following:The content of the step. An array of Content objects.
contentStringstring
The content of the step. A single string.
| JSON representation |
|---|
{
// content
"contentList": {
object ( |
ModelOutputStep
Output generated by the model.
| JSON representation |
|---|
{
"content": [
{
object ( |
ResponseFormatList
| JSON representation |
|---|
{
"responseFormats": [
{
object ( |
ResponseFormat
typeUnion type
type can be only one of the following:Multi-discriminator values is already enabled in GAOS
| JSON representation |
|---|
{ // type "audio": { object ( |
TextResponseFormat
Configuration for text output format.
mimeTypeenum (MimeType)
The MIME type of the text output.
The JSON schema that the output should conform to. Only applicable when mimeType is application/json.
| JSON representation |
|---|
{ "mimeType": enum ( |
ImageResponseFormat
Configuration for image output format.
mimeTypeenum (MimeType)
The MIME type of the image output.
deliveryenum (Delivery)
The delivery mode for the image output.
The aspect ratio for the image output.
The size of the image output.
| JSON representation |
|---|
{ "mimeType": enum ( |
VideoResponseFormat
Configuration for video output format.
deliveryenum (Delivery)
The delivery mode for the video output.
aspectRatioenum (AspectRatio)
The aspect ratio for the video output.
The duration for the video output.
A duration in seconds with up to nine fractional digits, ending with 's'. Example: "3.5s".
| JSON representation |
|---|
{ "delivery": enum ( |
ModelInteraction
Interaction for generating the completion using models.
modelstring
The name of the Model used for generating the completion.
Input only. Configuration parameters for the model interaction.
| JSON representation |
|---|
{
"model": string,
"generationConfig": {
object ( |
GenerationConfig
Configuration parameters for model interactions.
temperaturenumber
Controls the randomness of the output.
topPnumber
The maximum cumulative probability of tokens to consider when sampling.
seedinteger
Seed used in decoding for reproducibility.
stopSequences[]string
A list of character sequences that will stop output interaction.
The level of thought tokens that the model should generate.
Whether to include thought summaries in the response.
maxOutputTokensinteger
The maximum number of tokens to include in the response.
Configuration for speech interaction.
Configuration for image interaction.
Configuration for video generation.
tool_choiceUnion type
tool_choice can be only one of the following:The mode of the tool choice.
The config for the tool choice.
| JSON representation |
|---|
{ "temperature": number, "topP": number, "seed": integer, "stopSequences": [ string ], "thinkingLevel": enum ( |
ToolChoiceConfig
The tool choice configuration containing allowed tools.
The allowed tools.
| JSON representation |
|---|
{
"allowedTools": {
object ( |
SpeechConfig
The configuration for speech interaction.
voicestring
The voice of the speaker.
languagestring
The language of the speech.
speakerstring
The speaker's name, it should match the speaker name given in the prompt.
| JSON representation |
|---|
{ "voice": string, "language": string, "speaker": string } |
ImageConfig
The configuration for image interaction.
aspectRatiostring
The aspect ratio of the image to generate. Supported aspect ratios: 1:1, 2:3, 3:2, 3:4, 4:3, 9:16, 16:9, 21:9.
If not specified, the model will choose a default aspect ratio based on any reference images provided.
imageSizestring
Specifies the size of generated images. Supported values are 1K, 2K, 4K. If not specified, the model will use default value 1K.
| JSON representation |
|---|
{ "aspectRatio": string, "imageSize": string } |
VideoConfig
Configuration options for video generation.
Optional task mode for video generation. If not specified, the model automatically determines the appropriate mode based on the provided text prompt and input media.
| JSON representation |
|---|
{
"task": enum ( |
EnvironmentConfig
Configuration for a custom environment.
environmentIdstring
Optional. The environment ID for the interaction. If specified, the request will update the existing environment instead of creating a new one.
networkUnion type
network can be only one of the following:Allow only specific domains.
Network egress mode.
| JSON representation |
|---|
{ "sources": [ { object ( |
EnvironmentNetworkEgressAllowlist
Network egress configuration for the environment.
List of allowed domains and their configurations.
| JSON representation |
|---|
{
"allowlist": [
{
object ( |
EgressRule
A network egress rule that controls which external domains the environment is allowed to reach. Each rule identifies a target domain and, optionally, a set of HTTP headers to inject into every matching outbound request.
domainstring
The domain pattern to match for this rule. Use an exact hostname (e.g., github.com), a wildcard prefix (e.g., *.googleapis.com), or * to match all domains.
transformmap (key: string, value: string)
Headers to inject into requests matching this rule. Key: header name (e.g., "Authorization"). Value: header value (e.g., "Bearer your-token").
An object containing a list of "key": value pairs. Example: { "name": "wrench", "mass": "1.3kg", "count": "3" }.
| JSON representation |
|---|
{ "domain": string, "transform": { string: string, ... } } |
Source
A source to be mounted into the environment.
typeenum (Type)
sourcestring
The source of the environment. For GCS, this is the GCS path. For GitHub, this is the GitHub path.
targetstring
Where the source should appear in the environment.
contentstring
The inline content if type is INLINE.
encodingstring
Optional encoding for inline content (e.g. base64).
| JSON representation |
|---|
{
"type": enum ( |
LocalEnvironmentConfig
This type has no fields.
Configuration for an environment that lives on the client connection rather than in a server-managed sandbox.
When set (via Interaction.local_environment), the agent's filesystem and shell are treated as living on the client: the agent's built-in environment operations (e.g. reading/listing/editing files and running commands) are suspended on the server and yielded back to the client to execute, with their results returned on a subsequent turn. This is mutually exclusive with a server-managed EnvironmentConfig (remoteEnvironment), since the environment is either on the client or in a server sandbox, never both.
This governs only the agent's built-in environment. Client-declared function tools are always executed on the client regardless of this field.
Tool
A tool that can be used by the model.
typeUnion type
type can be only one of the following:A function that can be used by the model.
A tool that can be used by the model to execute code.
A tool that can be used by the model to fetch URL context.
Tool to support the model interacting directly with the computer.
A MCPServer is a server that can be called by the model to perform actions.
A tool that can be used by the model to search Google.
A tool that can be used by the model to search files.
A tool that can be used by the model to search Google Maps.
retrievalobject (Retrieval)
A tool that can be used by the model to retrieve files.
| JSON representation |
|---|
{ // type "function": { object ( |
Function
A tool that can be used by the model.
namestring
The name of the function.
descriptionstring
A description of the function.
The JSON Schema for the function's parameters.
| JSON representation |
|---|
{
"name": string,
"description": string,
"parameters": {
object ( |
UrlContext
This type has no fields.
A tool that can be used by the model to fetch URL context.
McpServer
A MCPServer is a server that can be called by the model to perform actions.
namestring
The name of the MCPServer.
urlstring
The full URL for the MCPServer endpoint. Example: "https://api.example.com/mcp"
headersmap (key: string, value: string)
Optional: Fields for authentication headers, timeouts, etc., if needed.
An object containing a list of "key": value pairs. Example: { "name": "wrench", "mass": "1.3kg", "count": "3" }.
The allowed tools.
| JSON representation |
|---|
{
"name": string,
"url": string,
"headers": {
string: string,
...
},
"allowedTools": [
{
object ( |
GoogleSearch
A tool that can be used by the model to search Google.
The types of search grounding to enable.
| JSON representation |
|---|
{
"searchTypes": [
enum ( |
FileSearch
A tool that can be used by the model to search files.
fileSearchStoreNames[]string
The file search store names to search.
topKinteger
The number of semantic retrieval chunks to retrieve.
metadataFilterstring
Metadata filter to apply to the semantic retrieval documents and chunks.
| JSON representation |
|---|
{ "fileSearchStoreNames": [ string ], "topK": integer, "metadataFilter": string } |
GoogleMaps
A tool that can be used by the model to call Google Maps.
enableWidgetboolean
Whether to return a widget context token in the tool call result of the response.
latitudenumber
The latitude of the user's location.
longitudenumber
The longitude of the user's location.
| JSON representation |
|---|
{ "enableWidget": boolean, "latitude": number, "longitude": number } |
Usage
Statistics on the interaction request's token usage.
totalInputTokensinteger
Number of tokens in the prompt (context).
A breakdown of input token usage by modality.
totalCachedTokensinteger
Number of tokens in the cached part of the prompt (the cached content).
A breakdown of cached token usage by modality.
totalOutputTokensinteger
Total number of tokens across all the generated responses.
A breakdown of output token usage by modality.
totalToolUseTokensinteger
Number of tokens present in tool-use prompt(s).
A breakdown of tool-use token usage by modality.
totalThoughtTokensinteger
Number of tokens of thoughts for thinking models.
totalTokensinteger
Total token count for the interaction request (prompt + responses + other internal tokens).
Grounding tool count.
| JSON representation |
|---|
{ "totalInputTokens": integer, "inputTokensByModality": [ { object ( |
ModalityTokens
The token count for a single response modality.
The modality associated with the token count.
tokensinteger
Number of tokens for the modality.
| JSON representation |
|---|
{
"modality": enum ( |
GroundingToolCount
The number of grounding tool counts.
typeenum (Type)
The grounding tool type associated with the count.
countinteger
The number of grounding tool counts.
| JSON representation |
|---|
{
"type": enum ( |
WebhookConfig
Message for configuring webhook events for a request.
uris[]string
Optional. If set, these webhook URIs will be used for webhook events instead of the registered webhooks.
Optional. The user metadata that will be returned on each event emission to the webhooks.
| JSON representation |
|---|
{ "uris": [ string ], "userMetadata": { object } } |
SafetySetting
A safety setting that affects the safety-blocking behavior.
A [SafetySetting][google.cloud.aiplatform.master.SafetySetting] consists of a harm [category][google.cloud.aiplatform.master.SafetySetting.category] and a [threshold][google.cloud.aiplatform.master.SafetySetting.threshold] for that category.
Required. The type of harm category to be blocked.
Required. The threshold for blocking content. If the harm probability exceeds this threshold, the content will be blocked.
Optional. The method for blocking content. If not specified, the default behavior is to use the probability score.
| JSON representation |
|---|
{ "type": enum ( |
Delivery
Delivery mode for audio output.
| Enums | |
|---|---|
DELIVERY_UNSPECIFIED |
Default value. This value is unused. |
INLINE |
Audio data is returned inline in the response. |
URI |
Audio data is returned as a URI. |
Environment
Represents the environment being operated, such as a web browser.
| Enums | |
|---|---|
ENVIRONMENT_UNSPECIFIED |
Defaults to browser. |
BROWSER |
Operates in a web browser. |
MOBILE |
Operates in a mobile environment. |
DESKTOP |
Operates in a desktop environment. |
HarmBlockMethod
The method for blocking content.
| Enums | |
|---|---|
HARM_BLOCK_METHOD_UNSPECIFIED |
The harm block method is unspecified. |
SEVERITY |
The harm block method uses both probability and severity scores. |
PROBABILITY |
The harm block method uses the probability score. |
HarmBlockThreshold
Thresholds for blocking content based on harm probability.
| Enums | |
|---|---|
HARM_BLOCK_THRESHOLD_UNSPECIFIED |
The harm block threshold is unspecified. |
BLOCK_LOW_AND_ABOVE |
Block content with a low harm probability or higher. |
BLOCK_MEDIUM_AND_ABOVE |
Block content with a medium harm probability or higher. |
BLOCK_ONLY_HIGH |
Block content with a high harm probability. |
BLOCK_NONE |
Do not block any content, regardless of its harm probability. |
OFF |
Turn off the safety filter entirely. |
ImageSize
Supported image sizes for image output.
| Enums | |
|---|---|
IMAGE_SIZE_UNSPECIFIED |
Default value. This value is unused. |
IMAGE_SIZE_FIVE_TWELVE |
512px image size. |
IMAGE_SIZE_ONE_K |
1K image size. |
IMAGE_SIZE_TWO_K |
2K image size. |
IMAGE_SIZE_FOUR_K |
4K image size. |
Language
Supported programming languages for the generated code.
| Enums | |
|---|---|
LANGUAGE_UNSPECIFIED |
Unspecified language. This value should not be used. |
PYTHON |
Python >= 3.10, with numpy and simpy available. |
MediaResolution
Resolution for input media (images/video).
| Enums | |
|---|---|
MEDIA_RESOLUTION_UNSPECIFIED |
Default value. This value is unused. |
LOW |
Low resolution. |
MEDIUM |
Medium resolution. |
HIGH |
High resolution. |
ULTRA_HIGH |
Ultra high resolution. |
MimeType
| Enums | |
|---|---|
TYPE_UNSPECIFIED |
|
TYPE_WAV |
WAV audio format |
TYPE_MP3 |
MP3 audio format |
TYPE_AIFF |
AIFF audio format |
TYPE_AAC |
AAC audio format |
TYPE_OGG |
OGG audio format |
TYPE_FLAC |
FLAC audio format |
TYPE_MPEG |
MPEG audio format |
TYPE_M4A |
M4A audio format |
TYPE_L16 |
L16 audio format |
TYPE_OPUS |
OPUS audio format |
TYPE_ALAW |
ALAW audio format |
TYPE_MULAW |
MULAW audio format |
Mode
Defines the depth and thoroughness of the find session.
| Enums | |
|---|---|
MODE_UNSPECIFIED |
Default value. This value is unused. |
MODE_SCAN |
Fast scan using only the initial classifier. |
MODE_VERIFY |
Performs classification followed by detailed investigation. |
NetworkMode
Network egress mode for non-allowlist configurations.
| Enums | |
|---|---|
NETWORK_MODE_UNSPECIFIED |
Default value. Unused. |
DISABLED |
All network egress is blocked. |
ResponseModality
The modality of the response.
| Enums | |
|---|---|
RESPONSE_MODALITY_UNSPECIFIED |
Default value. This value is unused. |
TEXT |
Indicates the model should return text. |
IMAGE |
Indicates the model should return images. |
AUDIO |
Indicates the model should return audio. |
VIDEO |
Indicates the model should return video. |
DOCUMENT |
Indicates the model should return documents. |
ReviewSnippet
Encapsulates a snippet of a user review that answers a question about the features of a specific place in Google Maps.
titlestring
Title of the review.
urlstring
A link that corresponds to the user review on Google Maps.
reviewIdstring
The ID of the review snippet.
| JSON representation |
|---|
{ "title": string, "url": string, "reviewId": string } |
SafetyPolicy
| Enums | |
|---|---|
SAFETY_POLICY_UNSPECIFIED |
Unspecified safety policy. |
FINANCIAL_TRANSACTIONS |
Safety policy for financial transactions. |
SENSITIVE_DATA_MODIFICATION |
Safety policy for sensitive data modification. |
COMMUNICATION_TOOL |
Safety policy for communication tools (e.g. Gmail, Chat, Meet). |
ACCOUNT_CREATION |
Safety policy for account creation. |
DATA_MODIFICATION |
Safety policy for data modification. |
USER_CONSENT_MANAGEMENT |
Safety policy for user consent management. |
LEGAL_TERMS_AND_AGREEMENTS |
Safety policy for legal terms and agreements. |
Schema
The Schema object allows the definition of input and output data types. These types can be objects, but also primitives and arrays. Represents a select subset of an OpenAPI 3.0 schema object.
Required. Data type.
formatstring
Optional. The format of the data. Any value is allowed, but most do not trigger any special functionality.
titlestring
Optional. The title of the schema.
descriptionstring
Optional. A brief description of the parameter. This could contain examples of use. Parameter description may be formatted as Markdown.
nullableboolean
Optional. Indicates if the value may be null.
enum[]string
Optional. Possible values of the element of Type.STRING with enum format. For example we can define an Enum Direction as : {type:STRING, format:enum, enum:["EAST", NORTH", "SOUTH", "WEST"]}
Optional. Maximum number of the elements for Type.ARRAY.
Optional. Minimum number of the elements for Type.ARRAY.
Optional. Properties of Type.OBJECT.
An object containing a list of "key": value pairs. Example: { "name": "wrench", "mass": "1.3kg", "count": "3" }.
required[]string
Optional. Required properties of Type.OBJECT.
Optional. Minimum number of the properties for Type.OBJECT.
Optional. Maximum number of the properties for Type.OBJECT.
Optional. SCHEMA FIELDS FOR TYPE STRING Minimum length of the Type.STRING
Optional. Maximum length of the Type.STRING
patternstring
Optional. Pattern of the Type.STRING to restrict a string to a regular expression.
Optional. Example of the object. Will only populated when the object is the root.
Optional. The value should be validated against any (one or more) of the subschemas in the list.
propertyOrdering[]string
Optional. The order of the properties. Not a standard field in open api spec. Used to determine the order of the properties in the response.
Optional. Default value of the field. Per JSON Schema, this field is intended for documentation generators and doesn't affect validation. Thus it's included here and ignored so that developers who send schemas with a default field don't get unknown-field errors.
Optional. Schema of the elements of Type.ARRAY.
minimumnumber
Optional. SCHEMA FIELDS FOR TYPE INTEGER and NUMBER Minimum value of the Type.INTEGER and Type.NUMBER
maximumnumber
Optional. Maximum value of the Type.INTEGER and Type.NUMBER
| JSON representation |
|---|
{ "type": enum ( |
Type
Type contains the list of OpenAPI data types as defined by https://spec.openapis.org/oas/v3.0.3#data-types
| Enums | |
|---|---|
TYPE_UNSPECIFIED |
Not specified, should not be used. |
STRING |
String type. |
NUMBER |
Number type. |
INTEGER |
Integer type. |
BOOLEAN |
Boolean type. |
ARRAY |
Array type. |
OBJECT |
Object type. |
NULL |
Null type. |
SearchType
The types of search grounding to enable.
| Enums | |
|---|---|
SEARCH_TYPE_UNSPECIFIED |
Unspecified search type. This value should not be used. |
SEARCH_TYPE_WEB_SEARCH |
Setting this field enables web search. Only text results are returned. |
SEARCH_TYPE_IMAGE_SEARCH |
Setting this field enables image search. Image bytes are returned. |
Struct
Struct represents a structured data value, consisting of fields which map to dynamically typed values.
Dynamically typed fields. List instead of map because LLMs are sensitive to ordering, and we want to give users full control.
| JSON representation |
|---|
{
"fields": [
{
object ( |
Field
Task
Supported video generation tasks.
| Enums | |
|---|---|
TASK_UNSPECIFIED |
Unspecified task. The task is inferred from the input prompt and media. |
TEXT_TO_VIDEO |
Generates video solely from a text prompt. |
IMAGE_TO_VIDEO |
Generates video from one or two source images. The first image defines the starting frame, and the optional second image defines the ending frame. |
REFERENCE_TO_VIDEO |
Generates video using reference media (such as images, audio, or video). |
EDIT |
Modifies an existing input video. |
ThinkingLevel
The level of thought tokens that the model should generate.
| Enums | |
|---|---|
THINKING_LEVEL_UNSPECIFIED |
Default value. This value is unused. |
THINKING_LEVEL_MINIMAL |
Little to no thinking. |
THINKING_LEVEL_LOW |
Low thinking level. |
THINKING_LEVEL_MEDIUM |
Medium thinking level. |
THINKING_LEVEL_HIGH |
High thinking level. |
ThinkingSummaries
Whether to include thought summaries in the response.
| Enums | |
|---|---|
THINKING_SUMMARIES_UNSPECIFIED |
Default value. This value is unused. |
THINKING_SUMMARIES_AUTO |
Auto thinking summaries. |
THINKING_SUMMARIES_NONE |
No thinking summaries. |
Tool
- JSON representation
- FunctionDeclaration
- Behavior
- GoogleSearchRetrieval
- DynamicRetrievalConfig
- Mode
- CodeExecution
- GoogleSearch
- Interval
- SearchTypes
- WebSearch
- ImageSearch
- ComputerUse
- Environment
- SafetyPolicy
- UrlContext
- FileSearch
- McpServer
- StreamableHttpTransport
- GoogleMaps
Tool details that the model may use to generate response.
A Tool is a piece of code that enables the system to interact with external systems to perform an action, or set of actions, outside of knowledge and scope of the model.
Next ID: 16
Optional. A list of FunctionDeclarations available to the model that can be used for function calling.
The model or system does not execute the function. Instead the defined function may be returned as a FunctionCall with arguments to the client side for execution. The model may decide to call a subset of these functions by populating FunctionCall in the response. The next conversation turn may contain a FunctionResponse with the Content.role "function" generation context for the next model turn.
Optional. Retrieval tool that is powered by Google search.
Optional. Enables the model to execute code as part of generation.
Optional. GoogleSearch tool type. Tool to support Google Search in Model. Powered by Google.
Optional. Tool to support the model interacting directly with the computer. If enabled, it automatically populates computer-use specific Function Declarations.
Optional. Tool to support URL context retrieval.
Optional. FileSearch tool type. Tool to retrieve knowledge from Semantic Retrieval corpora.
Optional. MCP Servers to connect to.
Optional. Tool that allows grounding the model's response with geospatial context related to the user's query.
| JSON representation |
|---|
{ "functionDeclarations": [ { object ( |
FunctionDeclaration
Structured representation of a function declaration as defined by the OpenAPI 3.03 specification. Included in this declaration are the function name and parameters. This FunctionDeclaration is a representation of a block of code that can be used as a Tool by the model and executed by the client.
namestring
Required. The name of the function. Must be a-z, A-Z, 0-9, or contain underscores, colons, dots, and dashes, with a maximum length of 128.
descriptionstring
Required. A brief description of the function.
Optional. Specifies the function Behavior. Currently only supported by the BidiGenerateContent method.
Optional. Describes the parameters to this function. Reflects the Open API 3.03 Parameter Object string Key: the name of the parameter. Parameter names are case sensitive. Schema Value: the Schema defining the type used for the parameter.
Optional. Describes the parameters to the function in JSON Schema format. The schema must describe an object where the properties are the parameters to the function. For example:
{
"type": "object",
"properties": {
"name": { "type": "string" },
"age": { "type": "integer" }
},
"additionalProperties": false,
"required": ["name", "age"],
"propertyOrdering": ["name", "age"]
}
This field is mutually exclusive with parameters.
Optional. Describes the output from this function in JSON Schema format. Reflects the Open API 3.03 Response Object. The Schema defines the type used for the response value of the function.
Optional. Describes the output from this function in JSON Schema format. The value specified by the schema is the response value of the function.
This field is mutually exclusive with response.
Behavior
Defines the function behavior. Defaults to BLOCKING.
| Enums | |
|---|---|
UNSPECIFIED |
This value is unused. |
BLOCKING |
If set, the system will wait to receive the function response before continuing the conversation. |
NON_BLOCKING |
If set, the system will not wait to receive the function response. Instead, it will attempt to handle function responses as they become available while maintaining the conversation between the user and the model. |
GoogleSearchRetrieval
Tool to retrieve public web data for grounding, powered by Google.
Specifies the dynamic retrieval configuration for the given source.
| JSON representation |
|---|
{
"dynamicRetrievalConfig": {
object ( |
DynamicRetrievalConfig
Describes the options to customize dynamic retrieval.
The mode of the predictor to be used in dynamic retrieval.
dynamicThresholdnumber
The threshold to be used in dynamic retrieval. If not set, a system default value is used.
| JSON representation |
|---|
{
"mode": enum ( |
Mode
The mode of the predictor to be used in dynamic retrieval.
| Enums | |
|---|---|
MODE_UNSPECIFIED |
Always trigger retrieval. |
MODE_DYNAMIC |
Run retrieval only when system decides it is necessary. |
CodeExecution
This type has no fields.
Tool that executes code generated by the model, and automatically returns the result to the model.
See also ExecutableCode and CodeExecutionResult which are only generated when using this tool.
GoogleSearch
GoogleSearch tool type. Tool to support Google Search in Model. Powered by Google.
Optional. Filter search results to a specific time range. If customers set a start time, they must set an end time (and vice versa).
Optional. The set of search types to enable. If not set, web search is enabled by default.
| JSON representation |
|---|
{ "timeRangeFilter": { object ( |
Interval
Represents a time interval, encoded as a Timestamp start (inclusive) and a Timestamp end (exclusive).
The start must be less than or equal to the end. When the start equals the end, the interval is empty (matches no time). When both start and end are unspecified, the interval matches any time.
Optional. Inclusive start of the interval.
If specified, a Timestamp matching this interval will have to be the same or after the start.
Uses RFC 3339, where generated output will always be Z-normalized and use 0, 3, 6 or 9 fractional digits. Offsets other than "Z" are also accepted. Examples: "2014-10-02T15:01:23Z", "2014-10-02T15:01:23.045123456Z" or "2014-10-02T15:01:23+05:30".
Optional. Exclusive end of the interval.
If specified, a Timestamp matching this interval will have to be before the end.
Uses RFC 3339, where generated output will always be Z-normalized and use 0, 3, 6 or 9 fractional digits. Offsets other than "Z" are also accepted. Examples: "2014-10-02T15:01:23Z", "2014-10-02T15:01:23.045123456Z" or "2014-10-02T15:01:23+05:30".
| JSON representation |
|---|
{ "startTime": string, "endTime": string } |
SearchTypes
Different types of search that can be enabled on the GoogleSearch tool.
Optional. Enables web search. Only text results are returned.
Optional. Enables image search. Image bytes are returned.
| JSON representation |
|---|
{ "webSearch": { object ( |
WebSearch
This type has no fields.
Standard web search for grounding and related configurations.
ImageSearch
This type has no fields.
Image search for grounding and related configurations.
ComputerUse
Computer Use tool type.
Required. The environment being operated.
excludedPredefinedFunctions[]string
Optional. By default, predefined functions are included in the final model call. Some of them can be explicitly excluded from being automatically included. This can serve two purposes: 1. Using a more restricted / different action space. 2. Improving the definitions / instructions of predefined functions.
enablePromptInjectionDetectionboolean
Optional. Whether enable the prompt injection detection check on computer-use request.
Optional. Disabled safety policies for computer use.
| JSON representation |
|---|
{ "environment": enum ( |
Environment
Represents the environment being operated, such as a web browser.
| Enums | |
|---|---|
ENVIRONMENT_UNSPECIFIED |
Defaults to browser. |
ENVIRONMENT_BROWSER |
Operates in a web browser. |
ENVIRONMENT_MOBILE |
Operates in a mobile environment. |
ENVIRONMENT_DESKTOP |
Operates in a desktop environment. |
SafetyPolicy
Predefined safety policies for computer use.
| Enums | |
|---|---|
SAFETY_POLICY_UNSPECIFIED |
Unspecified safety policy. |
FINANCIAL_TRANSACTIONS |
Safety policy for financial transactions. |
SENSITIVE_DATA_MODIFICATION |
Safety policy for sensitive data modification. |
COMMUNICATION_TOOL |
Safety policy for communication tools (e.g. Gmail, Chat, Meet). |
ACCOUNT_CREATION |
Safety policy for account creation. |
DATA_MODIFICATION |
Safety policy for data modification. |
USER_CONSENT_MANAGEMENT |
Safety policy for user consent management. |
LEGAL_TERMS_AND_AGREEMENTS |
Safety policy for legal terms and agreements. |
UrlContext
This type has no fields.
Tool to support URL context retrieval.
FileSearch
The FileSearch tool that retrieves knowledge from Semantic Retrieval corpora. Files are imported to Semantic Retrieval corpora using the ImportFile API.
fileSearchStoreNames[]string
Required. The names of the fileSearchStores to retrieve from. Example: fileSearchStores/my-file-search-store-123
metadataFilterstring
Optional. Metadata filter to apply to the semantic retrieval documents and chunks.
topKinteger
Optional. The number of semantic retrieval chunks to retrieve.
| JSON representation |
|---|
{ "fileSearchStoreNames": [ string ], "metadataFilter": string, "topK": integer } |
McpServer
A MCPServer is a server that can be called by the model to perform actions. It is a server that implements the MCP protocol. Next ID: 6
namestring
The name of the MCPServer.
transportUnion type
transport can be only one of the following:A transport that can stream HTTP requests and responses.
| JSON representation |
|---|
{
"name": string,
// transport
"streamableHttpTransport": {
object ( |
StreamableHttpTransport
A transport that can stream HTTP requests and responses. Next ID: 6
urlstring
The full URL for the MCPServer endpoint. Example: "https://api.example.com/mcp"
headersmap (key: string, value: string)
Optional: Fields for authentication headers, timeouts, etc., if needed.
An object containing a list of "key": value pairs. Example: { "name": "wrench", "mass": "1.3kg", "count": "3" }.
HTTP timeout for regular operations.
A duration in seconds with up to nine fractional digits, ending with 's'. Example: "3.5s".
Timeout for SSE read operations.
A duration in seconds with up to nine fractional digits, ending with 's'. Example: "3.5s".
terminateOnCloseboolean
Whether to close the client session when the transport closes.
| JSON representation |
|---|
{ "url": string, "headers": { string: string, ... }, "timeout": string, "sseReadTimeout": string, "terminateOnClose": boolean } |
GoogleMaps
The GoogleMaps Tool that provides geospatial context for the user's query.
enableWidgetboolean
Optional. Whether to return a widget context token in the GroundingMetadata of the response. Developers can use the widget context token to render a Google Maps widget with geospatial context related to the places that the model references in the response.
| JSON representation |
|---|
{ "enableWidget": boolean } |
ToolChoiceType
The type of tool choice.
| Enums | |
|---|---|
TOOL_CHOICE_TYPE_UNSPECIFIED |
Default value. This value is unused. |
AUTO |
Auto tool choice. |
ANY |
Any tool choice. |
NONE |
No tool choice. |
VALIDATED |
Validated tool choice. |
Value
Value represents a dynamically typed value which can be either null, a number, a string, a boolean, a recursive struct value, or a list of values. A producer of value is expected to set one of these variants. Absence of any variant indicates an error.
kindUnion type
kind can be only one of the following:nullValuenull
Represents a null value.
numberValuenumber
Represents a double value.
stringValuestring
Represents a string value.
boolValueboolean
Represents a boolean value.
Represents a structured value.
Represents a repeated Value.
Represents rich content (text, image, etc.).
ListValue
VisualizationMode
Enum for visualization mode. Eventually we will support an interactive mode where the user can choose whether to include HTML visualizations in the response.
| Enums | |
|---|---|
UNSPECIFIED |
The default visualization mode. Will default to AUTO. |
OFF |
Do not include visualizations. |
AUTO |
Automatically include visualizations. |
REST Resource: auth_tokens
- Resource: AuthToken
- BidiGenerateContentSetup
- GenerationConfig
- Modality
- SpeechConfig
- VoiceConfig
- PrebuiltVoiceConfig
- MultiSpeakerVoiceConfig
- SpeakerVoiceConfig
- ThinkingConfig
- ThinkingLevel
- ImageConfig
- MediaResolution
- ResponseFormatConfig
- TextResponseFormat
- MimeType
- AudioResponseFormat
- MimeType
- Delivery
- ImageResponseFormat
- MimeType
- Delivery
- AspectRatio
- ImageSize
- TranslationConfig
- RealtimeInputConfig
- AutomaticActivityDetection
- StartSensitivity
- EndSensitivity
- ActivityHandling
- TurnCoverage
- SessionResumptionConfig
- ContextWindowCompressionConfig
- SlidingWindow
- AudioTranscriptionConfig
- LanguageAuto
- LanguageHints
- HistoryConfig
- Methods
Resource: AuthToken
A request to create an ephemeral authentication token.
namestring
Output only. Identifier. The token itself.
Optional. Input only. Immutable. An optional time after which, when using the resulting token, messages in BidiGenerateContent sessions will be rejected. (Gemini may preemptively close the session after this time.)
If not set then this defaults to 30 minutes in the future. If set, this value must be less than 20 hours in the future.
Uses RFC 3339, where generated output will always be Z-normalized and use 0, 3, 6 or 9 fractional digits. Offsets other than "Z" are also accepted. Examples: "2014-10-02T15:01:23Z", "2014-10-02T15:01:23.045123456Z" or "2014-10-02T15:01:23+05:30".
Optional. Input only. Immutable. The time after which new Live API sessions using the token resulting from this request will be rejected.
If not set this defaults to 60 seconds in the future. If set, this value must be less than 20 hours in the future.
Uses RFC 3339, where generated output will always be Z-normalized and use 0, 3, 6 or 9 fractional digits. Offsets other than "Z" are also accepted. Examples: "2014-10-02T15:01:23Z", "2014-10-02T15:01:23.045123456Z" or "2014-10-02T15:01:23+05:30".
Optional. Input only. Immutable. If fieldMask is empty, and bidiGenerateContentSetup is not present, then the effective BidiGenerateContentSetup message is taken from the Live API connection.
If fieldMask is empty, and bidiGenerateContentSetup is present, then the effective BidiGenerateContentSetup message is taken entirely from bidiGenerateContentSetup in this request. The setup message from the Live API connection is ignored.
If fieldMask is not empty, then the corresponding fields from bidiGenerateContentSetup will overwrite the fields from the setup message in the Live API connection.
This is a comma-separated list of fully qualified names of fields. Example: "user.displayName,photo".
configUnion type
config can be only one of the following:Optional. Input only. Immutable. Configuration specific to BidiGenerateContent.
usesinteger
Optional. Input only. Immutable. The number of times the token can be used. If this value is zero then no limit is applied. Resuming a Live API session does not count as a use. If unspecified, the default is 1.
| JSON representation |
|---|
{
"name": string,
"expireTime": string,
"newSessionExpireTime": string,
"fieldMask": string,
// config
"bidiGenerateContentSetup": {
object ( |
BidiGenerateContentSetup
Message to be sent in the first (and only in the first) BidiGenerateContentClientMessage. Contains configuration that will apply for the duration of the streaming RPC.
Clients should wait for a BidiGenerateContentSetupComplete message before sending any additional messages.
modelstring
Required. The model's resource name. This serves as an ID for the Model to use.
Format: models/{model}
Optional. Generation config.
The following fields are not supported:
responseLogprobsresponseMimeTypelogprobsresponseSchemaresponseJsonSchemastop_sequenceskipResponseCacherouting_configaudio_timestamp
systemInstructionobject (Content)
Optional. The user provided system instructions for the model.
Note: Only text should be used in parts and content in each part will be in a separate paragraph.
Optional. A list of Tools the model may use to generate the next response.
A Tool is a piece of code that enables the system to interact with external systems to perform an action, or set of actions, outside of knowledge and scope of the model.
Optional. Configures the handling of realtime input.
Optional. Configures session resumption mechanism.
If included, the server will send SessionResumptionUpdate messages.
Optional. Configures a context window compression mechanism.
If included, the server will automatically reduce the size of the context when it exceeds the configured length.
Optional. If set, enables transcription of voice input. The transcription aligns with the input audio language, if configured.
Optional. If set, enables transcription of the model's audio output. The transcription aligns with the language code specified for the output audio, if configured.
Optional. Configures the exchange of history between the client and the server.
| JSON representation |
|---|
{ "model": string, "generationConfig": { object ( |
GenerationConfig
Configuration options for model generation and outputs. Not all parameters are configurable for every model.
stopSequences[]string
Optional. The set of character sequences (up to 5) that will stop output generation. If specified, the API will stop at the first appearance of a stop_sequence. The stop sequence will not be included as part of the response.
responseMimeTypestring
Optional. MIME type of the generated candidate text. Supported MIME types are: text/plain: (default) Text output. application/json: JSON response in the response candidates. text/x.enum: ENUM as a string response in the response candidates. Refer to the docs for a list of all supported text MIME types.
Optional. Output schema of the generated candidate text. Schemas must be a subset of the OpenAPI schema and can be objects, primitives or arrays.
If set, a compatible responseMimeType must also be set. Compatible MIME types: application/json: Schema for JSON response. Refer to the JSON text generation guide for more details.
Optional. Output schema of the generated response. This is an alternative to responseSchema that accepts JSON Schema.
If set, responseSchema must be omitted, but responseMimeType is required.
While the full JSON Schema may be sent, not all features are supported. Specifically, only the following properties are supported:
$id$defs$ref$anchortypeformattitledescriptionenum(for strings and numbers)itemsprefixItemsminItemsmaxItemsminimummaximumanyOfoneOf(interpreted the same asanyOf)propertiesadditionalPropertiesrequired
The non-standard propertyOrdering property may also be set.
Cyclic references are unrolled to a limited degree and, as such, may only be used within non-required properties. (Nullable properties are not sufficient.) If $ref is set on a sub-schema, no other properties, except for than those starting as a $, may be set.
Optional. An internal detail. Use responseJsonSchema rather than this field.
Optional. The requested modalities of the response. Represents the set of modalities that the model can return, and should be expected in the response. This is an exact match to the modalities of the response.
A model may have multiple combinations of supported modalities. If the requested modalities do not match any of the supported combinations, an error will be returned.
An empty list is equivalent to requesting only text.
candidateCountinteger
Optional. Number of generated responses to return. If unset, this will default to 1. Please note that this doesn't work for previous generation models (Gemini 1.0 family)
maxOutputTokensinteger
Optional. The maximum number of tokens to include in a response candidate.
Note: The default value varies by model, see the Model.output_token_limit attribute of the Model returned from the getModel function.
temperaturenumber
Optional. Controls the randomness of the output.
Note: The default value varies by model, see the Model.temperature attribute of the Model returned from the getModel function.
Values can range from [0.0, 2.0].
topPnumber
Optional. The maximum cumulative probability of tokens to consider when sampling.
The model uses combined Top-k and Top-p (nucleus) sampling.
Tokens are sorted based on their assigned probabilities so that only the most likely tokens are considered. Top-k sampling directly limits the maximum number of tokens to consider, while Nucleus sampling limits the number of tokens based on the cumulative probability.
Note: The default value varies by Model and is specified by theModel.top_p attribute returned from the getModel function. An empty topK attribute indicates that the model doesn't apply top-k sampling and doesn't allow setting topK on requests.
topKinteger
Optional. The maximum number of tokens to consider when sampling.
Gemini models use Top-p (nucleus) sampling or a combination of Top-k and nucleus sampling. Top-k sampling considers the set of topK most probable tokens. Models running with nucleus sampling don't allow topK setting.
Note: The default value varies by Model and is specified by theModel.top_p attribute returned from the getModel function. An empty topK attribute indicates that the model doesn't apply top-k sampling and doesn't allow setting topK on requests.
seedinteger
Optional. Seed used in decoding. If not set, the request uses a randomly generated seed.
presencePenaltynumber
Optional. Presence penalty applied to the next token's logprobs if the token has already been seen in the response.
This penalty is binary on/off and not dependant on the number of times the token is used (after the first). Use frequencyPenalty for a penalty that increases with each use.
A positive penalty will discourage the use of tokens that have already been used in the response, increasing the vocabulary.
A negative penalty will encourage the use of tokens that have already been used in the response, decreasing the vocabulary.
frequencyPenaltynumber
Optional. Frequency penalty applied to the next token's logprobs, multiplied by the number of times each token has been seen in the respponse so far.
A positive penalty will discourage the use of tokens that have already been used, proportional to the number of times the token has been used: The more a token is used, the more difficult it is for the model to use that token again increasing the vocabulary of responses.
Caution: A negative penalty will encourage the model to reuse tokens proportional to the number of times the token has been used. Small negative values will reduce the vocabulary of a response. Larger negative values will cause the model to start repeating a common token until it hits the maxOutputTokens limit.
responseLogprobsboolean
Optional. If true, export the logprobs results in response.
logprobsinteger
Optional. Only valid if responseLogprobs=True. This sets the number of top logprobs, including the chosen candidate, to return at each decoding step in the Candidate.logprobs_result. The number must be in the range of [0, 20].
enableEnhancedCivicAnswersboolean
Optional. Enables enhanced civic answers. It may not be available for all models.
Optional. The speech generation config.
Optional. Config for thinking features. An error will be returned if this field is set for models that don't support thinking.
Optional. Config for image generation. An error will be returned if this field is set for models that don't support these config options.
Optional. If specified, the media resolution specified will be used.
enableAffectiveDialogboolean
Optional. If enabled, the model will detect emotions and adapt its responses accordingly.
Optional. Configuration for the response output format. Allows specifying output configuration per modality (text, audio, image) in a flat structure.
Optional. Config for translation.
| JSON representation |
|---|
{ "stopSequences": [ string ], "responseMimeType": string, "responseSchema": { object ( |
Modality
Supported modalities of the response.
| Enums | |
|---|---|
MODALITY_UNSPECIFIED |
Default value. |
TEXT |
Indicates the model should return text. |
IMAGE |
Indicates the model should return images. |
AUDIO |
Indicates the model should return audio. |
SpeechConfig
Config for speech generation and transcription.
The configuration in case of single-voice output.
Optional. The configuration for the multi-speaker setup. It is mutually exclusive with the voiceConfig field.
languageCodestring
Optional. The IETF BCP-47 language code that the user configured the app to use. Used for speech recognition and synthesis.
Valid values are: de-DE, en-AU, en-GB, en-IN, en-US, es-US, fr-FR, hi-IN, pt-BR, ar-XA, es-ES, fr-CA, id-ID, it-IT, ja-JP, tr-TR, vi-VN, bn-IN, gu-IN, kn-IN, ml-IN, mr-IN, ta-IN, te-IN, nl-NL, ko-KR, cmn-CN, pl-PL, ru-RU, and th-TH.
| JSON representation |
|---|
{ "voiceConfig": { object ( |
VoiceConfig
The configuration for the voice to use.
voice_configUnion type
voice_config can be only one of the following:The configuration for the prebuilt voice to use.
| JSON representation |
|---|
{
// voice_config
"prebuiltVoiceConfig": {
object ( |
PrebuiltVoiceConfig
The configuration for the prebuilt speaker to use.
voiceNamestring
The name of the preset voice to use.
| JSON representation |
|---|
{ "voiceName": string } |
MultiSpeakerVoiceConfig
The configuration for the multi-speaker setup.
Required. All the enabled speaker voices.
| JSON representation |
|---|
{
"speakerVoiceConfigs": [
{
object ( |
SpeakerVoiceConfig
The configuration for a single speaker in a multi speaker setup.
speakerstring
Required. The name of the speaker to use. Should be the same as in the prompt.
Required. The configuration for the voice to use.
| JSON representation |
|---|
{
"speaker": string,
"voiceConfig": {
object ( |
ThinkingConfig
Config for thinking features.
includeThoughtsboolean
Indicates whether to include thoughts in the response. If true, thoughts are returned only when available.
thinkingBudgetinteger
The number of thoughts tokens that the model should generate.
Optional. Controls the maximum depth of the model's internal reasoning process before it produces a response. The default value is model-dependent. Refer to the Thinking levels guide for more details. Recommended for Gemini 3 or later models. Use with earlier models results in an error.
| JSON representation |
|---|
{
"includeThoughts": boolean,
"thinkingBudget": integer,
"thinkingLevel": enum ( |
ThinkingLevel
Allow user to specify how much to think using enum instead of integer budget.
| Enums | |
|---|---|
THINKING_LEVEL_UNSPECIFIED |
Default value. |
MINIMAL |
Little to no thinking. |
LOW |
Low thinking level. |
MEDIUM |
Medium thinking level. |
HIGH |
High thinking level. |
ImageConfig
Config for image generation features.
aspectRatiostring
Optional. The aspect ratio of the image to generate. Supported aspect ratios: 1:1, 1:4, 4:1, 1:8, 8:1, 2:3, 3:2, 3:4, 4:3, 4:5, 5:4, 9:16, 16:9, or 21:9.
If not specified, the model will choose a default aspect ratio based on any reference images provided.
imageSizestring
Optional. Specifies the size of generated images. Supported values are 512, 1K, 2K, 4K. If not specified, the model will use default value 1K.
| JSON representation |
|---|
{ "aspectRatio": string, "imageSize": string } |
MediaResolution
Media resolution for the input media.
| Enums | |
|---|---|
MEDIA_RESOLUTION_UNSPECIFIED |
Media resolution has not been set. |
MEDIA_RESOLUTION_LOW |
Media resolution set to low (64 tokens). |
MEDIA_RESOLUTION_MEDIUM |
Media resolution set to medium (256 tokens). |
MEDIA_RESOLUTION_HIGH |
Media resolution set to high (zoomed reframing with 256 tokens). |
ResponseFormatConfig
Configuration for the response output format. This is a flat object where each optional sub-field configures a specific output modality.
Optional. Text output format configuration.
Optional. Audio output format configuration.
Optional. Image output format configuration.
| JSON representation |
|---|
{ "text": { object ( |
TextResponseFormat
Configuration for text output format.
Optional. The MIME type of the text output.
Optional. The JSON schema that the output should conform to. Only applicable when mimeType is APPLICATION_JSON.
| JSON representation |
|---|
{
"mimeType": enum ( |
MimeType
Supported MIME types for text output.
| Enums | |
|---|---|
MIME_TYPE_UNSPECIFIED |
Default value. This value is unused. |
APPLICATION_JSON |
JSON output format. |
TEXT_PLAIN |
Plain text output format. |
AudioResponseFormat
Configuration for audio output format.
Optional. The MIME type of the audio output.
Optional. The delivery mode for the audio output.
sampleRateinteger
Optional. Sample rate in Hz.
bitRateinteger
Optional. Bit rate in bits per second (bps). Only applicable for compressed formats (MP3, Opus).
MimeType
Supported MIME types for audio output.
| Enums | |
|---|---|
MIME_TYPE_UNSPECIFIED |
Default value. This value is unused. |
AUDIO_MP3 |
MP3 audio format. |
AUDIO_OGG_OPUS |
OGG Opus audio format. |
AUDIO_L16 |
Raw PCM (L16) audio format. |
AUDIO_WAV |
WAV audio format. |
AUDIO_ALAW |
A-law audio format. |
AUDIO_MULAW |
Mu-law audio format. |
Delivery
Delivery mode for audio output.
| Enums | |
|---|---|
DELIVERY_UNSPECIFIED |
Default value. This value is unused. |
INLINE |
Audio data is returned inline in the response. |
URI |
Audio data is returned as a URI. |
ImageResponseFormat
Configuration for image output format.
Optional. The MIME type of the image output.
Optional. The delivery mode for the image output.
Optional. The aspect ratio for the image output.
Optional. The size of the image output.
| JSON representation |
|---|
{ "mimeType": enum ( |
MimeType
Supported MIME types for image output.
| Enums | |
|---|---|
MIME_TYPE_UNSPECIFIED |
Default value. This value is unused. |
IMAGE_JPEG |
JPEG image format. |
Delivery
Delivery mode for image output.
| Enums | |
|---|---|
DELIVERY_UNSPECIFIED |
Default value. This value is unused. |
INLINE |
Image data is returned inline in the response. |
URI |
Image data is returned as a URI. |
AspectRatio
Supported aspect ratios for image output.
| Enums | |
|---|---|
ASPECT_RATIO_UNSPECIFIED |
Default value. This value is unused. |
ASPECT_RATIO_ONE_BY_ONE |
1:1 aspect ratio. |
ASPECT_RATIO_TWO_BY_THREE |
2:3 aspect ratio. |
ASPECT_RATIO_THREE_BY_TWO |
3:2 aspect ratio. |
ASPECT_RATIO_THREE_BY_FOUR |
3:4 aspect ratio. |
ASPECT_RATIO_FOUR_BY_THREE |
4:3 aspect ratio. |
ASPECT_RATIO_FOUR_BY_FIVE |
4:5 aspect ratio. |
ASPECT_RATIO_FIVE_BY_FOUR |
5:4 aspect ratio. |
ASPECT_RATIO_NINE_BY_SIXTEEN |
9:16 aspect ratio. |
ASPECT_RATIO_SIXTEEN_BY_NINE |
16:9 aspect ratio. |
ASPECT_RATIO_TWENTY_ONE_BY_NINE |
21:9 aspect ratio. |
ASPECT_RATIO_ONE_BY_EIGHT |
1:8 aspect ratio. |
ASPECT_RATIO_EIGHT_BY_ONE |
8:1 aspect ratio. |
ASPECT_RATIO_ONE_BY_FOUR |
1:4 aspect ratio. |
ASPECT_RATIO_FOUR_BY_ONE |
4:1 aspect ratio. |
ImageSize
Supported image sizes for image output.
| Enums | |
|---|---|
IMAGE_SIZE_UNSPECIFIED |
Default value. This value is unused. |
IMAGE_SIZE_FIVE_TWELVE |
512px image size. |
IMAGE_SIZE_ONE_K |
1K image size. |
IMAGE_SIZE_TWO_K |
2K image size. |
IMAGE_SIZE_FOUR_K |
4K image size. |
TranslationConfig
Config for translation features.
targetLanguageCodestring
Required. The target language for translation. Supported values are BCP-47 language codes (e.g. "en", "es", "fr").
echoTargetLanguageboolean
Optional. If true, the model will generate audio when the target language is spoken, essentially it will parrot the input. If false, we will not produce audio for the target language.
| JSON representation |
|---|
{ "targetLanguageCode": string, "echoTargetLanguage": boolean } |
RealtimeInputConfig
Configures the realtime input behavior in BidiGenerateContent.
Optional. If not set, automatic activity detection is enabled by default. If automatic voice detection is disabled, the client must send activity signals.
Optional. Defines what effect activity has.
Optional. Defines which input is included in the user's turn.
| JSON representation |
|---|
{ "automaticActivityDetection": { object ( |
AutomaticActivityDetection
Configures automatic detection of activity.
disabledboolean
Optional. If enabled (the default), detected voice and text input count as activity. If disabled, the client must send activity signals.
Optional. Determines how likely speech is to be detected.
prefixPaddingMsinteger
Optional. The required duration of detected speech before start-of-speech is committed. The lower this value, the more sensitive the start-of-speech detection is and shorter speech can be recognized. However, this also increases the probability of false positives.
Optional. Determines how likely detected speech is ended.
silenceDurationMsinteger
Optional. The required duration of detected non-speech (e.g. silence) before end-of-speech is committed. The larger this value, the longer speech gaps can be without interrupting the user's activity but this will increase the model's latency.
| JSON representation |
|---|
{ "disabled": boolean, "startOfSpeechSensitivity": enum ( |
StartSensitivity
Determines how start of speech is detected.
| Enums | |
|---|---|
START_SENSITIVITY_UNSPECIFIED |
The default is START_SENSITIVITY_HIGH. |
START_SENSITIVITY_HIGH |
Automatic detection will detect the start of speech more often. |
START_SENSITIVITY_LOW |
Automatic detection will detect the start of speech less often. |
EndSensitivity
Determines how end of speech is detected.
| Enums | |
|---|---|
END_SENSITIVITY_UNSPECIFIED |
The default is END_SENSITIVITY_HIGH. |
END_SENSITIVITY_HIGH |
Automatic detection ends speech more often. |
END_SENSITIVITY_LOW |
Automatic detection ends speech less often. |
ActivityHandling
The different ways of handling user activity.
| Enums | |
|---|---|
ACTIVITY_HANDLING_UNSPECIFIED |
If unspecified, the default behavior is START_OF_ACTIVITY_INTERRUPTS. |
START_OF_ACTIVITY_INTERRUPTS |
If true, start of activity will interrupt the model's response (also called "barge in"). The model's current response will be cut-off in the moment of the interruption. This is the default behavior. |
NO_INTERRUPTION |
The model's response will not be interrupted. |
TurnCoverage
Options about which input is included in the user's turn.
| Enums | |
|---|---|
TURN_COVERAGE_UNSPECIFIED |
If unspecified, a default behavior is selected based on the model. E.g., for Gemini 2.5, the default is TURN_INCLUDES_ONLY_ACTIVITY, while for Gemini 3.1 and onwards, it's TURN_INCLUDES_AUDIO_ACTIVITY_AND_ALL_VIDEO. |
TURN_INCLUDES_ONLY_ACTIVITY |
Includes activity since the last turn, excluding inactivity (e.g. silence on the audio stream). |
TURN_INCLUDES_ALL_INPUT |
Includes all realtime input since the last turn, including inactivity (e.g. silence on the audio stream). |
TURN_INCLUDES_AUDIO_ACTIVITY_AND_ALL_VIDEO |
Includes audio activity and all video since the last turn. With automatic activity detection, audio activity means speech and excludes silence. |
SessionResumptionConfig
Session resumption configuration.
This message is included in the session configuration as BidiGenerateContentSetup.session_resumption. If configured, the server will send SessionResumptionUpdate messages.
handlestring
The handle of a previous session. If not present then a new session is created.
Session handles come from SessionResumptionUpdate.token values in previous connections.
| JSON representation |
|---|
{ "handle": string } |
ContextWindowCompressionConfig
Enables context window compression — a mechanism for managing the model's context window so that it does not exceed a given length.
compression_mechanismUnion type
compression_mechanism can be only one of the following:A sliding-window mechanism.
The number of tokens (before running a turn) required to trigger a context window compression.
This can be used to balance quality against latency as shorter context windows may result in faster model responses. However, any compression operation will cause a temporary latency increase, so they should not be triggered frequently.
If not set, the default is 80% of the model's context window limit. This leaves 20% for the next user request/model response.
| JSON representation |
|---|
{
// compression_mechanism
"slidingWindow": {
object ( |
SlidingWindow
The SlidingWindow method operates by discarding content at the beginning of the context window. The resulting context will always begin at the start of a USER role turn. System instructions and any BidiGenerateContentSetup.prefix_turns will always remain at the beginning of the result.
The target number of tokens to keep. The default value is triggerTokens/2.
Discarding parts of the context window causes a temporary latency increase so this value should be calibrated to avoid frequent compression operations.
| JSON representation |
|---|
{ "targetTokens": string } |
AudioTranscriptionConfig
The audio transcription configuration.
adaptationPhrases[]
(deprecated)string
Optional. A list of phrases used for speech adaptation, which biases the ASR model to improve recognition of these specific terms.
customVocabulary[]string
Optional. A list of custom vocabulary phrases to bias the speech recognition model toward recognizing specific terms (product names, proper nouns, jargon).
language_configUnion type
language_config can be only one of the following:Optional. The model will detect the language automatically.
Optional. Specifies one or more languages in the audio.
| JSON representation |
|---|
{ "adaptationPhrases": [ string ], "customVocabulary": [ string ], // language_config "languageAuto": { object ( |
LanguageAuto
This type has no fields.
Indicates the language of the audio should be automatically detected.
LanguageHints
Provides hints to the model about possible languages present in the audio.
languageCodes[]string
Required. BCP-47 language codes.
| JSON representation |
|---|
{ "languageCodes": [ string ] } |
HistoryConfig
History configuration.
This message is included in the session configuration as BidiGenerateContentSetup.history_config. Configures the exchange of history messages.
initialHistoryInClientContentboolean
Optional. If true, after sending setupComplete, the server will wait and at first process clientContent messages until turnComplete is true. This initial history will not trigger a model call and may end with role MODEL. After turnComplete is true, the client can start the realtime conversation via realtimeInput.
| JSON representation |
|---|
{ "initialHistoryInClientContent": boolean } |
Method: auth_tokens.create
Creates a token that can be used to constrain the behavior of a BidiGenerateContent session.
Endpoint
posthttps: / /generativelanguage.googleapis.com /v1beta /auth_tokens
Request body
The request body contains an instance of AuthToken.
Optional. Input only. Immutable. An optional time after which, when using the resulting token, messages in BidiGenerateContent sessions will be rejected. (Gemini may preemptively close the session after this time.)
If not set then this defaults to 30 minutes in the future. If set, this value must be less than 20 hours in the future.
Uses RFC 3339, where generated output will always be Z-normalized and use 0, 3, 6 or 9 fractional digits. Offsets other than "Z" are also accepted. Examples: "2014-10-02T15:01:23Z", "2014-10-02T15:01:23.045123456Z" or "2014-10-02T15:01:23+05:30".
Optional. Input only. Immutable. The time after which new Live API sessions using the token resulting from this request will be rejected.
If not set this defaults to 60 seconds in the future. If set, this value must be less than 20 hours in the future.
Uses RFC 3339, where generated output will always be Z-normalized and use 0, 3, 6 or 9 fractional digits. Offsets other than "Z" are also accepted. Examples: "2014-10-02T15:01:23Z", "2014-10-02T15:01:23.045123456Z" or "2014-10-02T15:01:23+05:30".
Optional. Input only. Immutable. If fieldMask is empty, and bidiGenerateContentSetup is not present, then the effective BidiGenerateContentSetup message is taken from the Live API connection.
If fieldMask is empty, and bidiGenerateContentSetup is present, then the effective BidiGenerateContentSetup message is taken entirely from bidiGenerateContentSetup in this request. The setup message from the Live API connection is ignored.
If fieldMask is not empty, then the corresponding fields from bidiGenerateContentSetup will overwrite the fields from the setup message in the Live API connection.
This is a comma-separated list of fully qualified names of fields. Example: "user.displayName,photo".
configUnion type
config can be only one of the following:Optional. Input only. Immutable. Configuration specific to BidiGenerateContent.
usesinteger
Optional. Input only. Immutable. The number of times the token can be used. If this value is zero then no limit is applied. Resuming a Live API session does not count as a use. If unspecified, the default is 1.
Response body
If successful, the response body contains a newly created instance of AuthToken.