Gemini Robotics ER 2 API Tutorial: Streaming Robots, Function Calling, and Migration

Google will open the Gemini Robotics ER 2 Public Preview on July 30, 2026. This time, instead of just replacing one model name, it simultaneously offers both standard inference and real-time streaming access routes:

• gemini-robotics-er-2-preview: Suitable for single-shot image and video analysis and multi-step tool orchestration.
• gemini-robotics-er-2-streaming-preview: Continuously receives audio, video, and text via the Live API, suitable for low-latency bot interactions.

Legacy gemini-robotics-er-1.6-preview will cease service on August 31, 2026. Projects already using the old model should complete model name replacement and regression testing as soon as possible, and do not wait until the shutdown day to migrate.

Quick Conclusion: Which endpoint should be chosen?

If the robot plans actions after receiving a photo, or needs to analyze an already recorded video, it prioritizes using standard endpoints. If the robot needs to continuously observe the camera, receive voice commands, and repeatedly call tools like movement and grab during sessions, select the real-time streaming endpoint.

Both can receive text, images, video, and audio, but the output is text. If the robot needs to speak, it must also hand the text over to an external TTS service or declare the TTS as a callable tool.

What is Embodied Reasoning in ER 2?

ER stands for Embodied Reasoning. Ordinary vision models focus more on what’s in the image, while embodied reasoning models need to understand where the object is, how it arrives, what step the task has taken, and what robot action to call next. The standard ER 2 is built on Gemini 3.5 Flash, with official highlights including:

• Spatial reasoning: Identify points, track objects, generate bounding boxes, and plan trajectories.
• Agentic Vision: Combines code execution and image processing to perform visual analysis.
• Video Understanding: Identify critical moments to assess task progress and completion status.
• Tool orchestration: Combining custom bot APIs into a growth process.
• Multi-robot collaboration: coordinate different devices based on task status.

This does not mean the model can directly and securely control the real hardware. The model is responsible for understanding and making recommendations, while developers still have to limit speed, scope, permissions, and stop conditions at the tool execution layer.

Functional Differences Between the Standard and Streaming Endpoints

Although the two models have similar names, the Gemini API features they support are not exactly the same.

The standard version also supports Computer Use, Google Maps Grounding, and other features, but does not support Live API. The streaming version is optimized for persistent connections and sensor inputs; don’t assume that just because the name includes ER 2, the configuration parameters on both sides can be swapped as is. Both have input token caps of 131,072 and output token caps of 65,536. High-resolution input and higher Thinking levels increase latency, so when balancing speed and inference quality, The official recommendation is to start testing with Medium Thinking.

Prepare the Python SDK and API Key

Install or update the Google Gen AI SDK:

1

pip install -U google-genai

Then, configure the API Key in the environment variables:

1

$env:GEMINI_API_KEY = "你的_API_Key"

Do not write keys directly into the code repository. If the request returns 403, Also check whether the API Key is completely unrestricted. The Robotics API requires adding appropriate API restrictions for keys, Unrestricted keys may be rejected.

Example 1: Use the Standard Endpoint to Locate Objects in an Image

For example, upload an image first, Then the model is required to return up to ten target points. The coordinate order is [y, x], and normalized to 0 to 1000.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

from google import genai

PROMPT = """
Point to no more than 10 items in the image.
Return JSON objects with point coordinates in [y, x] order,
normalized to 0-1000, and an identifying label.
"""

client = genai.Client()
uploaded_file = client.files.upload(file="my-image.png")

response = client.interactions.create(
    model="gemini-robotics-er-2-preview",
    input=[
        {
            "type": "image",
            "uri": uploaded_file.uri,
            "mime_type": uploaded_file.mime_type,
        },
        {"type": "text", "text": PROMPT},
    ],
    generation_config={"thinking_level": "high"},
)

print(response.output_text)

After receiving the output, do not immediately drive the robotic arm. At minimum, the following verifications should be added:

1. The acknowledgment result can be parsed into the expected JSON structure.
2. Make sure each coordinate is within the range of 0 to 1000.
3. Convert normalized coordinates into prime coordinates of the original image.
4. Reject unknown labels, empty labels, and abnormal quantity results.
5. Convert two-dimensional positions into robot coordinate systems through camera calibration.
6. Check workspace, collision risk, and confidence conditions before execution.

If the object is too small or heavily obstructed, You can crop and enlarge the target area first. Light, contrast, and camera angle all affect spatial judgment, High-precision tasks can be repeatedly queried and use consistent results, However, this still cannot replace sensors and safety controllers.

Example 2: Declaring robot actions as tools

Models should not be directly stitched together and executed by the robot SDK commands. A safer approach is to expose only the tools that have been reviewed, Parameters are also whitelisted for verification. Physical actions in the streaming Robotics Live API must be declared as blocking calls:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

tools = [
    {
        "function_declarations": [
            {
                "name": "navigate",
                "description": "Navigate to a named safe waypoint.",
                "behavior": "BLOCKING",
                "parameters": {
                    "type": "OBJECT",
                    "properties": {
                        "name": {"type": "STRING"},
                    },
                    "required": ["name"],
                },
            }
        ]
    }
]

"behavior": "BLOCKING" means: After the model issues an action call, You must wait for the execution side to return the result, Only then can we continue planning for the next steps. For example, when a robot is moving to a safe point, The model cannot be moved before the move is complete, Assume it is already in place and call the grab tool. The tool execution layer should also ensure:

• Only predefined action names and safe spots are allowed.
• Limiting the robotic arm’s speed, force, joint angle, and range of motion.
• Timeout and cancellation mechanisms are set for each action.
• Retains hardware emergency stop and independent safety interlock.
• Clearly return success, failure, timeout, and sensor status to the model.
• Adding manual confirmation or policy engine approval for dangerous actions.
• Record model requests, tool parameters, execution results, and timestamps.

The natural language output by the model can only serve as suggestions, It cannot bypass the tool layer and directly convert into motor commands.

How Real-Time Streaming Endpoints Work

gemini-robotics-er-2-streaming-preview Use dedicated Live APIs, Maintain sessions through persistent, stateful WebSocket connections. A typical cycle consists of three steps:

1. Declare the bot action tool in the session configuration.
2. Continuously sending camera, microphone, or text input.
3. Receive the tool call, execute the robot SDK, and then return the tool result.

There are also clear restrictions on audio and video input:

• Audio uses native PCM, 16-bit, 16 kHz, small-end format.
• Videos are sent as JPEG image frames, up to 1 FPS.
• The video frame itself does not automatically trigger model inference.
• Requires triggering responses with text or audio commands.
• For ongoing proactive monitoring, regularly send heartbeat alerts.

The last point is easy to fall into. Continuously uploading camera footage, This does not mean the model will actively open or call tools when it sees an exception. Warehouse inspections, production line monitoring, and other scenarios, Clear heartbeat commands should be designed, For example, ask the model to check for blockages, falls, or personnel entering danger zones in the latest footage. Heart rate should also not be increased indefinitely. Considering the 1 FPS video limit, latency, quota, and actual risk window, Determine a reasonable inspection cycle.

Migrating from ER 1.6 to ER 2

The smallest code change is to replace the model string:

1
2
3

OLD_MODEL = "gemini-robotics-er-1.6-preview"
NEW_MODEL = "gemini-robotics-er-2-preview"
STREAMING_MODEL = "gemini-robotics-er-2-streaming-preview"

But formal migration cannot be based solely on whether the request returns 200. It is recommended to handle them in the following order:

1. Take stock of all code, environment variables, and old model names in configuration files.
2. Choose the standard or streaming version according to the business type to avoid mechanical replacement errors.
3. Check whether the API capabilities currently depended on are supported by the target endpoint.
4. Update the SDK and use the restricted API Key in the test environment.
5. Revalidate tool schemas, blocking behaviors, and error handling.
6. Compare the results of the previous version with ER 2 using a fixed test set.
7. Low-traffic grayscale online to observe latency, failure rate, and operation abortion rate.
8. Remove the old version rollback path by August 31, 2026.

The regression test set should at least cover:

• Pointer coordinates and bounding boxes in images.
• Small objects, obscured objects, and low-contrast scenes.
• Instrument readings and object orientation determination.
• Video key moments, task progress, and completion status.
• The order and stop conditions for multi-step function calls.
• Tool failure, timeout, and return abnormal formats.
• Refusal or security responses after personnel enter the work area.
• Latency and quality at different Thinking levels.

During the Public Preview phase, behaviors or interfaces may still be adjusted, Production projects should be locked to the SDK version, Record model identification and test results, And continue to follow the update log.

Security, Privacy, and Production Deployment

Errors in the bot model can result in real property damage or personal risk. Developers need to be responsible for the physical environment and the final actions, You can’t treat “correct model judgment” as security certification. The production environment suggests breaking down the system into three layers:

• Understanding Layer: ER 2 is responsible for perception, reasoning, and proposing tool calls.
• Policy layer: Validates permissions, parameters, status, and security rules.
• Execution layer: Robot controllers perform actions and provide hardware protection.

When it comes to camera and microphone data involving personnel, Clear notification and necessary consent should be obtained. Minimize personal data collection, It can avoid images of identifiable people, Or perform facial blur or other processing before uploading. Do not store original audio and video, exact location, or identity information indefinitely in logs. Access control, retention period, and deletion mechanisms should be set, Compliance assessments are conducted according to deployment regions.

Frequently Asked Questions

Can ER 2 output robot actions directly?

It can select the robot actions defined by the developer through function calls, But the real action is still performed by your tool’s execution layer. Physical operations should use blocking calls, Parameter validation and safety interlocking have been passed.

Can the Standard Endpoint Use the Live API?

No. Real-time streaming sessions must be used gemini-robotics-er-2-streaming-preview。

Does streaming automatically monitor all video frames?

No. Video frames do not trigger inference alone, Text, audio commands, or regular heartbeats are required.

Does streaming generate voice directly?

No, the model output is text. Voice broadcasts require external TTS, TTS can also be packaged as tools.

Why does the API Key return 403?

First, check if the key has any restrictions. The official explanation states that API keys without restrictions will be rejected, Appropriate API restrictions should be configured for them, At the same time, confirm that both project and API permissions are correct.

Should You Migrate Now?

It should be. Old version gemini-robotics-er-1.6-preview Service will cease on August 31, 2026. First, we conducted the comparison test in a fixed scene, Then gradually switch production flow.

Summary

Gemini Robotics ER 2 divides the robot API into two clear routes: The standard version handles more complete spatial, video, and multi-step reasoning, Streaming is responsible for low-latency, continuous audio and video interaction. What truly determines project reliability, Not only choosing the correct model name, It also includes blocking function calls, tool whitelists, state returns, hardware interlocking, and establishing repeatable regression tests for real-world scenarios. If you are still using ER 1.6, The migration should start now, Finalize the grayscale launch and rollback plan before the shutdown date.

Official Resources

• Gemini API Update Log
• Gemini Robotics ER Overview
• Gemini Robotics ER Real-Time Streaming Guide