Mortdecai
eecebe7ef5
Five-lane parallel research pass. Each subdir under tooling/ has its own README indexing downloaded files with verified upstream sources. - google-official/: deepmind-gemma JAX examples, gemma_pytorch scripts, gemma.cpp API server docs, google-gemma/cookbook notebooks, ai.google.dev HTML snapshots, Gemma 3 tech report - huggingface/: 8 gemma-4-* model cards, chat-template .jinja files, tokenizer_config.json, transformers gemma4/ source, launch blog posts, official HF Spaces app.py - inference-frameworks/: vLLM/llama.cpp/MLX/Keras-hub/TGI/Gemini API/Vertex AI comparison, run_commands.sh with 8 working launches, 9 code snippets - gemma-family/: 12 per-variant briefs (ShieldGemma 2, CodeGemma, PaliGemma 2, Recurrent/Data/Med/TxGemma, Embedding/Translate/Function/Dolphin/SignGemma) - fine-tuning/: Unsloth Gemma 4 notebooks, Axolotl YAMLs (incl 26B-A4B MoE), TRL scripts, Google cookbook fine-tune notebooks, recipe-recommendation.md Findings that update earlier CORPUS_* docs are flagged in tooling/README.md (not applied) — notably the new <|turn>/<turn|> prompt format, gemma_pytorch abandonment, gemma.cpp Gemini-API server, transformers AutoModelForMultimodalLM, FA2 head_dim=512 break, 26B-A4B MoE quantization rules, no Gemma 4 tech report PDF yet, no Gemma-4-generation specialized siblings yet. Pre-commit secrets hook bypassed per user authorization — flagged "secrets" are base64 notebook cell outputs and example Ed25519 keys in the HDP agentic-security demo, not real credentials. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1002 lines
38 KiB
Plaintext
1002 lines
38 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e5ed3a20",
|
|
"metadata": {
|
|
"id": "-u7xRR3DeFXz"
|
|
},
|
|
"source": [
|
|
"##### Copyright 2025 Google LLC."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "87fc2129",
|
|
"metadata": {
|
|
"cellView": "form",
|
|
"id": "oed1Dh9SeIlD"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
|
"# you may not use this file except in compliance with the License.\n",
|
|
"# You may obtain a copy of the License at\n",
|
|
"#\n",
|
|
"# https://www.apache.org/licenses/LICENSE-2.0\n",
|
|
"#\n",
|
|
"# Unless required by applicable law or agreed to in writing, software\n",
|
|
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
|
|
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
|
|
"# See the License for the specific language governing permissions and\n",
|
|
"# limitations under the License."
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"id": "be48d087",
|
|
"metadata": {
|
|
"id": "gdkDG20KtfsH"
|
|
},
|
|
"source": [
|
|
"# Function calling with Gemma 4"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b06867b0",
|
|
"metadata": {
|
|
"id": "DzkEdu7YxUOM"
|
|
},
|
|
"source": [
|
|
"<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
|
|
" <td>\n",
|
|
" <a target=\"_blank\" href=\"https://ai.google.dev/gemma/docs/capabilities/text/function-calling-gemma4\"><img src=\"https://ai.google.dev/static/site-assets/images/docs/notebook-site-button.png\" height=\"32\" width=\"32\" />View on ai.google.dev</a>\n",
|
|
" </td>\n",
|
|
" <td>\n",
|
|
" <a target=\"_blank\" href=\"https://colab.research.google.com/github/google-gemma/cookbook/blob/main/docs/capabilities/text/function-calling-gemma4.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
|
|
" </td>\n",
|
|
" <td>\n",
|
|
" <a target=\"_blank\" href=\"https://kaggle.com/kernels/welcome?src=https://github.com/google-gemma/cookbook/blob/main/docs/capabilities/text/function-calling-gemma4.ipynb\"><img src=\"https://www.kaggle.com/static/images/logos/kaggle-logo-transparent-300.png\" height=\"32\" width=\"70\"/>Run in Kaggle</a>\n",
|
|
" </td>\n",
|
|
" <td>\n",
|
|
" <a target=\"_blank\" href=\"https://console.cloud.google.com/vertex-ai/colab/import/https%3A%2F%2Fraw.githubusercontent.com%2Fgoogle-gemma%2Fcookbook%2Fmain%2Fdocs%2Fcapabilities%2Ftext%2Ffunction-calling-gemma4.ipynb\"><img src=\"https://ai.google.dev/images/cloud-icon.svg\" width=\"40\" />Open in Vertex AI</a>\n",
|
|
" </td>\n",
|
|
" <td>\n",
|
|
" <a target=\"_blank\" href=\"https://github.com/google-gemma/cookbook/blob/main/docs/capabilities/text/function-calling-gemma4.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
|
|
" </td>\n",
|
|
"</table>"
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"id": "c9252776",
|
|
"metadata": {
|
|
"id": "4A2wX6qLPAFc"
|
|
},
|
|
"source": [
|
|
"When using a generative artificial intelligence (AI) model such as Gemma, you\n",
|
|
"may want to use the model to operate programming interfaces in order to complete\n",
|
|
"tasks or answer questions. Instructing a model by defining a programming\n",
|
|
"interface and then making a request that uses that interface is called *function\n",
|
|
"calling*.\n",
|
|
"\n",
|
|
"> Important: *A Gemma model cannot execute code on its own.* When you\n",
|
|
"generate code with function calling, you must run the generated code yourself or\n",
|
|
"run it as part of your application. Always put safeguards in place to validate\n",
|
|
"any generated code before executing it.\n",
|
|
"\n",
|
|
"This guide shows the process of using Gemma 4 within the Hugging Face ecosystem."
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"id": "b5711fa8",
|
|
"metadata": {
|
|
"id": "brMXesIJu6_0"
|
|
},
|
|
"source": [
|
|
"This notebook will run on T4 GPU."
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"id": "1a138a16",
|
|
"metadata": {
|
|
"id": "JYq4z39uu-1a"
|
|
},
|
|
"source": [
|
|
"## Install Python packages\n",
|
|
"\n",
|
|
"Install the Hugging Face libraries required for running the Gemma model and making requests."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "be2ea88b",
|
|
"metadata": {
|
|
"id": "BKczkppWvBAD"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Install PyTorch & other libraries\n",
|
|
"!pip install torch accelerate\n",
|
|
"\n",
|
|
"# Install the transformers library\n",
|
|
"!pip install transformers"
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"id": "5ec0919e",
|
|
"metadata": {
|
|
"id": "eylDAA5LxVXr"
|
|
},
|
|
"source": [
|
|
"## Load Model\n",
|
|
"\n",
|
|
"Use the `transformers` libraries to create an instance of a `processor` and `model` using the `AutoProcessor` and `AutoModelForImageTextToText` classes as shown in the following code example:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "6c8a4ae4",
|
|
"metadata": {
|
|
"id": "vNLDpsrxxW76"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"application/vnd.jupyter.widget-view+json": {
|
|
"model_id": "ebbd88f6ff4f4c29913db37e2b461389",
|
|
"version_major": 2,
|
|
"version_minor": 0
|
|
},
|
|
"text/plain": [
|
|
"Loading weights: 0%| | 0/2011 [00:00<?, ?it/s]"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"MODEL_ID = \"google/gemma-4-E2B-it\" # @param [\"google/gemma-4-E2B-it\",\"google/gemma-4-E4B-it\", \"google/gemma-4-31B-it\", \"google/gemma-4-26B-A4B-it\"]\n",
|
|
"\n",
|
|
"from transformers import AutoProcessor, AutoModelForMultimodalLM\n",
|
|
"\n",
|
|
"model = AutoModelForMultimodalLM.from_pretrained(MODEL_ID, dtype=\"auto\", device_map=\"auto\")\n",
|
|
"processor = AutoProcessor.from_pretrained(MODEL_ID)"
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"id": "84d87d6e",
|
|
"metadata": {
|
|
"id": "9XeOTt3Tslrk"
|
|
},
|
|
"source": [
|
|
"## Passing Tools\n",
|
|
"\n",
|
|
"You can pass tools to the model using the `apply_chat_template()` function via the `tools` argument. There are two methods for defining these tools:\n",
|
|
"\n",
|
|
"- **JSON schema**: You can manually construct a JSON dictionary defining the function name, description, and parameters (including types and required fields).\n",
|
|
"- **Raw Python Functions**: You can pass actual Python functions. The system automatically generates the required JSON schema by parsing the function's type hints, arguments, and docstrings. For best results, docstrings should adhere to the [Google Python Style Guide](https://google.github.io/styleguide/pyguide.html#38-comments-and-docstrings).\n",
|
|
"\n",
|
|
"Below is the example with the JSON schema."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"id": "d123c11c",
|
|
"metadata": {
|
|
"id": "Goe0xrndM8xx"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"<bos><|turn>system\n",
|
|
"You are a helpful assistant.<|tool>declaration:get_current_temperature{description:<|\"|>Gets the current temperature for a given location.<|\"|>,parameters:{properties:{location:{description:<|\"|>The city name, e.g. San Francisco<|\"|>,type:<|\"|>STRING<|\"|>}},required:[<|\"|>location<|\"|>],type:<|\"|>OBJECT<|\"|>}}<tool|><turn|>\n",
|
|
"<|turn>user\n",
|
|
"What's the temperature in London?<turn|>\n",
|
|
"<|turn>model\n",
|
|
"<|tool_call>call:get_current_temperature{location:<|\"|>London<|\"|>}<tool_call|><|tool_response>\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from transformers import TextStreamer\n",
|
|
"\n",
|
|
"weather_function_schema = {\n",
|
|
" \"type\": \"function\",\n",
|
|
" \"function\": {\n",
|
|
" \"name\": \"get_current_temperature\",\n",
|
|
" \"description\": \"Gets the current temperature for a given location.\",\n",
|
|
" \"parameters\": {\n",
|
|
" \"type\": \"object\",\n",
|
|
" \"properties\": {\n",
|
|
" \"location\": {\n",
|
|
" \"type\": \"string\",\n",
|
|
" \"description\": \"The city name, e.g. San Francisco\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" \"required\": [\"location\"],\n",
|
|
" },\n",
|
|
" }\n",
|
|
"}\n",
|
|
"\n",
|
|
"message = [\n",
|
|
" {\n",
|
|
" \"role\": \"system\", \"content\": \"You are a helpful assistant.\"\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"role\": \"user\", \"content\": \"What's the temperature in London?\"\n",
|
|
" }\n",
|
|
"]\n",
|
|
"\n",
|
|
"text = processor.apply_chat_template(message, tools=[weather_function_schema], tokenize=False, add_generation_prompt=True)\n",
|
|
"inputs = processor(text=text, return_tensors=\"pt\").to(model.device)\n",
|
|
"streamer = TextStreamer(processor)\n",
|
|
"outputs = model.generate(**inputs, streamer=streamer, max_new_tokens=64)"
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"id": "b089527f",
|
|
"metadata": {
|
|
"id": "8yRB855BuiQ6"
|
|
},
|
|
"source": [
|
|
"And the same example with the raw Python function."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"id": "eb63e062",
|
|
"metadata": {
|
|
"id": "pRjDUMBGuk0s"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"<bos><|turn>system\n",
|
|
"<|tool>declaration:get_current_temperature{description:<|\"|>Gets the current temperature for a given location.<|\"|>,parameters:{properties:{location:{description:<|\"|>The city name, e.g. San Francisco<|\"|>,type:<|\"|>STRING<|\"|>}},required:[<|\"|>location<|\"|>],type:<|\"|>OBJECT<|\"|>}}<tool|><turn|>\n",
|
|
"<|turn>user\n",
|
|
"What's the temperature in London?<turn|>\n",
|
|
"<|turn>model\n",
|
|
"<|tool_call>call:get_current_temperature{location:<|\"|>London<|\"|>}<tool_call|><|tool_response>\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from transformers.utils import get_json_schema\n",
|
|
"\n",
|
|
"def get_current_temperature(location: str):\n",
|
|
" \"\"\"\n",
|
|
" Gets the current temperature for a given location.\n",
|
|
"\n",
|
|
" Args:\n",
|
|
" location: The city name, e.g. San Francisco\n",
|
|
" \"\"\"\n",
|
|
" return \"15°C\"\n",
|
|
"\n",
|
|
"message = [\n",
|
|
" {\n",
|
|
" \"role\": \"user\", \"content\": \"What's the temperature in London?\"\n",
|
|
" }\n",
|
|
"]\n",
|
|
"\n",
|
|
"text = processor.apply_chat_template(message, tools=[get_json_schema(get_current_temperature)], tokenize=False, add_generation_prompt=True)\n",
|
|
"inputs = processor(text=text, return_tensors=\"pt\").to(model.device)\n",
|
|
"streamer = TextStreamer(processor)\n",
|
|
"outputs = model.generate(**inputs, streamer=streamer, max_new_tokens=256)"
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"id": "422d1ce4",
|
|
"metadata": {
|
|
"id": "Fcr4EZWMMg6L"
|
|
},
|
|
"source": [
|
|
"## Full function calling sequence\n",
|
|
"\n",
|
|
"This section demonstrates a three-stage cycle for connecting the model to external tools: the **Model's Turn** to generate function call objects, the **Developer's Turn** to parse and execute code (such as a weather API), and the **Final Response** where the model uses the tool's output to answer the user."
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"id": "86ee44ab",
|
|
"metadata": {
|
|
"id": "_bk7y3hKNMMO"
|
|
},
|
|
"source": [
|
|
"### Model's Turn\n",
|
|
"\n",
|
|
"Here's the user prompt `\"Hey, what's the weather in Tokyo right now?\"`, and the tool `[get_current_weather]`. Gemma generates a function call object as follows."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 6,
|
|
"id": "d74862bc",
|
|
"metadata": {
|
|
"id": "mqAOVhFAMsrm"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Prompt: Hey, what's the weather in Tokyo right now?\n",
|
|
"Tools: [<function get_current_weather at 0x7cef824ece00>]\n",
|
|
"Output: <|tool_call>call:get_current_weather{location:<|\"|>Tokyo, JP<|\"|>}<tool_call|><|tool_response>\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# Define a function that our model can use.\n",
|
|
"def get_current_weather(location: str, unit: str = \"celsius\"):\n",
|
|
" \"\"\"\n",
|
|
" Gets the current weather in a given location.\n",
|
|
"\n",
|
|
" Args:\n",
|
|
" location: The city and state, e.g. \"San Francisco, CA\" or \"Tokyo, JP\"\n",
|
|
" unit: The unit to return the temperature in. (choices: [\"celsius\", \"fahrenheit\"])\n",
|
|
"\n",
|
|
" Returns:\n",
|
|
" temperature: The current temperature in the given location\n",
|
|
" weather: The current weather in the given location\n",
|
|
" \"\"\"\n",
|
|
" return {\"temperature\": 15, \"weather\": \"sunny\"}\n",
|
|
"\n",
|
|
"prompt = \"Hey, what's the weather in Tokyo right now?\"\n",
|
|
"tools = [get_current_weather]\n",
|
|
"\n",
|
|
"message = [\n",
|
|
" {\n",
|
|
" \"role\": \"system\", \"content\": \"You are a helpful assistant.\"\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"role\": \"user\", \"content\": prompt\n",
|
|
" },\n",
|
|
"]\n",
|
|
"\n",
|
|
"text = processor.apply_chat_template(message, tools=tools, tokenize=False, add_generation_prompt=True)\n",
|
|
"inputs = processor(text=text, return_tensors=\"pt\").to(model.device)\n",
|
|
"out = model.generate(**inputs, max_new_tokens=128)\n",
|
|
"generated_tokens = out[0][len(inputs[\"input_ids\"][0]):]\n",
|
|
"output = processor.decode(generated_tokens, skip_special_tokens=False)\n",
|
|
"\n",
|
|
"print(f\"Prompt: {prompt}\")\n",
|
|
"print(f\"Tools: {tools}\")\n",
|
|
"print(f\"Output: {output}\")"
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"id": "87d67184",
|
|
"metadata": {
|
|
"id": "lBnyE8JqOnDc"
|
|
},
|
|
"source": [
|
|
"### Developer's Turn\n",
|
|
"\n",
|
|
"Your application should parse the model's response to extract the function name and argments, and append `tool_calls` and `tool_responses` with the `assistant` role.\n",
|
|
"\n",
|
|
"> NOTE: Always validate function names and arguments before execution."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 7,
|
|
"id": "0810d0e9",
|
|
"metadata": {
|
|
"id": "xpRfHIVlOuFx"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"{\n",
|
|
" \"role\": \"assistant\",\n",
|
|
" \"tool_calls\": [\n",
|
|
" {\n",
|
|
" \"function\": {\n",
|
|
" \"name\": \"get_current_weather\",\n",
|
|
" \"arguments\": {\n",
|
|
" \"location\": \"Tokyo, JP\"\n",
|
|
" }\n",
|
|
" }\n",
|
|
" }\n",
|
|
" ],\n",
|
|
" \"tool_responses\": [\n",
|
|
" {\n",
|
|
" \"name\": \"get_current_weather\",\n",
|
|
" \"response\": {\n",
|
|
" \"temperature\": 15,\n",
|
|
" \"weather\": \"sunny\"\n",
|
|
" }\n",
|
|
" }\n",
|
|
" ]\n",
|
|
"}\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"import re\n",
|
|
"import json\n",
|
|
"\n",
|
|
"def extract_tool_calls(text):\n",
|
|
" def cast(v):\n",
|
|
" try: return int(v)\n",
|
|
" except:\n",
|
|
" try: return float(v)\n",
|
|
" except: return {'true': True, 'false': False}.get(v.lower(), v.strip(\"'\\\"\"))\n",
|
|
"\n",
|
|
" return [{\n",
|
|
" \"name\": name,\n",
|
|
" \"arguments\": {\n",
|
|
" k: cast((v1 or v2).strip())\n",
|
|
" for k, v1, v2 in re.findall(r'(\\w+):(?:<\\|\"\\|>(.*?)<\\|\"\\|>|([^,}]*))', args)\n",
|
|
" }\n",
|
|
" } for name, args in re.findall(r\"<\\|tool_call>call:(\\w+)\\{(.*?)\\}<tool_call\\|>\", text, re.DOTALL)]\n",
|
|
"\n",
|
|
"calls = extract_tool_calls(output)\n",
|
|
"if calls:\n",
|
|
" # Call the function and get the result\n",
|
|
" #####################################\n",
|
|
" # WARNING: This is a demonstration. #\n",
|
|
" #####################################\n",
|
|
" # Using globals() to call functions dynamically can be dangerous in\n",
|
|
" # production. In a real application, you should implement a secure way to\n",
|
|
" # map function names to actual function calls, such as a predefined\n",
|
|
" # dictionary of allowed tools and their implementations.\n",
|
|
" results = [\n",
|
|
" {\"name\": c['name'], \"response\": globals()[c['name']](**c['arguments'])}\n",
|
|
" for c in calls\n",
|
|
" ]\n",
|
|
"\n",
|
|
" message.append({\n",
|
|
" \"role\": \"assistant\",\n",
|
|
" \"tool_calls\": [\n",
|
|
" {\"function\": call} for call in calls\n",
|
|
" ],\n",
|
|
" \"tool_responses\": results\n",
|
|
" })\n",
|
|
" print(json.dumps(message[-1], indent=2))\n"
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"id": "22347163",
|
|
"metadata": {
|
|
"id": "_wXyQtm8SRfi"
|
|
},
|
|
"source": [
|
|
"> Note: For optimal results, append the tool execution result to your message history using the specific format below. This ensures the chat template correctly generates the required token structure (e.g., `response:get_current_weather{temperature:15,weather:<|\"|>sunny<|\"|>}`).\n",
|
|
"\n",
|
|
"```python\n",
|
|
"\"tool_responses\": [\n",
|
|
" {\n",
|
|
" \"name\": function_name,\n",
|
|
" \"response\": function_response\n",
|
|
" }\n",
|
|
"]\n",
|
|
"```\n",
|
|
"\n",
|
|
"In case of multiple independent requests:\n",
|
|
"\n",
|
|
"```python\n",
|
|
"\"tool_responses\": [\n",
|
|
" {\n",
|
|
" \"name\": function_name_1,\n",
|
|
" \"response\": function_response_1\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"name\": function_name_2,\n",
|
|
" \"response\": function_response_2\n",
|
|
" }\n",
|
|
"]\n",
|
|
"```"
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"id": "a6cbcd72",
|
|
"metadata": {
|
|
"id": "qpJrjXgtSh3w"
|
|
},
|
|
"source": [
|
|
"### Final Response\n",
|
|
"\n",
|
|
"Finally, Gemma reads the tool response and reply to the user."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 8,
|
|
"id": "1c8961e8",
|
|
"metadata": {
|
|
"id": "tS6IBGaGSm0i"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Output: The current weather in Tokyo is 15 degrees and sunny.\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"text = processor.apply_chat_template(message, tools=tools, tokenize=False, add_generation_prompt=True)\n",
|
|
"inputs = processor(text=text, return_tensors=\"pt\").to(model.device)\n",
|
|
"out = model.generate(**inputs, max_new_tokens=128)\n",
|
|
"generated_tokens = out[0][len(inputs[\"input_ids\"][0]):]\n",
|
|
"output = processor.decode(generated_tokens, skip_special_tokens=True)\n",
|
|
"print(f\"Output: {output}\")\n",
|
|
"message[-1][\"content\"] = output"
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"id": "3196dd09",
|
|
"metadata": {
|
|
"id": "7jCc58grS81c"
|
|
},
|
|
"source": [
|
|
"You can see the full chat history below."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 9,
|
|
"id": "46cdfec1",
|
|
"metadata": {
|
|
"id": "k1LQOKusS-BF"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"[\n",
|
|
" {\n",
|
|
" \"role\": \"system\",\n",
|
|
" \"content\": \"You are a helpful assistant.\"\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"role\": \"user\",\n",
|
|
" \"content\": \"Hey, what's the weather in Tokyo right now?\"\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"role\": \"assistant\",\n",
|
|
" \"tool_calls\": [\n",
|
|
" {\n",
|
|
" \"function\": {\n",
|
|
" \"name\": \"get_current_weather\",\n",
|
|
" \"arguments\": {\n",
|
|
" \"location\": \"Tokyo, JP\"\n",
|
|
" }\n",
|
|
" }\n",
|
|
" }\n",
|
|
" ],\n",
|
|
" \"tool_responses\": [\n",
|
|
" {\n",
|
|
" \"name\": \"get_current_weather\",\n",
|
|
" \"response\": {\n",
|
|
" \"temperature\": 15,\n",
|
|
" \"weather\": \"sunny\"\n",
|
|
" }\n",
|
|
" }\n",
|
|
" ],\n",
|
|
" \"content\": \"The current weather in Tokyo is 15 degrees and sunny.\"\n",
|
|
" }\n",
|
|
"]\n",
|
|
"--------------------------------------------------------------------------------\n",
|
|
"Output: <bos><|turn>system\n",
|
|
"You are a helpful assistant.<|tool>declaration:get_current_weather{description:<|\"|>Gets the current weather in a given location.<|\"|>,parameters:{properties:{location:{description:<|\"|>The city and state, e.g. \"San Francisco, CA\" or \"Tokyo, JP\"<|\"|>,type:<|\"|>STRING<|\"|>},unit:{description:<|\"|>The unit to return the temperature in.<|\"|>,enum:[<|\"|>celsius<|\"|>,<|\"|>fahrenheit<|\"|>],type:<|\"|>STRING<|\"|>}},required:[<|\"|>location<|\"|>],type:<|\"|>OBJECT<|\"|>}}<tool|><turn|>\n",
|
|
"<|turn>user\n",
|
|
"Hey, what's the weather in Tokyo right now?<turn|>\n",
|
|
"<|turn>model\n",
|
|
"<|tool_call>call:get_current_weather{location:<|\"|>Tokyo, JP<|\"|>}<tool_call|><|tool_response>response:get_current_weather{temperature:15,weather:<|\"|>sunny<|\"|>}<tool_response|>The current weather in Tokyo is 15 degrees and sunny.<turn|>\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# full history\n",
|
|
"print(json.dumps(message, indent=2))\n",
|
|
"\n",
|
|
"print(\"-\"*80)\n",
|
|
"output = processor.decode(out[0], skip_special_tokens=False)\n",
|
|
"print(f\"Output: {output}\")"
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"id": "e249b292",
|
|
"metadata": {
|
|
"id": "4sDRdwLclbsH"
|
|
},
|
|
"source": [
|
|
"### Function calling with Thinking\n",
|
|
"\n",
|
|
"By utilizing an internal reasoning process, the model significantly enhances its function-calling accuracy. This allows for more precise decision-making regarding when to trigger a tool and how to define its parameters."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 14,
|
|
"id": "925328f2",
|
|
"metadata": {
|
|
"id": "S9-nXig6lsMt"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Role: assistant\n",
|
|
"\n",
|
|
"=== Thoughts ===\n",
|
|
"1. **Analyze the Request:** The user is asking if it's \"good for running now\" in \"Seoul\".\n",
|
|
"\n",
|
|
"2. **Identify Necessary Information:** To determine if it's good for running, I need current weather information (temperature, precipitation, etc.) for Seoul.\n",
|
|
"\n",
|
|
"3. **Examine Available Tools:** The available tool is `get_current_weather(location, unit)`.\n",
|
|
"\n",
|
|
"4. **Determine Tool Arguments:**\n",
|
|
" * `location`: The user specified \"Seoul\".\n",
|
|
" * `unit`: The user did not specify a unit (Celsius or Fahrenheit).\n",
|
|
"\n",
|
|
"5. **Formulate the Tool Call:** I need to call `get_current_weather` with the location. Since the user didn't specify a unit, I can either omit it (if the tool defaults are acceptable) or choose a common one. However, the tool definition requires `location` but `unit` is optional.\n",
|
|
"\n",
|
|
"6. **Construct the Response Strategy:**\n",
|
|
" * Call the tool to get the weather data for Seoul.\n",
|
|
" * Once the data is received, I can advise the user on whether it's suitable for running.\n",
|
|
"\n",
|
|
"7. **Generate Tool Call:**\n",
|
|
" ```json\n",
|
|
" {\n",
|
|
" \"toolSpec\": {\n",
|
|
" \"name\": \"get_current_weather\",\n",
|
|
" \"args\": {\n",
|
|
" \"location\": \"Seoul\"\n",
|
|
" }\n",
|
|
" }\n",
|
|
" }\n",
|
|
" ```\n",
|
|
" (Self-correction: The `unit` parameter is optional in the definition, so just providing the location is sufficient to proceed.)\n",
|
|
"\n",
|
|
"8. **Final Output Generation:** Present the tool call to the user/system.\n",
|
|
"\n",
|
|
"=== Tool Calls ===\n",
|
|
"[{'type': 'function', 'function': {'name': 'get_current_weather', 'arguments': {'location': 'Seoul'}}}]\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"prompt = \"Hey, I'm in Seoul. Is it good for running now?\"\n",
|
|
"message = [\n",
|
|
" {\n",
|
|
" \"role\": \"system\", \"content\": \"You are a helpful assistant.\"\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"role\": \"user\", \"content\": prompt\n",
|
|
" },\n",
|
|
"]\n",
|
|
"\n",
|
|
"text = processor.apply_chat_template(message, tools=tools, tokenize=False, add_generation_prompt=True, enable_thinking=True)\n",
|
|
"inputs = processor(text=text, return_tensors=\"pt\").to(model.device)\n",
|
|
"input_len = inputs[\"input_ids\"].shape[-1]\n",
|
|
"\n",
|
|
"out = model.generate(**inputs, max_new_tokens=1024)\n",
|
|
"output = processor.decode(out[0][input_len:], skip_special_tokens=False)\n",
|
|
"result = processor.parse_response(output)\n",
|
|
"\n",
|
|
"for key, value in result.items():\n",
|
|
" if key == \"role\":\n",
|
|
" print(f\"Role: {value}\")\n",
|
|
" elif key == \"thinking\":\n",
|
|
" print(f\"\\n=== Thoughts ===\\n{value}\")\n",
|
|
" elif key == \"content\":\n",
|
|
" print(f\"\\n=== Answer ===\\n{value}\")\n",
|
|
" elif key == \"tool_calls\":\n",
|
|
" print(f\"\\n=== Tool Calls ===\\n{value}\")\n",
|
|
" else:\n",
|
|
" print(f\"\\n{key}: {value}...\\n\")\n"
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"id": "013a4826",
|
|
"metadata": {
|
|
"id": "JwBOHXWmpSyE"
|
|
},
|
|
"source": [
|
|
"Process the tool call and get the final answer."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 15,
|
|
"id": "1d7dd4b5",
|
|
"metadata": {
|
|
"id": "tPgZ2gjWpWoq"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Output: The current weather in Seoul is 15 degrees Celsius and sunny. That sounds like great weather for a run!\n",
|
|
"--------------------------------------------------------------------------------\n",
|
|
"Full History\n",
|
|
"--------------------------------------------------------------------------------\n",
|
|
"[\n",
|
|
" {\n",
|
|
" \"role\": \"system\",\n",
|
|
" \"content\": \"You are a helpful assistant.\"\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"role\": \"user\",\n",
|
|
" \"content\": \"Hey, I'm in Seoul. Is it good for running now?\"\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"role\": \"assistant\",\n",
|
|
" \"tool_calls\": [\n",
|
|
" {\n",
|
|
" \"function\": {\n",
|
|
" \"name\": \"get_current_weather\",\n",
|
|
" \"arguments\": {\n",
|
|
" \"location\": \"Seoul\"\n",
|
|
" }\n",
|
|
" }\n",
|
|
" }\n",
|
|
" ],\n",
|
|
" \"tool_responses\": [\n",
|
|
" {\n",
|
|
" \"name\": \"get_current_weather\",\n",
|
|
" \"response\": {\n",
|
|
" \"temperature\": 15,\n",
|
|
" \"weather\": \"sunny\"\n",
|
|
" }\n",
|
|
" }\n",
|
|
" ],\n",
|
|
" \"content\": \"The current weather in Seoul is 15 degrees Celsius and sunny. That sounds like great weather for a run!\"\n",
|
|
" }\n",
|
|
"]\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"calls = extract_tool_calls(output)\n",
|
|
"if calls:\n",
|
|
" # Call the function and get the result\n",
|
|
" #####################################\n",
|
|
" # WARNING: This is a demonstration. #\n",
|
|
" #####################################\n",
|
|
" # Using globals() to call functions dynamically can be dangerous in\n",
|
|
" # production. In a real application, you should implement a secure way to\n",
|
|
" # map function names to actual function calls, such as a predefined\n",
|
|
" # dictionary of allowed tools and their implementations.\n",
|
|
" results = [\n",
|
|
" {\"name\": c['name'], \"response\": globals()[c['name']](**c['arguments'])}\n",
|
|
" for c in calls\n",
|
|
" ]\n",
|
|
"\n",
|
|
" message.append({\n",
|
|
" \"role\": \"assistant\",\n",
|
|
" \"tool_calls\": [\n",
|
|
" {\"function\": call} for call in calls\n",
|
|
" ],\n",
|
|
" \"tool_responses\": results\n",
|
|
" })\n",
|
|
"\n",
|
|
"text = processor.apply_chat_template(message, tools=tools, tokenize=False, add_generation_prompt=True)\n",
|
|
"inputs = processor(text=text, return_tensors=\"pt\").to(model.device)\n",
|
|
"out = model.generate(**inputs, max_new_tokens=128)\n",
|
|
"generated_tokens = out[0][len(inputs[\"input_ids\"][0]):]\n",
|
|
"output = processor.decode(generated_tokens, skip_special_tokens=True)\n",
|
|
"print(f\"Output: {output}\")\n",
|
|
"message[-1][\"content\"] = output\n",
|
|
"\n",
|
|
"print(\"-\"*80)\n",
|
|
"print(\"Full History\")\n",
|
|
"print(\"-\"*80)\n",
|
|
"print(json.dumps(message, indent=2))"
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"id": "3e91bbb9",
|
|
"metadata": {
|
|
"id": "DXar9v3Ew75B"
|
|
},
|
|
"source": [
|
|
"## Important Caveat: Automatic vs. Manual Schemas\n",
|
|
"\n",
|
|
"When relying on automatic conversion from Python functions to JSON schema, the generated output may not always meet specific expectations regarding complex parameters.\n",
|
|
"\n",
|
|
"If a function uses a custom object (like a Config class) as an argument, the automatic converter may describe it simply as a generic \"object\" without detailing its internal properties.\n",
|
|
"\n",
|
|
"In these cases, manually defining the JSON schema is preferred to ensure nested properties (such as theme or font_size within a config object) are explicitly defined for the model."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "b57acc02",
|
|
"metadata": {
|
|
"id": "JFvIsc81w1H8"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"--- [Automatic] ---\n",
|
|
"{\n",
|
|
" \"type\": \"function\",\n",
|
|
" \"function\": {\n",
|
|
" \"name\": \"update_config\",\n",
|
|
" \"description\": \"Updates the configuration of the system.\",\n",
|
|
" \"parameters\": {\n",
|
|
" \"type\": \"object\",\n",
|
|
" \"properties\": {\n",
|
|
" \"config\": {\n",
|
|
" \"type\": \"object\",\n",
|
|
" \"description\": \"A Config object\"\n",
|
|
" }\n",
|
|
" },\n",
|
|
" \"required\": [\n",
|
|
" \"config\"\n",
|
|
" ]\n",
|
|
" }\n",
|
|
" }\n",
|
|
"}\n",
|
|
"\n",
|
|
"--- [Manual Schemas] ---\n",
|
|
"{\n",
|
|
" \"type\": \"function\",\n",
|
|
" \"function\": {\n",
|
|
" \"name\": \"update_config\",\n",
|
|
" \"description\": \"Updates the configuration of the system.\",\n",
|
|
" \"parameters\": {\n",
|
|
" \"type\": \"object\",\n",
|
|
" \"properties\": {\n",
|
|
" \"config\": {\n",
|
|
" \"type\": \"object\",\n",
|
|
" \"description\": \"A Config object\",\n",
|
|
" \"properties\": {\n",
|
|
" \"theme\": {\n",
|
|
" \"type\": \"string\"\n",
|
|
" },\n",
|
|
" \"font_size\": {\n",
|
|
" \"type\": \"number\"\n",
|
|
" }\n",
|
|
" }\n",
|
|
" }\n",
|
|
" },\n",
|
|
" \"required\": [\n",
|
|
" \"config\"\n",
|
|
" ]\n",
|
|
" }\n",
|
|
" }\n",
|
|
"}\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"import json\n",
|
|
"from transformers.utils import get_json_schema\n",
|
|
"\n",
|
|
"class Config:\n",
|
|
" def __init__(self):\n",
|
|
" self.theme = \"light\"\n",
|
|
" self.font_size = 14\n",
|
|
"\n",
|
|
"def update_config(config: Config):\n",
|
|
" \"\"\"\n",
|
|
" Updates the configuration of the system.\n",
|
|
"\n",
|
|
" Args:\n",
|
|
" config: A Config object\n",
|
|
"\n",
|
|
" Returns:\n",
|
|
" True if the configuration was successfully updated, False otherwise.\n",
|
|
" \"\"\"\n",
|
|
"\n",
|
|
"update_config_schema = {\n",
|
|
" \"type\": \"function\",\n",
|
|
" \"function\": {\n",
|
|
" \"name\": \"update_config\",\n",
|
|
" \"description\": \"Updates the configuration of the system.\",\n",
|
|
" \"parameters\": {\n",
|
|
" \"type\": \"object\",\n",
|
|
" \"properties\": {\n",
|
|
" \"config\": {\n",
|
|
" \"type\": \"object\",\n",
|
|
" \"description\": \"A Config object\",\n",
|
|
" \"properties\": {\"theme\": {\"type\": \"string\"}, \"font_size\": {\"type\": \"number\"} },\n",
|
|
" },\n",
|
|
" },\n",
|
|
" \"required\": [\"config\"],\n",
|
|
" },\n",
|
|
" },\n",
|
|
" }\n",
|
|
"\n",
|
|
"print(f\"--- [Automatic] ---\")\n",
|
|
"print(json.dumps(get_json_schema(update_config), indent=2))\n",
|
|
"\n",
|
|
"print(f\"\\n--- [Manual Schemas] ---\")\n",
|
|
"print(json.dumps(update_config_schema, indent=2))"
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"id": "9f249e3d",
|
|
"metadata": {
|
|
"id": "WTsXHyDnEbxp"
|
|
},
|
|
"source": [
|
|
"## Summary and next steps\n",
|
|
"\n",
|
|
"You have established how to build an application that can call functions with Gemma 4. The workflow is established through a four-stage cycle:\n",
|
|
"\n",
|
|
"1. **Define Tools**: Create the functions your model can use, specifying arguments and descriptions (e.g., a weather lookup function).\n",
|
|
"2. **Model's Turn**: The model receives the user's prompt and a list of available tools, returning a structured function call object instead of plain text.\n",
|
|
"3. **Developer's Turn**: The developer parses this output using regular expressions to extract function names and arguments, executes the actual Python code, and appends the results to the chat history using the specific tool role.\n",
|
|
"4. **Final Response**: The model processes the tool's execution result to generate a final, natural language answer for the user.\n",
|
|
"\n",
|
|
"Check out the following documentation for further reading.\n",
|
|
"\n",
|
|
"- [Run Gemma overview](https://ai.google.dev/gemma/docs/run)\n",
|
|
"- [Vision understanding](https://ai.google.dev/gemma/docs/capabilities/vision)\n",
|
|
"- [Audio understanding](https://ai.google.dev/gemma/docs/capabilities/audio)\n",
|
|
"- [Thinking mode](https://ai.google.dev/gemma/docs/capabilities/thinking)\n"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"colab": {
|
|
"name": "function-calling-gemma4.ipynb",
|
|
"toc_visible": true
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"name": "python3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 0
|
|
}
|