Google expands on-device AI Edge to Gemma 3n, adds RAG and function calling

Google published the details on its developer blog, alongside two new libraries: one for on-device Retrieval Augmented Generation (RAG) and one for function calling. Both ship on Android first, with other platforms to follow.

Google argues that small language models (SLMs) have moved past the demo stage. The blog, written by senior product manager Mark Sherwood and four engineers, describes field technicians photographing a part with no service, warehouse staff updating inventory by voice, and a medical app that turns dictated patient history into filled form fields. Each example assumes the model runs locally, with no round trip to a server. The numbers are specific. Gemma 3 1B, introduced earlier this year, is 529MB and runs up to 2,585 tokens per second pre-fill on a mobile GPU. The company says this is enough to process a page of content in under a second.

Its small footprint keeps device coverage broad and limits the download an app forces on a user.

The newer Gemma 3n arrives as an early preview in 2B and 4B parameter variants, both native to text, image, video and audio. Only text and image are on Hugging Face for now; audio is to follow. Google also points to new quantization schemes that cut model size by a factor of 2.5 to 4 times against bf16 while reducing latency and peak memory use.

Apple and others have been circling the same problem

Google is not first. Ars Technica reported on 25 April 2024 that Apple released eight source-available models under the name OpenELM, ranging from 270 million to 3 billion parameters, small enough to run on a smartphone. Apple published model weights, training code and reproducible recipes, though the Apple Sample Code License carries restrictions that may not meet a strict definition of open source. Apple's white paper claimed a 2.36 percent accuracy improvement over Allen AI's OLMo 1B while using half the pre-training tokens, via what it called a layer-wise scaling strategy. For scale, Ars Technica noted that Meta's largest Llama 3 model at the time held 70 billion parameters and OpenAI's GPT-3 from 2020 shipped with 175 billion. A 529MB model is a different category of object entirely.

The reproducibility and transparency of large language models are crucial for advancing open research, ensuring the trustworthiness of results, and enabling investigations into data and model biases, as well as potential risks.

That is from Apple's OpenELM paper abstract. The company also warned that models trained on public datasets may produce inaccurate, harmful, biased or objectionable output.

What the research says about limits

A November 2024 paper on arXiv introduced SlimLM, a series of models from 125M to 7B parameters tuned for document assistance on a Samsung Galaxy S24. Its authors reported comparable or superior performance against existing small models and released an Android app, while noting that on-device processing can cut server costs and improve privacy. The paper's own framing is that real-world performance on phones remains underexplored.

There is also a more uncomfortable finding. A separate arXiv paper by Prashant Mudgal tested seven confidence approaches across 7 model pairs and 5 benchmarks and found that token-level entropy is effectively blind in models under 3 billion parameters: in 91 percent of dataset-model combinations, mean token entropy sat near zero regardless of whether the answer was right. Semantic entropy, computed by sampling multiple answers and clustering them by meaning, recovered a usable signal. Routing uncertain queries to a larger expert model improved accuracy by up to 50 percentage points, and cross-family routing averaged +22.0 percent against +6.8 percent for same-family routing.

Google's function calling library makes a similar concession in practice. The blog notes it can be paired with a Python tool simulation library that generates synthetic data to improve function calling accuracy, which is an admission that small models need help before they can be trusted to act.

Google has not said when Gemma 3n will leave preview, or which platforms the RAG and function calling libraries will reach next.