On-device GenAI APIs as a part of ML Equipment assist you simply construct with Gemini Nano

Posted by Caren Chang – Developer Relations Engineer, Chengji Yan – Software program Engineer, Taj Darra – Product Supervisor

We’re excited to announce a set of on-device GenAI APIs, as a part of ML Equipment, that will help you combine Gemini Nano in your Android apps.

To start out, we’re releasing 4 new APIs:

• Summarization: to summarize articles and conversations
• Proofreading: to shine brief textual content
• Rewriting: to reword textual content in numerous types
• Picture Description: to offer brief description for photographs

Key advantages of GenAI APIs

GenAI APIs are excessive degree APIs that enable for straightforward integration, much like present ML Equipment APIs. This implies you possibly can anticipate high quality outcomes out of the field with out additional effort for immediate engineering or high-quality tuning for particular use instances.

GenAI APIs run on-device and thus present the next advantages:

• Enter, inference, and output information is processed domestically
• Performance stays the identical with out dependable web connection
• No extra value incurred for every API name

To stop misuse, we additionally added security safety in varied layers, together with base mannequin coaching, safety-aware LoRA fine-tuning, enter and output classifiers and security evaluations.

How GenAI APIs are constructed

There are 4 foremost parts that make up every of the GenAI APIs.

1. Gemini Nano is the bottom mannequin, as the muse shared by all APIs.
2. Small API-specific LoRA adapter fashions are educated and deployed on prime of the bottom mannequin to additional enhance the standard for every API.
3. Optimized inference parameters (e.g. immediate, temperature, topK, batch dimension) are tuned for every API to information the mannequin in returning the very best outcomes.
4. An analysis pipeline ensures high quality in varied datasets and attributes. This pipeline consists of: LLM raters, statistical metrics and human raters.

Collectively, these parts make up the high-level GenAI APIs that simplify the trouble wanted to combine Gemini Nano in your Android app.

Evaluating high quality of GenAI APIs

For every API, we formulate a benchmark rating primarily based on the analysis pipeline talked about above. This rating is predicated on attributes particular to a activity. For instance, when evaluating the summarization activity, one of many attributes we take a look at is “grounding” (ie: factual consistency of generated abstract with supply content material).

To offer out-of-box high quality for GenAI APIs, we utilized function particular fine-tuning on prime of the Gemini Nano base mannequin. This resulted in a rise for the benchmark rating of every API as proven beneath:

As well as, this can be a fast reference of how the APIs carry out on a Pixel 9 Professional:

Pattern utilization

That is an instance of implementing the GenAI Summarization API to get a one-bullet abstract of an article:

val articleToSummarize = "We're excited to announce a set of on-device generative AI APIs..."

// Outline activity with desired enter and output format
val summarizerOptions = SummarizerOptions.builder(context)
    .setInputType(InputType.ARTICLE)
    .setOutputType(OutputType.ONE_BULLET)
    .setLanguage(Language.ENGLISH)
    .construct()
val summarizer = Summarization.getClient(summarizerOptions)

droop enjoyable prepareAndStartSummarization(context: Context) {
    // Verify function availability. Standing will likely be one of many following: 
    // UNAVAILABLE, DOWNLOADABLE, DOWNLOADING, AVAILABLE
    val featureStatus = summarizer.checkFeatureStatus().await()

    if (featureStatus == FeatureStatus.DOWNLOADABLE) {
        // Obtain function if obligatory.
        // If downloadFeature just isn't referred to as, the primary inference request will 
        // additionally set off the function to be downloaded if it is not already
        // downloaded.
        summarizer.downloadFeature(object : DownloadCallback {
            override enjoyable onDownloadStarted(bytesToDownload: Lengthy) { }

            override enjoyable onDownloadFailed(e: GenAiException) { }

            override enjoyable onDownloadProgress(totalBytesDownloaded: Lengthy) {}

            override enjoyable onDownloadCompleted() {
                startSummarizationRequest(articleToSummarize, summarizer)
            }
        })    
    } else if (featureStatus == FeatureStatus.DOWNLOADING) {
        // Inference request will robotically run as soon as function is      
        // downloaded.
        // If Gemini Nano is already downloaded on the gadget, the   
        // feature-specific LoRA adapter mannequin will likely be downloaded very  
        // shortly. Nevertheless, if Gemini Nano just isn't already downloaded, 
        // the obtain course of might take longer.
        startSummarizationRequest(articleToSummarize, summarizer)
    } else if (featureStatus == FeatureStatus.AVAILABLE) {
        startSummarizationRequest(articleToSummarize, summarizer)
    } 
}

enjoyable startSummarizationRequest(textual content: String, summarizer: Summarizer) {
    // Create activity request  
    val summarizationRequest = SummarizationRequest.builder(textual content).construct()

    // Begin summarization request with streaming response
    summarizer.runInference(summarizationRequest) { newText -> 
        // Present new textual content in UI
    }

    // You may as well get a non-streaming response from the request
    // val summarizationResult = summarizer.runInference(summarizationRequest)
    // val abstract = summarizationResult.get().abstract
}

// Be sure you launch the useful resource when not wanted
// For instance, on viewModel.onCleared() or exercise.onDestroy()
summarizer.shut()

For extra examples of implementing the GenAI APIs, take a look at the official documentation and samples on GitHub:

Use instances

Right here is a few steering on the right way to greatest use the present GenAI APIs:

For Summarizationthink about:

• Dialog messages or transcripts that contain 2 or extra customers
• Articles or paperwork lower than 4000 tokens (or about 3000 English phrases). Utilizing the primary few paragraphs for summarization is normally adequate to seize an important info.

For Proofreading and Rewriting APIs, think about using them in the course of the content material creation course of for brief content material beneath 256 tokens to assist with duties akin to:

• Refining messages in a selected tone, akin to extra formal or extra informal
• Sharpening private notes for simpler consumption later

For the Picture Description API, think about it for:

• Producing titles of photographs
• Producing metadata for picture search
• Using descriptions of photographs in use instances the place the pictures themselves can’t be displayed, akin to inside an inventory of chat messages
• Producing various textual content to assist visually impaired customers higher perceive content material as an entire

GenAI API in manufacturing

Envision is an app that verbalizes the visible world to assist people who find themselves blind or have low imaginative and prescient lead extra impartial lives. A standard use case within the app is for customers to take an image to have a doc learn out loud. Using the GenAI Summarization API, Envision is now in a position to get a concise abstract of a captured doc. This considerably enhances the person expertise by permitting them to shortly grasp the details of paperwork and decide if a extra detailed studying is desired, saving them effort and time.

Supported gadgets

GenAI APIs can be found on Android gadgets utilizing optimized MediaTek Dimensity, Qualcomm Snapdragon, and Google Tensor platforms by AICore. For a complete record of gadgets that assist GenAI APIs, check with our official documentation.

Study extra

Begin implementing GenAI APIs in your Android apps at present with steering from our official documentation and samples on GitHub: AI Catalog GenAI API Samples with Compose, ML Equipment GenAI APIs Quickstart.