Enhancing Android Apps with On-Device Intelligence

In the latest installment of our ongoing series, “Build Intelligent Android Apps,” we delve into the transformative capabilities of on-device inference using Gemini Nano through ML Kit’s Prompt API. This post focuses on enhancing the Jetpacker demo app by incorporating intelligent features that operate directly on users’ devices.

What is On-Device Inference?

On-device inference allows Android apps to process prompts and data locally, eliminating the need to send sensitive information to external servers. This approach not only enhances privacy but also improves performance by reducing latency and reliance on internet connectivity.

Key Features for Jetpacker

We identified three essential features that can significantly enhance user experience in the Jetpacker app:

• Trip Itinerary Summarization: Users can receive a concise overview of their travel plans, complete with packing tips and useful local phrases.
• Expense Management: An integrated tool that simplifies the process of tracking spending and managing budgets while ensuring sensitive data remains secure.
• Voice Notes Recording: A feature that allows users to capture audio memos, which are then transcribed to text and associated with specific activities in their itinerary.

Summarizing Trip Itineraries

The itinerary screen can become cluttered with information, making it challenging for users to prepare for their trips. To address this, we can implement a new section titled ‘Get ready for your trip’ at the top of the screen. By utilizing Gemini Nano, we can input the trip itinerary into the model and generate a summary that includes essential packing advice and local phrases tailored to the destination.

Managing Expenses Efficiently

Another valuable addition is the expense manager, which automates the sorting of receipts and budget calculations. Given that receipts often contain sensitive information, processing this data on-device ensures that users’ privacy is maintained. Gemini Nano 4 excels in multimodal capabilities, particularly in image understanding tasks such as Optical Character Recognition (OCR). This allows the app to analyze receipt images and extract pertinent details like title, amount spent, and expense category.

Recording Voice Notes

To enrich the travel experience further, we are introducing a voice notes feature that operates entirely on-device. By leveraging ML Kit’s Speech Recognition API, users can easily record short audio notes during their trips. The recorded audio will be transcribed into text, enabling users to associate their voice notes with specific trip activities for easy reference.

The Speech Recognition API offers two modes: a basic mode available on most Android devices with API level 31 and higher, and an advanced mode that utilizes Gemini Nano for enhanced language support and transcription quality, currently available on Pixel 10 devices.

Conclusion

By integrating these intelligent features using ML Kit’s GenAI APIs and Gemini Nano, the Jetpacker app can provide users with a seamless and enriching travel experience. The ability to process information on-device not only enhances user privacy but also delivers a faster, more responsive application. For developers interested in implementing similar features, the complete source code for Jetpacker is available on GitHub, along with a video tutorial on integrating intelligent capabilities into Android applications.

Source for the original facts: Original source.