stream-chat-android-ai-ondevice/io.getstream.chat.android.ai.ondevice/AIOnDeviceModel

AIOnDeviceModel

class AIOnDeviceModel(context: Context, val maximumResponseTokens: Int = DEFAULT_RESPONSE_TOKENS) : AILocalModel, AutoCloseable

Gemini Nano, the model in Android's AICore, through ML Kit's GenAI Prompt API. It answers without a network connection, and the conversation never leaves the device.

It runs on devices whose AICore offers Gemini Nano (recent Pixel, Samsung and other flagship phones), not on emulators. The model may need downloading first: status says whether it is Status.Downloadable or Status.Downloading, and download starts it.

It is a small model, good for short answers and for working with what is in the conversation. Its context holds a few thousand tokens, so the oldest turns of a long conversation are left out.

val model = AIOnDeviceModel(context)
if (model.isAvailable()) {
    model.reply(instructions, turns).collect { answer -> show(answer) }
}

Parameters

context

Any context; the application context is kept.

maximumResponseTokens

The longest answer, in tokens. The model writes at most MODEL_MAXIMUM_RESPONSE_TOKENS, so larger values are capped.

Constructors

AIOnDeviceModel

constructor(context: Context, maximumResponseTokens: Int = DEFAULT_RESPONSE_TOKENS)

Types

Companion

object Companion

Download

abstract class Download

How a download of the model is going. More states may be added.

Failure

class Failure(cause: Throwable) : Exception

Thrown by reply when the model fails while answering.

Status

enum Status : Enum<AIOnDeviceModel.Status>

Where the model is on this device.

Unavailable

class Unavailable(cause: Throwable? = null) : Exception

Thrown by reply when the model can't answer on this device.

Properties

maximumResponseTokens

val maximumResponseTokens: Int

Functions

open override fun close()

Releases the model's resources.

download

fun download(): Flow<AIOnDeviceModel.Download>

Downloads the model, when status is Status.Downloadable. The download continues while the flow is collected.

isAvailable

open suspend override fun isAvailable(): Boolean

Whether the model can answer now.

open override fun reply(instructions: String, turns: List<AIConversationTurn>): Flow<String>

Streams the answer to the last of turns. Each emission is the whole answer so far, and cancelling the collection stops the model.

status

suspend fun status(): AIOnDeviceModel.Status

Where the model is on this device. A device without AICore, or an error, reads as unavailable.