[Android] LlmGenerationConfig default temperature (0.8) overrides LlmModule temperature in the config generate overload

[alpharomercoma]

🐛 Describe the bug

Summary

On Android, temperature can be set on the LlmModule constructor and on LlmGenerationConfig. The native layer uses a per-call temperature when it is >= 0 and otherwise falls back to the constructor's (#16728). The simple generate(...) overloads pass -1.0f, so they use the constructor's value. LlmGenerationConfig.Builder defaults temperature to 0.8f, so generate(prompt, config, callback) replaces the constructor's value unless the config sets temperature explicitly.

• Expected: LlmModule(..., temperature = 0.0f) decodes greedily whichever generate overload is used.
• Actual: through the config overload it samples at 0.8 unless .temperature(...) is set on the config.

To reproduce

Environment: executorch-android 1.4.0 from Maven Central (the latest published), with fbjni 0.7.0 and nativeloader 0.10.5; the test program also uses kotlin-stdlib 2.3.20. Models (both attention only, so #23262 cannot affect them):

• larryliu0820/Qwen3-1.7B-INT8-INT4-ExecuTorch-XNNPACK: model.pte (sha256 074d87c3cba37ed89ea968201f847b37a5a7d92461a477bad650e28aab188296), tokenizer.json (aeb13307a71acd8fe81861d94ad54ab689df773318809eed3cbe794b4492dae4)
• larryliu0820/Gemma3-1B-IT-INT8-INT4-ExecuTorch-XNNPACK: model.pte (8659099ef466bfc85e5a5a566323c9edb39071dfe339ea574d9365fd6ce17ac6), tokenizer.json (4667f2089529e8e7657cfb6d1c19910ae71ff5f28aa7ab2ff2763330affad795)

The module is built for greedy decoding, then each way of calling generate runs twice from resetContext():

val module = LlmModule(LlmModule.MODEL_TYPE_TEXT, modelPath, tokenizerPath, 0.0f)   // greedy intended
module.load()
fun config() = LlmGenerationConfig.create().seqLen(160).echo(false)

// A: config, temperature not set
module.generate(prompt, config().build(), callback)
// B: config, temperature set to 0
module.generate(prompt, config().temperature(0.0f).build(), callback)
// C: config, temperature set to -1 ("use the module's")
module.generate(prompt, config().temperature(-1.0f).build(), callback)
// D: simple overload
module.generate(prompt, 160, callback, false)

Were the two runs identical?

Call	Qwen3, POCO X8 Pro Max	Gemma3, POCO X8 Pro Max	Qwen3, Android emulator
A: config, temperature not set	no	no	no
B: config, temperature(0.0f)	yes	yes	yes
C: config, temperature(-1.0f)	yes	yes	yes
D: simple overload	yes	yes	yes

Devices: Xiaomi POCO X8 Pro Max (MT6991, Android 16) and an Android 16 emulator (android-36 google_apis arm64-v8a image) on an Apple silicon Mac. On the phone the Qwen3 A/B ran twice with the same result, and an earlier version comparing only A and D gave the same result in two more runs. Example from A (Qwen3, phone):

run 1: In the quiet town of Cape Cod, where the sea whispered secrets to the
run 2: In the quiet coastal town of Elmhurst, there stood a lighthouse that h

Within each B, C and D pair, the two runs are identical, and the first 70 characters are also the same across B, C and D on each device. Runs under A start with the same first word and diverge shortly after, because the first token is chosen during prefill with temperature 0 (text_prefiller.cpp#L32, called at text_llm_runner.cpp#L181) and config.temperature only applies to the decode loop (text_llm_runner.cpp#L240-L245). That makes the problem easy to mistake for small numerical noise.

Code path

Permalinks at 2f78245. The cited lines are the same at v1.5.1, and javap on the 1.4.0 AAR shows the same Kotlin behaviour (builder default 0.8f, config overload passes the config's temperature, simple overloads pass -1.0f).

• LlmGenerationConfig.kt#L49: private var temperature: Float = 0.8f
• LlmModule.kt#L378-L391: the config overload passes config.temperature. The simple overloads (#L257-L324) pass DEFAULT_TEMPERATURE = -1.0f (#L880).
• jni_layer_llama.cpp#L153 stores the constructor's temperature, #L197 builds the text runner with -1, and #L258 picks temperature >= 0 ? temperature : temperature_.

run-on-android.md#L173-L197 shows a config with .temperature(0.8f) and says the defaults match the C++ GenerationConfig, but does not say that a config temperature replaces the one given to the constructor.

Suggested fix

Either:

1. Default LlmGenerationConfig.Builder.temperature to -1.0f ("use the module's"), matching the simple overloads. Variant C above shows this restores the constructor's temperature. It changes behaviour for anyone relying on the 0.8 default while passing a different constructor value, so it is worth a release note.
2. Keep 0.8 and document the precedence on the LlmModule constructor, on LlmGenerationConfig.temperature and in run-on-android.md.

#17637 (consolidating the LlmModule overloads) may be a natural place to settle this. Scope: the generic text path (MODEL_TYPE_TEXT to TextLLMRunner); I have not checked the multimodal or vendor runners.

Full program

ReproAB.kt, verbatim. Built against the AAR's classes.jar and run with app_process, so no app is needed. Commands as run, from a directory holding ReproAB.kt, deps/ (the AAR classes.jar files and nativeloader-0.10.5.jar) and kotlinc/:

kotlinc ReproAB.kt \
  -cp deps/et/classes.jar:deps/fbjni/classes.jar:deps/nativeloader-0.10.5.jar:$ANDROID_HOME/platforms/android-36/android.jar \
  -jvm-target 11 -d reproab-classes.jar
d8 --release --min-api 26 --lib $ANDROID_HOME/platforms/android-36/android.jar --output ab \
  reproab-classes.jar kotlinc/lib/kotlin-stdlib.jar deps/et/classes.jar deps/fbjni/classes.jar deps/nativeloader-0.10.5.jar
# Push ab/classes.dex as /data/local/tmp/et-temp-repro/reproab.dex, plus the model files, plus lib/ with
# libexecutorch.so (AAR jni/arm64-v8a) and libfbjni.so, libc++_shared.so (fbjni AAR).
adb shell "cd /data/local/tmp/et-temp-repro && LD_LIBRARY_PATH=/data/local/tmp/et-temp-repro/lib \
  CLASSPATH=/data/local/tmp/et-temp-repro/reproab.dex app_process -Djava.library.path=/data/local/tmp/et-temp-repro/lib \
  /data/local/tmp/et-temp-repro ReproABKt \
  /data/local/tmp/et-temp-repro/qwen3.pte /data/local/tmp/et-temp-repro/qwen3.tokenizer.json qwen3"   # gemma: its files, "gemma"

// Issue-2 A/B: which way of calling generate() samples when LlmModule was built with temperature 0?
import org.pytorch.executorch.extension.llm.LlmCallback
import org.pytorch.executorch.extension.llm.LlmGenerationConfig
import org.pytorch.executorch.extension.llm.LlmModule
import kotlin.system.exitProcess

fun collect(block: (LlmCallback) -> Unit): String {
    val sb = StringBuilder()
    block(object : LlmCallback { override fun onResult(result: String) { sb.append(result) } })
    return sb.toString()
}

fun main(args: Array<String>) {
    val (modelPath, tokenizerPath, family) = Triple(args[0], args[1], args[2])
    val q = "Tell me a short story about a lighthouse keeper."
    val prompt = when (family) {
        "gemma" -> "<bos><start_of_turn>user\n$q<end_of_turn>\n<start_of_turn>model\n"
        else -> "<|im_start|>user\n$q<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n"
    }
    val module = LlmModule(LlmModule.MODEL_TYPE_TEXT, modelPath, tokenizerPath, 0.0f)   // greedy intended
    module.load()
    val seqLen = 160
    val variants = listOf<Pair<String, (LlmCallback) -> Unit>>(
        "A config, temperature not set" to { cb -> module.generate(prompt, LlmGenerationConfig.create().seqLen(seqLen).echo(false).build(), cb) },
        "B config, temperature(0.0f)" to { cb -> module.generate(prompt, LlmGenerationConfig.create().seqLen(seqLen).echo(false).temperature(0.0f).build(), cb) },
        "C config, temperature(-1.0f)" to { cb -> module.generate(prompt, LlmGenerationConfig.create().seqLen(seqLen).echo(false).temperature(-1.0f).build(), cb) },
        "D simple overload" to { cb -> module.generate(prompt, seqLen, cb, false) },
    )
    val out = StringBuilder()
    for ((name, gen) in variants) {
        module.resetContext(); val r1 = collect(gen)
        module.resetContext(); val r2 = collect(gen)
        out.append("RESULT $family | $name | identical: ${r1 == r2}\n")
        out.append("   run 1: ${r1.take(70).replace("\n", "\\n")}\n")
        out.append("   run 2: ${r2.take(70).replace("\n", "\\n")}\n")
    }
    print(out)
    System.out.flush()
    exitProcess(0)
}

Versions

executorch-android 1.4.0 (Maven Central), fbjni 0.7.0, nativeloader 0.10.5, kotlin-stdlib 2.3.20 (test program). Source checked at main 2f7824593d0d14f9d5d73faa540b681b6b2b507c and v1.5.1. Devices: Xiaomi POCO X8 Pro Max (MT6991, Android 16); Android 16 emulator (android-36 google_apis arm64-v8a) on an Apple silicon Mac.