Does prewarming a short-lived LanguageModelSession benefit a later session?

Question

Created 10h

Replies 1

Boosts 0

Participants 2

I’m building Summon (https://github.com/NakliTechie/summon), an open-source native macOS launcher that uses the on-device SystemLanguageModel.

Summon creates a fresh LanguageModelSession for each query and attaches only the read-only tools relevant to that query. It currently calls prewarm() after the first keystroke using a temporary session, then creates a different session for generation.

The documentation describes prewarm(promptPrefix:) as loading the resources required “for this session.” I would value guidance on four points:

Is the prewarming benefit scoped to that exact LanguageModelSession instance?
Does a later session using the same SystemLanguageModel receive any benefit?
For an ephemeral launcher, is retaining one session preferable to creating a fresh session per query?
Which Foundation Models Instrument signal identifies an ineffective prewarm or cache invalidation?

Thank You

Chirag

Answer 1

Frameworks Engineer OP

Apple

6h

Prewarming is only guaranteed to apply to the instance of LanguageModelSession that you call it on. And prewarming is best-effort, so it's not truly guaranteed at all. The system may ignore your call to prewarm if the model is being used by other apps.
Related to the answer above; there are times when it might, but it's dependent on the state of the system and generally you cannot know.
Generally, keeping one session is better if you only support one query at a time. If you support parallel queries, you need multiple sessions though.
The tracks indicate model loading. If you see model loading appear after your prompt is submitted, that implies that the model was not warm in memory. Calling prewarm earlier in the request lifecycle may fix that, but it's not fully within your control. The model is a shared system resource, and the OS may ignore your calls to prewarm it's tied up servicing requests from other apps. It may also unload the model between requests to serve another app, which can cause your app to incur another load on a future request.

Hope that helps!