Post

Replies

Boosts

Views

Activity

Does prewarming a short-lived LanguageModelSession benefit a later session?
I’m building Summon (https://github.com/NakliTechie/summon), an open-source native macOS launcher that uses the on-device SystemLanguageModel. Summon creates a fresh LanguageModelSession for each query and attaches only the read-only tools relevant to that query. It currently calls prewarm() after the first keystroke using a temporary session, then creates a different session for generation. The documentation describes prewarm(promptPrefix:) as loading the resources required “for this session.” I would value guidance on four points: Is the prewarming benefit scoped to that exact LanguageModelSession instance? Does a later session using the same SystemLanguageModel receive any benefit? For an ephemeral launcher, is retaining one session preferable to creating a fresh session per query? Which Foundation Models Instrument signal identifies an ineffective prewarm or cache invalidation? Thank You Chirag
1
0
34
7h
Does prewarming a short-lived LanguageModelSession benefit a later session?
I’m building Summon (https://github.com/NakliTechie/summon), an open-source native macOS launcher that uses the on-device SystemLanguageModel. Summon creates a fresh LanguageModelSession for each query and attaches only the read-only tools relevant to that query. It currently calls prewarm() after the first keystroke using a temporary session, then creates a different session for generation. The documentation describes prewarm(promptPrefix:) as loading the resources required “for this session.” I would value guidance on four points: Is the prewarming benefit scoped to that exact LanguageModelSession instance? Does a later session using the same SystemLanguageModel receive any benefit? For an ephemeral launcher, is retaining one session preferable to creating a fresh session per query? Which Foundation Models Instrument signal identifies an ineffective prewarm or cache invalidation? Thank You Chirag
Replies
1
Boosts
0
Views
34
Activity
7h