Keep getting exceededContextWindowSize with Foundation Models

Question

Created 3w

Replies 2

Boosts 0

Participants 2

I'm a bit new to the LLM stuff and with Foundation Models. My understanding is that there is a token limit of around 4K.

I want to process the contents of files which may be quite large. I first tried going the Tool route but that didn't work out so I then tried manually chunking the text to keep things under the limit.

It mostly works except that every now and then it'll exceed the limit. This happens even when the chunks are less than 100 characters. Instructions themselves are about 500 characters but still overall, well below 1000 characters per prompt, all told, which, in my limited understanding, should not result in 4K tokens being parsed.

Any ideas on what is going on here?

Answered by Mr_Noodle in 854600022

Ok, it seems that the limit being exceeded was for the whole session and not per request/prompt. Misunderstanding on my part. Disregard.

Boost

Answer 1

Mr_Noodle OP

3w

Accepted Answer

Ok, it seems that the limit being exceeded was for the whole session and not per request/prompt. Misunderstanding on my part. Disregard.

1

Answer 2

mi12 OP

2w

I am glad you were able to solve your issue. We’ve encountered similar challenges and have built a library that is intentionally designed to prevent us from falling into similar traps.

https://github.com/mi12labs/SwiftAI

We have an example where we use a divide and conquer strategy for summarization.

0