[Solved] Local models don't acknowledge context + current file

[cxar]

Seems like local models don't receive the context + current file if the file is too long, or just fail to acknowledge it. Seems to work for smaller files or remote models. For some reason it also doesn't work for highlights. I believe the long context may be causing these smaller local models to get confused.

It could be nice to have a config to limit the max code context size sent to the model.

Opening this discussion to see if others have experienced this + have any workarounds

Update: I had to set the ollama max context to match the model's max context using a Modelfile. It is now working as-expected.

[MaxOSchulte]

Sounds similar to: #592

[rheinardkorf]

@cxar , any chance you can give an example?

[cxar]

Sure!

The problem was due to Ollama default modelfiles using the default context window of 2048 tokens. I have some larger code files (on the order of a few hundred LoC), and when conversing using Avante it claimed it could not read the file.

This is solvable by either setting the num_ctx field in the request to match the model's max context (although there's some debate about whether that is an actual solution), or by using a custom Modelfile.

Here's the content of an example modelfile:

FROM qwen2.5-coder:32b-instruct-q8_0
PARAMETER num_ctx 32768

You then load that into Ollama like so (docs):
ollama create choose-a-model-name -f <location of the file e.g. ./Modelfile>

and use the new model name in Avante.

[rheinardkorf]

Thanks. That solved my issue and I have my qwen2.5 14b model working well.