Cody Local Ollama: Autocomplete doesn't use Ollama (BUT IT CAN!)

Cody VS Code
, ,
1

Problem: Out of the box, Cody doesn’t use Ollama offline for autocomplete, inline commands (opt-k), or Command Palette. But: Local autocomplete and inline commands work if you fiddle with your JSON settings - see fix below).

Note: To ensure you’re running against Ollama while testing, go offline.

This lack of automagic Ollama autocomplete support has downstream effects:

  1. Can’t use a local offline provider… offline. Yes, chat still works, but imagine you’re in the back of an Uber and fire up VSC. Your local chat is working, but… no autocomplete? The user isn’t going to understand why chat works offline but autocomplete doesn’t.
  2. If you start a new VSC session, Cody will sign in (Pro users need connectivity, so this makes sense). If you’re offline when starting a new VSC session, then Cody won’t run at all because it can’t sign in (“Connection Issues”). This means no local GPT even when using the fix that enables local Ollama chat+autocomplete+inline.

Fix to use autocomplete and inline commands right now!
To use Ollama offline for chat, autocomplete, and inline (opt-k on Mac), you still have to add this to your settings (manually specify which local model you want to use - I’m using llama3):

"cody.autocomplete.experimental.ollamaOptions": {
   "url": "http://localhost:11434",
     "model": "llama3"
},
"cody.experimental.ollamaChat": true,
"cody.autocomplete.advanced.provider": "experimental-ollama"
5 Likes
2

Thanks Rory, trying this approach and for some reason it’s showing this:

image

i.e., it’s somehow replying with the chat rather than the autocompletion. Tried multiple 5-6 local models with Ollama such as Codegemma2, Codellama, etc., but they all seem to do the same thing.

Have you experienced anything similar, and do you know what might be the issue and fix?

1 Like
3

Hey @AAR

did you try out to delete the config setting "cody.experimental.ollamaChat": true,. This may help.

Let me know.

4

Thanks for the quick reply. Yes I have tried it both with and without the "cody.experimental.ollamaChat": true option

5

Please switch back to another version of Cody if the issue persists.

Report which version works for you.

6

Hi!

When I use the cody.autocomplete.experimental.ollamaOptions I get an error saying that the configuration param is unknown. I’m under version v1.30.3. Could you help? Thanks!

Capture d’écran 2024-08-14 à 22.32.11

1 Like
7

Hello @ZeFifi

Does it still works even showing the error?

8

Thanks for getting back to me!

It works for the chat, but for the auto-completion, which interests me more, it’s messy : when I start typing code I get nothing or something inappropriate.

Capture d’écran 2024-08-16 à 10.35.48
Capture d’écran 2024-08-16 à 10.35.48775×88 14.9 KB

1 Like
9

Please put in the model key a completion model as value instead of the llama3 chat model. That’s why in the completions appears chat responses.

1 Like
10

Is "cody.autocomplete.experimental.ollamaChat": true deprecated? My VSC shows it as Unknown Configuration Setting.

11

I used "cody.autocomplete.advanced.provider": "experimental-ollama" together with cody.autocomplete.experimental.ollamaOptions but can’t get it to work properly with models that work fine with Continue. If I use granite-code:8b-base-q8_0 with Cody, I only get autocomplete suggestions with Cody at the root indentation level. For example, when the cursor is at the next level in a function, I need to hit backspace before I get a suggestion from Cody. In Continue, the same model works just fine. If I use deepseek-coder:6.7b-base-q8_0 with Cody autocomplete, I get a suggestion at the correct indentation level, but only the first token. If I want to get the entire line suggested, I need to go back to the root indentation level.

12

I tried a few more models, and the results were similar. I filed a GitHub issue report:

1 Like