Gemini 1.5 Pro support (experimental feature)

[sqs]

Cody (in VS Code) works with Google’s latest Gemini 1.5 Pro, announced on 2024 May 14.

To try it:

First, get an API key from https://makersuite.google.com/app/apikey.

Next, ensure you’re using the latest Cody prerelease build from the VS Code marketplace: right-click on the Cody extension in the VS Code extension sidebar and choose Switch to Prerelease Version. Your extension version should be v1.17.1715723071 (pre-release) or newer. Reload VS Code after the extension upgrade.

Then add the following to your VS Code user settings, replacing $GEMINI_API_KEY with your Gemini API key:

  "cody.dev.models": [
    // Google Gemini 1.5 Pro
    {
      "provider": "google",
      "model": "gemini-1.5-pro-latest",
      "tokens": 1000000,
      "apiKey": "$GEMINI_API_KEY"
    },
  ]

Start a new Cody chat (Opt+/ or Alt+/), and select gemini 1.5 pro latest in the model dropdown. (If you don’t see it there, try reloading VS Code.) Report any issues or feedback here.

---

Troubleshooting steps: Set the VS Code setting "cody.debug.verbose": true and check the Cody by Sourcegraph output channel in VS Code to see the full HTTP error response (line starting with googleChatClient: in the output channel).

[PriNova]

What an amazing feature addition and how fast it is implemented.

I would love to see an indicator under the chat input text box of how many tokens are used up for context fetching and for the chat transcript per chat session to have control over the rate limits.

This gives better control for the user of how many tokens remain, especially to stay in the free rate limits.

Nice work. Keep going

[Kynlo]

Did I miss something here?

[sqs]

@Kynlo Check your VS Code settings syntax. It looks like you accidentally pasted the cody.dev.models inside of the cody.customHeaders setting.

[Kynlo]

Let’s pretend this never happened - It’s showing now haha

Is this gonna be limited for those of us in the UK (where we can’t actually use it without a VPN), or does it run through your systems first?

I mainly ask cause now I get:

[sqs]

@Kynlo Can you check if you are on the latest Cody for VS Code prerelease build from the VS Code marketplace? Version string is v1.17.1715723071 (pre-release).

[Kynlo]

(Edit): Tried changing to a new API key I just generated, still getting “Request Failed: HTTP 400 Error: Bad Request”

[sqs]

@Kynlo

Is this gonna be limited for those of us in the UK (where we can’t actually use it without a VPN), or does it run through your systems first?

Right now, Cody’s experimental Gemini support uses your own Gemini API key and goes directly from your machine to Google’s Gemini API. This is different from other Cody Pro models (like Anthropic, OpenAI, Mixtral, StarCoder, etc.) where we provide inference through our servers and don’t require a separate API token per provider. We do plan to make Cody’s support for Gemini non-experimental and more like the other models soon.

[sqs]

@Kynlo

(Edit): Tried changing to a new API key I just generated, still getting “Request Failed: HTTP 400 Error: Bad Request”

Hmm, can you try reloading VS Code after your extension upgrade, and then opening a new Cody chat? It may be using a Cody chat window from before the upgrade.

[Kynlo]

Tried multiple API keys on multiple Google billing accounts.

█ SimpleChatPanelProvider: postError: HTTP 400 Error: Bad Request
█ logEvent: CodyVSCodeExtension:chatResponse:noCode {
“details”: “VSCode”,
“properties”: {
“charCount”: 0,
“lineCount”: 0,
“requestID”: “1ad8fb39-9d47-45d9-a118-4684e34f2b05”,
“chatModel”: “google/gemini-1.5-pro-latest”
},
“opts”: {
“hasV2Event”: true
}
}

From the error docs for Gemini:

HTTP Code	Status	Description	Solution
400	INVALID_ARGUMENT	The request body is malformed.	Check the API reference for request format, examples, and supported versions. Using features from a newer API version with an older endpoint can cause errors.

From my settings.json:

"cody.dev.models": [
    {
    "provider": "google",
    "model": "gemini-1.5-pro-latest",
    "tokens": 1000000,
    "apiKey": "[Redacted]"
    },
],

I also attempted to change the model to “flash-latest” instead of “pro-latest” and got the same issue.

I also fully rebooted my computer to check if it was something messing up locally.

For more context, the chat model shows in the dropdown (and changes based on the models I inputted while testing), but always responds with 400.

The following is from googles AI Studio to show the models I have access to:

[Kynlo]

Also ignore the previous about regional restrictions, the UK and EU are now allowed to use this without sneaking through VPNs. I checked my API key was working using some local code and I get a reply from Gemini, so there’s something with the setup I have going on with VSCode (I heavily suspect this is a me problem, not a cody problem haha)

[PriNova]

You need to activate billing with a new API key if not done already. Gemini API is now available in Europe too.

[Kynlo]

Already did this - Checked my API Key was working via a different method and Gemini responds (see message above yours)

[PriNova]

It works for me flawless.

[Kynlo]

A LOT of messing around later…a new google account, a new billing account, a new project on google cloud console and a new API key for Gemini

…It WORKS!

I have absolutely NO idea why it wasn’t working with my previous one (which the key worked elsewhere)…but we can scratch everything I said as “google did something, no clue what”

BUT IT WORKS!

<3

[ado]

I was able to get it working on my end, but another person I was helping was not. They did not have billing setup on their GCP account though. I’ve also had access to Gemini Pro for a bit on the account I did it with, so not sure if it’s gated at all based on account-age.

[Kynlo]

Sorry for thinking it was a Cody/Code issue - I genuinely don’t have a clue why it refuses to work with my previous google cloud account (which actually has funding already sitting there doing nothing)

[PriNova]

People in Europe including UK need a billing account linked with the API in a Google Cloud Project, because they only have the ‘pay-as-you-go’ option. I like the politics in Europoor [ironic-mode-off].

By using VPN you can use an API key as free tier.

[sqs]

Thanks for trying a bunch of stuff! We’ll try to figure out what kind of configuration or token setup is needed on the Google side and document that. And I also added better logging of the errors in log errors from Google Gemini API by sqs · Pull Request #4176 · sourcegraph/cody · GitHub, which should be in the latest prerelease build on the VS Code marketplace in a couple hours.

[Kynlo]

I have a vague feeling that it might have something to do with the UK being previously regionally restricted and perhaps that’s caused some lingering “lock out” issues on my account, where creating a new one with a new billing account etc, didn’t have those issues.