Hello,
I am using Cody as a VSCode extension and would like to know if it is possible to use Azure OpenAI as a custom API provider. If so, could you provide guidance on how to configure it properly?
Thank you!
Hey @OMS90
You can configure Cody for VS Code to provide your own API key for OpenAI compatible models like this: Experimental Models.
If openai as provider does not work then replace the provider key value with groq.
I hope that helps.
Thanks for the reply. I am not familiar with these settings, so let me check a little more. In particular, I am not sure of the value to set for the endpoints section. (I know the value of
I do know the value “path”.)
I’m not sure if I should add the following to the “setting.json” file?
{
"cody.dev.models": [
{
"provider": "openai",
"model": "gpt-XX",
"inputTokens": 32000,
"outputTokens": 4000,
"apiKey": "XXXXXXXXXXXXX",
"options": {
"temperature": 0.0
},
"apiEndpoint": "https://XXXXXXXXXXXXX.openai.azure.com"
},
]
}
Hey @OMS90
For using the correct API endpoint you can read more here in the Azure OpenAI Rest API on how to set your parameters correctly.
In your case, you need to modify the apiEndpoint key.
I tried to set it up, but got the error shown below.
Bad Request: status 400, reason model "openai/gpt-XX" is not available on sourcegraph.com
I have set the value of Azure’s deployment-id as the model value.
The model actually used in this ID is “gpt-4o”.
Is this wrong?
Thanks for your kind support.
Then try to use as provider ‘groq’ instead of ‘OpenAI’ which is also OpenAI compatible.
I have tried that too, but I get an error.
Request Failed: HTTP 404 Resource Not Found: {"error":{"code":"404","message": "Resource not found"}}
I think I have the wrong settings because I am able to use this model on another service.
But, I don’t know which part is wrong.
Another concern is that I am trying this tool for free, implementing only account registration. Is that the cause of the problem?
@OMS90 Can you give the latest config which you are trying in which is not working? Why in your previous config example you use gpt-XX (if it’s a placeholder then please also provide the actual value or replace with existing model name).
@arvis
This is the current setup. gpt4o is the deployed name.
setting.json
{
"cody.dev.models": [
{
"provider": "groq",
"model": "gpt4o",
"inputTokens": 32000,
"outputTokens": 4000,
"apiKey": "XXXXXXXXXXXXXXXXXXXXXXXXXXXX",
"options": {
"temperature": 0.0
},
"apiEndpoint": "https://XXXXXXX.openai.azure.com"
}
]
}
Sorry for the hassle.
What we actually want to check is “whether AzureOpenAI API can be used with Cody as a VScode extension” and “if so, under what settings and conditions”.
@OMS90 In general it should be possible to use with Cody, Cody should work with all OpenAI compatible APIs.
Sadly I don’t use Azure services so I can’t do the tests to identify the possible problem but maybe you could try to check if your credetials works with other tools (like your own scripts, REST API, cURL etc.) - just to rull out the possibility that there is a issue in config or access rights.
Maybe I am wrong but one another thing I see in - Azure OpenAI Service models - Azure OpenAI | Microsoft Learn - is that there is gpt-4o model name but not the gpt4o so try to replace model name with gpt-4o in your config. I think deployment name is not used in Cody’s API integration (it’s more for you to identify different deployments etc.). But as I said - I may be wrong, that even may be the reason API calls is not working… so I would suggest you to try to get your Azure OpenAI deployment working with cURL, REST API calls first.
@arvis
Thanks a lot.
In case you are wondering, I have confirmed that the other extensions in VScode are communicating with the same values. In that tool, there is a field in the settings to enter the AzureAPI URL and key, so I was able to set it up without any confusion.
I have changed the settings in the model section from the above settings, but the same error message was returned.
Request Failed: HTTP 404 Resource Not Found:
{"error":{"code":"404","message": "Resource not found"}}
@OMS90 Have you also tried specifying end-point as mentioned in Azure OpenAI Service REST API reference - Azure OpenAI | Microsoft Learn ?
So in your case the model name would be gpt-4o but the endpoint will be something like https://[YOUR_RESOURCE_NAME].openai.azure.com/openai/deployments/[YOUR_DEPLOYMENT_NAME]/chat/completion → https://[YOUR_RESOURCE_NAME].openai.azure.com/openai/deployments/gpt4o/chat/completion
^ It looks like you may need to plug deployment name of gpt4o in the Azure end-point.
{
"cody.dev.models": [
{
"provider": "groq",
"model": "gpt-4o",
"inputTokens": 32000,
"outputTokens": 4000,
"apiKey": "XXXXXXXXXXXXXXXXXXXXXXXXXXXX",
"options": {
"temperature": 0.0
},
"apiEndpoint": "https://XXXXXXX.openai.azure.com/openai/deployments/gpt4o/chat/completion"
}
]
}
I also tried that setting.
In that case, the error message also changes.
（Set the endpoint to include version information. If the version information is removed, the message returns to “Not Found”.）
“apiEndpoint”: “https://XXXXXX.openai.azure.com/openai/deployments/gpt4o/chat/completions?api-version=2024-06-01”,
Request Failed:
HTTP 401 Unauthorized: { “statusCode”: 401, “message”: “Unauthorized. Access token is missing, invalid, audience is incorrect (https://cognitiveservices.azure.com), or have expired.” }
The “Not found” error is gone, but the key authentication does not seem to be working.
I also added this statement, but the result did not change.（Value is the same as apiKey.）
"headers": {
"api-key": "XXXXXXXXXXXXXXXXXXXXXXXXXXXX"
},
or
"headers": {
"Authorization": "Bearer XXXXXXXXXXXXXXXXXXXXXXXXXXXX"
}
@OMS90 Good catches about needing to specify also api-version=2024-06-01 and about api-key. Hmmm…, yes, looks like the main issue now is authorization problem. I don’t think that you can provide that api-key in a headers structure like you mentioned…
Take a look at Azure OpenAI Service REST API reference - Azure OpenAI | Microsoft Learn - especially at Microsoft Entra ID authentication part - it looks like that you need use this Micosft Entra ID in Cody’s apiKey parameter (to authenticate with Bearer - which is the standart method of Cody doing requests to all other OpenAI compatible services). This is probably different and new ID than you was previously using in api-key.
As for Microsoft Entra ID, it seems to me that one or the other would be good, acting as a replacement for the API key.
By the way, I upgraded from a free plan after reading the following description.
You also get additional capabilities like BYOLLM (Bring Your Own LLM), supporting Single-Tenant and Self Hosted setups for flexible coding environments. Your site administrator determines the LLM, and cannot be changed within the editor. However, Cody Enterprise users when using Cody Gateway have the ability to configure custom models Anthropic (like Claude 2.0 and Claude Instant), OpenAI (GPT 3.5 and GPT 4) and Google Gemini 1.5 models (Flash and Pro).
How should I recognize this description? Does it mean that calling the gpt-4o model as a custom API can only be done by subscribing to a paid plan?
Then the chat worked with the following settings, but it also worked with wrong URL and key values.
In other words, it seems to be using the default GPT-4o.
It does not refer to any URL or key I have set up.
（If the provider value is set to groq, it will result in an authentication error.）
{
"cody.dev.models": [
{
"provider": "openai",
"apiEndpoint": "https://XXXXX.openai.azure.com/openai/deployments/gpt4o/chat/completions?api-version=2024-06-01",
"apiKey": "XXXXXXXXXXXXXXXXXXXXX",
"model": "gpt-4o",
"inputTokens": 32000,
"outputTokens": 4000,
"options": {
"temperature": 0.0,
},
},
],
}
How do I understand this?
I have looked at the page you gave me, but I am confused.
- Azure Open API not available after all?
- If it available, how to set it up in
setting.json?
BYOK should work for cody.dev.models in free plan too (I am not 100% sure for Azure deployments but OpenRouter and other custum API end-points work also in free-plan), I think that your mentioned BYOLLM is more for Enterprise admins to set-up/host LLMs for all their devs in one centralized place.
^ Also make sure that you can see and are selecting GPT-4o model under the “Other” section (which should be your Azure deployment) because the “GPT-4o” model listed at the beginning is provided in Pro/paid plans.
Anyways, maybe @PriNova has some additional comments about this whole situation.
1 Like
@arvis
Thank you for teaching me so much.