Skip to main content
Version: Next

OpenAI (or Compatible) Language Models

To use a language model hosted on OpenAI (or compatible), specify the openai path in the from field.

For a specific model, include it as the model ID in the from field (see example below). The default model is gpt-4o-mini.

models:
- from: openai:gpt-4o-mini
name: openai_model
params:
openai_api_key: ${ secrets:OPENAI_API_KEY } # Required for official OpenAI models, unless the model uses workload identity federation
tools: auto # Optional. Connect the model to datasets via SQL query/vector search tools
system_prompt: 'You are a helpful assistant.' # Optional.

# Optional parameters
endpoint: https://api.openai.com/v1 # Override to use a compatible provider (i.e. NVidia NIM)
openai_org_id: ${ secrets:OPENAI_ORG_ID }
openai_project_id: ${ secrets:OPENAI_PROJECT_ID }

# Override default chat completion request parameters
openai_temperature: 0.1
openai_response_format: { 'type': 'json_object' }

# OpenAI Responses API configuration
responses_api: enabled
openai_responses_tools: web_search, code_interpreter

Configuration​

from​

The from field takes the form openai:model_id where model_id is the model ID of the OpenAI model, valid model IDs are found in the {endpoint}/v1/models API response.

Example:

curl -H "Authorization: Bearer $OPENAI_API_KEY" https://api.openai.com/v1/models
{
"object": "list",
"data": [
{
"id": "gpt-4o-mini",
"object": "model",
"created": 1727389042,
"owned_by": "system"
},
...
}

name​

The model name. This will be used as the model ID within Spice and Spice's endpoints (i.e. http://localhost:8090/v1/models). This can be set to the same value as the model ID in the from field.

params​

ParamDescriptionDefault
endpointThe OpenAI API base endpoint. Can be overridden to use a compatible provider (i.e. NVIDIA NIM).https://api.openai.com/v1
toolsWhich tools should be made available to the model. Set to auto to use all available tools.-
system_promptAn additional system prompt used for all chat completions to this model.-
openai_api_keyThe OpenAI API key.-
openai_org_idThe OpenAI organization ID.-
openai_project_idThe OpenAI project ID.-
openai_identity_provider_idThe OpenAI workload identity provider (idp_...) that verifies the workload's identity token. With openai_service_account_id, the model authenticates with workload identity federation instead of openai_api_key.-
openai_service_account_idThe OpenAI service account (user-...) that the workload identity provider's mapping resolves the workload to. Required for workload identity federation.-
openai_identity_token_audienceThe audience Spice requests the workload's identity token for, from GitHub Actions or the workload's cloud identity. Not used with openai_identity_token_file. The workload identity provider's Audience must match the aud claim of the token issued for this value, which is usually this value but, for a Microsoft Entra app registration that issues v2.0 access tokens, is the application ID.https://api.openai.com/v1
openai_identity_token_fileA file that holds the workload's identity token, such as a Kubernetes projected service account token. Spice reads it again for every exchange. When unset, Spice requests the token from GitHub Actions, then from the workload's cloud identity.-
openai_temperatureSet the default temperature to use on chat completions.-
openai_response_formatAn object specifying the format that the model must output, see structured outputs.-
openai_reasoning_effortFor reasoning models, like o1, this parameter specifies the reasoning effort used for the model.-
openai_usage_tierThe OpenAI usage tier for the account. This parameter sets the maximum number of concurrent requests based on OpenAI's published limits per tier. Valid values are free, tier1, tier2, tier3, tier4, or tier5.tier1
responses_apienabled or disabled. Whether to enable invoking this model from the /v1/responses HTTP endpoint using OpenAI's Responses API. When using OpenAI-compatible providers, ensure the provider supports OpenAI's Responses API.disabled
openai_responses_toolsComma-separated list of OpenAI-hosted tools exposed via the Responses API for this model. These hosted tools are not available from the /v1/chat/completions HTTP endpoint. Supported tools: code_interpreter, web_search, web_search_preview.-

web_search selects OpenAI's current web search tool. web_search_preview selects the legacy tool of the same name, retained for older models that do not support the current one — prefer web_search unless the model rejects it.

See Large Language Models for additional configuration options.

Workload Identity Federation​

An OpenAI model can authenticate with workload identity federation instead of an API key. Spice exchanges the identity token that the platform gives the workload for a short-lived OpenAI access token (RFC 8693), and exchanges again before each access token expires, so no OpenAI API key is stored with Spice. The same parameters apply to OpenAI embedding models.

models:
- from: openai:gpt-4o-mini
name: gpt
params:
openai_identity_provider_id: ${ secrets:OPENAI_IDENTITY_PROVIDER_ID }
openai_service_account_id: ${ secrets:OPENAI_SERVICE_ACCOUNT_ID }
openai_identity_token_file: /var/run/secrets/tokens/openai

openai_identity_provider_id and openai_service_account_id are both required. The identity token comes from the first of these sources that answers:

  1. openai_identity_token_file, read again for every exchange.
  2. The GitHub Actions job Spice runs in, which needs permissions: id-token: write (see OpenID Connect in GitHub Actions).
  3. The workload's cloud identity, tried in this order: the AWS IAM role, through STS GetWebIdentityToken, which needs outbound web identity federation enabled on the AWS account and sts:GetWebIdentityToken granted to the role; the Google Cloud metadata server; then the workload's Microsoft Entra identity. A source counts only once it issues a token for the audience, so AWS credentials without sts:GetWebIdentityToken pass to the next source. A model on EC2 or Compute Engine therefore needs no token file. Microsoft Entra ID issues a token only for an app registration of the tenant, so on Azure, create an app registration that represents the provider, set openai_identity_token_audience to its identifier URI (api://<app ID>). Configure the provider's federation to accept the issuer and the aud claim of the token Entra issues. For an app registration that issues v2.0 access tokens, aud is the application ID, not the api:// URI (access token claims). With the default audience, the Azure source fails with AADSTS500011 or AADSTS50001, and, as the last source, leaves the model unable to load.

Spice requests the token from GitHub Actions or the cloud identity for openai_identity_token_audience, which defaults to https://api.openai.com/v1. The audience of a token file is set where the token is issued, so a model that sets both openai_identity_token_file and openai_identity_token_audience fails to load. When no source answers, the model fails to load with an error that lists what each cloud identity reported.

Spice exchanges again one minute before the access token expires, or, for a token that lasts less than four minutes, after three quarters of its lifetime. When an exchange after the first one fails, Spice keeps using the current access token and retries, waiting at most 15 seconds between attempts.

A model that sets a federation parameter and openai_api_key in the Spicepod fails to load with an error that names both, and a model that sets only some of the required parameters fails with an error that names the missing one. An openai_api_key loaded automatically from a secret store does not conflict with federation. When OpenAI refuses the first exchange, the model fails to load with OpenAI's reason.

Supported OpenAI Compatible Providers​

Spice supports several OpenAI compatible providers. Specify the appropriate endpoint in the params section.

Azure OpenAI​

Follow Azure AI Models instructions.

Groq​

Groq provides OpenAI compatible endpoints. Use the following configuration:

models:
- from: openai:llama3-groq-70b-8192-tool-use-preview
name: groq-llama
params:
endpoint: https://api.groq.com/openai/v1
openai_api_key: ${ secrets:SPICE_GROQ_API_KEY }

NVIDIA NIM​

NVIDIA NIM models are OpenAI compatible endpoints. Use the following configuration:

models:
- from: openai:my_nim_model_id
name: my_nim_model
params:
endpoint: https://my_nim_host.com/v1
openai_api_key: ${ secrets:SPICE_NIM_API_KEY }

For examples of setting up NVIDIA NIM with Spice, see the Nvidia NIM on AWS EC2 and Nvidia NIM on Kubernetes cookbook recipes.

Parasail​

Parasail also offers OpenAI compatible endpoints. Use the following configuration:

models:
- from: openai:parasail-model-id
name: parasail_model
params:
endpoint: https://api.parasail.com/v1
openai_api_key: ${ secrets:SPICE_PARASAIL_API_KEY }

Refer to the respective provider documentation for more details on available models and configurations.

Cookbook​