Skip to main content

Language Model Overrides

Chat Completion Parameter Overrides​

The v1/chat/completion endpoint is compatible with OpenAI's API. It supports a subset of request body parameters defined in the OpenAI reference documentation. Spice helps configure different defaults for these request parameters.

Supported parameters:

Example: Setting Default Overrides​

To specify a default override for a parameter, use the openai_ prefix followed by the parameter name. For example, to set the temperature parameter to 0.1 for all requests with this model, use openai_temperature: 0.1. A temperature parameter in the request body will still override the default.

models:
  - name: pirate-haikus
    from: openai:gpt-4o
    params:
      openai_temperature: 0.1
      openai_response_format: { 'type': 'json_object' }

When sending this payload to spice /v1/chat/completions:

{
  "model": "pirate-haikus",
  "messages": [
    {
      "role": "user",
      "content": "What is the capital of France?"
    }
  ],
  "temperature": 0.5
}

Will be passed to the OpenAI API as:

{
  "model": "gpt-4",
  "messages": [
    {
      "role": "user",
      "content": "What is the capital of France?"
    }
  ],
  "temperature": 0.5,                          // temperature overriden by value in request body
  "response_format": { "type": "json_object" } // default response format from model configuration
}

System Prompt​

In addition to any system prompts provided in message dialogue, or added by model providers, Spice can configure an additional system prompt.

models:
  - name: pirate-haikus
    from: openai:gpt-4o
    params:
      system_prompt: |
        Write everything in Haiku like a pirate

Any request to HTTP v1/chat/completion will include the configured system prompt.

Example: Enforcing default structured output and using system prompt​

This example demonstrates how to create a specialized math tutoring model by combining system prompts with structured JSON output. The configuration ensures consistent, step-by-step mathematical solutions in a machine-readable format.

models:
  - name: math-tutor
    from: openai:gpt-4o
    params:
      system_prompt: |
        You are a helpful math tutor. Guide the user through the solution step by step.
      openai_response_format:
        type: json_schema
        json_schema:
          name: math_reasoning
          schema:
            type: object
            properties:
              steps:
                type: array
                items:
                  type: object
                  properties:
                    explanation:
                      type: string
                    output:
                      type: string
                  required:
                    - explanation
                    - output
                  additionalProperties: false
              final_answer:
                type: string
            required:
              - steps
              - final_answer
            additionalProperties: false
          strict: true

To use the configured math tutor, send a simple request to the chat completions endpoint:

curl -s -XPOST http://localhost:8090/v1/chat/completions -H "Content-Type: application/json" -d \
    '{
      "model": "math-tutor",
      "messages": [{
        "role": "user",
        "content" :"how can I solve 8x + 7 = -23"
      }]
    }' \
  | jq '.choices[0].message.content | fromjson'

Example response:

{
  "final_answer": "x = -3.75",
  "steps": [
    {
      "explanation": "We start with the given equation that we need to solve.",
      "output": "8x + 7 = -23"
    },
    {
      "explanation": "Our goal is to solve for x. We can start by isolating the term with x on one side of the equation. To do this, we need to eliminate the constant term (7) on the left side. We subtract 7 from both sides of the equation in order to keep it balanced.",
      "output": "8x + 7 - 7 = -23 - 7"
    },
    {
      "explanation": "Subtracting 7 from both sides simplifies the equation. On the left side, the +7 and -7 cancel out, leaving just the term with the variable.",
      "output": "8x = -30"
    },
    {
      "explanation": "Now, we have 8 times x equals -30. To solve for x, we divide both sides of the equation by the coefficient of x, which is 8.",
      "output": "8x / 8 = -30 / 8"
    },
    {
      "explanation": "Dividing both sides results in x on the left side and simplifies the fraction on the right side. The fraction -30/8 can be simplified further by dividing both the numerator and the denominator by their greatest common divisor, which is 2.",
      "output": "x = -3.75"
    },
    {
      "explanation": "The solution has been simplified completely, giving us the value of x.",
      "output": "x = -3.75"
    }
  ]
}

Visit OpenAI Structured Outputs for more information on how to use structured output formats.