OpenAI Responses API: A Practical Guide to State, Tools, Streaming and Migration

TL;DR

• The OpenAI Responses API (POST /v1/responses) is OpenAI’s recommended interface for new projects. It takes input items and returns a typed output array: reasoning, messages, tool calls and results.
• State is optional: replay output items yourself, chain turns with previous_response_id, or attach turns to a durable conversation object. Stored responses are kept for 30 days by default, and earlier input tokens in a chain are still billed on every turn.
• Hosted tools (web search, file search, code interpreter, remote MCP servers, computer use) run inside one request, while your own functions return a function_call item that you answer with a matching call_id.
• There is no separate API fee. Cost comes from model tokens (reasoning bills as output), tool calls, containers and storage; OpenAI reports 40% to 80% better cache utilization than Chat Completions.
• Azure, xAI and DeepSeek serve Responses-compatible endpoints, the Open Responses specification standardizes the format, and gateways translate it for chat-only providers.

The OpenAI Responses API is the interface OpenAI now recommends for every new integration, and in 2026 it stopped being optional for some workloads: tool calling on GPT-6 Astra and GPT-6.1 Sol runs only through Responses, and the Assistants API it replaced was shut down on August 26, 2026. This guide covers how the OpenAI Responses API differs from Chat Completions, what requests and responses look like in Python and curl, how conversation state and storage work, the built-in tools, function calling, streaming events, reasoning items, what changes on the bill, a migration checklist, and how to run the same request format against other providers.

OpenAI Responses API at a Glance

The OpenAI Responses API is a stateful-capable endpoint for generating model output that accepts text, images and files as input items, can run built-in and custom tools within a single request, and returns a list of typed output items rather than a single message. It is served at POST /v1/responses.

OpenAI’s migration guide states the position plainly: while Chat Completions remains supported, Responses is recommended for all new projects. The API launched in March 2025, and OpenAI’s deprecations page records that it announced the plan to fold Assistants features into it at the same time.

The current model lineup is built around it. OpenAI’s models page names GPT-6 Astra as the flagship for complex reasoning and coding, GPT-6.1 Sol to balance cost and GPT-6 Luna for high-volume work. For GPT-6.1 Sol, the GPT-6 guide says to use the Responses API for tool calling, because Chat Completions supports requests without tools only.

Chat Completions vs Responses API

The core difference between Chat Completions and the Responses API is the unit of context. Chat Completions passes messages; Responses passes items, where a message is one type alongside reasoning, function calls, function outputs and hosted tool calls. That lets one request run several tools and carry reasoning between turns.

The practical consequences: the Responses API drops the n parameter, so it returns only one generation per request. Structured Outputs moved from response_format to text.format. Function definitions are internally tagged rather than nested under a function key, and in Chat Completions, functions are non-strict by default, whereas Responses attempts strict mode when strict is omitted.

Reasoning models widen the gap. Starting with GPT-5.4, Chat Completions does not support tool calling with any reasoning effort other than none. OpenAI also claims a quality difference: its internal evals reveal a 3% improvement in SWE-bench with the same prompt and setup on Responses.

Chat Completions still fits an existing integration with no tools, code that depends on n, or a provider that only implements the chat format.

Request and Response Shape

A minimal Responses API request needs only model and input. System guidance goes into a top-level instructions field, and the response is an object with an id, a status and an output array; SDKs add output_text, which joins the text of every message item.

In Python, with the official SDK:

from openai import OpenAI

client = OpenAI()

response = client.responses.create(
    model="gpt-6.1-sol",
    instructions="You are a concise technical assistant.",
    input="Explain what an idempotency key is in two sentences.",
)

print(response.id)
print(response.output_text)

The same request with curl:

curl https://api.openai.com/v1/responses \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "model": "gpt-6.1-sol",
    "instructions": "You are a concise technical assistant.",
    "input": "Explain what an idempotency key is in two sentences."
  }'

Most migration bugs come from the response body. A reasoning model usually returns a reasoning item first and a message item second, so code that reads output[0] as the answer gets the reasoning item. Use output_text when you only need text, and branch on each item’s type when the flow uses tools or reasoning.

Text-only messages arrays can also be passed straight into input, so a first migration step can be as small as changing the method name.

Conversation State: previous_response_id, Conversations and store

The Responses API gives three ways to carry context between turns: replay the previous output items yourself, pass previous_response_id so OpenAI loads the prior turn, or attach every turn to a Conversations API object with a durable ID. None of them makes earlier context free.

Figure 1: Chaining and conversations move storage to OpenAI, but every earlier input token in the chain is still billed on each turn.

As Figure 1 shows, the options differ in who stores what:

• Manual replay. Append the previous response’s output items to the next input. This is the only option that works under Zero Data Retention and with providers that do not store responses.
• previous_response_id. Reference the last response and send only the new input. OpenAI’s migration guide warns that this does not carry over the previous response’s top-level instructions, so resend them on every request.
• Conversations API. Create a conversation once and pass it on each request. The conversation state guide describes it as a way to persist conversation state as a long-running object with its own durable identifier.

first = client.responses.create(model="gpt-6.1-sol", input="Name three SQL isolation levels.")
second = client.responses.create(
    model="gpt-6.1-sol",
    previous_response_id=first.id,
    input="Which one prevents phantom reads?",
)


conversation = client.conversations.create()
reply = client.responses.create(
    model="gpt-6.1-sol",
    conversation=conversation.id,
    input="Name three SQL isolation levels.",
)

Storage, Retention and store: false

Responses are stored by default; Chat Completions are too, but only for new accounts. OpenAI’s documentation says response objects are saved for 30 days by default, while conversation objects and items in them are not subject to the 30 day TTL. Set store: false to opt out.

Opting out does not mean losing reasoning continuity. For stateless and Zero Data Retention use, OpenAI offers encrypted reasoning items: the API returns reasoning in an opaque encrypted_content field, you replay it on the next request, and OpenAI decrypts it in memory and discards it. Data-residency trade-offs are covered in agitracker’s explainer on sovereign AI.

Built-in Tools: Web Search, File Search, Code Interpreter, MCP and Computer Use

Built-in tools are OpenAI-hosted capabilities that you enable in the tools array and that run on OpenAI’s side within the same request. The tools guide lists web search, file search, tool search, remote MCP servers, computer use, code execution and image generation, alongside your own function definitions.

OpenAI’s own one-line definitions:

• Web search allows models to access up-to-date information from the internet and provide answers with sourced citations.
• File search lets models retrieve information in a knowledge base of previously uploaded files stored in vector stores.
• Code interpreter allows models to write and run Python code in a sandboxed environment, using a container you create or one created automatically.
• Remote MCP servers can be any server on the public Internet that implements the Model Context Protocol, and a Secure MCP Tunnel covers private servers.
• Computer use lets a model operate a browser or desktop interface, with your application supplying the driver that takes each action.

A request that combines web search with a remote MCP server looks like this:

response = client.responses.create(
    model="gpt-6.1-sol",
    tools=[
        {"type": "web_search"},
        {
            "type": "mcp",
            "server_label": "openai_docs",
            "server_url": "https://developers.openai.com/mcp",
            "require_approval": "never",
        },
    ],
    input="Find the current Responses API streaming event names and cite sources.",
)

Figure 2: Hosted tools finish inside a single request, while a function call always costs a second request carrying the matching call_id.

Figure 2 is the mental model to keep: hosted tools loop on the server, so one request can search, read files and call MCP tools before answering. Remote MCP calls can either be allowed automatically, or restricted with explicit approval, and approval matters because a remote server sees whatever the model sends it.

The risks are covered in agitracker’s guide to MCP server security, and the layering with agent-to-agent protocols in A2A vs MCP. A longer treatment of putting tool traffic behind a control point is in enterprise MCP governance for AI agents.

OpenAI Function Calling in the Responses API

OpenAI function calling in the Responses API works as a two-request loop: the model returns a function_call item with a call_id and JSON arguments, your code runs the function, and you send back a function_call_output item with the same call_id. The model then answers or calls more tools.

The function calling guide breaks this into five steps, and the migration guide adds the rule that trips most ports: in Responses, tool calls and their outputs are two distinct types of Items correlated by call_id, not a single assistant message with a tool_calls array.

import json

tools = [{
    "type": "function",
    "name": "get_order_status",
    "parameters": {
        "type": "object",
        "properties": {"order_id": {"type": "string"}},
        "required": ["order_id"],
        "additionalProperties": False,
    },
}]

first = client.responses.create(
    model="gpt-6.1-sol", tools=tools, input="Where is order A-1042?"
)

follow_up = []
for item in first.output:
    if item.type == "function_call":
        args = json.loads(item.arguments)
        result = {"order_id": args["order_id"], "status": "shipped"}  # your code
        follow_up.append({
            "type": "function_call_output",
            "call_id": item.call_id,
            "output": json.dumps(result),
        })

final = client.responses.create(
    model="gpt-6.1-sol",
    tools=tools,
    previous_response_id=first.id,
    input=follow_up,
)
print(final.output_text)

With reasoning models and manual state, pass back every reasoning, function call and function output item since the last user message. The definition above is also flat (name at the top level); a Chat Completions definition nested under "function" will not validate.

Streaming Events

Streaming in the Responses API uses typed server-sent events instead of the anonymous delta chunks of Chat Completions. Each event carries a type such as response.created, response.output_text.delta or response.completed, so a client subscribes to the events it needs and ignores the rest.

OpenAI’s streaming guide describes this as semantic events with a predefined schema per type. The migration guide’s point is that Responses streaming uses typed server-sent events, so old chunk handlers have to be rewritten rather than repointed.

stream = client.responses.create(
    model="gpt-6.1-sol",
    input="Write a haiku about rate limits.",
    stream=True,
)

for event in stream:
    if event.type == "response.output_text.delta":
        print(event.delta, end="", flush=True)
    elif event.type == "response.function_call_arguments.done":
        print("\n[tool call]", event.arguments)
    elif event.type == "response.completed":
        print("\nusage:", event.response.usage)

The core text events are response.created, response.output_text.delta, response.completed and error. Function calls add response.function_call_arguments.delta and .done. Background jobs can also stream and resume from a sequence_number cursor after a disconnect.

Reasoning Items, Effort and Background Mode

Reasoning items are the Responses API’s container for a reasoning model’s hidden thinking. The raw reasoning is never returned; instead the API returns an item that can be passed back on the next turn, optionally with a readable summary or, in stateless mode, an encrypted payload. Effort is set per request with reasoning.effort.

Three facts from OpenAI’s reasoning guide drive cost and reliability:

• Reasoning tokens are billed as output. They are not visible via the API, but they occupy context and are billed as output tokens.
• Budget for them. OpenAI recommends reserving at least 25,000 tokens for reasoning and outputs when you start; a response that hits max_output_tokens during reasoning returns incomplete with no visible text but still bills for the tokens used.
• Summaries are opt-in. A reasoning summary will not be included unless you explicitly opt in through reasoning.summary.

Effort values are model-dependent, ranging from none through max. According to the reasoning guide, GPT-6.1 Sol, GPT-6 Sol, and GPT-6 Luna also default to medium, and GPT-6 Astra and GPT-6.1 Sol do not accept none. In stateless mode, reasoning items include an encrypted_content property by default, so replaying the full output array keeps the chain intact.

For work that runs for minutes, set background: true. OpenAI’s background mode guide says it kicks off tasks asynchronously so developers can poll the response object until it leaves the queued or in_progress state, and cancel it if needed. For Zero Data Retention projects, background response data is temporarily stored to disk for roughly 10 minutes to make polling possible.

Responses API Pricing and Cost Implications

The Responses API has no separate access fee: a request is billed for the model’s input, cached input and output tokens, plus any built-in tool calls, containers and storage it uses. The format still changes the bill, because state, caching, tools and reasoning all behave differently from Chat Completions.

Standard rates per million tokens on OpenAI’s pricing page for the current lineup:

Long-context prompts cost more, and Batch and Flex cost half. The detail behind the Sol and Luna price cuts is in agitracker’s GPT-6 pricing analysis. The Responses-specific cost effects are:

• Chaining is not free. Even with previous_response_id, all previous input tokens for responses in the chain are billed as input tokens. Long conversations need compaction or trimming either way.
• Caching improves. OpenAI attributes lower costs to improved cache utilization (40% to 80% improvement versus Chat Completions in internal tests). In the Responses API, you can choose a 12-hour minimum lifetime for cached prefixes at a higher write rate.
• Tools add line items. Tokens used for built-in tools are billed at the chosen model’s rates on top of per-call fees, and file search tool call pricing applies to the Responses API only.
• Reasoning is output. Output costs five times input on all three GPT-6 tiers, so effort settings often matter more than the per-token price.

Because the provider dashboard reports one number per organization, teams usually add attribution of their own. A primer on monitoring LLM API costs in production explains why native provider dashboards give a single number per month, and a separate guide on cutting LLM token costs covers output-heavy workloads.

Migration Checklist from Chat Completions and the Assistants API

Migrating to the Responses API is three changes: send requests to /v1/responses, read output from the typed output array, and decide how each flow carries state. Because Chat Completions remains supported, you can migrate one user flow at a time, starting with plain text generation.

A working checklist, adapted from OpenAI’s rollout list and common errors:

1. Endpoint and body. Change chat.completions.create to responses.create and messages to input; move the system prompt into instructions.
2. Output handling. Replace choices[0].message.content with output_text, or iterate output by type.
3. State. Pick manual replay, previous_response_id or a conversation per flow, and resend instructions on chained turns.
4. Storage. Add store: false where data must not persist, and replay encrypted reasoning items.
5. Functions. Flatten definitions, decide on strict, and return every result with its matching call_id.
6. Structured Outputs. Structured Outputs definitions have moved from response_format to text.format.
7. Streaming. Rewrite stream consumers to switch on event type.
8. Hosted tools. Replace home-grown search or code execution where OpenAI’s tools fit.
9. Compare. Check behavior, latency, tokens and errors before shifting traffic.

OpenAI Assistants API Sunset

For OpenAI Assistants API users the migration is no longer optional. OpenAI’s Assistants migration guide states that the Assistants API was officially sunset on August 26, 2026, and is no longer available. The mapping is direct: assistants become prompts (versioned in the dashboard), threads become conversations that store any item type, and runs become responses. Any integration still calling /v1/assistants or /v1/threads has been failing since late August.

Open Responses and Multi-Provider Portability

The Responses format is no longer OpenAI-only. Open Responses is an open specification derived from the OpenAI Responses API, and several providers now serve Responses-compatible endpoints directly. Compatibility is partial, though: stateless request and response shapes travel well, while stored state, hosted tools and background mode often do not.

The Open Responses project describes itself as an open-source specification and ecosystem for building multi-provider, interoperable LLM interfaces based on the OpenAI Responses API. Its specification separates externally hosted tools (functions and MCP) from internally hosted ones that live inside the provider’s system, and it lets providers choose never to expose raw reasoning, returning an opaque encrypted payload instead.

Provider support as documented in October 2026:

Microsoft’s Azure documentation says to use the Azure OpenAI Responses API to generate stateful, multi-turn responses. xAI, whose documentation now carries the SpaceXSI name, says the Responses API is the recommended way to interact with SpaceXSI models, and that the responses will be stored for 30 days by default.

DeepSeek’s change log says the DeepSeek API now natively supports the OpenAI Responses API format, adapted for Codex. Its compatibility notes list previous_response_id, conversation, store and background as unsupported, and fields that depend on unsupported capabilities always take fixed values such as store: false. Code that relies on server-side chaining breaks against it; code that replays items does not.

The model side is covered in agitracker’s DeepSeek V4 explainer and the companion DeepSeek API guide.

The portable subset is therefore clear: input items, instructions, function tools with call_id round-trips, streaming text events and client-side state. Anything beyond that needs either a provider that implements it or a translation layer.

Running the Responses API Across Providers Through a Gateway

A gateway lets one Responses-format client reach providers that implement the format natively and providers that only speak a chat API. The gateway accepts /v1/responses, picks the provider from the model name, and either forwards the call or translates it, while adding keys, budgets, fallbacks and logs in one place.

This is the OpenAI-compatible API pattern applied to Responses, and endpoint coverage differs sharply between gateways. Three approaches with documented Responses support:

• LiteLLM. The proxy states that LiteLLM provides an endpoint in the spec of OpenAI’s /responses API. For OpenAI-compatible providers without their own Responses support, it bridges /v1/responses through /v1/chat/completions. It is Python-based and widely deployed; teams weighing a move are covered in agitracker’s LiteLLM alternatives comparison.
• Vercel AI Gateway. A managed option: point your OpenAI SDK to AI Gateway’s base URL and use provider/model identifiers to route Responses requests to OpenAI, Anthropic, Google and others. Vercel also states that AI Gateway supports the OpenResponses API specification.
• Bifrost. An open-source, self-hosted gateway, covered in the worked example below because its documentation spells out how Responses requests are handled for each provider.

When comparing any of them, check the provider list, which Responses fields survive translation, and where the gateway runs. agitracker’s comparison of AI gateways for Anthropic, OpenAI and Gemini covers reasoning translation, and a comparison of multi-provider AI gateways for OpenAI, Anthropic and Bedrock widens the field.

Worked Example: Bifrost

Bifrost is an open-source AI gateway from Maxim AI that serves 25+ providers and 10,000+ models through one OpenAI-compatible API, with Responses on the same base URL as Chat Completions. Its provider support table marks Responses and Responses streaming for providers including OpenAI, Azure, Anthropic, Bedrock, Vertex AI, Gemini, xAI, DeepSeek, Groq, Mistral and vLLM, and explains that, depending on the provider, Bifrost either uses a native responses endpoint or maps to an equivalent chat API.

Figure 3: The client keeps one request shape; the gateway decides per provider whether to pass the call through natively or translate it.

As Figure 3 shows, the client change is the base URL. Bifrost’s drop-in replacement docs describe it as a single-line change: update your base_url to point to the gateway. Its OpenAI SDK integration then lets you use multiple providers through the same OpenAI SDK format by prefixing model names with the provider:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8080/openai",  # Bifrost gateway
    api_key="sk-bf-...",                      # Bifrost virtual key
)

for model in ["openai/gpt-6.1-sol", "xai/grok-4.7", "deepseek/deepseek-flash"]:
    r = client.responses.create(
        model=model,
        instructions="Answer in one sentence.",
        input="What does an idempotency key prevent?",
    )
    print(model, "->", r.output_text)

In curl, the same request goes to http://localhost:8080/openai/v1/responses with the virtual key as a bearer token. Around that call, the gateway adds:

• Access and budgets. Virtual keys are the primary governance entity in Bifrost, carrying per-consumer provider permissions, budgets and rate limits. The background on why provider-side limits are not enough is in a guide to managing OpenAI rate limits at scale.
• Failover. When a provider fails, Bifrost moves on to the next provider in your fallback chain, with retries and key rotation within each provider first; a walkthrough is in automatic fallback with Bifrost.
• Async jobs. For providers without OpenAI’s background mode, the gateway’s async inference lets clients submit a normal inference payload to an async endpoint, including /v1/async/responses, and poll for the result.
• Coding agents. For Codex CLI, Bifrost connects it through a named custom provider backed by the Responses API, and Codex’s session header means a session sticks to the provider and key that served it, keeping prompt-cache hits. A setup walkthrough is in running Codex CLI with multiple model providers.

On performance, Bifrost’s published benchmark reports 11 µs of added overhead per request at 5,000 RPS on an AWS t3.xlarge; agitracker’s guide to LLM gateway benchmarks explains how to measure that kind of figure on your own traffic.

Teams on LiteLLM can follow the LiteLLM migration guide, the OpenAI provider guide covers key setup, and an article on using any provider with the OpenAI SDK describes the gateway as a protocol adapter.

Deployment note. The gateway starts from an npx package or a Docker image, so the Responses endpoint can run inside the same network as the applications calling it.

What the Responses API Means for Teams Building Agents

The OpenAI Responses API is now the default way to build on OpenAI models: typed items make tool loops and reasoning continuity explicit, hosted tools remove orchestration code, and the newest models reserve tool calling for it. Teams on Chat Completions without tools can move at their own pace; Assistants API users already had to.

The decisions that matter are about state and portability. Server-side chaining is convenient but ties a workload to providers that store responses, and still bills the full history each turn. Client-side replay with store: false runs unchanged on OpenAI, Azure, xAI, DeepSeek and any Open Responses implementation. A sound default for multi-provider systems is to treat that stateless subset as the contract, use hosted tools where OpenAI-specific behavior is acceptable, and keep translation, keys and failover in one layer rather than in each application, a pattern agitracker’s API integration guide covers in more depth.