OpenAI Decisions API & Luna Model: Ultra-Low Latency Agentic Routing & 90% Prompt Cache Economics

A technical architectural deep-dive into OpenAI’s newly unveiled Decisions API and the Luna micro-reasoning model: analyzing sub-25ms deterministic classification, constrained decoding mathematics, and optimizing 90% prompt caching hit rates for high-throughput enterprise agent swarms.

High-Throughput Optical Switches — The physical network fabric supporting sub-25ms decision routing across enterprise cloud clusters
Photo: Wikimedia Commons / Creative Commons

The Latency and Cost Tax in Modern Agent Orchestration

In high-throughput enterprise architectures, autonomous agent systems frequently suffer from a crippling architectural anti-pattern: using massive, general-purpose frontier models (such as GPT-6 Astra, Claude 3.5 Sonnet, or Llama 3.1 405B) to make simple, discrete binary or categorical routing choices.

Consider a production customer support or DevOps triage pipeline ingesting 100,000 requests per hour. The vast majority of incoming events are routine: classifying whether an incoming email is a billing query or technical bug, deciding which specialized agent should receive a ticket, or verifying whether a tool argument satisfies authorization boundaries.

When an architecture invokes a 200-billion-parameter model for these discrete decisions, it incurs an unacceptable double tax:

• Latency Overhead: Even with modern speculative decoding, full frontier models exhibit a Time-To-First-Token (TTFT) of 400ms to 1,200ms, introducing crippling bottlenecks in multi-agent handoff chains.
• Economic Inefficiency: Re-sending identical multi-page system prompts and tool schemas to heavy models costs thousands of dollars daily for simple classification logic that could be performed with a fraction of the compute.

To resolve this systemic inefficiency, OpenAI introduced the Decisions API powered by a specialized, ultra-compact classification foundation model codenamed Luna, alongside sweeping upgrades to its Prompt Caching engine delivering up to 90% cost discounts.

Inside the Luna Model: Constrained Decoding at Sub-25ms Speeds

Unlike standard chat completion models that generate free-form conversational text, Luna is engineered strictly for constrained, structured decision-making.

1. Logit-Masked Constrained Decoding

When an application calls the Decisions API, the developer supplies an input text alongside an explicit enumerated array of allowed outcomes (for example: ["route_to_sales", "route_to_support", "mark_as_spam", "escalate_to_tier2"]).

Luna enforces deterministic selection by modifying the final softmax layer during generation:

• During the forward pass, the model’s vocabulary logit vector is masked so that only tokens corresponding to the valid enumerated target choices can be assigned non-zero probabilities.
• This mathematical constraint guarantees that Luna cannot produce conversational filler, polite acknowledgments, or invalid schemas. Output is 100% syntactically valid and arrives in a single decoding step.

2. Sub-25ms Execution Throughput

By pruning model parameters to under 3 billion and optimizing KV-cache layouts for single-token prediction, OpenAI delivers inference latencies under 25 milliseconds on standard cloud instances—approaching the speed of local embedded neural classifiers while retaining the broad semantic understanding of large foundation models.

Comparative Architectural Benchmark: Intent Routing Paradigms

Engineering Guide: Optimizing 90% Prompt Caching Hit Rates

Prompt caching allows an inference server to store the key-value (KV) states of static prompt prefixes in GPU High Bandwidth Memory (HBM). When subsequent requests arrive with identical initial token prefixes, the engine skips the compute-heavy attention prefill stage, streaming outputs almost instantaneously and passing massive cost savings back to the developer.

To achieve the newly announced 90% prompt caching discount in production, architectures must adhere to strict payload alignment rules:

1. Deterministic Prefix Construction

OpenAI’s cache lookup operates on an exact cryptographic hash of token IDs from the beginning of the prompt. If any dynamic variable (such as a timestamp, request ID, or ephemeral user token) is placed early in the payload, the entire cache key is invalidated:

# ANTI-PATTERN: Breaks Prompt Caching (Cache Miss Rate: 100%)
prompt_payload = [
    {"role": "system", "content": f"Current Time: {datetime.utcnow()} \n You are an SRE agent..."},
    {"role": "user", "content": "Check cluster metrics."}
]

# OPTIMAL PATTERN: Maximizes 90% Prompt Caching (Cache Hit Rate: >88%)
prompt_payload = [
    # Static prefix > 1,024 tokens: Base system instructions + tool definitions
    {"role": "system", "content": STATIC_SYSTEM_PROMPT_WITH_TOOLS},
    # Dynamic user context and timestamps pushed strictly to the final leaf block
    {"role": "user", "content": f"[Timestamp: {datetime.utcnow()}] Check cluster metrics."}
]

Mathematical Formulation of Constrained Softmax Decoding

To understand how the Decisions API achieves zero schema hallucinations, consider the mathematical mechanics of token selection under constrained decoding. In standard autoregressive generation, the probability of selecting the next token from a complete vocabulary of 128,000 tokens is calculated across all heads.

In the Luna runtime, the denominator is not evaluated across the entire 128,000-token vocabulary. Instead, an active logit mask is computed dynamically from the Trie representation of the target decision enum. This eliminates 99.99% of compute operations in the final projection layer, allowing the GPU tensor core to resolve the forward pass in under 5 milliseconds and guaranteeing mathematical impossibility of generating any token outside the allowed decision set.

Python Implementation: Integrating Decisions API & Prompt Caching

Below is an operational implementation demonstrating how to use the Decisions API (Luna) for sub-25ms event triage, routing complex requests to a cached GPT-6.1 Sol agent session:

import openai
import os
import time

client = openai.OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))

def triage_and_dispatch(incoming_event: str):
    start_time = time.perf_counter()
    
    # 1. Sub-25ms fast triage via Decisions API (Luna model)
    decision = client.decisions.create(
        model="luna-001",
        input=incoming_event,
        decisions=["fast_ack", "escalate_to_sre_agent", "reject_malformed"],
        explanation=False # Disable explanation to guarantee single-token latency
    )
    
    routing_latency_ms = (time.perf_counter() - start_time) * 1000
    chosen_action = decision.selected_decision
    print(f"[Decisions API] Selected '{chosen_action}' in {routing_latency_ms:.2f}ms")
    
    # 2. If triage requires action, dispatch to cached GPT-6.1 Sol agent
    if chosen_action == "escalate_to_sre_agent":
        response = client.chat.completions.create(
            model="gpt-6.1-sol",
            messages=[
                # Large static prefix automatically hits 90% prompt cache
                {"role": "system", "content": os.environ.get("STATIC_SRE_RUNBOOK_PROMPT")},
                {"role": "user", "content": f"Investigate alert: {incoming_event}"}
            ]
        )
        # Inspect usage to verify cache hit
        cached_tokens = getattr(response.usage, "prompt_tokens_details", {}).get("cached_tokens", 0)
        print(f"[GPT-6.1 Sol] Executed with {cached_tokens} cached tokens.")
        return response.choices[0].message.content
        
    return {"status": chosen_action}

if __name__ == "__main__":
    triage_and_dispatch("Database connection timeout on pool primary-replica-02.")

Production Checklist: Engineering High-Throughput Routing

Primary Citations & Technical References

• OpenAI. (2026). Decisions API & Luna Model Technical Specifications: Constrained Decoding and Low-Latency Classification. OpenAI Platform Documentation.
• OpenAI. (September 29, 2026). Prompt Caching Architecture and Developer Economics at Scale. DevDay Engineering Whitepaper.
• Kwon, W., et al. (2023–2026). PagedAttention and Efficient Memory Management for Large Language Model Serving. Proceedings of SOSP & ACM SIGOPS.
• NVIDIA Corporation. (2026). Optimizing KV-Cache Memory Access and TensorRT-LLM Throughput on Blackwell HBM3e Architectures. NVIDIA Technical Whitepaper.