feat: completions API improvements, gemini endpoint, response types
This commit is contained in:
Nikketryhard
@@ -1,464 +1,128 @@
|
||||
# Endpoint Gap Analysis
|
||||
|
||||
> **Generated:** 2026-02-15 (updated)
|
||||
> **Proxy Version:** 3.1.0
|
||||
> **Scope:** All three API endpoints vs official OpenAI / Gemini specifications
|
||||
> **Updated:** 2026-02-15
|
||||
> **Sources:** [OpenAI Chat Completions API](https://platform.openai.com/docs/api-reference/chat/create), [OpenAI Responses API](https://platform.openai.com/docs/api-reference/responses), [Gemini Thinking Mode](https://ai.google.dev/gemini-api/docs/thinking-mode), proxy source code
|
||||
> **Method:** Full source audit cross-referenced against context7 OpenAI API docs
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
## What's Implemented
|
||||
|
||||
- [Endpoint Overview](#endpoint-overview)
|
||||
- [Feature Parity Matrix](#feature-parity-matrix)
|
||||
- [Detailed Endpoint Analysis](#detailed-endpoint-analysis)
|
||||
- [Responses API (`/v1/responses`)](#responses-api-v1responses)
|
||||
- [Chat Completions API (`/v1/chat/completions`)](#chat-completions-api-v1chatcompletions)
|
||||
- [Gemini API (`/v1/gemini`)](#gemini-api-v1gemini)
|
||||
- [Priority Gaps](#priority-gaps)
|
||||
- [Architecture Notes](#architecture-notes)
|
||||
### All Endpoints
|
||||
|
||||
- ✅ Sync + streaming modes
|
||||
- ✅ Model selection + validation
|
||||
- ✅ OAuth auth check
|
||||
- ✅ Timeout control
|
||||
- ✅ Tool definitions, tool choice, tool results (OpenAI → Gemini auto-conversion)
|
||||
- ✅ MITM bypass path for custom tools
|
||||
- ✅ Thinking/reasoning in both sync and streaming
|
||||
- ✅ Generation params forwarded via MITM (`temperature`, `top_p`, `top_k`, `max_output_tokens`, `stop_sequences`, `frequency_penalty`, `presence_penalty`)
|
||||
- ✅ `reasoning_effort` / `thinkingLevel` — forwarded as `generationConfig.thinkingConfig.thinkingLevel`
|
||||
- ✅ `response_format: {type: "json_object"}` — injected as `responseMimeType: "application/json"`
|
||||
- ✅ Google Search grounding — `web_search: true` (Completions), `tools: [{type: "web_search_preview"}]` (Responses), `google_search: true` (Gemini)
|
||||
- ✅ `/v1/search` endpoint — dedicated web search via Google Search grounding, returns structured results + citations
|
||||
|
||||
### Reasoning Effort → Thinking Level Mapping
|
||||
|
||||
| OpenAI `reasoning_effort` | Google `thinkingLevel` | Gemini 3 Pro | Gemini 3 Flash |
|
||||
| :-----------------------: | :--------------------: | :----------: | :------------: |
|
||||
| `"low"` | `"low"` | ✅ | ✅ |
|
||||
| `"medium"` | `"medium"` | ❌ | ✅ |
|
||||
| `"high"` | `"high"` | ✅ (default) | ✅ (default) |
|
||||
| — | `"minimal"` | ❌ | ✅ |
|
||||
|
||||
### Completions-Specific
|
||||
|
||||
- ✅ `stream_options.include_usage` — final chunk with usage before `[DONE]`
|
||||
- ✅ `completion_tokens_details.reasoning_tokens` — thinking token count
|
||||
- ✅ `prompt_tokens_details.cached_tokens` — cache read tokens
|
||||
- ✅ `temperature`, `top_p`, `max_tokens`, `max_completion_tokens`, `frequency_penalty`, `presence_penalty`
|
||||
- ✅ `reasoning_effort`
|
||||
- ✅ `stop` — string or array, forwarded as `generationConfig.stopSequences`
|
||||
- ✅ `response_format: {type: "json_object"}` — injects `responseMimeType`
|
||||
- ✅ `response_format: {type: "json_schema", json_schema: {...}}` — injects `responseMimeType` + `responseSchema` via MITM
|
||||
- ✅ `n` (multiple choices) — fires N parallel cascades, collects into `choices[]` (sync only, capped at 5)
|
||||
- ✅ `conversation` — session ID for multi-turn cascade reuse (custom extension)
|
||||
- ✅ `reasoning_content` — thinking text in assistant message
|
||||
- ✅ `system_fingerprint` — `fp_<version>` in sync + all streaming chunks
|
||||
- ✅ `service_tier` — `"default"` in sync + all streaming chunks
|
||||
- ✅ `logprobs: null` — in every choice (sync + streaming)
|
||||
- ✅ `metadata` — accepted in request, ignored
|
||||
- ✅ `finish_reason` — correctly maps Google's `MAX_TOKENS`→`"length"`, `SAFETY`→`"content_filter"`, etc.
|
||||
- ✅ Full `messages[]` history — all user, assistant, system, tool messages forwarded
|
||||
|
||||
### Responses-Specific
|
||||
|
||||
- ✅ Full streaming event set (all `response.*` events including reasoning summary)
|
||||
- ✅ `temperature`, `top_p`, `max_output_tokens`
|
||||
- ✅ `reasoning_effort` — echoed from client request
|
||||
- ✅ `thinking_signature` for multi-turn thinking chains
|
||||
- ✅ `instructions`, `metadata`, `user` — echoed in response
|
||||
- ✅ Usage with MITM-intercepted real tokens
|
||||
- ✅ `max_tool_calls` — limits tool calls returned per response
|
||||
- ✅ `conversation` — session reuse
|
||||
- ✅ `previous_response_id`, `store`, `parallel_tool_calls`, `truncation`, `text.format`, `tool_choice` — echoed
|
||||
- ✅ `tools` — echoed from client request (was previously always `[]`)
|
||||
- ✅ `text.format` — `{format: {type: "json_schema", ...}}` injects `responseMimeType` + `responseSchema` via MITM, echoed in response
|
||||
|
||||
### Gemini-Specific
|
||||
|
||||
- ✅ Native tool format (no conversion needed)
|
||||
- ✅ `usageMetadata` in sync **and streaming** responses
|
||||
- ✅ `temperature`, `topP`, `topK`, `maxOutputTokens`, `stopSequences`
|
||||
- ✅ `thinkingLevel`
|
||||
- ✅ Session/conversation reuse
|
||||
- ✅ Array/multipart `input` — strings, string arrays, `{text: "..."}` object arrays
|
||||
|
||||
---
|
||||
|
||||
## Endpoint Overview
|
||||
## Fixed Bugs
|
||||
|
||||
The proxy exposes three main API endpoints, each serving different client ecosystems:
|
||||
| # | Bug | Fix |
|
||||
| --- | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| B1 | Messages history dropped | `extract_chat_input` now calls `build_conversation_with_tools` with ALL messages — full multi-turn via `messages[]` works. |
|
||||
| B2 | `finish_reason` never `"length"` | `google_to_openai_finish_reason()` helper maps `MAX_TOKENS`→`"length"`, `SAFETY`/`RECITATION`/etc→`"content_filter"`. Applied to all paths. |
|
||||
| B3 | `reasoning` always null | `build_response_object` now echoes client's `reasoning_effort` from `RequestParams`. |
|
||||
| B4 | `tool_choice` always `"auto"` | Changed from `&'static str` to `serde_json::Value`. Echoes whatever the client sent. |
|
||||
| B5 | `tools` always `[]` | Echoes the client's tools array in the response. |
|
||||
| B7 | `temperature`/`top_p` wrong | Already defaults to `1.0` via `unwrap_or(1.0)`. Was a false positive — no fix needed. |
|
||||
|
||||
| Endpoint | Protocol | Primary Clients | Spec Reference |
|
||||
| --------------------------- | --------------------------- | ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
|
||||
| `POST /v1/responses` | OpenAI Responses API | Claude Code, Antigravity-native clients | [platform.openai.com/docs/api-reference/responses](https://platform.openai.com/docs/api-reference/responses) |
|
||||
| `POST /v1/chat/completions` | OpenAI Chat Completions API | OpenCode, Vercel AI SDK, any OpenAI-compatible client | [platform.openai.com/docs/api-reference/chat](https://platform.openai.com/docs/api-reference/chat/create) |
|
||||
| `POST /v1/gemini` | Custom Gemini-native API | Direct Gemini-format consumers | [ai.google.dev/api](https://ai.google.dev/api) (loosely based) |
|
||||
### Acceptable / Won't Fix
|
||||
|
||||
All three endpoints share the same backend pipeline:
|
||||
|
||||
```
|
||||
Client Request → Proxy Endpoint → LS (Language Server) → Google API
|
||||
↓
|
||||
MITM Proxy (captures real usage + injects generation params + tool calls)
|
||||
```
|
||||
| # | Bug | Status |
|
||||
| --- | ----------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
|
||||
| B6 | `Usage::estimate` fake tokens as fallback | Only triggers on timeout/error paths. Heuristic `len/4` is reasonable for timeouts where output tokens = 0. |
|
||||
|
||||
---
|
||||
|
||||
## Feature Parity Matrix
|
||||
## TODO — New Features
|
||||
|
||||
### Core Features
|
||||
### Trivial (all done ✅)
|
||||
|
||||
| Feature | Responses | Completions | Gemini |
|
||||
| -------------------- | :-------: | :---------: | :----: |
|
||||
| Sync mode | ✅ | ✅ | ✅ |
|
||||
| Streaming mode (SSE) | ✅ | ✅ | ✅ |
|
||||
| Model selection | ✅ | ✅ | ✅ |
|
||||
| Model validation | ✅ | ✅ | ✅ |
|
||||
| Auth check (OAuth) | ✅ | ✅ | ✅ |
|
||||
| Timeout control | ✅ | ✅ | ✅ |
|
||||
All trivial response shape fixes have been implemented.
|
||||
|
||||
### Generation Parameters (MITM-injected)
|
||||
### Medium (schema injection via MITM) — all done ✅
|
||||
|
||||
| Feature | Responses | Completions | Gemini |
|
||||
| ------------------- | :-------: | :---------: | :----: |
|
||||
| `temperature` | ✅ | ✅ | ✅ |
|
||||
| `top_p` / `topP` | ✅ | ✅ | ✅ |
|
||||
| `top_k` / `topK` | ❌ | ❌ | ✅ |
|
||||
| `max_output_tokens` | ✅ | ✅ | ✅ |
|
||||
| `stop_sequences` | ❌ | ❌ | ✅ |
|
||||
| `frequency_penalty` | ❌ | ✅ | ❌ |
|
||||
| `presence_penalty` | ❌ | ✅ | ❌ |
|
||||
All structured output features have been implemented.
|
||||
|
||||
> **Note:** All generation parameters are forwarded to Google's API via MITM injection into `request.generationConfig`. They override the LS defaults.
|
||||
### Hard (new features)
|
||||
|
||||
### Thinking / Reasoning
|
||||
| # | Gap | API | Notes |
|
||||
| --- | ------------------------- | ---- | ---------------------------------------------------------- |
|
||||
| 7 | **`parallel_tool_calls`** | Both | Accept param, echo in response. Can't enforce server-side. |
|
||||
|
||||
| Feature | Responses | Completions | Gemini |
|
||||
| ---------------------------------- | :-------------------------------: | :-------------------------------: | :---------------------: |
|
||||
| Thinking — LS path (streaming) | ✅ `reasoning_summary_text.delta` | ✅ `reasoning_content` delta | ✅ `thought: true` part |
|
||||
| Thinking — LS path (sync) | ✅ `reasoning` output item | ✅ `reasoning_content` in message | ✅ `thought: true` part |
|
||||
| Thinking — Bypass path (streaming) | ✅ | ✅ | ✅ |
|
||||
| Thinking — Bypass path (sync) | ✅ | ✅ | ✅ |
|
||||
| Thinking signature (multi-turn) | ✅ `thinking_signature` field | ❌ Not applicable | ❌ Not applicable |
|
||||
### Stretch (research needed)
|
||||
|
||||
### Tool Calls
|
||||
|
||||
| Feature | Responses | Completions | Gemini |
|
||||
| ---------------------------- | :-----------------------------: | :------------------------: | :-------------------------------------: |
|
||||
| Tool definitions input | ✅ OpenAI format → Gemini | ✅ OpenAI format → Gemini | ✅ Native Gemini format |
|
||||
| Tool choice control | ✅ `tool_choice` | ✅ `tool_choice` | ✅ `tool_config` |
|
||||
| Tool call output (streaming) | ✅ `function_call` items | ✅ `tool_calls` in delta | ✅ `functionCall` parts |
|
||||
| Tool call output (sync) | ✅ `function_call` items | ✅ `tool_calls` in message | ✅ `functionCall` parts |
|
||||
| Tool result input | ✅ `function_call_output` items | ✅ `tool` role messages | ✅ `functionResponse` in `tool_results` |
|
||||
| MITM bypass (custom tools) | ✅ | ✅ | ✅ |
|
||||
| Stale state protection | ✅ | ✅ | ✅ |
|
||||
|
||||
### Session Management
|
||||
|
||||
| Feature | Responses | Completions | Gemini |
|
||||
| ------------------------------------ | :---------------------: | :--------------: | :---------------------: |
|
||||
| Session/conversation reuse | ✅ `conversation` field | ❌ Not supported | ✅ `conversation` field |
|
||||
| Session listing (`GET /v1/sessions`) | ✅ Shared | ✅ Shared | ✅ Shared |
|
||||
| Session deletion | ✅ Shared | ✅ Shared | ✅ Shared |
|
||||
|
||||
### Usage / Token Tracking
|
||||
|
||||
| Feature | Responses | Completions | Gemini |
|
||||
| -------------------------------- | :---------------------------: | :-------------------------------: | :--------------------------: |
|
||||
| Usage in sync response | ✅ MITM real tokens | ✅ MITM real tokens | ✅ `usageMetadata` |
|
||||
| Usage in streaming (final chunk) | ❌ Not emitted | ✅ `stream_options.include_usage` | ❌ Not emitted |
|
||||
| `reasoning_tokens` in usage | ✅ In `output_tokens_details` | ✅ In `completion_tokens_details` | ✅ `thoughtsTokenCount` |
|
||||
| Cache tokens | ✅ `cached_tokens` | ✅ `cached_tokens` | ✅ `cachedContentTokenCount` |
|
||||
| # | Gap | API | Notes |
|
||||
| --- | -------------------------- | ---- | ---------------------------------------------------------------------------------------------------------------------------- |
|
||||
| 12 | **Image/audio modalities** | Both | LS `sendMessage` is text-only. Need to reverse-engineer proto format for binary payloads. Gemini 3 supports vision natively. |
|
||||
|
||||
---
|
||||
|
||||
## Detailed Endpoint Analysis
|
||||
## Won't Implement
|
||||
|
||||
### Responses API (`/v1/responses`)
|
||||
|
||||
**Spec:** [OpenAI Responses API](https://platform.openai.com/docs/api-reference/responses)
|
||||
|
||||
#### Request Fields
|
||||
|
||||
| Field | Spec | Status | Implementation Details |
|
||||
| ---------------------------- | ------------- | :----: | -------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `model` | Required | ✅ | Mapped to internal model enum via `lookup_model()` |
|
||||
| `input` | Required | ✅ | String or array. Array supports `message` items and `function_call_output` items |
|
||||
| `instructions` | Optional | ✅ | Prepended to user text as system instructions |
|
||||
| `stream` | Optional | ✅ | SSE stream with `response.*` events |
|
||||
| `tools` | Optional | ✅ | OpenAI function format → auto-converted to Gemini `functionDeclarations` via `openai_tools_to_gemini()` |
|
||||
| `tool_choice` | Optional | ✅ | `"auto"`, `"required"`, `"none"`, or `{"type":"function","function":{"name":"X"}}` → converted to Gemini `functionCallingConfig` |
|
||||
| `store` | Optional | ✅ | Accepted, echoed in response. Not actually persisted. |
|
||||
| `temperature` | Optional | ✅ | **Forwarded** to Google via MITM `generationConfig` injection. |
|
||||
| `top_p` | Optional | ✅ | **Forwarded** to Google via MITM. |
|
||||
| `max_output_tokens` | Optional | ✅ | **Forwarded** to Google via MITM. |
|
||||
| `previous_response_id` | Optional | ✅ | Accepted, echoed. Not used for chaining (use `conversation` instead). |
|
||||
| `metadata` | Optional | ✅ | Accepted, echoed back in response. |
|
||||
| `user` | Optional | ✅ | Accepted, echoed. |
|
||||
| `conversation` | **Extension** | ✅ | Proxy-specific: session ID for multi-turn cascade reuse. |
|
||||
| `timeout` | **Extension** | ✅ | Proxy-specific: request timeout in seconds (default 120). |
|
||||
| `reasoning.effort` | Optional | ❌ | Could map to model variant selection (e.g., `"high"` → Opus, `"low"` → Flash). |
|
||||
| `reasoning.generate_summary` | Optional | ❌ | Not implemented. Could control thinking output inclusion. |
|
||||
| `truncation` | Optional | ❌ | Not applicable — LS manages context window. |
|
||||
| `parallel_tool_calls` | Optional | ✅ | Hardcoded `true` in response. |
|
||||
|
||||
#### Response Object
|
||||
|
||||
| Field | Spec | Status | Notes |
|
||||
| ---------------------- | ------------- | :----: | ---------------------------------------------------------------- |
|
||||
| `id` | Required | ✅ | `resp_` + UUID |
|
||||
| `object` | Required | ✅ | Always `"response"` |
|
||||
| `created_at` | Required | ✅ | Unix timestamp |
|
||||
| `status` | Required | ✅ | `"completed"` or `"incomplete"` |
|
||||
| `completed_at` | Required | ✅ | Unix timestamp or null |
|
||||
| `error` | Required | ✅ | null on success |
|
||||
| `incomplete_details` | Required | ✅ | null |
|
||||
| `instructions` | Required | ✅ | Echoed from request |
|
||||
| `max_output_tokens` | Required | ✅ | Echoed or null |
|
||||
| `model` | Required | ✅ | Model name string |
|
||||
| `output` | Required | ✅ | Array of `reasoning` and/or `message` items |
|
||||
| `parallel_tool_calls` | Required | ✅ | `true` |
|
||||
| `previous_response_id` | Required | ✅ | Echoed or null |
|
||||
| `reasoning` | Required | ✅ | `{effort: null, summary: null}` |
|
||||
| `store` | Required | ✅ | Echoed |
|
||||
| `temperature` | Required | ✅ | Echoed (default 1.0) |
|
||||
| `text` | Required | ✅ | `{format: {type: "text"}}` |
|
||||
| `tool_choice` | Required | ✅ | `"auto"` |
|
||||
| `tools` | Required | ✅ | Echoed or `[]` |
|
||||
| `top_p` | Required | ✅ | Echoed (default 1.0) |
|
||||
| `truncation` | Required | ✅ | `"disabled"` |
|
||||
| `usage` | Required | ✅ | MITM-intercepted real tokens when available, estimated otherwise |
|
||||
| `user` | Required | ✅ | Echoed or null |
|
||||
| `metadata` | Required | ✅ | Echoed or `{}` |
|
||||
| `thinking_signature` | **Extension** | ✅ | Proxy-specific: opaque blob for multi-turn thinking chain |
|
||||
|
||||
#### Streaming Events
|
||||
|
||||
| Event | Spec | Status | Notes |
|
||||
| ---------------------------------------- | --------- | :----: | ------------------------------ |
|
||||
| `response.created` | Required | ✅ | Initial response shell |
|
||||
| `response.in_progress` | Required | ✅ | |
|
||||
| `response.output_item.added` | Required | ✅ | For reasoning + message items |
|
||||
| `response.content_part.added` | Required | ✅ | |
|
||||
| `response.output_text.delta` | Required | ✅ | Progressive text deltas |
|
||||
| `response.output_text.done` | Required | ✅ | |
|
||||
| `response.content_part.done` | Required | ✅ | |
|
||||
| `response.output_item.done` | Required | ✅ | |
|
||||
| `response.completed` | Required | ✅ | Final event with full response |
|
||||
| `response.reasoning_summary_text.delta` | Required | ✅ | Progressive thinking deltas |
|
||||
| `response.reasoning_summary_text.done` | Required | ✅ | |
|
||||
| `response.function_call_arguments.delta` | For tools | ✅ | Tool call argument streaming |
|
||||
| `response.function_call_arguments.done` | For tools | ✅ | |
|
||||
|
||||
---
|
||||
|
||||
### Chat Completions API (`/v1/chat/completions`)
|
||||
|
||||
**Spec:** [OpenAI Chat Completions API](https://platform.openai.com/docs/api-reference/chat/create)
|
||||
|
||||
#### Request Fields
|
||||
|
||||
| Field | Spec | Status | Implementation Details |
|
||||
| ------------------------- | ------------- | :----: | -------------------------------------------------------------------- |
|
||||
| `model` | Required | ✅ | Mapped to internal model enum |
|
||||
| `messages` | Required | ✅ | Supports `system`, `developer`, `user`, `assistant`, `tool` roles |
|
||||
| `messages[].content` | Required | ✅ | String or array of `{type: "text", text: "..."}` objects |
|
||||
| `messages[].tool_calls` | Optional | ✅ | For assistant messages with tool calls |
|
||||
| `messages[].tool_call_id` | Optional | ✅ | For tool result messages |
|
||||
| `stream` | Optional | ✅ | SSE with `chat.completion.chunk` events |
|
||||
| `stream_options` | Optional | ✅ | `include_usage: true` emits final usage chunk before `[DONE]` |
|
||||
| `tools` | Optional | ✅ | OpenAI function format → auto-converted to Gemini |
|
||||
| `tool_choice` | Optional | ✅ | `"auto"`, `"none"`, `"required"`, or specific function |
|
||||
| `timeout` | **Extension** | ✅ | Proxy-specific (default 120s) |
|
||||
| `temperature` | Optional | ✅ | **Forwarded** to Google via MITM `generationConfig.temperature` |
|
||||
| `top_p` | Optional | ✅ | **Forwarded** to Google via MITM `generationConfig.topP` |
|
||||
| `max_tokens` | Optional | ✅ | **Forwarded** to Google via MITM `generationConfig.maxOutputTokens` |
|
||||
| `max_completion_tokens` | Optional | ✅ | **Forwarded** (same as `max_tokens`, newer OpenAI param) |
|
||||
| `frequency_penalty` | Optional | ✅ | **Forwarded** to Google via MITM `generationConfig.frequencyPenalty` |
|
||||
| `presence_penalty` | Optional | ✅ | **Forwarded** to Google via MITM `generationConfig.presencePenalty` |
|
||||
| `user` | Optional | ✅ | Accepted, not used |
|
||||
| `n` | Optional | ❌ | N/A — single generation only |
|
||||
| `logprobs` | Optional | ❌ | N/A |
|
||||
| `top_logprobs` | Optional | ❌ | N/A |
|
||||
| `logit_bias` | Optional | ❌ | N/A |
|
||||
| `response_format` | Optional | ❌ | Could be useful for JSON mode |
|
||||
| `seed` | Optional | ❌ | N/A |
|
||||
| `stop` | Optional | ❌ | Could be forwarded as `stopSequences` |
|
||||
|
||||
#### Sync Response Object
|
||||
|
||||
| Field | Spec | Status | Notes |
|
||||
| ------------------------------------------------------------ | ----------- | :----: | -------------------------------------------- |
|
||||
| `id` | Required | ✅ | `chatcmpl-` + UUID |
|
||||
| `object` | Required | ✅ | `"chat.completion"` |
|
||||
| `created` | Required | ✅ | Unix timestamp |
|
||||
| `model` | Required | ✅ | Model name |
|
||||
| `choices[0].index` | Required | ✅ | `0` |
|
||||
| `choices[0].message.role` | Required | ✅ | `"assistant"` |
|
||||
| `choices[0].message.content` | Required | ✅ | Response text |
|
||||
| `choices[0].message.reasoning_content` | Extension | ✅ | Thinking text (when model produces thinking) |
|
||||
| `choices[0].message.tool_calls` | Conditional | ✅ | When model returns tool calls |
|
||||
| `choices[0].message.refusal` | Optional | ❌ | Not implemented |
|
||||
| `choices[0].message.annotations` | Optional | ❌ | Not implemented |
|
||||
| `choices[0].logprobs` | Optional | ❌ | Not implemented |
|
||||
| `choices[0].finish_reason` | Required | ✅ | `"stop"` or `"tool_calls"` |
|
||||
| `usage.prompt_tokens` | Required | ✅ | MITM real or estimated |
|
||||
| `usage.completion_tokens` | Required | ✅ | MITM real or estimated |
|
||||
| `usage.total_tokens` | Required | ✅ | Sum |
|
||||
| `usage.prompt_tokens_details.cached_tokens` | Optional | ✅ | MITM cache read tokens |
|
||||
| `usage.completion_tokens_details.reasoning_tokens` | Optional | ✅ | MITM thinking token count |
|
||||
| `usage.completion_tokens_details.accepted_prediction_tokens` | Optional | ❌ | N/A |
|
||||
| `usage.completion_tokens_details.rejected_prediction_tokens` | Optional | ❌ | N/A |
|
||||
| `system_fingerprint` | Deprecated | ❌ | Cosmetic, not needed |
|
||||
| `service_tier` | Optional | ❌ | Cosmetic, not needed |
|
||||
|
||||
#### Streaming Chunk Object
|
||||
|
||||
| Field | Spec | Status | Notes |
|
||||
| ------------------------------------ | --------------- | :----: | ----------------------------------------------------- |
|
||||
| `id` | Required | ✅ | Same across all chunks |
|
||||
| `object` | Required | ✅ | `"chat.completion.chunk"` |
|
||||
| `created` | Required | ✅ | Same across all chunks |
|
||||
| `model` | Required | ✅ | |
|
||||
| `choices[0].index` | Required | ✅ | `0` |
|
||||
| `choices[0].delta.role` | First chunk | ✅ | `"assistant"` in first chunk |
|
||||
| `choices[0].delta.content` | Text chunks | ✅ | Progressive text deltas |
|
||||
| `choices[0].delta.reasoning_content` | Thinking chunks | ✅ | Progressive thinking deltas |
|
||||
| `choices[0].delta.tool_calls` | Tool chunks | ✅ | Tool call data |
|
||||
| `choices[0].delta` | Final chunk | ✅ | Empty `{}` |
|
||||
| `choices[0].finish_reason` | Final chunk | ✅ | `"stop"` or `"tool_calls"` |
|
||||
| `choices[0].logprobs` | Optional | ❌ | Not implemented |
|
||||
| `usage` (final chunk) | Optional | ✅ | Emitted when `stream_options.include_usage` is `true` |
|
||||
| `data: [DONE]` | Required | ✅ | Stream termination signal |
|
||||
|
||||
---
|
||||
|
||||
### Gemini API (`/v1/gemini`)
|
||||
|
||||
**Spec:** Custom endpoint loosely based on [Gemini REST API](https://ai.google.dev/api/generate-content)
|
||||
|
||||
> **Note:** This is NOT a 1:1 Gemini API replica. It's a simplified proxy-native endpoint that uses Gemini's `functionDeclarations` / `functionCall` / `functionResponse` format directly, avoiding OpenAI ↔ Gemini format conversion overhead.
|
||||
|
||||
#### Request Fields
|
||||
|
||||
| Field | Spec | Status | Implementation Details |
|
||||
| --------------------------------------- | -------- | :----: | --------------------------------------------------------------- |
|
||||
| `model` | Required | ✅ | Mapped to internal model enum |
|
||||
| `input` | Required | ✅ | String only (no array/multipart) |
|
||||
| `tools` | Optional | ✅ | Native Gemini `[{functionDeclarations: [...]}]` format |
|
||||
| `tool_config` | Optional | ✅ | Native Gemini `{functionCallingConfig: {mode: "AUTO"}}` |
|
||||
| `tool_results` | Optional | ✅ | Array of `{functionResponse: {name, response}}` |
|
||||
| `conversation` | Optional | ✅ | Session ID for cascade reuse |
|
||||
| `stream` | Optional | ✅ | SSE streaming |
|
||||
| `timeout` | Optional | ✅ | Default 120s |
|
||||
| `temperature` | Optional | ✅ | **Forwarded** to Google via MITM `generationConfig.temperature` |
|
||||
| `top_p` / `topP` | Optional | ✅ | **Forwarded** to Google via MITM `generationConfig.topP` |
|
||||
| `top_k` / `topK` | Optional | ✅ | **Forwarded** to Google via MITM `generationConfig.topK` |
|
||||
| `max_output_tokens` / `maxOutputTokens` | Optional | ✅ | **Forwarded** via MITM `generationConfig.maxOutputTokens` |
|
||||
| `stop_sequences` / `stopSequences` | Optional | ✅ | **Forwarded** via MITM `generationConfig.stopSequences` |
|
||||
|
||||
#### Sync Response Object
|
||||
|
||||
| Field | Spec | Status | Notes |
|
||||
| -------------------------------------------- | --------- | :----: | -------------------------------- |
|
||||
| `candidates[0].content.parts` | Required | ✅ | Array of text/functionCall parts |
|
||||
| `candidates[0].content.parts[].text` | Text | ✅ | Response text |
|
||||
| `candidates[0].content.parts[].thought` | Extension | ✅ | `true` for thinking parts |
|
||||
| `candidates[0].content.parts[].functionCall` | Tool call | ✅ | `{name, args}` |
|
||||
| `candidates[0].content.role` | Required | ✅ | `"model"` |
|
||||
| `candidates[0].finishReason` | Required | ✅ | `"STOP"` |
|
||||
| `modelVersion` | Required | ✅ | Model name string |
|
||||
| `usageMetadata` | Optional | ✅ | MITM-intercepted token counts |
|
||||
|
||||
`usageMetadata` fields:
|
||||
|
||||
| Field | Status | Notes |
|
||||
| ------------------------- | :----: | ------------------------- |
|
||||
| `promptTokenCount` | ✅ | Input tokens |
|
||||
| `candidatesTokenCount` | ✅ | Output tokens |
|
||||
| `totalTokenCount` | ✅ | Input + output |
|
||||
| `thoughtsTokenCount` | ✅ | Thinking/reasoning tokens |
|
||||
| `cachedContentTokenCount` | ✅ | Cache read tokens |
|
||||
|
||||
#### Streaming Format
|
||||
|
||||
Each SSE `data:` chunk is a complete Gemini-format JSON object with progressive `candidates[0].content.parts`:
|
||||
|
||||
```
|
||||
data: {"candidates":[{"content":{"parts":[{"text":"thinking...","thought":true}],"role":"model"}}],"modelVersion":"opus-4.6"}
|
||||
|
||||
data: {"candidates":[{"content":{"parts":[{"text":"Hello!"}],"role":"model"}}],"modelVersion":"opus-4.6"}
|
||||
|
||||
data: {"candidates":[{"content":{"parts":[{"text":""}],"role":"model"},"finishReason":"STOP"}],"modelVersion":"opus-4.6"}
|
||||
|
||||
data: [DONE]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Priority Gaps
|
||||
|
||||
### 🔴 High Priority — RESOLVED ✅
|
||||
|
||||
All high-priority gaps have been addressed:
|
||||
|
||||
1. ~~**Completions: `stream_options.include_usage`**~~ → ✅ Implemented
|
||||
2. ~~**Completions: `completion_tokens_details.reasoning_tokens`**~~ → ✅ Implemented
|
||||
3. ~~**Completions: Accept `temperature`, `top_p`, `max_tokens`**~~ → ✅ Forwarded via MITM
|
||||
4. ~~**Gemini: `usageMetadata`**~~ → ✅ Implemented
|
||||
|
||||
### 🟡 Medium Priority
|
||||
|
||||
5. **Responses: `reasoning.effort`**
|
||||
- **What:** Map reasoning effort levels (`"high"`, `"medium"`, `"low"`) to model variant selection
|
||||
- **Why:** Could automatically select Opus vs Flash based on reasoning needs
|
||||
- **Effort:** Medium — needs model selection logic changes
|
||||
|
||||
6. **Completions: Session/conversation support**
|
||||
- **What:** Add session reuse similar to Responses and Gemini endpoints
|
||||
- **Why:** Would allow multi-turn conversations via the completions API
|
||||
- **Effort:** Medium — need a way to pass session ID (maybe via `user` field or custom header)
|
||||
|
||||
7. **Completions: `stop` sequences**
|
||||
- **What:** Forward `stop` to Google as `stopSequences` in `generationConfig`
|
||||
- **Why:** Some clients use stop sequences to control generation
|
||||
- **Effort:** Trivial — just add to `CompletionRequest` and `GenerationParams`
|
||||
|
||||
8. **Completions: `response_format` (JSON mode)**
|
||||
- **What:** Forward `response_format: {"type": "json_object"}` to Google's `responseMimeType`
|
||||
- **Why:** Useful for structured output
|
||||
- **Effort:** Low — inject `responseMimeType: "application/json"` in generationConfig
|
||||
|
||||
### 🟢 Low Priority
|
||||
|
||||
Cosmetic or not applicable to our architecture:
|
||||
|
||||
9. **`system_fingerprint`** — OpenAI-specific field, meaningless for our proxy
|
||||
10. **`service_tier`** — OpenAI billing concept, not applicable
|
||||
11. **`n` > 1** — Multiple completions per request; our backend only generates one
|
||||
12. **`logprobs`** — Would require token-level access we don't have
|
||||
13. **`seed`** — Deterministic sampling not controllable through our proxy
|
||||
|
||||
---
|
||||
|
||||
## Architecture Notes
|
||||
|
||||
### Generation Parameter Injection
|
||||
|
||||
Client-specified sampling parameters are forwarded to Google's API via the MITM request modification pipeline:
|
||||
|
||||
```
|
||||
Client sends temperature=0.5 → API handler stores in MitmStore.generation_params
|
||||
↓
|
||||
LS sends request to Google API
|
||||
↓
|
||||
MITM intercepts request
|
||||
↓
|
||||
modify_request() reads generation_params
|
||||
↓
|
||||
Injects into request.generationConfig:
|
||||
temperature, topP, topK, maxOutputTokens,
|
||||
stopSequences, frequencyPenalty, presencePenalty
|
||||
↓
|
||||
Forwards modified request to Google
|
||||
```
|
||||
|
||||
This approach overrides whatever defaults the LS sets, giving clients direct control over sampling parameters.
|
||||
|
||||
### Dual Path Architecture
|
||||
|
||||
All three endpoints share a dual-path architecture:
|
||||
|
||||
```
|
||||
┌─────────────────┐
|
||||
│ Has custom │
|
||||
Request ────────────► │ tools? │
|
||||
└────────┬────────┘
|
||||
│
|
||||
┌──── Yes ──┴── No ────┐
|
||||
│ │
|
||||
┌─────▼─────┐ ┌─────▼─────┐
|
||||
│ MITM │ │ LS Steps │
|
||||
│ Bypass │ │ Polling │
|
||||
│ Path │ │ Path │
|
||||
└─────┬─────┘ └─────┬─────┘
|
||||
│ │
|
||||
┌─────▼─────┐ ┌─────▼─────┐
|
||||
│ Poll │ │ Poll │
|
||||
│ MitmStore │ │ get_steps │
|
||||
│ directly │ │ from LS │
|
||||
└─────┬─────┘ └─────┬─────┘
|
||||
│ │
|
||||
└──────────┬────────────┘
|
||||
│
|
||||
┌─────▼─────┐
|
||||
│ Response │
|
||||
│ to client │
|
||||
└───────────┘
|
||||
```
|
||||
|
||||
- **Bypass Path:** When custom tools are present, the handler polls `MitmStore` directly for response text, thinking text, and function calls. The MITM proxy captures these from the Google API response before the LS processes them.
|
||||
|
||||
- **LS Path:** When no custom tools are present, the handler polls the LS's `get_steps` API for progressive response data (text, thinking, status).
|
||||
|
||||
### Stale State Protection
|
||||
|
||||
All bypass paths include protection against stale `response_complete` flags from previous requests:
|
||||
|
||||
```rust
|
||||
if complete && text.is_empty() && thinking.is_none() {
|
||||
warn!("stale response_complete detected — clearing");
|
||||
state.mitm_store.clear_response_async().await;
|
||||
continue; // or retry
|
||||
}
|
||||
```
|
||||
|
||||
This handles the race condition where a previous request's MITM handler calls `mark_response_complete()` after the new request has already called `clear_response_async()`.
|
||||
|
||||
### Tool Format Conversion
|
||||
|
||||
```
|
||||
OpenAI tools ──► openai_tools_to_gemini() ──► Gemini functionDeclarations
|
||||
│
|
||||
MitmStore.set_tools()
|
||||
│
|
||||
MITM proxy injects into
|
||||
outgoing LS request
|
||||
```
|
||||
|
||||
The Gemini endpoint skips this conversion entirely — tools are stored in native Gemini format.
|
||||
| # | Gap | Reason |
|
||||
| --- | ------------------------------- | ------------------------------------------------------------------------ |
|
||||
| 9 | `prediction` (Predicted Output) | Inference-level speculative decoding optimization. No Gemini equivalent. |
|
||||
| 10 | `logprobs` / `top_logprobs` | Gemini never exposes token-level log probabilities. |
|
||||
|
||||
Reference in New Issue
Block a user