Capabilities
Multimodal Input
Sending images, documents, audio, and video as user content
Read this when
- Building a feature that accepts non-text user content
- Wiring a new input modality on a provider adapter
ai.Content defines the input variants pi-go can carry. Today: Text, Image (base64 + MIME), File (Data/URL/FileID + filename + MIME). No audio or video variants exist.
Images
| Provider | API | pi-go | Notes |
|---|---|---|---|
| Anthropic | ✅ jpeg/png/gif/webp; base64 or URL | ⚠️ base64 only | NewImageBlockBase64 (anthropic.go:71) |
| OpenAI Chat | ✅ base64 or URL with detail | ⚠️ base64 only as data: URL | (convert.go:74-85) |
| OpenAI Responses | ✅ base64 or URL with detail | ⚠️ base64 only | detail: auto (convert.go:66-80) |
| Google Gemini | ✅ InlineData base64; Files URI; media_resolution per image | ✅ base64; URL via File block | (convert.go:66-71) |
| Claude CLI | ✅ via @filepath | ❌ | image content silently dropped |
Provider docs
Documents (PDF, plain text, DOCX, CSV, XLSX, MD)
| Provider | API | pi-go | Notes |
|---|---|---|---|
| Anthropic | ✅ PDF (URL/base64), plain text via DocumentBlock; DOCX/CSV/XLSX/MD via Files API | ⚠️ PDF + plain text only; FileID-only files dropped | NewDocumentBlock (convert.go:88-105); plain text base64-decoded into PlainTextSourceParam (convert.go:92-101) |
| OpenAI Chat | ✅ FileID or inline base64 | ✅ FileID + base64; URL form skipped | (convert.go:107-133) |
| OpenAI Responses | ✅ FileID, URL, or base64 | ✅ all three | (convert.go:102-129) |
| Google Gemini | ✅ Files URI/FileID, or base64 InlineData | ✅ all three | (convert.go:86-117) |
| Claude CLI | ⚠️ via filepath arg | ❌ | not forwarded |
Provider docs
Audio
| Provider | API | pi-go | Notes |
|---|---|---|---|
| Anthropic | ❌ | ❌ | not supported by API |
| OpenAI Chat | ✅ gpt-4o-audio-preview input_audio | ❌ | no Go content type |
| OpenAI Responses | ✅ | ❌ | |
| Google Gemini | ✅ WAV, MP3, AIFF, AAC, OGG, FLAC | ❌ | API supports inline + Files API |
| Gemini Live | ✅ realtime audio | ❌ | see realtime-api.md |
| Claude CLI | ❌ | — |
Provider docs
Video
| Provider | API | pi-go | Notes |
|---|---|---|---|
| Anthropic | ❌ | ❌ | |
| OpenAI Chat | ⚠️ via frame sampling | ❌ | passes frames as images |
| OpenAI Responses | ✅ | ❌ | |
| Google Gemini | ✅ inline or Files API; up to 1h; videoMetadata for offsets/FPS | ❌ | strongest video support |
| Claude CLI | ❌ | — |
Provider docs
pi-go Gaps
- URL images not used by any provider;
ai.Imageonly carriesData. Anthropic, OpenAI, and Gemini all accept URL form natively. - Image detail / media_resolution hint is hard-coded to
auto; not exposed. - Anthropic FileID-only documents are silently dropped (convert.go:86-108).
- OpenAI Chat URL-only files silently dropped (convert.go:108-110).
- No
Audiocontent variant, no audio conversion code anywhere. - No
Videocontent variant; Gemini’svideoMetadata(start/end offset, FPS) cannot be expressed. - Claude CLI drops all non-text content.
- Files API uploads missing — see files-api.md.