LLM Provider Integrations — Octopus Framework

ILLMProvider Interface

Every LLM integration implements this contract:

public interface ILLMProvider
{
    string ProviderName { get; }   // "Anthropic", "OpenAI", etc.

    Task<LLMResponse> CompleteAsync(
        IReadOnlyList<LLMMessage> messages,
        IReadOnlyList<ToolDefinition>? tools,
        LLMOptions options,
        CancellationToken ct = default);

    Task StreamAsync(
        IReadOnlyList<LLMMessage> messages,
        IReadOnlyList<ToolDefinition>? tools,
        LLMOptions options,
        Func<LLMStreamChunk, Task> onChunk,
        CancellationToken ct = default);

    Task<int> CountTokensAsync(
        IReadOnlyList<LLMMessage> messages,
        CancellationToken ct = default);
}

Supported Providers

Provider	Models	Tool Calling	Streaming	Notes
Anthropic Claude	claude-sonnet-4-6, claude-opus-4	Yes	SSE	Primary — longest context window
OpenAI	gpt-4o, gpt-4-turbo	Yes	SSE	Via direct API or SemanticKernel
Azure OpenAI	gpt-4o (deployed)	Yes	SSE	Requires endpoint + deployment name
Ollama (local)	llama3, mistral, etc.	Partial	Yes	For air-gapped / on-prem deployments

Anthropic Claude (Primary)

Anthropic Claude is the primary LLM for Octopus. It is configured directly without the Semantic Kernel plugin:

// appsettings.json (no raw keys — CredentialId references the vault)
{
  "Octopus": {
    "DefaultLLM": {
      "Provider": "Anthropic",
      "Model": "claude-sonnet-4-6",
      "CredentialId": 42,
      "MaxContextTokens": 200000,
      "MaxOutputTokens": 8192,
      "Temperature": 0.3
    }
  }
}

// AnthropicLLMProvider implementation
public class AnthropicLLMProvider : ILLMProvider
{
    public async Task<LLMResponse> CompleteAsync(
        IReadOnlyList<LLMMessage> messages,
        IReadOnlyList<ToolDefinition>? tools,
        LLMOptions options, CancellationToken ct)
    {
        var apiKey = await _credentialResolver.GetPasswordAsync(options.CredentialId);
        var request = BuildAnthropicRequest(messages, tools, options);
        var response = await _httpClient.PostAsync("/v1/messages", request, ct);
        return ParseAnthropicResponse(await response.Content.ReadAsStringAsync(ct));
    }
}

Azure OpenAI Configuration

// Per-agent LLM config for Azure OpenAI
{
  "Provider": "AzureOpenAI",
  "Model": "gpt-4o",
  "CredentialId": 55,          // API key in vault
  "Endpoint": "https://mycompany.openai.azure.com/",
  "DeploymentName": "gpt-4o-prod",
  "MaxContextTokens": 128000,
  "Temperature": 0.2
}

Local Models via Ollama

For air-gapped deployments or cost-sensitive workloads, Octopus supports local LLMs via Ollama. Note that tool calling support varies by model:

{
  "Provider": "Ollama",
  "Model": "llama3:70b",
  "Endpoint": "http://ollama-server:11434",
  "MaxContextTokens": 8192,
  "Temperature": 0.4
  // CredentialId: not required for local Ollama without auth
}

Tool Calling with Local Models

Not all Ollama models support structured tool calling. Models that do not natively support function calling will fall back to prompt-based tool invocation, which is less reliable. Use llama3 or mistral-nemo for best tool calling support with Ollama.

LLMResponse Structure

public class LLMResponse
{
    public string Content { get; set; }         // the text response
    public List<ToolCallDirective> ToolCalls { get; set; } // structured tool calls
    public LLMUsage Usage { get; set; }          // input/output token counts
    public string StopReason { get; set; }       // "end_turn", "tool_use", "max_tokens"
}

public class ToolCallDirective
{
    public string ToolName { get; set; }
    public JsonDocument Arguments { get; set; }
    public string CallId { get; set; }           // for correlating results
}

← UserComposite Next: Flow Studio Integration →