AI Orchestrator SDK - Interactive Documentation

What is AI Orchestrator?

A flexible, multi-provider AI SDK for Node.js that dynamically routes requests to different models (OpenAI, Google, etc.) based on cost, latency, or quality. It simplifies using multiple AI services by providing a unified interface, automatic fallbacks, and powerful features like streaming, embeddings, and context-aware chat sessions, giving you full control over your AI operations.

Key Features

Unified API

Access any model from any provider through a single, consistent set of methods.

Intelligent Routing

Automatically select the best model based on tiered strategies like `['cost', 'quality']`.

Full Generation Control

Specify `systemPrompt`, `temperature`, and other parameters for fine-tuned results.

Stateful Chat Sessions

Manage conversational context effortlessly with `sessionId` to reduce token usage.

Real-time Streaming

Stream responses token-by-token for interactive, real-time user experiences.

Text Embeddings

Built-in support for creating vector embeddings, the foundation for RAG and semantic search.

🔄 Automatic Key Rotation

Avoid rate limits with automatic key rotation. Learn more →

Installation

npm install ai-orchestrator

Quick Start

Define your available AI providers and models, create an orchestrator instance, and you're ready to make your first call.

import { AIOrchestrator } from 'ai-orchestrator';

const config = {
  providers: [
    {
      name: 'google',
      apiKey: process.env.GOOGLE_API_KEY,
      models: [
        { id: 'gemini-1.5-flash-latest', type: 'text', cost: 0.00, quality: 'medium' }
      ]
    },
    {
      name: 'openai',
      apiKey: process.env.OPENAI_API_KEY,
      models: [
        { id: 'gpt-4o-mini', type: 'text', cost: 0.15, quality: 'high' }
      ]
    }
  ]
};

const orchestrator = new AIOrchestrator(config);

async function main() {
  // Automatically routes to Google's model based on the default 'cost' strategy
  const response = await orchestrator.generate({
    type: 'text',
    prompt: 'Write a short poem about TypeScript.'
  });

  if (response.status === 'completed') {
    console.log(`Response from ${response.provider} (${response.model}):`);
    console.log(response.data);
  }
}

main();

API Reference

Explore the core methods of the AI Orchestrator SDK. Try the interactive examples to see how they work.

.generate()

The primary method for making a standard, non-streaming request. It intelligently routes the request to the best provider based on your strategy and returns a single, complete response.

const response = await orchestrator.generate({
  type: 'text',
  prompt: 'What are the three most popular dog breeds in Germany?',
  systemPrompt: 'You are a helpful assistant specializing in canine facts.',
  strategy: ['cost', 'quality'] // Prioritize cost, then quality as a tie-breaker
});

Click "Run Example" to see a simulated API response.

.generateStream()

For real-time applications. This method returns an async generator that yields chunks of the response as they become available, perfect for creating a "typing" effect in a UI.

const stream = orchestrator.generateStream({
  type: 'text',
  prompt: 'Write a short, three-sentence story about a robot who discovers music.',
});

for await (const chunk of stream) {
  if (chunk.status === 'streaming') {
    // Append chunk.data to your UI
  }
}

Click "Run Example" to see a simulated stream.

.embedContent()

Transforms text into numerical vector representations (embeddings). This is the core function for building RAG, semantic search, or text clustering applications.

const response = await orchestrator.embedContent({
  texts: [
    'What a beautiful day!',
    'The weather today is very pleasant.'
  ],
  model: 'text-embedding-ada-002' // Specify an embedding model
});

Click "Run Example" to see a simulated embedding response.

.countTokens()

A utility method to calculate how many tokens a piece of text will consume for a specific provider. Essential for managing context window limits and estimating costs before sending a request.

const response = await orchestrator.countTokens({
  text: 'This is a sample sentence.',
  provider: 'openai',
  model: 'gpt-4o-mini'
});

Click "Run Example" to see a simulated token count.

Advanced Configuration

The SDK's power lies in its configuration. Below is a detailed guide to configuring the highly flexible `CustomAdapter` to connect to any RESTful AI API.

Custom Adapter Deep Dive

The `custom` provider type allows you to integrate with self-hosted models, internal APIs, or third-party services that don't have a dedicated adapter. You control every aspect of the API call through the configuration object.

const config = {
  providers: [
    {
      name: 'my-custom-api',
      // The base URL for all API calls
      baseUrl: 'https://api.my-custom-ai.com/v1',
      apiKey: 'your-secret-api-key',
      
      // Optional: A specific endpoint for health checks.
      // The orchestrator will ping this before sending a request.
      healthCheckEndpoint: '/healthz',

      // --- Define Authentication ---
      // Optional: Defaults to 'Authorization'.
      authenticationHeader: 'X-Api-Key', 
      // Optional: A scheme prepended to the key. Defaults to 'Bearer '.
      // Set to '' for keys that don't need a scheme.
      authenticationScheme: '',

      // --- Define Request & Response Formats ---
      // A stringified JSON template for the request body.
      requestBodyTemplate: JSON.stringify({
        model: '{{model}}', // '{{model}}' is a required placeholder
        prompt: '{{prompt}}', // '{{prompt}}' is a required placeholder
        system_prompt: '{{systemPrompt}}', // Optional placeholder
        max_tokens: '{{maxTokens}}', // Optional placeholder
        stream: false
      }),

      // Dot-notation path to extract the final text from the response JSON.
      responseExtractor: 'choices[0].message.content',

      // --- Define Models Served by this Provider ---
      models: [
        { 
          id: 'my-llama3-_8b', 
          type: 'text', 
          cost: 0.0, 
          quality: 'medium',
          supportsStreaming: true 
        }
      ]
    }
  ]
};

Practical Examples

See how to use the orchestrator to build common AI-powered features.

Stateful Chatbot with Context Caching

Use the `caching.sessionId` property to maintain conversational context. The SDK automatically manages the session, so you don't need to re-send the `systemPrompt` on every turn, saving significant token costs.

const sessionId = 'user-123-conversation-abc';

// First turn: establishes the session and caches the system prompt
const response1 = await orchestrator.generate({
  type: 'text',
  prompt: 'What is the capital of France?',
  systemPrompt: 'You are a helpful geography expert.',
  caching: { sessionId }
});

// Second turn: Re-uses the cached system prompt automatically
const response2 = await orchestrator.generate({
  type: 'text',
  prompt: 'And what is its population?',
  caching: { sessionId } // No systemPrompt needed here
});

// Clean up the session from memory when the conversation is over
await orchestrator.endChatSession(sessionId);

Click "Run Example" to see a simulated chat conversation.