Introduction to Semantic Router

Go Semantic Router is a decision-making layer for your large language models (LLMs) and agents written in pure Go. It routes requests using semantic meaning rather than waiting for slow LLM generations to make tool-use decisions.

What is Semantic Routing?

Semantic routing uses vector embeddings to understand and categorize the meaning of text. Instead of relying on keyword matching or complex rules, it captures the semantic essence of queries and routes them to the appropriate handlers based on similarity in vector space.

Key Features

  • Fast Decision Making: Make routing decisions in milliseconds rather than waiting for LLM generations
  • Multiple Similarity Methods: Support for various similarity metrics including:
    • Dot Product Similarity
    • Euclidean Distance
    • Manhattan Distance
    • Jaccard Similarity
    • Pearson Correlation
  • Pluggable Architecture:
    • Multiple encoder options (Ollama, OpenAI, Google, VoyageAI)
    • Various vector storage backends (in-memory, MongoDB, Valkey)
  • Concurrent Processing: Process multiple routes in parallel for maximum performance

Use Cases

  • Conversational Agents: Route user queries to appropriate response handlers
  • Tool Selection: Determine which tools an agent should use based on query semantics
  • Intent Classification: Classify user intents without relying on slow LLM reasoning
  • Multi-agent Systems: Route requests to specialized agents based on query content

Installation

Install using Go

go get github.com/conneroisu/semanticrouter-go

This will add the package to your Go project.

Install via Nix

If you're using Nix or NixOS, you can use the provided flake:

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    flake-utils.url = "github:numtide/flake-utils";
    flake-utils.inputs.systems.follows = "systems";
    systems.url = "github:nix-systems/default";
    semanticrouter-go.url = "github:conneroisu/semanticrouter-go";
    semanticrouter-go.inputs.nixpkgs.follows = "nixpkgs";
  };

  outputs = { self, semanticrouter-go, nixpkgs, flake-utils, ... }:
    flake-utils.lib.eachSystem [
      "x86_64-linux"
      "i686-linux" 
      "x86_64-darwin"
      "aarch64-linux"
      "aarch64-darwin"
    ] (system: let
      pkgs = import nixpkgs { inherit system; };
    in
      {
        devShells.default = pkgs.mkShell {
          buildInputs = with pkgs; [
            semanticrouter-go.packages.${system}.semanticrouter-go
          ];
        };
      });
}

Building from Source

git clone https://github.com/conneroisu/semanticrouter-go.git
cd semanticrouter-go
go build

Development Shell

The project includes a Nix development shell with all necessary tools:

nix develop

This provides access to commands like:

  • tests - Run all tests
  • format - Format code files
  • lint - Run linting tools

Configuration

Basic Configuration

The Semantic Router can be configured using various options when creating a new router instance:

router, err := semanticrouter.NewRouter(
    []semanticrouter.Route{route1, route2},
    encoder,
    store,
    // Optional configuration options
    semanticrouter.WithSimilarityDotMatrix(1.0),
    semanticrouter.WithWorkers(4),
)

Routes

Routes represent the destinations to which utterances can be matched:

route := semanticrouter.Route{
    Name: "greeting",
    Utterances: []semanticrouter.Utterance{
        {Utterance: "hello there"},
        {Utterance: "hi, how are you?"},
        {Utterance: "good morning"},
    },
}

Encoders

Encoders convert text into vector embeddings. The library supports multiple encoder implementations:

Ollama

import (
    "github.com/conneroisu/semanticrouter-go/encoders/ollama"
    "github.com/ollama/ollama/api"
)

client, _ := api.ClientFromEnvironment()
encoder := &ollama.Encoder{
    Client: client,
    Model: "mxbai-embed-large",
}

OpenAI

import "github.com/conneroisu/semanticrouter-go/encoders/closedai"

encoder := &closedai.Encoder{
    APIKey: "your-api-key",
    Model: "text-embedding-ada-002",
}

Google

import "github.com/conneroisu/semanticrouter-go/encoders/google"

encoder := &google.Encoder{
    APIKey: "your-api-key",
    Model: "textembedding-gecko@latest",
}

VoyageAI

import "github.com/conneroisu/semanticrouter-go/encoders/voyageai"

encoder := &voyageai.Encoder{
    APIKey: "your-api-key",
    Model: "voyage-large-2",
}

Storage

Storage interfaces allow you to persist vector embeddings. The library provides multiple storage options:

In-Memory Storage

import "github.com/conneroisu/semanticrouter-go/stores/memory"

store := memory.NewStore()

MongoDB Storage

import "github.com/conneroisu/semanticrouter-go/stores/mongo"

store, err := mongo.NewStore(ctx, "mongodb://localhost:27017", "database", "collection")

Valkey Storage

import "github.com/conneroisu/semanticrouter-go/stores/valkey"

store, err := valkey.NewStore("localhost:6379", "", 0)

Similarity Methods

The router supports multiple similarity calculation methods that can be used together:

router, err := semanticrouter.NewRouter(
    routes,
    encoder,
    store,
    semanticrouter.WithSimilarityDotMatrix(1.0),     // Cosine similarity (default)
    semanticrouter.WithEuclideanDistance(0.5),       // Euclidean distance
    semanticrouter.WithManhattanDistance(0.3),       // Manhattan/city block distance
    semanticrouter.WithJaccardSimilarity(0.2),       // Jaccard similarity
    semanticrouter.WithPearsonCorrelation(0.1),      // Pearson correlation coefficient
)

Concurrency

You can configure the number of worker goroutines used for computing similarity scores:

semanticrouter.WithWorkers(8) // Use 8 worker goroutines

Basic Usage

Quick Start

Here's a simple example of using the semantic router:

package main

import (
    "context"
    "fmt"

    "github.com/conneroisu/semanticrouter-go"
    "github.com/conneroisu/semanticrouter-go/encoders/ollama"
    "github.com/conneroisu/semanticrouter-go/stores/memory"
    "github.com/ollama/ollama/api"
)

func main() {
    // Create routes
    greetingRoute := semanticrouter.Route{
        Name: "greeting",
        Utterances: []semanticrouter.Utterance{
            {Utterance: "hello there"},
            {Utterance: "hi, how are you?"},
            {Utterance: "good morning"},
        },
    }

    weatherRoute := semanticrouter.Route{
        Name: "weather",
        Utterances: []semanticrouter.Utterance{
            {Utterance: "what's the weather like?"},
            {Utterance: "is it going to rain today?"},
            {Utterance: "temperature forecast"},
        },
    }

    // Create encoder (you need Ollama running locally)
    client, _ := api.ClientFromEnvironment()
    encoder := &ollama.Encoder{
        Client: client,
        Model:  "mxbai-embed-large",
    }

    // Create store
    store := memory.NewStore()

    // Create router
    router, err := semanticrouter.NewRouter(
        []semanticrouter.Route{greetingRoute, weatherRoute},
        encoder,
        store,
    )
    if err != nil {
        panic(err)
    }

    // Match an utterance
    ctx := context.Background()
    bestRoute, score, err := router.Match(ctx, "hi there, how are you doing?")
    if err != nil {
        panic(err)
    }

    fmt.Printf("Best route: %s with score: %.4f\n", bestRoute.Name, score)
}

Understanding the Results

The Match function returns three values:

  1. bestRoute *Route - A pointer to the best matching route
  2. score float64 - The similarity score (higher is better)
  3. err error - An error, if any occurred

The score is calculated based on the similarity methods configured. By default, the router uses cosine similarity (via dot product similarity).

Handling No Match

If no suitable route is found, the router will return an ErrNoRouteFound error:

bestRoute, score, err := router.Match(ctx, "completely unrelated text")
if err != nil {
    if _, ok := err.(semanticrouter.ErrNoRouteFound); ok {
        fmt.Println("No matching route found!")
    } else {
        fmt.Printf("Error: %v\n", err)
    }
}

Multiple Similarity Methods

You can configure the router to use multiple similarity methods:

router, err := semanticrouter.NewRouter(
    routes,
    encoder,
    store,
    semanticrouter.WithSimilarityDotMatrix(0.6),
    semanticrouter.WithEuclideanDistance(0.4),
)

In this example, the final score will be a weighted combination of both similarity methods.

Extending Semantic Router

This section explains how to extend Semantic Router with custom implementations of core interfaces.

Creating a Custom Encoder

To create a custom encoder, implement the Encoder interface:

type Encoder interface {
    Encode(ctx context.Context, utterance string) ([]float64, error)
}

Example implementation using a hypothetical embedding service:

package myencoder

import (
    "context"
    "encoding/json"
    "fmt"
    "net/http"
    "strings"
)

type MyEncoder struct {
    APIKey string
    URL    string
}

func (e *MyEncoder) Encode(ctx context.Context, utterance string) ([]float64, error) {
    req, err := http.NewRequestWithContext(
        ctx, 
        "POST", 
        e.URL,
        strings.NewReader(fmt.Sprintf(`{"text":"%s"}`, utterance)),
    )
    if err != nil {
        return nil, err
    }
    
    req.Header.Add("Authorization", "Bearer "+e.APIKey)
    req.Header.Add("Content-Type", "application/json")
    
    client := &http.Client{}
    resp, err := client.Do(req)
    if err != nil {
        return nil, err
    }
    defer resp.Body.Close()
    
    var result struct {
        Embedding []float64 `json:"embedding"`
    }
    
    if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
        return nil, err
    }
    
    return result.Embedding, nil
}

Creating a Custom Storage Backend

To create a custom storage backend, implement the Store interface:

type Store interface {
    Setter
    Getter
    io.Closer
}

type Setter interface {
    Set(ctx context.Context, keyValPair Utterance) error
}

type Getter interface {
    Get(ctx context.Context, key string) ([]float64, error)
}

Example implementation using SQLite:

package sqlitestore

import (
    "context"
    "database/sql"
    "encoding/json"
    
    "github.com/conneroisu/semanticrouter-go"
    _ "github.com/mattn/go-sqlite3"
)

type Store struct {
    db *sql.DB
}

func NewStore(dbPath string) (*Store, error) {
    db, err := sql.Open("sqlite3", dbPath)
    if err != nil {
        return nil, err
    }
    
    // Create table if it doesn't exist
    _, err = db.Exec(`CREATE TABLE IF NOT EXISTS embeddings 
        (utterance TEXT PRIMARY KEY, embedding BLOB)`)
    if err != nil {
        db.Close()
        return nil, err
    }
    
    return &Store{db: db}, nil
}

func (s *Store) Set(ctx context.Context, utterance semanticrouter.Utterance) error {
    embJSON, err := json.Marshal(utterance.Embed)
    if err != nil {
        return err
    }
    
    _, err = s.db.ExecContext(
        ctx,
        "INSERT OR REPLACE INTO embeddings (utterance, embedding) VALUES (?, ?)",
        utterance.Utterance,
        embJSON,
    )
    return err
}

func (s *Store) Get(ctx context.Context, key string) ([]float64, error) {
    var embJSON []byte
    err := s.db.QueryRowContext(
        ctx,
        "SELECT embedding FROM embeddings WHERE utterance = ?",
        key,
    ).Scan(&embJSON)
    if err != nil {
        return nil, err
    }
    
    var embedding []float64
    if err := json.Unmarshal(embJSON, &embedding); err != nil {
        return nil, err
    }
    
    return embedding, nil
}

func (s *Store) Close() error {
    return s.db.Close()
}

Creating a Custom Similarity Method

You can define a custom similarity function and use it with the router:

package main

import (
    "github.com/conneroisu/semanticrouter-go"
    "gonum.org/v1/gonum/mat"
)

// Define a custom similarity function
func customSimilarity(queryVec *mat.VecDense, indexVec *mat.VecDense) (float64, error) {
    // Implementation of your custom similarity algorithm
    // ...
    return score, nil
}

// Create an option to use this function
func WithCustomSimilarity(coefficient float64) semanticrouter.Option {
    return func(r *semanticrouter.Router) {
        r.AddComparisonFunc(customSimilarity, coefficient)
    }
}

// Use it with your router
router, err := semanticrouter.NewRouter(
    routes,
    encoder,
    store,
    WithCustomSimilarity(1.0),
)

Best Practices

When extending the Semantic Router:

  1. Handle context cancellation appropriately
  2. Implement proper error handling and propagation
  3. Make implementations thread-safe when appropriate
  4. Consider performance implications, especially for high-traffic applications
  5. Add unit tests for your custom implementations

Ollama Encoder

The Ollama encoder uses Ollama to generate text embeddings.

Installation

First, ensure you have Ollama installed and running:

# Install Ollama
curl -fsSL https://ollama.ai/install.sh | sh

# Start Ollama
ollama serve

Then add the Ollama encoder to your Go project:

go get github.com/conneroisu/semanticrouter-go/encoders/ollama

Configuration

import (
    "github.com/conneroisu/semanticrouter-go/encoders/ollama"
    "github.com/ollama/ollama/api"
)

// Create client using environment variables (OLLAMA_HOST and OLLAMA_SCHEMA)
client, err := api.ClientFromEnvironment()
if err != nil {
    // Handle error
}

// Alternatively, create client with specific address
client, err := api.NewClient("http://localhost:11434")
if err != nil {
    // Handle error
}

// Create encoder
encoder := &ollama.Encoder{
    Client: client,
    Model: "mxbai-embed-large", // or another embedding model
}

Supported Models

The Ollama encoder supports any embedding model available in Ollama. Here are some recommended models:

ModelSizeDescription
mxbai-embed-large~1.5GBHigh quality embeddings, excellent performance
nomic-embed-text~270MBGood quality with smaller resource footprint
all-minilm~134MBSmall model with good quality for basic tasks

Pull your desired model before using it:

ollama pull mxbai-embed-large

Example Usage

package main

import (
    "context"
    "fmt"
    "log"

    "github.com/conneroisu/semanticrouter-go/encoders/ollama"
    "github.com/ollama/ollama/api"
)

func main() {
    client, err := api.ClientFromEnvironment()
    if err != nil {
        log.Fatalf("Failed to create Ollama client: %v", err)
    }

    encoder := &ollama.Encoder{
        Client: client,
        Model:  "mxbai-embed-large",
    }

    ctx := context.Background()
    embedding, err := encoder.Encode(ctx, "Hello, world!")
    if err != nil {
        log.Fatalf("Failed to encode text: %v", err)
    }

    fmt.Printf("Embedding has %d dimensions\n", len(embedding))
}

Error Handling

The Ollama encoder may return errors in the following situations:

  • The Ollama service is not running
  • The specified model is not available
  • The context is canceled
  • Network issues occur

Always handle these errors appropriately in production code.

OpenAI Encoder

The OpenAI encoder uses OpenAI's API to generate text embeddings.

Coming soon: Full documentation

Basic Usage

import "github.com/conneroisu/semanticrouter-go/encoders/closedai"

encoder := &closedai.Encoder{
    APIKey: "your-api-key",
    Model: "text-embedding-ada-002",
}

Google Encoder

The Google encoder uses Google's API to generate text embeddings.

Coming soon: Full documentation

Basic Usage

import "github.com/conneroisu/semanticrouter-go/encoders/google"

encoder := &google.Encoder{
    APIKey: "your-api-key",
    Model: "textembedding-gecko@latest",
}

VoyageAI Encoder

The VoyageAI encoder uses VoyageAI's API to generate text embeddings.

Coming soon: Full documentation

Basic Usage

import "github.com/conneroisu/semanticrouter-go/encoders/voyageai"

encoder := &voyageai.Encoder{
    APIKey: "your-api-key",
    Model: "voyage-large-2",
}

In-Memory Store

The in-memory store is the simplest storage backend for Semantic Router, keeping all embeddings in memory.

Installation

The in-memory store is included with Semantic Router:

go get github.com/conneroisu/semanticrouter-go/stores/memory

Configuration

Using the in-memory store is straightforward:

import "github.com/conneroisu/semanticrouter-go/stores/memory"

// Create a new in-memory store
store := memory.NewStore()

The in-memory store doesn't require any additional configuration.

Example Usage

package main

import (
    "context"
    "fmt"
    "log"

    "github.com/conneroisu/semanticrouter-go"
    "github.com/conneroisu/semanticrouter-go/stores/memory"
)

func main() {
    // Create a new in-memory store
    store := memory.NewStore()
    
    // Store an embedding
    ctx := context.Background()
    err := store.Set(ctx, semanticrouter.Utterance{
        Utterance: "hello world",
        Embed:     []float64{0.1, 0.2, 0.3, 0.4, 0.5},
    })
    if err != nil {
        log.Fatalf("Failed to store embedding: %v", err)
    }
    
    // Retrieve the embedding
    embedding, err := store.Get(ctx, "hello world")
    if err != nil {
        log.Fatalf("Failed to get embedding: %v", err)
    }
    
    fmt.Printf("Retrieved embedding: %v\n", embedding)
    
    // Close the store when done
    store.Close()
}

Characteristics

Advantages

  • Very fast: No network or disk I/O overhead
  • Zero dependencies: No external services required
  • Simple: Easiest store to set up and use
  • Thread-safe: Implements mutex locking for concurrent access

Limitations

  • Non-persistent: Data is lost when the program exits
  • Limited capacity: Bounded by available RAM
  • Single-instance: Can't be shared across multiple processes

Best Practices

The in-memory store is ideal for:

  1. Development and testing environments
  2. Simple applications with a small number of routes
  3. Applications where startup time isn't critical (routes need to be re-encoded on startup)
  4. Low-latency requirements where every millisecond counts

For production use with many routes or where persistence is important, consider using the MongoDB or Valkey stores instead.

MongoDB Store

The MongoDB store uses MongoDB to persist embeddings.

Coming soon: Full documentation

Basic Usage

import "github.com/conneroisu/semanticrouter-go/stores/mongo"

store, err := mongo.NewStore(ctx, "mongodb://localhost:27017", "database", "collection")

Valkey Store

The Valkey store uses Valkey (Redis-compatible) to persist embeddings.

Coming soon: Full documentation

Basic Usage

import "github.com/conneroisu/semanticrouter-go/stores/valkey"

store, err := valkey.NewStore("localhost:6379", "", 0)

Frequently Asked Questions

General Questions

What is Semantic Router?

Semantic Router is a Go library that routes requests based on semantic meaning rather than explicit rules or keywords. It uses vector embeddings to understand the meaning of text and route it to the appropriate handler.

How does it compare to traditional routing approaches?

Traditional routing approaches rely on explicit rules, regular expressions, or keyword matching. Semantic Router uses the meaning of the text itself, allowing for more flexible and natural language understanding.

Technical Questions

Which embedding models work best?

For optimal performance, we recommend using models specifically designed for embeddings. For example:

  • OpenAI: text-embedding-3-large or text-embedding-3-small
  • Ollama: mxbai-embed-large or nomic-embed-text
  • Google: textembedding-gecko
  • VoyageAI: voyage-large-2

How many example utterances should I provide per route?

We recommend at least 3-5 utterances per route for good performance. More examples generally lead to better results, especially for routes with diverse semantic content.

How can I optimize performance?

  • Use the WithWorkers option to increase concurrency
  • Use the in-memory store for fastest performance
  • Consider precomputing and caching embeddings for known routes
  • Use a smaller embedding model if latency is critical

Which similarity method should I use?

The default dot product similarity (cosine similarity) works well for most cases. For specific use cases:

  • Dot product similarity: Good default for most text similarity tasks
  • Euclidean distance: Better for when the magnitude of vectors matters
  • Manhattan distance: Less sensitive to outliers than Euclidean
  • Jaccard similarity: Good for comparing sets of items
  • Pearson correlation: Good for finding linear relationships

Can I create my own encoder or storage backend?

Yes! You can implement the Encoder interface to create a custom encoder and the Store interface to create a custom storage backend.

Troubleshooting

My router isn't matching correctly

Check the following:

  1. Ensure your example utterances are diverse and representative
  2. Try different similarity methods or adjust their coefficients
  3. Make sure your embedding model is appropriate for the task
  4. Add more example utterances for challenging routes

I'm getting "context canceled" errors

This usually means the context passed to the Match function was canceled before the operation completed. Make sure your context has an appropriate timeout and isn't being canceled prematurely.

API Reference

This page provides a reference for the most important types and functions in the Semantic Router library.

Core Types

Router

type Router struct {
    Routes  []Route
    Encoder Encoder
    Storage Store
    // contains filtered or unexported fields
}

The Router is the main struct for the semantic router. It contains the routes, encoder, and storage.

Creating a Router

func NewRouter(routes []Route, encoder Encoder, store Store, opts ...Option) (router *Router, err error)

Creates a new semantic router with the given routes, encoder, and storage. Additional options can be provided to configure the router.

Matching an Utterance

func (r *Router) Match(ctx context.Context, utterance string) (bestRoute *Route, bestScore float64, err error)

Matches the given utterance against the routes in the router and returns the best matching route and its score.

Route

type Route struct {
    Name       string
    Utterances []Utterance
}

A Route represents a possible destination for an utterance. It contains a name and a list of example utterances.

Utterance

type Utterance struct {
    ID        int
    Utterance string
    Embed     []float64
}

An Utterance represents a text string and its embedding vector.

Interfaces

Encoder

type Encoder interface {
    Encode(ctx context.Context, utterance string) ([]float64, error)
}

The Encoder interface is implemented by encoder drivers to convert text into vector embeddings.

Store

type Store interface {
    Setter
    Getter
    io.Closer
}

type Setter interface {
    Set(ctx context.Context, keyValPair Utterance) error
}

type Getter interface {
    Get(ctx context.Context, key string) ([]float64, error)
}

The Store interface is implemented by storage backends to store and retrieve embeddings.

Configuration Options

WithSimilarityDotMatrix

func WithSimilarityDotMatrix(coefficient float64) Option

Sets the similarity dot matrix function with the given coefficient. This implements cosine similarity.

WithEuclideanDistance

func WithEuclideanDistance(coefficient float64) Option

Sets the Euclidean distance function with the given coefficient.

WithManhattanDistance

func WithManhattanDistance(coefficient float64) Option

Sets the Manhattan distance function with the given coefficient.

WithJaccardSimilarity

func WithJaccardSimilarity(coefficient float64) Option

Sets the Jaccard similarity function with the given coefficient.

WithPearsonCorrelation

func WithPearsonCorrelation(coefficient float64) Option

Sets the Pearson correlation function with the given coefficient.

WithWorkers

func WithWorkers(workers int) Option

Sets the number of worker goroutines to use for computing similarity scores.

Error Types

ErrNoRouteFound

type ErrNoRouteFound struct {
    Message   string
    Utterance string
}

Error returned when no route is found for an utterance.

ErrEncoding

type ErrEncoding struct {
    Message string
}

Error returned when an error occurs during encoding.

ErrGetEmbedding

type ErrGetEmbedding struct {
    Message string
    Storage Store
}

Error returned when an error occurs while retrieving an embedding from storage.