Custom Routers

​

Overview

​

What Are Custom Routers?

• Analyze each request to understand its requirements and complexity
• Select the optimal model from your model pool based on learned patterns
• Learn from your evaluations to continuously improve selection accuracy
• Optimize for your goals - whether that’s quality, cost, speed, or a balance
• Adapt to your domain using your proprietary datasets and benchmarks

​

The Power of Evaluation-Driven Routing

​

The Complete Workflow

1. Create Datasets → 2. Design Evaluation Templates → 3. Run Evaluations → 4. Build Custom Router

​

1. Create Your Datasets

• Manual Datasets: Curate specific test cases for your domain
• Learning Datasets: Extract successful interactions from your spaces
• Benchmark Datasets: Leverage open-source benchmarks from the Pulze Evals repository

​

2. Design Evaluation Templates

• Define metrics that matter to your use case (accuracy, helpfulness, tone, etc.)
• Configure evaluation criteria using “LLM-as-a-judge” methodology
• Set quality thresholds for passing evaluations
• Test different configurations and feature flags

​

3. Run Comprehensive Evaluations

• Test multiple AI models against your datasets
• Evaluate entire space configurations with assistants and tools
• Compare models across different task types and complexity levels
• Generate detailed performance metrics and scores

​

4. Build Your Custom Router

1. Select Evaluation Runs: Choose evaluations that represent your target use cases
2. Configure Router Logic: Define how the router should select models based on learned patterns
3. Train the Router: The system builds a KNN (K-Nearest Neighbors) router using your evaluation data
4. Deploy and Use: Activate your router for specific spaces, agents, teams, or your entire organization

​

Your AI Competitive Advantage

​

Proprietary Intelligence

• Domain-specific routing tailored to your industry and tasks
• Proprietary performance data that reflects your unique requirements
• Fine-tuned selection logic that understands your quality standards
• Private evaluation results that remain within your organization

​

Public and Private Options

• Use open-source benchmarks from the Pulze Evals repository
• Contribute your own benchmarks back to the community
• Leverage community-curated evaluation templates
• Benefit from collaborative benchmark development

• Keep your datasets, evaluations, and routing logic completely private
• Build defensible AI systems on proprietary data
• Maintain competitive advantages through private benchmarks
• Ensure sensitive data never leaves your organization

​

Mixture of Experts (MoE) Routing

​

How MoE Routing Works

• Different models for different tasks: Route simple queries to fast, cost-effective models and complex reasoning to powerful models
• Context-aware selection: Consider query complexity, domain, and requirements when selecting models
• Performance-based optimization: Learn from evaluation data which model is the “expert” for each task type
• Dynamic adaptation: Continuously improve routing decisions based on outcomes

​

The KNN Router Engine

1. Semantic Understanding: Analyze incoming queries to understand their intent and requirements
2. Similarity Matching: Find semantically similar queries from your evaluation dataset
3. Expert Selection: Identify which models performed best on similar queries
4. Weighted Scoring: Select the optimal model based on learned performance patterns

​

Accessing Router Management

​

Creating Custom Routers

​

Prerequisites

1. Completed evaluation runs with performance data
2. Multiple models tested to provide routing options
3. Representative datasets covering your use cases
4. Quality thresholds defined in your evaluation templates

​

Step-by-Step Router Creation

1. Click “Create Router” button in the Routers section
2. Select Evaluation Runs: Choose completed evaluation runs that represent your target performance
3. Configure Router Settings:
   • Name: Descriptive identifier (e.g., “Customer Support MoE Router”)
   • Description: Explain the router’s purpose and use cases
   • Model Pool: The router automatically includes models from selected evaluations
   • Routing Strategy: The system configures KNN-based routing from evaluation patterns
4. Review Configuration: Verify the router’s training data and model pool
5. Create and Activate: Save your router and make it available for use

​

Router Configuration Details

• Analyzes performance patterns across all selected evaluation runs
• Identifies query-model relationships that indicate which models excel at which tasks
• Builds a semantic understanding of query types and complexity levels
• Configures KNN routing logic to replicate successful model selections
• Sets up fallback strategies for edge cases and error handling

​

Using Custom Routers

​

In Spaces

1. Navigate to Space Settings
2. Go to Models & Routers section
3. Select your custom router
4. Save configuration
5. All new conversations in the space use intelligent routing

​

In API Calls

curl https://api.pulze.ai/v1/chat/completions \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "router": "your-custom-router-name",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

​

Scope Options

• Specific Agents/Assistants: Route intelligently for specialized AI agents
• Individual Spaces: Apply custom routing to specific teams or projects
• Team-Level: Provide consistent routing across team workspaces
• Organization-Wide: Deploy your router across your entire organization

​

Continuous Improvement

1. Run more evaluations on new datasets and scenarios
2. Update router training data with recent evaluation results
3. Add new models to the pool and evaluate their performance
4. Refine quality standards through updated evaluation templates
5. Analyze routing patterns and adjust strategies

​

Managing Routers

​

Editing Routers

1. Click the menu icon (⋮) on the router
2. Select “Edit”
3. Update evaluation runs or configuration
4. Re-train with new data if needed
5. Save and test updated router

​

Activating/Deactivating

• Activate: Make router available for spaces and API calls
• Deactivate: Temporarily disable without deleting training data
• Archive: Preserve router configuration but remove from active use

​

Deleting Routers

1. Click the menu icon (⋮)
2. Select “Delete”
3. Confirm deletion
4. Router and its configuration are permanently removed

​

Benefits of Evaluation-Driven Routing

​

Open Source and Community

​

Pulze Evals Repository

• Datasets: Curated benchmarks for testing AI performance
• Evaluation Templates: Community-contributed evaluation criteria
• Benchmark Results: Shared performance data across models
• KNN Router: Open-source routing technology

​

Contributing Back

1. Export your datasets as public benchmarks
2. Contribute evaluation templates that others can use
3. Share performance insights (without sensitive data)
4. Improve routing technology through open-source collaboration

​

Enterprise Use Cases

​

Private Evaluation Pipelines

• Maintain data privacy by keeping all evaluation data internal
• Build defensible AI that competitors cannot replicate
• Optimize speed across thousands or millions of requests
• Ensure compliance by controlling model selection based on data sensitivity
• Improve productivity through AI that understands domain-specific tasks

​

Industry-Specific Routing

• Legal: Route complex legal reasoning to specialized models
• Healthcare: Ensure medical queries use compliant, accurate models
• Finance: Balance speed and accuracy for financial analysis
• Customer Support: Optimize for helpfulness and response speed
• Software Development: Route coding tasks to programming-specialized models

​

Best Practices

​

Permission Requirements

• View: See router list and details
• Create: Build new routers from evaluation data
• Edit: Modify router configuration and training data
• Delete: Remove routers
• Assign: Set routers as space or organization defaults

​

Advanced Router Features

​

A/B Testing

1. Create two routers with different evaluation training sets
2. Deploy both to similar spaces
3. Compare performance metrics
4. Select the better-performing router

​

Multi-Router Strategies

• Domain-specific routers for specialized teams
• Cost-optimized routers for high-volume use cases
• Quality-focused routers for critical applications
• General-purpose routers for diverse tasks

​

Failover and Reliability

• Automatic fallback to reliable models on errors
• Confidence scoring to detect uncertain routing decisions
• Error handling to ensure robust performance
• Rate limiting awareness to avoid model capacity issues

​

Troubleshooting

​

Router Not Selectable

1. Verify router status is Active
2. Check you have permissions to use the router
3. Confirm evaluation runs are valid and complete
4. Ensure router training completed successfully

​

Unexpected Model Selection

1. Review the evaluation data the router learned from
2. Check if new query types differ from training examples
3. Verify model pool includes appropriate options
4. Consider re-training with more representative evaluations

​

Performance Issues

1. Analyze recent routing decisions and outcomes
2. Run new evaluations on current query patterns
3. Update router training with recent evaluation data
4. Review model availability and performance
5. Check for changes in query distribution

​

Integration Workflow Summary

1. Create Datasets: Build evaluation datasets from your use cases
2. Design Templates: Define evaluation criteria and quality standards
3. Run Evaluations: Test models across your datasets
4. Analyze Results: Understand which models perform best for which tasks
5. Build Router: Create a custom router from evaluation insights
6. Deploy and Monitor: Use the router and track its performance
7. Iterate: Run more evaluations and improve your router over time

​

Related Resources