Neural Network Intent Routing with UIUC's LLMRouter

David Pickett

— 1 min read

People: David

Idea: Tested UIUC's LLMRouter framework as an alternative to LiteLLM's semantic routing—this one trains an actual neural network for intent classification and can run on hardware as small as a Raspberry Pi with 4GB of ram.

Details:

  • Wanted to compare this against last week's LiteLLM semantic router setup to see which fits our edge routing vision better
  • LLMRouter comes out of UIUC research and takes a different approach - trains a small neural network classifier instead of using embedding similarity
  • Got it working with Ollama handling both the base models and the embeddings, keeping everything local
  • The framework lets you define intents and train a routing model on example prompts for each (and also a lot of other routing strategies - this prototype is to try a simple option that is close to what we tried with LiteLLM)
  • Training was straightforward once I figured out the config - just feed it examples of what each route should handle
  • Serving the trained router through Ollama means no external API calls for the routing decision itself
  • This matters for the long-term goal: sensors in the field deciding on their own whether to process locally, hit an on-site hub, or call out to cloud
  • Looks like it should be lightweight enough to deploy on a 4GB RAM Raspberry Pi
  • Routing accuracy was solid for basic intent classification, though more complex queries will need more training data
  • Included setup instructions for Pi deployment in case others want to replicate on minimal hardware, code/documents here: https://github.com/pickettd/local-uiuc-llmrouter-example

Read more