(Post-WIP) Self-hosted distributed local LLMs load balanced proxy endpoint
- People
- David
- Idea
- Single OpenAI-compatible LLM endpoint that can be reached over the internet, secured with an API key, bridges hardware options across 3+ GPU machines (Apple, Nvidia, etc)
- Details
- LiteLLM
- LM Studio
- TabbyAPI
- Caddy
- Cloudflare Tunnels+DNS