(Post-WIP) Self-hosted distributed local LLMs load balanced proxy endpoint

  • People
    • David
  • Idea
    • Single OpenAI-compatible LLM endpoint that can be reached over the internet, secured with an API key, bridges hardware options across 3+ GPU machines (Apple, Nvidia, etc)
  • Details
    • LiteLLM
    • LM Studio
    • TabbyAPI
    • Caddy
    • Cloudflare Tunnels+DNS

Read more