Last week I wrote about two paths to vLLM on Apple Silicon - comparing vllm-metal and vllm-mlx as options for local inference. This week the picture changed. LM Studio shipped concurrent request support, and suddenly the simplest option became the most practical one. People David Idea We spent the week
People Donavan Idea Set up MCP servers for CRM and Google Drive to connect them directly to Claude Desktop - giving Claude real-time access to our deal pipeline, contacts, and documents without copy-pasting or context-switching. Details * MCP (Model Context Protocol) for connecting AI assistants to external tools and data sources.
People David Idea I tested both vllm-metal and vllm-mlx on the M3 Ultra Mac Studio to see which one gives better multi-user LLM throughput on a high-memory Apple Silicon machine. Details * Both projects showed up in late 2025 - vllm on Metal/MLX is very exciting because it should let
People David Idea Testing Nvidia's v1 LLM Router blueprint as a third approach to intelligent query routing - this time using policy-based task classification instead of semantic similarity or neural network intent matching. Details * Nvidia's v1 LLM Router blueprint (main branch) takes a three-step approach: apply
People: David Idea: Tested UIUC's LLMRouter framework as an alternative to LiteLLM's semantic routing - this one trains an actual neural network for intent classification and can run on hardware as small as a Raspberry Pi with 4GB of ram. Details: * Wanted to compare this against
People: David Idea: Testing semantic routing as a way to automatically send LLM requests to different models based on what the user is asking—part of a bigger vision for smart edge-to-hub-to-cloud routing on our Maui cluster. Details: * Our LiteLLM proxy already bundles multiple machines into one API endpoint, so
People: David Idea: Testing KIRA, Krafton's open-source project that lets you run a full Claude Code instance through Slack, on our local PMF hardware. Details: * Nicole mentioned wanting more Slack bot and automation options for the team * Found Krafton's KIRA project on GitHub which does exactly
People: Me Idea: Tested whether local/open-source models can handle real marketing strategy work using CrewAI as the orchestration layer. Details: * Wanted to see how open source tools do for automating some tasks in marketing * Used gpt-oss-120b as the backbone model * Ran everything through CrewAI for agent orchestration * Got Joe&
People: Me Idea: I wanted to see how well the Qwen3-32B model runs on an AMD 7900XTX using different quantization formats and inference backends—spoiler: AWQ is not the move on this generation of consumer AMD card. Details: * Tested on a host system with Ubuntu 24.04 with ROCm 7.
People David Idea Part of our work in the Kumubot cluster involves being able to work on both text as well as audio recordings - the idea was the figure out the capabilities of Qwen3-Omni on our Blackwell hardware Details * Blackwell software support (at least for the RTX 6000 Pro)
People David Idea Joe wanted to know what the capabilities of https://github.com/HKUDS/AI-Trader look like in the context for local LLMs Details * Looks like the assumption of the repo is that you can test multiple AI in agentic fashion against historical stock market data over a period
People: David Idea: To setup our Kumubot cluster, I went down a rabbit hole benchmarking different quantization methods for Qwen3 8B and 32B models to see which ones actually deliver the best accuracy-to-speed tradeoff in real-world use Details: • ExLlamaV3-4bpw surprisingly beat even BF16 on LiveBench accuracy (60.0 vs 58.