FlashInfer: Kernel Library for LLM Serving
-
Updated
Apr 11, 2026 - Python
#
distributed-inference
Here are 24 public repositories matching this topic...
A GPU cluster manager that configures and orchestrates inference engines like vLLM and SGLang for high-performance AI model deployment.
cuda inference openai llama maas rocm ascend llm llm-serving vllm genai llm-inference qwen deepseek sglang distributed-inference high-performance-inference mindie
-
Updated
Apr 10, 2026 - Python
一個基於 llama.cpp 的分佈式 LLM 推理程式，讓您能夠利用區域網路內的多台電腦協同進行大型語言模型的分佈式推理，使用 Electron 的製作跨平台桌面應用程式操作 UI。
-
Updated
Aug 24, 2025 - JavaScript
Code for paper "JMDC: A Joint Model and Data Compression System for Deep Neural Networks Collaborative Computing in Edge-Cloud Networks"
compression deep-learning distributed-computing inference dnn neural-networks pruning quantization dnn-optimization edge-computing mobilenet edge-cloud ai-models dnn-partition distributed-inference
-
Updated
Aug 24, 2025 - Python
Analyze and generate unstructured data using LLMs, from quick experiments to billion token jobs.
csv s3 pandas data-engineering parquet data-processing data-pipelines observability synthetic-data unstructured-data mlops batch-inference polars llm-inference evals distributed-inference
-
Updated
Mar 17, 2026 - Python
Accelerate reproducible inference experiments for large language models with LLM-D! This lab automates the setup of a complete evaluation environment on OpenShift/OKD: GPU worker pools, core operators, observability, traffic control, and ready-to-run example workloads.
-
Updated
Apr 2, 2026 - Python
Source code of the paper "Private Collaborative Edge Inference via Over-the-Air Computation".
machine-learning ensemble-learning wireless-communications differential-privacy multi-view-learning edge-inference over-the-air-computation collaborative-inference distributed-inference
-
Updated
Jan 14, 2025 - Python
Super Ollama Load Balancer - Performance-aware routing for distributed Ollama deployments with Ray, Dask, and adaptive metrics
python caching machine-learning streaming performance ai multi-node http2 load-balancer rpc asyncio ray model-serving gpu-monitoring inference-optimization llm llama-cpp ollama distributed-inference hybrid-routing
-
Updated
Apr 10, 2026 - Python
Web UI for orchestrating distributed llama.cpp RPC GPU clusters with auto node discovery, telemetry, and one-click deployment.
python flask rpc rpc-server rpc-client gpu-cluster llm llama-cpp gguf distributed-inference ai-cluster gguf-models ai-cluster-automation
-
Updated
Mar 8, 2026 - JavaScript
Official impl. of ACM MM paper "Identity-Aware Attribute Recognition via Real-Time Distributed Inference in Mobile Edge Clouds". A distributed inference model for pedestrian attribute recognition with re-ID in an MEC-enabled camera monitoring system. Jointly training of pedestrian attribute recognition and Re-ID.
mec re-identification edge-computing re-id attribute-recognition mobile-edge-computing distributed-inference
-
Updated
Apr 26, 2020 - Python
Distributed LLM inference protocol where compute is currency. Compute Unit (CU) = 10^9 FLOPs of verified inference. Rust, OpenAI-compatible, no token, no ICO.
rust bitcoin p2p openai-api nostr llm llama-cpp distributed-inference compute-economy self-improving-agents
-
Updated
Apr 9, 2026 - Rust
Turn any Kubernetes Cluster into a private LLM endpoint. One Helm command deploys distributed inference across commodity hardware. Raspberry Pi's, old servers, mixed architectures. OpenAI-Compatible API Powered by llama.cpp RPC
docker kubernetes open-source raspberry-pi machine-learning ai gpu helm decentralized distributed-computing inference self-hosted homelab cluster-computing edge-computing large-language-models llm llm-inference ollama distributed-inference
-
Updated
Mar 4, 2026 - Dockerfile
Referral service for your LLM
-
Updated
Apr 11, 2026 - Rust
Mixed-vendor GPU inference cluster manager with speculative decoding
python machine-learning deep-learning metal cuda p2p homelab rocm gpu-cluster llama-cpp gpu-inference local-llm llm-inference ollama speculative-decoding distributed-inference gpu-pooling self-hosted-ai openai-compatible mixed-gpu
-
Updated
Apr 11, 2026 - Python
Encrypted Decentralized Inference and Learning (E.D.I.L.)
deep-neural-networks deep-learning he federated-learning decentralized-training distributed-inference ratio1
-
Updated
Jan 20, 2026 - Python
A comprehensive framework for multi-node, multi-GPU scalable LLM inference on HPC systems using vLLM and Ollama. Includes distributed deployment templates, benchmarking workflows, and chatbot/RAG pipelines for high-throughput, production-grade AI services
hpc rag ai-inference vllm llm-inference ollama distributed-inference vllm-serve ai-inference-server multinode-training
-
Updated
Dec 10, 2025 - Python
Mycellm iOS / iPadOS app for iPhone and iPad
swift open-source ios machine-learning ai metal peer-to-peer ipad swiftui llm llama-cpp distributed-inference
-
Updated
Mar 31, 2026 - Swift
Distributed LLM inference across multiple machines. A central server routes OpenAI-compatible requests to llama.cpp client nodes, with automatic model distribution and mutual TLS security.
docker golang grpc openai inference-server llm llm-serving llama-cpp llm-inference local-ai gguf distributed-inference
-
Updated
Mar 11, 2026 - Go
A cache-centric architecture, compatibility contracts, and protocols for KV cache handoff in LLM inference.
-
Updated
Mar 22, 2026
Improve this page
Add a description, image, and links to the distributed-inference topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the distributed-inference topic, visit your repo's landing page and select "manage topics."