Selected work · 2025—2026

Systems for faster,smarter inference.

Production-scale multimodal inference, speculative decoding and adaptive early-exit research — developed into complete engineering case studies.

03
case studies
11.58×
best RTF gain
64
H20 GPUs
Explore the case studies

Selected projects

Three systems. Measurable impact.

Each case study traces the system context, implementation path, measurable results and technology stack.

Multimodal inference · Project 01

MiMo-Audio on vLLM Omni

Oct 2025 — Feb 2026

vLLM OmniMiMo-AudioCUDA GraphsStreaming DecodeKV CacheBatchingGPU InferenceAudio Tokens

Adapted Xiaomi’s MiMo-Audio model to a community inference stack for high-performance deployment, creating a long-term serving path for fully immersive voice conversations.

Engineering approach

  • Migrated MiMo Transformers to the vLLM Omni multimodal inference framework and delivered both offline/online and multi-batching request paths, including request packaging, audio inputs and lifecycle management.
  • Implemented end-to-end asynchronous streaming decode for chunked audio-token generation and incremental return, then added Audio Tokenizer CUDA Graph capture to improve TTFAT, RTF and continuous-generation stability.
  • Upgraded the integration to vLLM Omni v0.16.0 by reconnecting GPUARRunner, GPUGenerationRunner, KV Cache, batching and scheduling, and reworking multimodal token organization for streaming and non-streaming requests.
  • Built a benchmark and stress-testing workflow covering throughput, P50/P95/P99 latency, TTFT, RTF, GPU memory and utilization, with comparisons against the Transformers baseline across batching, cache, attention and I/O.

Outcome

Offline throughput increased 25–40% (1.2k → 1.6k tokens/s). Online TTFT fell 25–40% (180 → 110 ms), RTF improved 11.58× (1.39 → 0.12; an 89–93% reduction), P95 fell 15–25% (890 → 658 ms), and P99 fell 10–20%. CUDA Graphs reduced P95 variance by roughly 15–25%. The MiMo-Audio model support was merged into vLLM Omni as PR #750; bug fixes #1510 and #1296 were also merged.

Speculative decoding · Project 02

UI Agent — Qwen3-VL 30B-A3B with EAGLE-3

Oct 2025 — Feb 2026

EAGLE-3Qwen3-VLSpeculative SamplingDeepSpeed ZeRO-2AccelerateINT464× H20CloudML

Reverse-engineered the EAGLE-3 speculative-sampling architecture and training flow to replace a Medusa-based path, then adapted it for production Qwen3-VL multimodal inference.

Engineering approach

  • Helped build the lightweight SpecuLite speculative-training framework, extending Qwen models with an EAGLE draft architecture that matches the target model and can initialize directly from target weights.
  • Created an online-distribution-aligned data pipeline: extracted production logs and regenerated labels with vLLM so training labels precisely matched inference behavior, preventing severe acceptance-rate degradation in RL workloads.
  • Built distributed training with Accelerate and DeepSpeed ZeRO-2 across 64 H20 GPUs, reducing the complete training time for a 0.2B-token UI Agent dataset to four hours.
  • Introduced EAGLE-3 to the VLM path, validated speedups without sacrificing visual-language quality, connected the training job to CloudML and shipped it into the UI Agent workload.

Outcome

Fourteen workloads prepared for EAGLE-3 training (9 Agent, 2 UI Agent and 3 Vision). On an RTX 5090 with INT4 quantization, QPS improved 1.9× (3 → 5.8), average latency fell 18–23% (500 → 405 ms), P50 fell 24–33% (557 → 394 ms), and P99 fell 9–13% (822 → 732 ms). Multi-step accuracy improved 16.42–20.15% over Medusa while meeting the UI Agent SLA.

Edge LLM systems · Project 03

RHRLSmartExit

Jun 2025 — Aug 2025

Early Exit LLMRecurrent PPOQAT + PTQTensorRT-LLMTritonRay ServeJetson AGX OrinSLA Scheduling

Designed hierarchical early-exit LLM inference with adaptive scheduling for mobile and edge GPUs, targeting predictable low latency under strict memory, power and SLA constraints.

Engineering approach

  • Implemented a confidence-and-consistency early-exit rule with Platt calibration and order-preserving calibration, producing offline per-layer threshold tables with a bounded false-exit rate.
  • Applied multi-exit distillation, progressive head/neuron pruning near exits, and token-importance scoring from attention accumulation and vector norms to skip low-value computation and reduce local FLOPs.
  • Combined QAT and PTQ: critical layers used fake quantization and observer statistics, non-critical layers used static calibration, and layer-wise sparse regularization controlled quantization drift.
  • Implemented SLA-aware recurrent PPO scheduling to adapt prefill micro-batches from predicted exit depth and SLA rewards, reducing GPU pipeline bubbles with offline warm-start and online fine-tuning.

Outcome

Validated on Jetson AGX Orin with Llama-2/3.1 7B–13B using Ray Serve, Triton Inference Server and TensorRT-LLM. Under mixed workloads the system held P95 within the target SLA and accuracy loss within 1%, while increasing tokens/s by 31.3%, reducing energy per token by 21.1%, lowering peak memory, and keeping false exits at or below 0.9%.

Open-source timeline

Building in public.

Live GitHub data with an offline-safe snapshot, ordered by the latest push.

Continue exploring

Code, experiments and work in progress.

View complete GitHubBack home