Flide: Accelerating Decision Inference with Exact State Forking

GitHub Docs Quickstart pip install flide

9.4×faster than vLLM and SGLang (QuALITY, 0.8B)
2.4×the throughput with 64 clients
0.1points from a float32 reference
1 passfor a state and all its questions

Fork the state into every question

Flide reads the state once and forks it into all of its questions. Each one answers exactly as if asked alone.

One request

3 questions

Generation engine

Multi-item scoring

Flide

computed from cache computed again delimiter questions block saved by a prefix cache

Fastest where a state carries many questions

Up to 9.4× faster than vLLM and SGLang. The more questions a state carries, the bigger the gap.

Benchmarks on one H200

Flide vLLM SGLang

Shared work, exact answers

Flide stays within 1.1 points of a float32 reference, the closest of any engine.

Distance from a float32 reference

Flide

0.1 to 1.1 pts

Multi-item scoring

0.9 to 14.8 pts

Decisions flipped at 0.5

Flide

up to 2.3%

Multi-item scoring

0.8 to 28.9%

Same bits at any load

With --batch-invariant, a request gets the same bits however it is batched.

Agent steps resume from an exact state

Flide keeps an exact state from each step, and the next step picks up right there.

Extra ms per step, per 1k tokens of history

Flide

0.6 to 1.0 ms

vLLM and SGLang

11 to 15 ms

Multi-item scoring

3 to 29 ms

What is inside

ForkAttention

Each question reads its state in place, exactly as if asked alone.

Exact on hybrid models

Gated DeltaNet layers start every question from the exact state of the shared context.

One flat stream

Requests of any mix run as one token stream, replayed by CUDA graphs.

The systemone API

Same /v1/systemone API as vLLM and SGLang, with a probability for every option.

Quickstart

Install with pip and serve. Kernels come prebuilt for A40, A10, RTX 30, H100 and H200.

Install

$ pip install flide

Serve

Option 1 · Qwen3.5 and Gemma 4

$ flide serve --model Qwen/Qwen3.5-0.8B

Option 2 · OneJev and Open-Jev

$ flide serve --model OmniJev/OneJev1.6-0.8B

Ask

curl http://localhost:8000/v1/systemone -H "Content-Type: application/json" -d '{
  "state": "Order #1182 arrived with a cracked screen. The customer asks for a refund and attached two photos.",
  "questions": {
    "action": {"type": "choice", "instructions": "What should support do next?",
               "criteria": {"refund": "refund the order", "replace": "send a replacement", "escalate": "ask a human"}},
    "urgent": {"type": "noul", "instructions": "Does this need a reply within the hour?"}
  }
}'

Models

Backbone	Sizes	Inputs
Qwen3.5	0.8B, 2B, 4B, 9B, 27B	text, images, video
Gemma 4	E2B, E4B, 12B, 26B-A4B, 31B	text, images

Switches

--quantize fp8: Runs Qwen3.5 in FP8 on Ada, Hopper and Blackwell GPUs.
--session-cache-gb 40: Keeps 40 GB of earlier states for agents to resume from.
--batch-invariant: Gives every request the same bits under any batching.
--frontends 2: Runs HTTP, rendering and tokenizing in two processes of their own, for many clients.
--host 127.0.0.1: Serves on this machine.