Action tokenization in a VLA, explained
How vision-language-action models turn continuous motion into discrete action tokens a transformer can predict, and why that choice caps resolution.
RT-2 taught a language model to move a robot arm by handing motion the same machinery it used for words. Each action dimension, the change in gripper position, the rotation, and the open-or-close command, was normalized and chopped into 256 bins. Those bins were mapped onto 256 token IDs borrowed from the model's own vocabulary. The arm moved because the transformer predicted the next token, exactly the way it would finish a sentence.
That trick sits underneath most vision-language-action (VLA) models today. A transformer speaks in discrete tokens drawn from a fixed vocabulary. A robot speaks in continuous numbers: joint angles, end-effector velocities, gripper forces, arriving tens to hundreds of times per second. Something has to translate between the two, and that something is the action tokenizer. It is easy to treat it as plumbing. It is not.
The tokenizer sets the resolution ceiling of everything downstream. It decides how much of a demonstration survives the trip into the model and back out to the motors, and it decides which tasks the model can even represent. A scheme that works cleanly on slow tabletop pick-and-place can quietly fall apart on fast, contact-rich manipulation. The interesting engineering is in why.
The tokenizer is a lossy contract
Strip away the branding and every action tokenizer answers the same four questions. How do continuous values become discrete symbols? How are multiple degrees of freedom packed together? How is a stream that arrives at 50 Hz turned into a token sequence a transformer will tolerate? And how do you invert all of it to get smooth motion back?
The default answer is uniform binning. Take each action dimension, clip it to a sensible range, and quantize it into N bins, commonly 256 so a bin fits in one byte. RT-2 and, later, the open-source OpenVLA reused rarely used entries in the language model's own token table for these bins, which let them fine-tune a pretrained VLM without growing the vocabulary. One timestep of a 7-DoF arm becomes seven tokens. A short trajectory becomes a few dozen. The model trains with the same next-token cross-entropy loss it always used.
This is elegant, and it is lossy on purpose. Quantization throws away sub-bin precision. Reusing language tokens means the geometry of the action space, the fact that bin 128 sits between 127 and 129, is not encoded anywhere the model can see; it has to relearn ordering from data. For coarse, slow tasks none of that hurts much. Push the control rate up and it starts to.
A tokenizer is a lossy contract between continuous physics and a discrete model. Whatever it discards at capture time, the policy can never learn to reproduce.
Why binning breaks when the hands get fast
Here is the failure mode that surprised a lot of people. Sample a dexterous manipulation demo at a high rate and consecutive actions look almost identical, because in ten milliseconds a hand barely moves. Per-timestep binning turns that into long runs of near-repeated tokens. An autoregressive model trained on that data discovers the cheapest possible strategy: predict that the next token equals the last one. The marginal information per token is tiny, the loss looks fine, and the policy learns to stall.
Physical Intelligence hit exactly this wall training high-frequency policies and answered it with FAST, a frequency-space tokenizer. Instead of binning raw samples, FAST takes a chunk of the action trajectory, applies a discrete cosine transform, keeps the coefficients that carry real signal, and byte-pair-encodes the result. A smooth chunk collapses into a short, dense token string with the redundancy squeezed out. That let an autoregressive VLA, their pi0-FAST line, train on dexterous data that plain binning could not fit. The lesson generalizes: the right token is not a raw sample, it is a compressed description of a motion.
The other road, continuous action heads
Not every VLA tokenizes actions at all. A second family skips discretization and bolts a continuous generative head onto the transformer's output. Physical Intelligence's pi0 uses a flow-matching action expert; NVIDIA's GR00T N1 pairs a vision-language backbone with a diffusion-transformer action head; Google DeepMind's Gemini Robotics runs a VLA that emits continuous low-level commands. In all three the transformer produces a latent, and a small head samples a chunk of continuous actions conditioned on it.
The payoff is no quantization error and smooth output at high control rates. The price is that you give up the clean single-vocabulary autoregressive loss, and sampling a diffusion or flow head is iterative, which costs inference time you have to engineer back. This is the live design fork in VLAs right now: compress motion into discrete tokens the transformer predicts directly, or keep it continuous and pay for a separate sampler. Both ship in serious systems, and the choice interacts with your data more than with your model.
Four schemes, side by side
The options are easier to weigh next to each other. The table below sketches where each representation earns its keep and what it costs.
| Scheme | How motion becomes output | Strength | Main cost |
|---|---|---|---|
| Uniform per-dimension binning (RT-2, OpenVLA) | Each dimension quantized to roughly 256 bins mapped onto reused language-token IDs | Simple; reuses the pretrained vocabulary and loss | Quantization error; degrades at high control rates |
| Frequency-space tokenization (pi0-FAST) | DCT over an action chunk, then byte-pair encoding of the kept coefficients | Trains autoregressively on fast, dexterous data | Extra encode and decode step; compression to tune |
| Learned vector-quantized codebook | Action segments mapped to entries in a learned discrete codebook | Compact, data-adaptive vocabulary | Codebook collapse; needs representative data |
| Continuous diffusion or flow head (pi0, GR00T N1) | A small head samples continuous action chunks from a transformer latent | No quantization error; smooth high-rate output | Iterative sampling; no single autoregressive loss |
Where tokenizers leak
Two practical pitfalls decide whether any of this survives contact with a real fleet. The first is normalization. Bin edges are set from the statistics of the training data, so a policy trained on one robot's action ranges will silently miscalibrate on another with different reach or gripper stroke. Aggregations like Open X-Embodiment, which pooled demonstrations across many robot embodiments, and large teleoperation sets like DROID exist partly to force tokenizers to generalize across those ranges rather than overfit one arm.
The second is chunking. Predicting one step at a time compounds error and floods the sequence with the redundancy that breaks binning. Most current systems predict a short chunk of future actions at once, which cuts compounding drift and gives the tokenizer a segment worth compressing. Open toolkits like Hugging Face's LeRobot now bundle these tokenizers and chunked action heads directly, which is turning what used to be a research decision into a config choice. That is good for velocity and slightly dangerous: the defaults encode assumptions about frequency and embodiment that may not match your robot.
The quiet variable
Action tokenization rarely makes the headline when a new VLA lands, and it should. It is the layer where a model's ambitions meet the resolution of its data, and it is the first place a system stops being able to do something it was never able to represent. Watch it the next time a lab reports a jump on dexterous tasks. The new trick is often not a bigger brain. It is a better way to write down what the hands did.