arXiv 2026

Learning Anisotropic Prediction Targets for Flow Models

State-of-the-art Pixel DiTs free of patch artifacts, without conv layers, even in 8-bit.

Rayhan Zirvi1,2*, Hansheng Chen1*, Gordon Wetzstein1, Leonidas Guibas1
1Stanford University 2California Institute of Technology
*Equal contribution
AnisoFlow pixel-space text-to-image generations

Pixel-space text-to-image generations by AnisoFlow-L/16-T2I, a diffusion transformer with only 918M parameters trained from scratch, using a learned anisotropic prediction target.

Abstract

Flow-based generative models are trained on a hand-chosen prediction target, such as the velocity, the noise, or the clean data. This choice strongly affects training and sample quality. We introduce Anisotropic Flow Models (AnisoFlow), which replaces a fixed prediction target with a fully learnable one. AnisoFlow learns to scale the noise component of the target anisotropically, assigning a separate nonnegative weight to each direction in a jointly learned basis. A closed-form map recovers the velocity from the network's prediction, leaving the architecture, flow-matching loss, and sampler unchanged. Since recovery is exact, training optimizes the target purely to make it easier for the network to predict. We theoretically analyze the resulting prediction error and predict that the preferred noise weight in each direction scales near-linearly with data variance, in agreement with the weights learned empirically in pixel space. AnisoFlow improves FID from 1.90 to 1.55 over the JiT baseline on ImageNet 256×256, by learning only the prediction target, without representation alignment or architectural changes. In pixel-space text-to-image generation, AnisoFlow improves image quality and, through better-conditioned velocity recovery, largely removes patch-seam artifacts under reduced-precision inference.

1.90 → 1.55 FID

Improvement over JiT on ImageNet

Same JiT-H/16 network and training recipe. Learning only the prediction target substantially improves FID, with no REPA, extra pretraining, or architectural changes.

Artifact-free

Even under 8-bit inference

A well-conditioned learned target fixes the patch seams of \(x\)-prediction pixel DiTs without extra conv layers, and stays clean even under INT8 and FP8 inference.

A Fully Learnable Prediction Target

Flow models can predict targets like clean data \(x\), noise \(\epsilon\), or velocity \(u = \epsilon - x\). This is one of the most critical design choices, yet it remains largely empirical, often hand-picked or tuned within restricted degrees of freedom.

AnisoFlow makes this choice fully learnable. It defines a generic linear target \(u_W = W\epsilon - x\), where \(W\) is a learned positive semidefinite operator. A single \(W\), shared across tokens, scales the noise component in the target anisotropically.

A closed-form formula recovers the per-token velocity \(\hat{u}\) from the prediction \(\hat{u}_W\), so the network and \(W\) are jointly trained with the standard flow-matching loss on the recovered velocity, and sampling is unchanged.

AnisoFlow prediction and closed-form velocity recovery

The network predicts the AnisoFlow target \(u_W = W\epsilon - x\), and a closed-form map recovers the standard velocity.

Standard Targets as Special Cases

Writing \(W = B\,\mathrm{diag}(w_1, \dots, w_D)\,B^\top\) with an orthonormal basis \(B\), each direction \(i\) gets its own noise weight \(w_i \ge 0\). Clean-data prediction (\(w_i = 0\)), velocity prediction (\(w_i = 1\)), scalar interpolations (\(w_i \equiv w\)), and the binary low-rank mask of AsymFlow (\(w_i \in \{0, 1\}\)) are all special cases with restricted weight profiles. AnisoFlow instead learns the full profile together with the basis.

AnisoFlow target family

Standard targets correspond to restricted noise-weight profiles. AnisoFlow learns an unrestricted nonnegative profile across directions.

Theory: Learned Weights Follow the Data Spectrum

A larger noise weight reduces how much prediction error is amplified when converting back to velocity, but puts more noise into the target that the network has to predict. Analyzing this trade-off in the PCA basis, we predict that the preferred weight in each direction scales near-linearly with the data variance \(\lambda_i\), i.e., \(w_i \approx c\lambda_i\), and that a single weight profile suffices across all timesteps.

The trained pixel-space models agree with our theory. Regardless of initialization and basis optimization, the learned weights follow the near-linear scaling, and a basis learned from scratch aligns closely with PCA.

Learned weights versus directional data variance
(a) Learned weights
Alignment between learned and PCA bases
(b) Alignment between learned and PCA bases

(a) Learned weights versus directional data variance on JiT-H/16. (b) Absolute inner products between PCA directions and learned directions. Both initializations yield near-diagonal alignment.

ImageNet 256×256 Results

The baseline is JiT-H/16, a plain pixel transformer with \(x\)-prediction. AnisoFlow keeps the same network and training recipe and changes only the prediction target, with no REPA, pretrained encoder, or extra pretraining.

Effect of the Learnable Target

Method Noise weights Noise basis Init FID↓
160 epochs
FID↓
600 epochs
JiTFrozen (zeros)--2.681.90
AsymFlowFrozen (binary)Frozen (PCA)PCA2.341.76
AnisoFlow
AnisoFlowLearnedFrozen (PCA)PCA2.251.63
AnisoFlowLearnedLearnedPCA2.281.63
AnisoFlowLearnedLearned-2.301.55

Learning continuous weights in a frozen PCA basis already improves considerably over JiT and AsymFlow. Learning the basis from an isotropic initialization gives the best final FID.

Pixel Diffusion Comparison

Method Target Params GFLOPs FID↓
Hierarchical CNNs (U-Net-like)
ADM-G\(\epsilon\)554M22404.59
Hierarchical transformers (U-ViT-like)
RIN\(\epsilon\)320M6683.42
SiD, UViT/2\(\epsilon\)2B11102.44
VDM++, UViT/2\(\epsilon\)2B11102.12
SiD2, UViT/2\(\epsilon\)-2741.73
PixelU-H/16†\(x\)1.2B2741.63*
EPG-G/16†\(x\)1.4B6421.58
SiD2, UViT/1\(\epsilon\)-13061.38
Hierarchical transformers (DDT-like)
PixNerd-XL/16†\(\epsilon - x\)700M2682.15
DiP-XL/16†\(\epsilon - x\)631M-1.79
DeCo-XL/16†\(\epsilon - x\)682M2451.62
PixelDiT-XL/16†\(\epsilon - x\)797M3111.61
FrequencyBooster-H/16†\(x\)1.1B-1.60*
HyperDiT-H†\(\epsilon - x\)952M4581.56*
Method Target Params GFLOPs FID↓
Plain transformers (DiT-like)
PixelFlow-XL/4\(\epsilon - x\)677M58181.98
JiT-H/16\(x\)953M3641.86*
PixelGen-XL/16†\(x\)676M2601.83
JiT-G/16\(x\)2B7661.82*
PixelREPA-H/16†\(x\)953M3641.81*
AsymFlow-H/16†\(P\epsilon - x\)953M3641.57
AnisoFlow-H/16 (ours)\(W\epsilon - x\)953M3651.55

FLOP estimation follows PixelDiT. * marks the JiT evaluation protocol, which can give up to 0.08 lower FID than the ADM protocol. † uses a pretrained representation encoder (e.g., REPA-style alignment or DINO-based losses) or self-supervised pretraining.

Pixel-Space Text-to-Image Generation

We train JiT-L/16-T2I, whose architecture is based on the MM-JiT denoiser of MiniT2I, with the text encoder, data, and training schedule of i1, and compare fixed \(x\)-prediction with the learned AnisoFlow target under matched training and evaluation. At 512×512, AnisoFlow improves over the \(x\)-prediction baseline on every metric except GenEval, with the largest gains in HPSv3, FID, and patch FID, and surpasses the larger latent i1-1B model on the detail- and style-sensitive metrics.

Text-to-Image 512×512

Method Params OneIG↑ LongText*↑ GenEval*↑ DPG*↑ HPSv3↑ FID↓ pFID↓
i1-1B (FLUX.2 VAE, \(u\)-pred.)1.3B0.5110.8530.82686.37.1162.362.4
Pixel space, matched training and evaluation
JiT-L/16-T2I (\(x\)-pred.)917M0.5120.8620.82085.77.7122.024.4
AnisoFlow-L/16-T2I (ours)918M0.5160.8640.81485.88.0320.122.7

All models are trained using i1's data, text encoder, and schedule, and tested with the same guidance and sampler settings. FID and pFID are computed on COCO-10k. * indicates using i1's rewritten prompts.

AnisoFlow (ours)
Wedding invitation, AnisoFlow Rescue helicopter, AnisoFlow Space-station portrait, AnisoFlow
JiT (x-pred.)
Wedding invitation, JiT x-prediction Rescue helicopter, JiT x-prediction Space-station portrait, JiT x-prediction
i1 (latent, u-pred.)
Wedding invitation, i1 latent Rescue helicopter, i1 latent Space-station portrait, i1 latent

512×512 generations with identical prompts and seeds. AnisoFlow renders finer texture than the \(x\)-prediction baseline (paper, lace, waves, skin) and a sharper, more distinct style than the latent model.

No Patch Artifacts, Even in 8-Bit

For \(x\)-prediction, recovering the velocity divides the prediction error by \(\sigma_t\), so errors at low noise levels are amplified into visible artifacts, especially under reduced-precision inference. AnisoFlow is trained on the recovered-velocity error directly, so it converges to a well-conditioned target. It stays clean even under 8-bit inference.

Conceptually, the rotate-then-scale form of AnisoFlow’s recovery resembles rotation-based quantization methods. These methods operate layer-wise on an already trained network, whereas AnisoFlow learns the preconditioning end-to-end for the entire network.

INT8
FP8
FP8 blockwise
x-pred.
x-prediction, INT8 x-prediction, FP8 x-prediction, FP8 blockwise x-prediction, FP8 blockwise x-prediction, FP8 blockwise
AnisoFlow
AnisoFlow, INT8 AnisoFlow, FP8 AnisoFlow, FP8 blockwise AnisoFlow, FP8 blockwise AnisoFlow, FP8 blockwise

W8A8 inference with the same prompt and seed. \(x\)-prediction develops strong noise or patch seams, while AnisoFlow stays clean.

Citation

BibTeX
@article{zirvi2026anisotropic,
  title={Learning Anisotropic Prediction Targets for Flow Models},
  author={Rayhan Zirvi and Hansheng Chen and Gordon Wetzstein and Leonidas Guibas},
  journal={arXiv preprint arXiv:2611.17767},
  url={https://arxiv.org/abs/2611.17767},
  year={2026},
}