FuRA: Full-Rank PEFT with Spectral Preconditioning

TL;DRFreeze each layer's pretrained block-wise SVD basis and train only a small core plus singular values. The update stays full-rank, is spectrally preconditioned, and costs the same as LoRA — yet beats Full FT on SFT, math RLVR and VLM tuning.

Avg. accuracy ↑ (%)

FuRA88.01

Full FT86.64

LoRA83.46

Step time ↓ (s/step)

FuRA0.046

Full FT0.201

LoRA0.043

Peak memory ↓ (GB)

FuRA22.7

Full FT76.9

LoRA26.8

LLaMA-3-8B, Commonsense-170K SFT, single H100. Accuracy bars span 60–90%.

Abstract

Both full fine-tuning and LoRA add weight updates without regard to the spectral structure established by pretraining, letting noisy gradients from a small fine-tuning set perturb robust pretrained features. We identify spectral preconditioning as the missing ingredient: reparameterizing each weight through its full-rank SVD and freezing one singular basis confines every update to the pretrained column space, and already outperforms unconstrained Full FT at the same parameter count.

To make this practical, FuRA (Full-Rank Adaptation) factorizes $\mathbf W$ with a block tensor-train decomposition $\mathbf W=\mathbf L\,\mathbf S\,\mathbf R$: the large core $\mathbf L$ is frozen at the pretrained block-wise SVD basis, and only the small core $\mathbf R$ and per-block singular values $\mathbf S$ are trained. One design choice gives full-rank preconditioning, full-rank update capacity, and LoRA-level parameter, time and memory cost. The 4-bit QFuRA also outperforms QLoRA.

Method

Full FT trains the dense W; FuRA splits W into column blocks W_k = L_k S_k R_k, freezes L and trains S and R, then merges back to a dense W' for inference — Each linear layer is losslessly re-expressed as $\mathbf W_k=\mathbf L_k\mathbf S_k\mathbf R_k$ via the full-rank SVD of each column block. $\mathbf L$ is frozen; $\mathbf S,\mathbf R$ are trained; cores merge back into one dense matrix after training (no inference overhead).

Why it works

One gradient step on block $k$ gives an effective update that is not plain SGD on $\mathbf W$:

$$\Delta\mathbf W_k=-\eta\,\underbrace{\mathbf U_k\,\mathrm{diag}(\mathbf S_k)^2\,\mathbf U_k^{\top}}_{\text{spectral preconditioner}}\,\mathbf g\mathbf x_k^{\top}\;+\;\mathbf U_k\,\mathrm{diag}(\Delta\mathbf S_k)\,\mathbf V_k^{\top}$$

Column-space projection. $\mathbf U_k\mathbf U_k^\top$ keeps every update inside the pretrained output subspace of that block.
Singular-value preconditioning. $\sigma^2$ scaling amplifies high-signal directions and damps the rest; trainable $\mathbf S$ decouples magnitude from direction.
Full-rank capacity. Across blocks the subspaces span $\mathrm{col}(\mathbf W)$, so $\mathrm{rank}(\Delta\mathbf W)$ is uncapped — no rank to tune.
Cheap. ~$d^{3/2}$ trainable parameters per layer (1.46% on LLaMA-3-8B); forward is one dense GEMM plus one batched GEMM.

Motivation: what Full FT does implicitly

(a) Full FT gradients are near the random baseline in col(U) while the weight stays at 0.9996. (b) Singular values barely change; singular vectors rotate broadly. (c) SVD FT beats Full FT on target and source accuracy — LLaMA-3-8B on Math-10K. (a) Gradients are spectrally diffuse, yet the fine-tuned weight stays in the pretrained column space. (b) Few singular values move; singular vectors rotate. (c) Enforcing this explicitly (SVD FT: freeze $\mathbf U$) beats Full FT on the target task and reduces forgetting on the source task.

(a) FuRA update effective rank is far above LoRA and below Full FT. (b) Heatmaps of R updates. (c) FuRA gets 71.6 target and 45.5 source accuracy versus Full FT 71.2/40.6 and LoRA 70.8/39.7 — (a) FuRA's update rank is far above LoRA's. (b) Updates concentrate on selected singular directions. (c) FuRA matches Full FT on target accuracy while preserving — and improving — source-domain accuracy.

Results

Commonsense reasoning SFT · 8-task avg.

Method	Trainable %	LLaMA-2-7B	LLaMA-3-8B
Full FT	100	83.53	86.64
LoRA (r=64)	1.39	81.25	83.46
DoRA (r=64)	1.42	81.80	85.04
PiSSA (r=128)	1.39	82.36	85.90
MiLoRA (r=64)	1.39	–	84.96
RandLoRA (r=64)	0.33	–	83.28
FuRA	1.46	84.41	88.01

Trainable % shown for LLaMA-3-8B (FuRA: 1.53% on LLaMA-2-7B). FuRA averaged over 3 seeds (std 0.09).

Math RLVR · GRPO on Qwen3-1.7B, 50 steps

Method	MATH-500	AMC23	AIME-24	AIME-25
Base	52.2	30.0	5.8	7.5
Full FT	61.5	46.7	13.2	14.6
LoRA	59.6	48.3	9.2	11.1
DoRA	62.3	45.0	13.6	14.7
PiSSA	49.1	30.8	2.2	4.7
MiLoRA	58.5	41.7	7.6	11.7
RandLoRA	62.2	55.0	13.8	18.3
FuRA	62.5	55.8	13.8	13.8

Mean over 3 seeds. AIME avg@8 at T=0.6; MATH-500/AMC23 greedy. FuRA also matches or beats Full FT on Qwen2.5-7B.

VLM instruction tuning · LLaVA-1.5-7B

Method	Params	7-task avg.
Full FT	100%	66.5
LoRA	4.61%	66.9
DoRA	4.63%	67.6
FuRA	1.37%	67.6

4-bit quantized fine-tuning

Method	LLaMA-3-8B CS avg.	LLaMA-3-70B GSM8K
QLoRA	83.89	81.27
QDoRA	86.34	81.80
QFuRA	87.30	83.78

Quick start

from fura import convert_linear_to_btt, configure_blocktt_trainability

convert_linear_to_btt(
    model,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    decomp_mode="output_one_block",   # freeze L (output subspace)
    rank="full",
    s_merged_to="keep_trainable",     # S as separate trainable vector
)
configure_blocktt_trainability(model, train_position="small")  # train R + S

Reproduction scripts for all four tracks (commonsense SFT, math RL, LLaVA, QFuRA) are in the repository.

BibTeX

@inproceedings{zhao2026fura,
  title     = {{FuRA}: Full-Rank Parameter-Efficient Fine-Tuning with Spectral Preconditioning},
  author    = {Zhao, Yequan and Zhang, Ruijie and Tan, Liyan and Moran, Niall and Qin, Tong and Zhang, Zheng},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026}
}