Add LoRA fork weight loading (pre-transformers-v5 base)#654
Open
Add LoRA fork weight loading (pre-transformers-v5 base)#654
Conversation
Adds three pieces needed for LocalBackend._experimental_fork_checkpoint to actually load the forked LoRA weights into the trainer: 1. UnslothState.load_lora_adapter — loads adapter_model.safetensors into the live peft model via set_peft_model_state_dict, replacing the freshly-initialized LoRA layers from from_pretrained. 2. UnslothService._forked_checkpoint_dir — stores the forked path so the first _train_dedicated / _train_shared call can apply it. 3. backend._experimental_fork_checkpoint — invalidates the _state cache after copytree, then records _forked_checkpoint_dir on the service. Built on 621e82b (pre-transformers-v5) because v5 introduces a bf16/fp16 mismatch in Unsloth's fused LoRA kernels that crashes every forward pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the pieces needed for `backend._experimental_fork_checkpoint` to actually
load the forked LoRA weights into the trainer (rather than just copying the
checkpoint directory and letting `from_pretrained` initialize a fresh LoRA).
Why the unusual base
This branch is based on commit `621e82b2` (last commit before the transformers-v5 upgrade in #629), not current main. On H200 + `load_in_4bit=True`, transformers v5 + Unsloth 2026.3.3 crash with `Half and BFloat16` in Unsloth's fused LoRA kernels on the first forward pass, before any rollouts. The v4 base avoids that.
Not expected to merge as-is — posting as a reference for the fork-weight-loading mechanics. Maintainers would likely want to:
Test plan
🤖 Generated with Claude Code