Repository navigation
Add an RFD4-Proteina LoRA training example #15155
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
sayakpaul
merged 14 commits into
huggingface:main
from
kierandidi:rfd4-proteina-lora-example
Nov 19, 2026
+915
−0
Merged
Changes from all commits
Commits
Show all changes
14 commits
Select commit
Hold shift + click to select a range
024d284
Add an RFD4-Proteina LoRA training example
kierandidi 2c0be9a
Follow the established LoRA save pattern
kierandidi 1f0f41e
Make RFD4 LoRA data preparation and checkpointing distributed-safe
kierandidi f1efe9a
Trim RFD4 example docs and keep focused dataset tests
kierandidi 838b370
Merge branch 'main' into rfd4-proteina-lora-example
kierandidi ca024cf
Address RFD4 example follow-up review
kierandidi 3376408
Show held-out validation for the RFD4 LoRA example
kierandidi c096196
Remove benchmark results from the example README
kierandidi d6edaca
Cyclic peptide check
kierandidi 552328a
Write a model card and support pushing to the Hub
kierandidi d0a695f
Add Hub support to the RFD4 LoRA example
kierandidi 82d34b5
Align RFD4 LoRA example conventions
kierandidi 59579a1
Merge branch 'main' into rfd4-proteina-lora-example
kierandidi 8fd30f6
Sync with main and fix LoRA inference and model card generation
kierandidi File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,176 @@ | ||
| # LoRA fine-tuning for RFD4-Proteina | ||
|
|
||
| This example fine-tunes [RFD4-Proteína](https://huggingface.co/docs/diffusers/main/en/api/pipelines/rfd4_proteina) | ||
| with LoRA on annotated CIFs from a local folder or a Hub dataset. The autoencoder stays frozen; only the denoiser adapters train. | ||
| Saved adapters load with `RFD4ProteinaPipeline.load_lora_weights`. | ||
|
|
||
| ## Setup | ||
|
|
||
| Install Diffusers from source, then this example's requirements: | ||
|
|
||
| ```bash | ||
| pip install -e . | ||
| pip install -r examples/rfd4_proteina/requirements.txt | ||
| ``` | ||
|
|
||
| Training reads structures as annotated CIF files, which are turned into model features by the | ||
| [`rfd4-proteina-prep`](https://huggingface.co/nvidia/RFD4-Proteina-prep) package — the same preprocessing the pipeline's | ||
| inference examples use. It builds on AtomWorks and is not part of Diffusers: | ||
|
|
||
| ```bash | ||
| pip install git+https://huggingface.co/nvidia/RFD4-Proteina-prep@c9959efdebd75ac7fe9129fbbf0e291a6e0487d5 | ||
| ``` | ||
|
|
||
| Initialize an [🤗 Accelerate](https://github.com/huggingface/accelerate) environment with: | ||
|
|
||
| ```bash | ||
| accelerate config | ||
| ``` | ||
|
|
||
| Or write the default configuration without answering questions: | ||
|
|
||
| ```bash | ||
| accelerate config default | ||
| ``` | ||
|
|
||
| ## Training | ||
|
|
||
| Point `--instance_data_dir` at a directory of CIF files; it is searched recursively, so a nested layout such as | ||
| PLINDER's `systems/<shard>/<system>/<system>.cif` works as it is. | ||
|
|
||
| The model repository ships the small PLINDER subset the released fine-tuning tutorial uses, fifty systems in about a | ||
| megabyte. Download and unpack it: | ||
|
|
||
| ```python | ||
| import tarfile | ||
|
|
||
| from huggingface_hub import hf_hub_download | ||
|
|
||
| archive = hf_hub_download("nvidia/RFD4-Proteina", "test_dataset/test_dataset.tar.gz") | ||
| with tarfile.open(archive) as tar: | ||
| tar.extractall("test_dataset") | ||
| ``` | ||
|
|
||
| That writes the structures under `test_dataset/cifs`. Train on them with: | ||
|
|
||
| ```bash | ||
| accelerate launch examples/rfd4_proteina/train_rfd4_proteina_lora.py \ | ||
| --pretrained_model_name_or_path nvidia/RFD4-Proteina-Diffusers \ | ||
| --instance_data_dir test_dataset/cifs \ | ||
| --features_cache_dir feature-cache \ | ||
| --output_dir rfd4-proteina-lora \ | ||
| --mixed_precision bf16 \ | ||
| --rank 32 --lora_alpha 64 \ | ||
| --learning_rate 5e-5 \ | ||
| --gradient_accumulation_steps 2 \ | ||
| --max_train_steps 500 \ | ||
| --seed 0 | ||
| ``` | ||
|
|
||
| To upload the final adapter and its model card, log in with `hf auth login` and add | ||
| `--push_to_hub --hub_model_id your-username/rfd4-proteina-lora`. Authentication also supports `HF_TOKEN` or `--hub_token` to override the logged-in account. | ||
| For W&B logging, use `hf auth login` or `HF_TOKEN` rather than passing `--hub_token`. | ||
| Only the final adapter and model card are uploaded. Every run writes a model card to `--output_dir`, even without upload. | ||
|
|
||
| For a Hub dataset, replace `--instance_data_dir` with `--dataset_name your-username/annotated-structures`. | ||
| Use `--dataset_config_name`, `--dataset_split` (default `train`), `--dataset_revision` and `--structure_column` | ||
| (default `structure`) to select the data. The column must contain annotated `.cif`, `.cif.gz` or `.cif.zst` files | ||
| stored with `AtomicStructure`. Original file bytes are cached without decoding so conditioning annotations are preserved; | ||
| ordinary unannotated structures still need the preparation described above. | ||
|
|
||
| `AtomicStructure` is currently in [Datasets PR #8895](https://github.com/huggingface/datasets/pull/8895). | ||
| Until it is released, `--dataset_name` requires this revision; local CIF training does not require Datasets: | ||
|
|
||
| ```bash | ||
| pip install git+https://github.com/huggingface/datasets.git@9ddf85d4fbd0aacba0b04911b12ee8527103c23a | ||
| ``` | ||
|
|
||
| Training uses rank 32 / alpha 64 adapters on `to_q`, `to_k`, `to_v` and `to_out`, following the released recipe. | ||
| The flow-matching loss scores the predicted clean backbone and local latents. It does not include the native | ||
| trainer's auxiliary sequence and insertion losses. | ||
|
|
||
| - Each batch holds one structure. Use `--gradient_accumulation_steps` to increase the effective batch size. | ||
| - Structures above `--max_tokens` (384 by default), rejected inputs, and structures without resolved target CA atoms | ||
| are filtered before training. Missing atoms are masked before encoding. | ||
| - `--features_cache_dir` avoids preparing the CIFs again on later runs. Without it, features stay in CPU memory. | ||
| - `--gradient_checkpointing` saves GPU memory. `--recycle_depth` controls trunk passes per training step (default 1). | ||
|
|
||
| For two GPUs, add `--multi_gpu --num_processes 2` to `accelerate launch`. | ||
| Resume with `--resume_from_checkpoint latest`, keeping the same output directory and LoRA configuration. | ||
| Use `--checkpointing_steps` and `--checkpoints_total_limit` to control checkpoint frequency and retention. | ||
|
|
||
| Pass `--validation_cif` to generate a structure every `--validation_steps`. Validation runs on the main process, | ||
| with dropout disabled, and writes a CIF next to the adapter. It uses the checkpoint's full sampling schedule. | ||
|
|
||
| ### Cyclic peptide check | ||
|
|
||
| Trained on 255 CPSea complexes for 1,000 steps on two H200s, starting from weights matched to the updated native | ||
| release. We tested the [PD-1 cyclic peptide inputs](https://github.com/RosettaCommons/RFD4-Proteina/pull/82) with the | ||
| same 16 seeds per chemistry at steps 0, 500 and 1,000. PD-1 was excluded from fine-tuning. | ||
|
|
||
|  | ||
|
|
||
| Median contacts (heavy-atom pairs within 5 Å) rose from 187.5→229.5, 150.5→262 and 160.5→220 for head-to-tail, | ||
| disulfide and isopeptide designs. Contacts per peptide atom also increased. Shading shows 95% seed-bootstrap intervals; | ||
| loss uses EMA α=0.02 over faint raw values. | ||
|
|
||
| This is one target and one training run. Head-to-tail closure stayed at 16/16, but disulfide fell from 12/16 to 6/16 | ||
| and isopeptide from 5/16 to 1/16. More contacts alone don't establish better binding or overall design quality. | ||
|
|
||
| ## Inference | ||
|
|
||
| The saved adapter loads into the pipeline: | ||
|
|
||
| ```python | ||
| import torch | ||
| from diffusers import RFD4ProteinaPipeline | ||
| from rfd4_proteina_prep import prepare_cif, write_cif | ||
|
|
||
| pipe = RFD4ProteinaPipeline.from_pretrained("nvidia/RFD4-Proteina-Diffusers", dtype=torch.bfloat16).to("cuda") | ||
| pipe.load_lora_weights("rfd4-proteina-lora") | ||
| pipe.transformer.eval() | ||
|
|
||
| features, atom_array = prepare_cif("input.cif", seed=0, device=pipe.device) | ||
| named = ( | ||
| "padding_mask", | ||
| "is_in_context_conditioning", | ||
| "supports_local_latent", | ||
| "token_type", | ||
| "atom37_mask", | ||
| "coords_nm", | ||
| ) | ||
| output = pipe( | ||
| **{name: features[name] for name in named}, | ||
| conditioning_features={name: value for name, value in features.items() if name not in named}, | ||
| generator=torch.Generator(device=pipe.device).manual_seed(0), | ||
| ) | ||
| write_cif(output, features, atom_array, "design.cif") | ||
| ``` | ||
|
|
||
| The [released native adapter](https://huggingface.co/nvidia/RFD4-Proteina/tree/main/adapter) is one example you can | ||
| convert. Download it first: | ||
|
|
||
| ```python | ||
| from huggingface_hub import snapshot_download | ||
|
|
||
| snapshot_download("nvidia/RFD4-Proteina", allow_patterns=["adapter/*"], local_dir="rfd4-native") | ||
| ``` | ||
|
|
||
| Conversion splits the fused `to_qkv` weights into `to_q`, `to_k` and `to_v`: | ||
|
|
||
| ```bash | ||
| python scripts/convert_rfd4_proteina_to_diffusers.py \ | ||
| --adapter \ | ||
| --source rfd4-native/adapter \ | ||
| --destination rfd4-proteina-lora-converted | ||
| ``` | ||
|
|
||
| This adapter goes with the `pretrain-base` revision. The default revision already has it merged: | ||
|
|
||
| ```python | ||
| pipe = RFD4ProteinaPipeline.from_pretrained( | ||
| "nvidia/RFD4-Proteina-Diffusers", revision="pretrain-base", dtype=torch.bfloat16 | ||
| ).to("cuda") | ||
| pipe.load_lora_weights("rfd4-proteina-lora-converted") | ||
| pipe.transformer.eval() | ||
| ``` | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| accelerate>=1.0.0 | ||
| peft>=0.14.0 | ||
| safetensors>=0.4.5 | ||
| torch>=2.4.0 | ||
| tensorboard | ||
| Jinja2 |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.