Add VBench benchmarking suite and scripts

[jitendra-jalwaniya]

Overview

This PR introduces a standardized benchmarks/ framework to MaxDiffusion, starting with the VBench video-generation benchmark integration. It provides an end-to-end, two-stage evaluation pipeline tailored for Wan text-to-video (T2V) models: generating videos on Google Cloud TPU and evaluating quality dimensions on GPU.

---

Key Changes

1. Benchmark Framework & Documentation

• benchmarks/README.md: Establishes the organizational guidelines for adding runnable benchmark suites and evaluation metadata across models.
• benchmarks/vbench/README.md: Detailed workflow documentation, parameter references, and usage examples for running locally or remotely via SSH.

2. Dataset & Metadata

• benchmarks/vbench/prompts_110.txt: A curated 110-prompt downsampled subset from the VBench benchmark.
• benchmarks/vbench/VBench_full_info_sub110.json: Aligned evaluation metadata covering all relevant VBench quality and semantic dimensions corresponding 1-to-1 with the prompt file.

3. Generation & Evaluation Automation

• benchmarks/vbench/run_tpu_generation.sh:
  • Automates Wan video generation on TPU VMs (local execution or remote via --ssh).
  • Sets recommended inference configurations (e.g., Ulysses custom attention, KV caching, context/data parallelism, bfloat16, experimental scheduler).
  • Handles external disk mounting, Hugging Face caching and uploads results directly to GCS at gs://<bucket>/<run-name>/videos.
• benchmarks/vbench/run_gpu_eval.sh:
  • Automates VBench evaluation on GPU VMs (local or via --ssh).
  • Clones VBench, sets up dependencies, downloads video outputs from GCS, and runs dimension scoring.
  • Copies results back and uploads to gs://<bucket>/<run-name>/vbench_results (handles SSH credential passthrough without requiring GPU VM GCS service account scopes).
• benchmarks/vbench/gpu_eval_utils.py:
  • Helper CLI providing dimension extraction, automated upstream VBench patching (PyTorch version checking, distributed rank device mapping), and video naming/symlink formatting matching VBench expectations.

---

Workflow

[TPU VM] run_tpu_generation.sh
   └─> Generates Wan videos -> uploads to gs://<bucket>/<run-name>/videos
                                      │
                                      ▼
[GPU VM] run_gpu_eval.sh
   └─> Downloads videos -> evaluates VBench metrics -> uploads to gs://<bucket>/<run-name>/vbench_results

---

How to Test

1. TPU Generation

Direct on TPU or remote via SSH:

bash benchmarks/vbench/run_tpu_generation.sh --ssh \
  GCS_BUCKET=<your-bucket> \
  TPU_NAME=<tpu-vm-name> \
  RUN_NAME=wan-vbench-test

2. GPU Evaluation

Direct on GPU or remote via SSH:

bash benchmarks/vbench/run_gpu_eval.sh --ssh \
  GCS_BUCKET=<your-bucket> \
  GPU_NAME=<gpu-vm-name> \
  RUN_NAME=wan-vbench-test

3. Python Helper Verification

python3 benchmarks/vbench/gpu_eval_utils.py dimensions benchmarks/vbench/VBench_full_info_sub110.json

[github-actions]

e2e testgrid: https://8bcf50593faf4ea38060e236169827e5-dot-us-central1.composer.googleusercontent.com/dags/maxdiffusion_tpu_e2e/grid

[gemini-code-assist]

Code Review

This pull request introduces a VBench evaluation workflow for MaxDiffusion, specifically tailored for Wan text-to-video models. It adds TPU generation and GPU evaluation scripts, utility helpers, and a downsampled 110-prompt benchmark dataset. The feedback suggests improving the robustness and portability of the scripts by using POSIX-compliant disk space checks, always enforcing virtual environment creation to avoid PEP 668 issues, and explicitly specifying UTF-8 encoding when reading and writing files.

[jitendra-jalwaniya]

This is just a lint fix, unintentional change from last PR.

[mbohlool]

Here is a code review covering items 1 through 9 for the VBench benchmarking suite integration: