Scalarizer Tracker

[ValerianRey]

This issue tracks the candidate methods that could maybe be implemented as Scalarizer in torchjd.scalarization.

Status	Name	Ref	Stateful	Existing implementations	Special Remarks
🟡 In progress (#725)	IMTL-L	Towards Impartial Multi-Task Learning (ICLR 2021, 279 citations)	Yes, trainable state	official, LibMTL (IMTL-L + IMTL-G)	Almost equivalent to UW, up to a 1/2 factor and a negative sign on the scale
🔵 To investigate	FAMO	FAMO: Fast Adaptive Multitask Optimization (NeurIPS 2023, 124 citations)	?	official, LibMTL	Not sure this is a Scalarizer, need more investigation.
🔵 To investigate	GradNorm	GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks (ICML 2018, 2334 citations)	Yes, trainable state	unofficial, LibMTL	Not sure this is a Scalarizer, need more investigation.
🔵 To investigate	DWA (Dynamic Weight Averaging)	End-to-End Multi-Task Learning with Attention (CVPR 2019, 1836 citations)	?	official, LibMTL	Not sure this is a Scalarizer, need more investigation.

[ValerianRey]

List of other methods in LibMTL that we don't support yet and that are not listed here or in #665

• MOML (bilevel)
• AutoLambda (bilevel)
• Forum (bilevel)
• ExcessMTL (probably a scalarizer or an aggregator)

[ValerianRey]

We could take a look at LibMoon's scalarizers for ideas of scalarizers that we may have missed.

@ppraneth FYI

[ppraneth]

@ValerianRey Sure will start working on those too

[ValerianRey]

Nice! I think you could start by finding the information (as in the rows of the above table) about each method, so we can easily decide which ones are worth implementing.