Empowering Source-Free Domain Adaptation via MLLM-Guided Reliability-Based Curriculum Learning

Dongjie Chen¹^*, Kartik Patwari¹^*, Zhengfeng Lai¹, Xiaoguang Zhu¹, Sen-ching Cheung² Chen-Nee Chuah¹

¹UC Davis ²University of Kentucky
WACV 2026
^*Equal Contribution

Reliability-based Curriculum Learning (RCL)

Our proposed RCL framework uses multiple frozen multimodal large language models as zero-shot teachers to guide source-free domain adaptation. RCL estimates the reliability of pseudo-labels by measuring inter-teacher agreement and progressively trains the target model through a three staged curriculum, from highly reliable samples to least reliable ones. This cooperative learning strategy allows robust knowledge distillation without access to source data, achieving state-of-the-art (SOTA) performance across standard SFDA benchmarks.

Abstract

Existing SFDA methods struggle to fully use pre-trained knowledge and often rely on a single model’s predictions or handcrafted prompts, limiting robustness under domain shift. Multimodal Large Language Models (MLLMs) offer a promising alternative: they encode rich visual-semantic knowledge and generalize well without task-specific tuning. However, their use in SFDA is hindered by instructionfollowing failures, inconsistent outputs, and high inference costs. We propose Reliability-based Curriculum Learning (RCL), a novel framework that distills robust supervision from multiple frozen MLLMs into a compact target model. RCL organizes adaptation as a three-stage curriculum that progressively incorporates pseudo-labels based on intermodel agreement and model confidence, enabling stable and noise-aware training. Our approach achieves stateof-the-art performance on standard SFDA datasets, OfficeHome, DomainNet-126, and VisDA-C, outperforming zeroshot MLLMs, their ensembles, all without accessing source data or tuning foundation models.

Key Features

1. We propose the first SFDA framework that distills knowledge from multiple frozen MLLMs.

2. We introduce STS, a simple semantic matching method to convert open-ended MLLM outputs into class predictions.

3. We design RCL, a three-stage curriculum (RKT, SMKE, MMR) that incorporates pseudo-labels based on MLLM agreement and model confidence.

4. We achieve state-of-the-art results on Office-Home, DomainNet-126, and VisDA-C, outperforming CLIP-based and zero-shot MLLM baselines.

Main SFDA Benchmarks

RCL achieves state-of-the-art performance across standard SFDA benchmarks, including Office-Home and DomainNet, and across all adaptation settings.

Office-Home

Accuracy (%) on **Office-Home** dataset. SF: source-free. CP: uses CLIP prompting. ViT: ViT backbone. * indicates zero-shot.
Method	SF	CP	ViT	A→C	A→P	A→R	C→A	C→P	C→R	P→A	P→C	P→R	R→A	R→C	R→P	Avg.
Source	-	✘	✘	44.7	64.2	69.4	48.3	57.9	60.3	49.5	40.3	67.2	59.7	45.6	73.0	56.7
PADCLIP-RN	✘	✔	✘	57.5	84.0	83.8	77.8	85.5	84.7	76.3	59.2	85.4	78.1	60.2	86.7	76.6
ADCLIP-RN	✘	✔	✘	55.4	85.2	85.6	76.1	85.8	86.2	76.7	56.1	85.4	76.8	56.1	85.5	75.9
ELR	✔	✘	✘	58.4	78.7	81.5	69.2	79.5	79.3	66.3	58.0	82.6	73.4	59.8	85.1	72.6
PLUE	✔	✘	✘	49.1	73.5	78.2	62.9	73.5	74.5	62.2	48.3	78.6	68.6	51.8	81.5	66.9
C-SFDA	✔	✘	✘	60.3	80.2	82.9	69.3	80.1	78.8	67.3	58.1	83.4	73.6	61.3	86.3	73.5
PSAT-GDA	✔	✘	✔	73.1	88.1	89.2	82.1	88.8	88.9	83.0	72.0	89.6	83.3	73.7	91.3	83.6
DIFO-C-RN	✔	✔	✘	62.6	87.5	87.1	79.5	87.9	87.4	78.3	63.4	88.1	80.0	63.3	87.7	79.4
DIFO-C-B32	✔	✔	✔	70.6	90.6	88.8	82.5	90.6	88.8	80.9	70.1	88.9	83.4	70.5	91.2	83.1
CLIP-RN*	-	✔	✘	51.7	85.0	83.7	69.3	85.0	83.7	69.3	51.7	83.7	69.3	51.7	85.0	72.4
LLaVA-34B (w/ STS)*	-	✔	✔	78.3	93.7	89.5	87.0	93.7	89.5	87.0	78.3	89.5	87.0	78.3	93.7	87.2
InstBLIP-XXL (w/ STS)*	-	✔	✔	82.0	91.6	88.8	82.2	91.6	88.8	82.2	82.0	88.8	82.2	82.0	91.6	86.2
ShrGPT4V-13B (w/ STS)*	-	✔	✔	66.7	85.8	84.8	83.2	85.8	84.8	83.2	66.7	84.8	83.2	66.7	85.8	80.1
RCL (Ours)	✔	✘	✘	82.5	95.3	93.3	89.1	95.3	92.7	89.3	82.4	92.8	89.4	82.1	95.4	90.0
RCL-ViT (Ours)	✔	✘	✔	83.1	95.7	93.1	89.2	95.3	92.6	89.2	82.3	92.9	90.0	83.2	95.5	90.2

DomainNet

Accuracy (%) on **DomainNet**. SF: source-free. CP: uses CLIP prompting. ViT: ViT backbone. * indicates zero-shot performance.
Method	SF	CP	ViT	C→P	C→R	C→S	P→C	P→R	P→S	R→C	R→P	R→S	S→C	S→P	S→R	Avg.
Source	-	✘	✘	42.6	53.7	51.9	52.9	66.7	51.6	49.1	56.8	43.9	60.9	48.6	53.2	52.7
DAPL-RN	✘	✔	✘	72.4	87.6	65.9	72.7	87.6	65.6	73.2	72.4	66.2	73.8	72.9	87.8	74.8
ADCLIP-RN	✘	✔	✘	71.7	88.1	66.0	73.2	86.9	65.2	73.6	73.0	68.4	72.3	74.2	89.3	75.2
PLUE	✔	✘	✘	59.8	74.0	56.0	61.6	78.5	57.9	61.6	65.9	53.8	67.5	64.3	76.0	64.7
TPDS	✔	✘	✘	62.9	77.1	59.8	65.6	79.0	61.5	66.4	67.0	58.2	68.6	64.3	75.3	67.1
DIFO-C-RN	✔	✔	✘	73.8	89.0	69.4	74.0	88.7	70.1	74.8	74.6	69.6	74.7	74.3	88.0	76.7
DIFO-C-B32	✔	✔	✔	76.6	87.2	74.9	80.0	87.4	75.6	80.8	77.3	75.5	80.5	76.7	87.3	80.0
LLaVA-34B (w/ STS)*	-	✔	✔	84.4	91.0	83.7	85.5	91.0	83.7	85.5	84.4	83.7	85.5	84.4	91.0	86.1
InstBLIP-XXL (w/ STS)*	-	✔	✔	82.5	89.0	83.0	86.7	89.0	83.0	86.7	82.5	83.0	86.7	82.5	89.0	85.3
ShrGPT4V-13B (w/ STS)*	-	✔	✔	79.7	87.9	79.2	79.9	87.9	79.2	79.9	79.7	79.2	79.9	79.7	87.9	81.7
RCL (Ours)	✔	✘	✘	87.6	92.8	87.9	89.2	92.7	87.8	89.6	87.7	87.6	89.4	87.5	92.7	89.4
RCL-ViT (Ours)	✔	✘	✔	88.1	93.3	88.0	89.7	93.3	88.0	89.7	88.0	87.8	89.7	88.1	93.3	89.7

Ablation Studies

We present select key ablation studies to analyze the contribution of individual RCL components and the robustness under different settings. See paper for all ablation experiments.

RCL Component Ablation

RKT	SMKE	MMR	→A	→C	→P	→R	Avg.
✔	✘	✘	82.8	73.3	89.3	88.1	83.3
✔	✘	✔	87.7	80.2	93.3	92.0	88.3
✔	✔	✘	88.5	80.9	95.1	92.5	89.3
✔	✔	✔	89.3	82.3	95.3	92.9	90.0

Takeaway: Each module contributes complementary progressive improvements, and combining all three components yields the strongest performance.

RCL Without MLLMs

Method	→C	→P	→R	→A	Avg.
TPDS (A)	59.1	81.7	81.7	71.6	73.5
LCFD-C-B32 (B)	72.2	90.2	89.7	81.0	83.3
DIFO-C-B32 (C)	70.4	90.8	88.8	82.3	83.1
RCL (A,B,C)	71.9	90.7	89.2	81.7	83.4

Takeaway: Combining non-MLLM teachers yields comparitive but limited gains.

Effect of MLLM Teacher Configuration

Takeaway: RCL consistently outperforms individual MLLMs and their ensembles, demonstrating robustness across teacher choices and maintaining strong performance even with weaker teachers.

Impact of Number of MLLM Teachers

Takeaway: Multiple teachers improve supervision diversity over a single teacher, with performance saturating beyond three teachers.

BibTeX

@article{chen2024empowering,
  title={Empowering Source-Free Domain Adaptation via MLLM-Guided Reliability-Based Curriculum Learning},
  author={Chen, Dongjie and Patwari, Kartik and Lai, Zhengfeng and Zhu, Xiaoguang and Cheung, Sen-ching and Chuah, Chen-Nee},
  journal={arXiv preprint arXiv:2405.18376},
  year={2024}
}