Challenge
Cross-Domain Few-Shot Learning (CD-FSL) Challenge
Figure 1: The cross-domain few-shot learning (CD-DSL) benchmark. ImageNet is used for source training, and domains of varying dissimilarity from ImageNet are used for target evaluation. No data is provided for meta-learning, and target classes are disjoint from the source classes.
LeaderBoard
Track 1: Cross-domain few-shot learning
1. Feature Transformation Ensemble Model with Batch Spectral Regularization for Cross-Domain Few-Shot Classification
https://arxiv.org/abs/2005.08463
Code: https://github.com/liubingyuu/FTEM_BSR_CDFSL
Bingyu Liu (DiDi); Zhen Zhao (DiDi); Zhenpeng Li (DiDi); Jianan Jiang (DiDi); Yuhong Guo (Didi Chuxing); Jieping Ye (Didi Chuxing)
Result: 73.94%
2. Cross-Domain Few-Shot Learning with Meta Fine-Tuning
https://arxiv.org/abs/2005.10544
Code: https://github.com/johncai117/Meta-Fine-Tuning
John Cai (Princeton University); Shengmei Shen (Pensees Pte Ltd)
Result: 73.78%
3. A Transductive Multi-Head Model for Cross-Domain Few-Shot Learning
Code: https://github.com/leezhp1994/TMHFS
Jianan Jiang (DiDi); Zhenpeng Li (DiDi); Yuhong Guo (Didi Chuxing); Jieping Ye (Didi Chuxing)
Result: 72.13%
4. Soft Pseudo-Label Teaching for Cross-Domain Few-shot Learning
Code: https://github.com/lynlynlyn96/cvprw2020-soft-pseudo-label-teaching
Yuning Lu (USTC); Yang Chen (University of Science and Technology of China); Yajing Liu (USTC); Xinmei Tian (USTC)
Result: 71.99%
5. Large Margin Mechanism and Pseudo Query Set on Cross-Domain Few-Shot Learning
https://arxiv.org/abs/2005.09218
Code: https://github.com/JiaFong/cvprw2020-cross-domain-few-shot-learning-challenge
Jia-Fong Yeh (National Taiwan University), Hsin-Ying Lee (National Taiwan University), Bing-Chen Tsai (National Taiwan University), Yi-Rong Chen (National Taiwan University), Ping-Chia Huang (National Taiwan University), Winston H. Hsu (National Taiwan University)
Result: 71.23%
6. Self-supervised learning in Cross-domain few-shot Learning
Da Chen (Alibaba Group), Feng Mao (Alibaba Group)
Result: 69.69%
7. Deep Transfer Meta-Learning
Code: https://github.com/Darkria8/Deep-Transfer-Meta-Learning
Yaoxing Wang (Ningxia University), Yu Xia (Ningxia University), Rongrong Du (Ningxia University)
Result: 67.38%
Track 2: Cross-domain few-shot learning with unlabeled data
1. Ensemble Model with Batch Spectral Regularization and Data Blending for Cross-Domain Few-Shot Learning with Unlabeled Data
Code: https://github.com/123zhen123/BSDB-CDFSL_2
Zhen Zhao (DiDi); Bingyu Liu (DiDi); Yuhong Guo (Didi Chuxing); Jieping Ye (Didi Chuxing)
Result: 73.38%
2. Self-supervised learning in Cross-domain few-shot Learning
Da Chen (Alibaba Group), Feng Mao (Alibaba Group)
Result: 70.69%
Challenge Submission Guidelines:
The Cross-Domain Few-Shot Learning (CD-FSL) challenge benchmark includes data from the CropDiseases [1], EuroSAT [2], ISIC2018 [3-4], and ChestX [5] datasets, which covers plant disease images, satellite images, dermoscopic images of skin lesions, and X-ray images, respectively. The selected datasets reflect real-world use cases for few-shot learning since collecting enough examples from above domains is often difficult, expensive, or in some cases not possible. In addition, they demonstrate the following spectrum of readily quantifiable domain shifts from ImageNet: 1) CropDiseases images are most similar as they include perspective color images of natural elements, but are more specialized than anything available in ImageNet, 2) EuroSAT images are less similar as they have lost perspective distortion, but are still color images of natural scenes, 3) ISIC2018 images are even less similar as they have lost perspective distortion and no longer represent natural scenes, and 4) ChestX images are the most dissimilar as they have lost perspective distortion, all color, and do not represent natural scenes. Example images from ImageNet and the proposed benchmark datasets are shown in Figure 1.
Participants are expected to run their own evaluations against the benchmark dataset according to the evaluation protocol, and submit the following 3 items:
1) Link to publicly accessible arXiv paper, minimum 2 pages and maximum 4 pages in length (including references) that describes the proposed method and the evaluation results.
2) Link to publicly accessible code on GitHub to reproduce all experiments (must also supply necessary models/resources as links).
3) Average accuracy across all tasks (determines challenge ranking).
>>NEW: There are now 2 tracks. See details below. One manuscript can be used to submit to both tracks. During submission, you must specify which tracks you are submitting to.
Submission Link: https://cmt3.research.microsoft.com/CDFSL2020/
Track 1: Cross-domain few-shot learning
In this track, participants perform few-shot evaluation using the benchmark as above (evaluation code provided below in "Evaluation Protocol"). However, for each dataset, participants may only use miniImageNet for learning. Particpants can use any resolution of images they wish, and any learning architecture.
Track 2: Cross-domain few-shot learning with unlabeled data
In this track, participants perform few-shot evaluation using the same benchmark code as below. However, for each dataset, participants may use the following image subsets (provided as text files listing images for each dataset below) as unlabeled data for un/self/semi-supervised learning. This learning can occur prior to the few-shot episodes, or during the few-shot episodes.
Participants must specify upon manuscript submission which tracks they are submitting to (can be both). A single manuscript can be used to describe both methods.
Text files listing images that can be used as unlabeled data:
ChestX: https://drive.google.com/open?id=1TnAWeKdUsWA8fXkpO8Pdenx3_Oz_sTWq
CropDisease: https://drive.google.com/open?id=1E5JnC2SLfKr1bmWESVtYHpVS6ViDhCZH
EuroSAT: https://drive.google.com/open?id=1Ldxg3NJMgWkTFnK2uLDo2XqScA6WgVnm
ISIC: https://drive.google.com/open?id=1c02C4OfDSjZjftMBopjEJlDEFxlHGrHy
Evaluation Protocol:
On all datasets in the evaluation benchmark, participants must report 5-way 5-shot, 5-way 20-shot, 5-way 50-shot evaluations of accuracy. For all evaluations, the test (query) set should have 15 images per class. For each evaluation, the same 600 randomly sampled few-shot episodes should be used for consistency, reporting the average accuracy and 95% confidence interval. Participants must employ the supplied code in order to conduct their evaluations. Additional details can be found in the supplied manuscript link below.
Data and Evaluation Code: https://github.com/IBM/cdfsl-benchmark
Important Dates:
Submission Deadline: May 20th, 11:59pm (PST) (NEW DATE)
Leaderboard Published / Invitations Sent: May 27th
References
[1] Sharada P Mohanty, David P Hughes, and Marcel Salathe. Using deep learning for image based plant disease detection. Frontiers in plant science, 7:1419, 2016
[2] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , 12(7):2217–2226, 2019.
[3] Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data, 5:180161, 2018.
[4] Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, et al. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). arXiv preprint. arXiv:1902.03368, 2019
[5] Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly supervised classification and localization of common thorax diseases. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2097–2106, 2017