PURSUhInT: In Search of Informative Hint Points Based on Layer Clustering for Knowledge Distillation

Schematic of the PURSUhInT workflow in three steps, drawn from the paper's Figure 1 caption. This is a schematic, not the paper's figure.

Abstract

One of the most efficient methods for model compression is hint distillation, where the student model is injected with information (hints) from several different layers of the teacher model. Although the selection of hint points can drastically alter the compression performance, conventional distillation approaches overlook this fact and use the same hint points as in the early studies. Therefore, we propose a clustering based hint selection methodology, where the layers of teacher model are clustered with respect to several metrics and the cluster centers are used as the hint points. Our method is applicable for any student network, once it is applied on a chosen teacher network. The proposed approach is validated in CIFAR-100 and ImageNet datasets, using various teacher-student pairs and numerous hint distillation methods. Our results show that hint points selected by our algorithm results in superior compression performance compared to state-of-the-art knowledge distillation algorithms on the same student models and datasets.

Method

Acquire layer representations

Feature maps for a number of samples are obtained from all sub-blocks of the pre-trained teacher and averaged to form matrices.

Cluster the layers

Layer representations are clustered with k-means; one hint point per cluster, at the cluster center, removes redundancy.

Distill with the chosen hints

The student is distilled with the determined hint positions, for any student network once applied on a chosen teacher.

Resources

Preprint: arXiv:2103.00053
Journal: ScienceDirect
DOI: 10.1016/j.eswa.2022.119040

BibTeX

@article{keser2023pursuhint,
  title = {{PURSUhInT}: In Search of Informative Hint Points Based on Layer Clustering for Knowledge Distillation},
  author = {Keser, Reyhan Kevser and Ayanzadeh, Aydin and Aghdam, Omid Abdollahi and Kilcioglu, Caglar and Toreyin, Behcet Ugur and Ure, Nazim Kemal},
  journal = {Expert Systems with Applications},
  volume = {213},
  pages = {119040},
  year = {2023},
  doi = {10.1016/j.eswa.2022.119040},
  url = {https://doi.org/10.1016/j.eswa.2022.119040},
  eprint = {2103.00053},
  archivePrefix = {arXiv},
  primaryClass = {cs.LG}
}