聚类辅助特征对齐的域适应方法

doi:10.11896/j.issn.1002-137X.2019.03.033

摘要/Abstract

摘要： 有监督深度学习在有大量标记数据的领域可以取得不错的效果,但实际上很多领域只有大量未标记的数据。如何利用大量无标记数据,成为了深度学习发展的一个关键问题,领域自适应就是解决这一问题的一种有效方法。目前,基于对抗训练的域适应方法取得了较好的效果,这类方法利用领域分类损失对源域和目标域的特征分布进行对齐,降低了两个领域特征表示的分布差异,使采用源域数据训练的模型可以应用在目标域数据上。现有的域适应方法是在适配后的特征上进行模型训练的,没有充分利用目标域数据的原始信息,当两个领域差异较大时,会降低目标域特征的域内可鉴别性。针对现有方法的弱点,文中基于对抗判别域适应方法(ADDA),提出了一种基于对目标域数据聚类辅助特征对齐的域适应方法(CAFA-DA)。该方法通过聚类获得目标域数据伪标记,并在域适应阶段约束特征编码器训练,利用目标域数据的原始信息提高目标域特征的可鉴别性;将聚类和域适应两个过程训练的分类器进行集成学习,用高置信度样本进行训练,以提升模型的最终效果。CAFA-DA可用在任何基于对抗损失的领域自适应方法上。最后,在领域自适应的4个标准数据集上将CAFA-DA方法与目前几种先进的领域自适应方法进行了对比实验,结果表明:CAFA-DA方法的实验结果比其他几种方法都好;相对于ADDA方法,该方法在2个对比实验上的准确率分别提升了3.2%和17.2%。

关键词: 集成学习, 聚类, 特征表示, 特征对齐, 域适应

Abstract: Supervised deep learning can reach good results in the areas with large amounts of labeled data,but the rea-lity is that there are only a lot of unlabeled data in many areas.How to take advantages of large amounts of unlabeled data has become a key issue in the development of deep learning.Domain adaptation is an effective way to solve this problem.At present,domain adaptation methods based on adversarial training have achieved a good effect.This method uses domain classification loss to align the feature distribution of source domain,and target domain and reduce the difference of distribution between the feature representations of two domain,so the classifier trained with source domain data can be applied to target domain data.The existing domain adaptation method trains the model on the features after domain adaptation and does not make full use of the original information of the target domain data.When the differences between two domains are large,the intra-domain discriminability of target domain features will be reduced.In view of the disadvantages of the present methods,this paper proposed a method for clustering target domain data to assist feature alignment(CAFA-DA) based on the adversarial discriminative domain adaptation (ADDA).Pseudo-labels of target domain data are obtained by clustering and the feature encoder training is constrained in the domain adaptation stage,and the original information of the target domain data is used to improve the discriminability of target domain features.Classifiers trained in the two processes of clustering and domain adaptation are used for ensemble learning and high confidence samples are trained to improve the final effect of the model.The CAFA-DA can be applied to any domain adaption method based on adversarial loss.Finally,this paper compared CAFA-DA with several advanced domain adaption methods on four standard domain adaption data sets.The results show that the accuracy of the CAFA-DA method is better than other methods.Compared with the ADDA method,the results of two comparative experiments are improved by 3.2% and 17.2% respectively.

Key words: Clustering, Domain adaption, Ensemble learning, Feature alignment, Feature representation

中图分类号:

TP181

袁丁,王茜,邓李维. 聚类辅助特征对齐的域适应方法[J]. 计算机科学, 2019, 46(3): 221-226. https://doi.org/10.11896/j.issn.1002-137X.2019.03.033

YUAN Ding, WANG Qian, DENG Li-wei. Clustering Assist Feature Alignment for Unsupervised Domain Adaptation[J]. Computer Science, 2019, 46(3): 221-226. https://doi.org/10.11896/j.issn.1002-137X.2019.03.033

参考文献

[1]BEN-DAVID S,BLITZER J,CRAMMER K,et al.A theory of learning from different domains.Machine learning,2010,79(1-2):151-175.
[2]TZENG E,HOFFMAN J,SAENKO K,et al.Adversarial dis-
criminative domain adaptation[C]∥Computer Vision and Pattern Recognition (CVPR).IEEE,2017:4.
[3]TZENG E,HOFFMAN J,ZHANG N,et al.Deep Domain Confusion:Maximizing for Domain Invariance.arXiv:1412.3474,2014.
[4]LONG M,CAO Y,WANG J,et al.Learning transferable fea-
tures with deep adaptation networks.arXiv:1502.02791,2015.
[5]LONG M,ZHU H,WANG J,et al.Deep transfer learning with joint adaptation networks[C]∥Proceedings of the 34th International Conference on Machine Learning-Volume 70.2017:2208-2217.
[6]GRETTON A,BORGWARDT K M,RASCH M J,et al.A kernel two-sample test.Journal of Machine Learning Research,2012,13(1):723-773.
[7]GANIN Y,USTINOVA E,AJAKAN H,et al.Domain-adver-
sarial training of neural networks.The Journal of Machine Learning Research,2016,17(1):2096-2030.
[8]BOUSMALIS K,TRIGEORGIS G,SILBERMAN N,et al.Domain separation networks[C]∥Advances in Neural Information Processing Systems.IEEE,2016:343-351.
[9]GOODFELLOW I,POUGET-ABADIE J,MIRZA M,et al.Gene-
rative adversarial nets[C]∥Advances in Neural Information Processing Systems.IEEE,2014:2672-2680.
[10]GULRAJANI I,AHMED F,ARJOVSKY M,et al.Improved
training of wasserstein gans[C]∥Advances in Neural Information Processing Systems.IEEE,2017:5767-5777.
[11]SHU R,BUI H H,NARUI H,et al.A DIRT-T Approach to Unsupervised Domain Adaptation.arXiv:1802.08735,2018.
[12]TAKERU M,SHIN-ICHI M,SHIN I,et al.Virtual Adversarial Training:A Regularization Method for Supervised and Semi-Supervised Learning.IEEE Transactions on Pattern Analysis and Machine Intelligence,2018:1.
[13]SAITO K,USHIKU Y,HARADA T.Asymmetric tri-training for unsupervised domain adaptation.arXiv:1702.08400,2017.
[14]CHEN C,XIE W,XU T,et al.Progressive Feature Alignment for Unsupervised Domain Adaptation.arXiv:1811.08585,2018.
[15]KUMAR A,SATTIGERI P,WADHAWAN K,et al.Co-regularized Alignment for Unsupervised Domain Adaptation[C]∥Advances in Neural Information Processing Systems.IEEE,2018:9367-9378.
[16]GRANDVALET Y,BENGIO Y.Semi-supervised learning by entropy minimization[C]∥Advances in Neural Information Processing Systems.IEEE,2005:529-536.
[17]SHIMODAIRA H.Improving predictive inference under cova-
riate shift by weighting the log-likelihood function.Journal of statistical planning and inference,2000,90(2):227-244.
[18]MANSOUR Y,MOHRI M,ROSTAMIZADEH A.Domain ada-
ptation:Learning bounds and algorithms.arXiv:0902.3430,2009.
[19]CHEN X,DUAN Y,HOUTHOOFT R,et al.Infogan:Interpretable representation learning by information maximizingge-nerative adversarial nets[C]∥Advances in Neural Information Processing Systems.IEEE,2016:2172-2180.
[20]TZENG E,HOFFMAN J,DARRELL T,et al.Simultaneous
deep transfer across domains and tasks[C]∥Proceedings of the IEEE International Conference on Computer Vision.IEEE,2015:4068-4076.
[21]GANIN Y,LEMPITSKY V.Unsupervised domain adaptation by backpropagation.arXiv:1409.7495,2014.
[22]GHIFARY M,KLEIJN W B,ZHANG M,et al.Deep reconstruction-classification networks for unsupervised domain adaptation[C]∥European Conference on Computer Vision.Cham:Springer,2016:597-613.
[23]LIU M Y,TUZEL O.Coupled generative adversarial networks[C]∥Advances in Neural Information Processing Systems.IEEE,2016:469-477.
[24]BOUSMALIS K,SILBERMAN N,DOHAN D,et al.Unsupervised pixel-level domain adaptation with generative adversarial networks[C]∥The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).IEEE,2017:7.
[25]TAIGMAN Y,POLYAK A,WOLF L.Unsupervised cross-do-
main image generation.arXiv:1611.02200,2016.
[26]CHAPELLE O,ZIEN A.Semi-Supervised Classification by Low Density Separation[C]∥AISTATS.2005:57-64.
[27]DAI Z,YANG Z,YANG F,et al.Good semi-supervised learning that requires a bad gan[C]∥Advances in Neural Information Processing Systems.IEEE,2017:6510-6520.
[28]ZHOU Z H,LI M.Tri-training:Exploiting unlabeled data using three classifiers.IEEE Transactions on knowledge and Data Engineering,2005,17(11):1529-1541.
[29]CARON M,BOJANOWSKI P,JOULIN A,et al.Deep Clustering for Unsupervised Learning of Visual Features//Computer Vision-ECCV 2018.Cham:Springer,2018:139-156.
[30]KRIZHEVSKY A,SUTSKEVER I,HINTON G E.ImageNet Classification with Deep Convolutional Neural Networks[C]∥Advances in Neural Information Processing Systems.IEEE,2012:1097-1105.
[31]SRIVASTAVA N,HINTON G,KRIZHEVSKY A,et al.Dropout:a simple way to prevent neural networks from overfitting.The Journal of Machine Learning Research,2014,15(1):1929-1958.
[32]JOHNSON J,DOUZE M,JGOU H.Billion-scale similarity
search with gpus.arXiv:1702.08734,2017.
[33]LECUN Y,BOTTOU L,BENGIO Y,et al.Gradient-based
learning applied to document recognition.Proceedings of the IEEE,1998,86(11):2278-2324.
[34]ARBELAEZ P,MAIRE M,FOWLKES C,et al.Contour detection and hierarchical image segmentation.IEEE Transactions on Pattern Analysis and Machine Intelligence,2011,33(5):898-916.
[35]NETZER Y,WANG T,COATES A,et al.Reading digits in na-
tural images with unsupervised feature learning[C]∥NIPS Workshop on Deep Learning and Unsupervised Feature Lear-ning.2011:5.
[36]WANG Y,WANG Q,LV Q F,et al.An Improved K-means Algorithm Based on Initial Clustering Center Optimization and Weighted Between Dimension.Journal of Chongqing University of Technology(Natural Science),2013,27(4):77-80.(in Chinese)
王越,王泉,吕奇峰,等. 基于初始聚类中心优化和维间加权的改进K-means算法. 重庆理工大学学报(自然科学),2013,27(4):77-80.

相关文章 15

[1]	鲁晨阳, 邓苏, 马武彬, 吴亚辉, 周浩浩. 基于分层抽样优化的面向异构客户端的联邦学习 Federated Learning Based on Stratified Sampling Optimization for Heterogeneous Clients 计算机科学, 2022, 49(9): 183-193. https://doi.org/10.11896/jsjkx.220500263
[2]	柴慧敏, 张勇, 方敏. 基于特征相似度聚类的空中目标分群方法 Aerial Target Grouping Method Based on Feature Similarity Clustering 计算机科学, 2022, 49(9): 70-75. https://doi.org/10.11896/jsjkx.210800203
[3]	刘丽, 李仁发. 医疗CPS协作网络控制策略优化 Control Strategy Optimization of Medical CPS Cooperative Network 计算机科学, 2022, 49(6A): 39-43. https://doi.org/10.11896/jsjkx.210300230
[4]	林夕, 陈孜卓, 王中卿. 基于不平衡数据与集成学习的属性级情感分类 Aspect-level Sentiment Classification Based on Imbalanced Data and Ensemble Learning 计算机科学, 2022, 49(6A): 144-149. https://doi.org/10.11896/jsjkx.210500205
[5]	康雁, 吴志伟, 寇勇奇, 张兰, 谢思宇, 李浩. 融合Bert和图卷积的深度集成学习软件需求分类 Deep Integrated Learning Software Requirement Classification Fusing Bert and Graph Convolution 计算机科学, 2022, 49(6A): 150-158. https://doi.org/10.11896/jsjkx.210500065
[6]	鲁晨阳, 邓苏, 马武彬, 吴亚辉, 周浩浩. 基于DBSCAN聚类的集群联邦学习方法 Clustered Federated Learning Methods Based on DBSCAN Clustering 计算机科学, 2022, 49(6A): 232-237. https://doi.org/10.11896/jsjkx.211100059
[7]	郁舒昊, 周辉, 叶春杨, 王太正. SDFA:基于多特征融合的船舶轨迹聚类方法研究 SDFA:Study on Ship Trajectory Clustering Method Based on Multi-feature Fusion 计算机科学, 2022, 49(6A): 256-260. https://doi.org/10.11896/jsjkx.211100253
[8]	毛森林, 夏镇, 耿新宇, 陈剑辉, 蒋宏霞. 基于密度敏感距离和模糊划分的改进FCM算法 FCM Algorithm Based on Density Sensitive Distance and Fuzzy Partition 计算机科学, 2022, 49(6A): 285-290. https://doi.org/10.11896/jsjkx.210700042
[9]	陈景年. 一种适于多分类问题的支持向量机加速方法 Acceleration of SVM for Multi-class Classification 计算机科学, 2022, 49(6A): 297-300. https://doi.org/10.11896/jsjkx.210400149
[10]	陈佳舟, 赵熠波, 徐阳辉, 马骥, 金灵枫, 秦绪佳. 三维城市场景中的小物体检测 Small Object Detection in 3D Urban Scenes 计算机科学, 2022, 49(6): 238-244. https://doi.org/10.11896/jsjkx.210400174
[11]	王宇飞, 陈文. 基于DECORATE集成学习与置信度评估的Tri-training算法 Tri-training Algorithm Based on DECORATE Ensemble Learning and Credibility Assessment 计算机科学, 2022, 49(6): 127-133. https://doi.org/10.11896/jsjkx.211100043
[12]	韩红旗, 冉亚鑫, 张运良, 桂婕, 高雄, 易梦琳. 基于共同子空间分类学习的跨媒体检索研究 Study on Cross-media Information Retrieval Based on Common Subspace Classification Learning 计算机科学, 2022, 49(5): 33-42. https://doi.org/10.11896/jsjkx.210200157
[13]	邢云冰, 龙广玉, 胡春雨, 忽丽莎. 基于SVM的类别增量人体活动识别方法 Human Activity Recognition Method Based on Class Increment SVM 计算机科学, 2022, 49(5): 78-83. https://doi.org/10.11896/jsjkx.210400024
[14]	朱哲清, 耿海军, 钱宇华. 面向化学结构的线段聚类算法 Line-Segment Clustering Algorithm for Chemical Structure 计算机科学, 2022, 49(5): 113-119. https://doi.org/10.11896/jsjkx.210700131
[15]	张宇姣, 黄锐, 张福泉, 隋栋, 张虎. 基于菌群优化的近邻传播聚类算法研究 Study on Affinity Propagation Clustering Algorithm Based on Bacterial Flora Optimization 计算机科学, 2022, 49(5): 165-169. https://doi.org/10.11896/jsjkx.210800218

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed