Computer Science ›› 2019, Vol. 46 ›› Issue (9): 291-297.doi: 10.11896/j.issn.1002-137X.2019.09.044

• Interdiscipline & Frontier • Previous Articles     Next Articles

Virtual Machine Placement Strategy with Energy Consumption Optimization under Reinforcement Learning

LU Hai-feng, GU Chun-hua, LUO Fei, DING Wei-chao, YUAN Ye, REN Qiang   

  1. (School of Information Science and Engineering,East China University of Science and Technology,Shanghai 200237,China)
  • Received:2018-08-28 Online:2019-09-15 Published:2019-09-02

Abstract: Although the rapid development of cloud data centers has brought very powerful computing power,the energy consumption problem has become increasingly serious.In order to reduce the energy consumption of physical servers in cloud data centers,firstly the virtual machine placement problem is modeled by reinforcement learning.Then,the Q-Learning(λ) algorithm is optimized from two aspects:state aggregation and time reliability.Finally,the virtual machine placement problem is simulated through cloud simulation platform CloudSim and actual data.The simulation results show that the optimized Q-Learning(λ) algorithm can effectively reduce the energy consumption of the cloud data center compared with the Greedy algorithm,PSO algorithm and Q-Learning algorithm,and can ensure better results for diffe-rent numbers of virtual machine placement requests.The proposed algorithm has strong practical value.

Key words: Cloud computing, Energy consumption optimization, Q-Learning(λ) algorithm, Reinforcement learning, Virtual machine placement

CLC Number: 

  • TP181
[1]GAI K,QIU M,ZHAO H,et al.Dynamic energy-aware cloudlet-based mobile cloud computing model for green computing[J].Journal of Network & Computer Applications,2016,59(C):46-54.
[2]HAMEED A,KHOSHKBARFOROUSHHA A,RANJAN R,et al.A survey and taxonomy on energy efficient resource allocation techniques for cloud computing systems[J].Computing,2016,98(7):751-774.
[3]GAI K,QIU M,ZHAO H.Cost-Aware Multimedia Data Allocation for Heterogeneous Memory Using Genetic Algorithm in Cloud Computing[J].IEEE Transactions on Cloud Computing,2016,PP(99):1-1.
[4]LINDBERG P,LEINGANG J,LYSAKER D,et al.Comparison and analysis of eight scheduling heuristics for the optimization of energy consumption and makespan in large-scale distributed systems[J].Journal of Supercomputing,2012,59(1):323-360.
[5]BELOGLAZOV A,ABAWAJY J,BUYYA R.Energy-aware resource alocation heuristics for eficient management of data centers for cloud computing[J].Future Generation Computer Systems,2012,28(5):755-768.
[6]GAO Y,GUAN H,QI Z,et al.A multi-objective ant colony system algorithm for virtual machine placement in cloud computing[J].Journal of Computer & System Sciences,2013,79(8):1230-1242.
[7]NEJAD M M,MASHAYEKHY L,GROSU D.Truthful GreedyMechanisms for Dynamic Virtual Machine Provisioning and Allocation in Clouds[J].IEEE Transactions on Parallel & Distri-buted Systems,2015,26(2):594-603.
[8]COUTINHO R D C,FROTA Y,OLIVEIRA D D.Optimizingvirtual machine allocation for parallel scientific workflows in federated clouds[J].Future Generation Computer Systems,2015,46(C):51-68.
[9]MAO H,ALIZADEH M,MENACHE I,et al.Resource Management with Deep Reinforcement Learning[C]//ACM Workshop on Hot Topics in Networks.ACM,2016:50-56.
[10]RUPASINGHE N,GÜVENÇ I.Reinforcement learning for licensed-assisted access of LTE in the unlicensed spectrum[C]//Wireless Communications and Networking Conference.IEEE,2015:1279-1284.
[11]SALEEM Y,YAU K L A,MOHAMAD H,et al.Clustering and Reinforcement-Learning-Based Routing for Cognitive Radio Networks[J].IEEE Wireless Communications,2017,24(4):146-151.
[12]MORADI M.A centralized reinforcement learning method formulti-agent job scheduling in Grid[C]//International Confe-rence on Computer and Knowledge Engineering.Mashhad:IEEE,2017.
[13]BOTVINICK M,WEINSTEIN A,SOLWAY A,et al.Rein-forcement learning,efficient coding,and the statistics of natural tasks[J].Current Opinion in Behavioral Sciences,2015,5:71-77.
[14]ZHENG Q,LI R,LI X,et al.A Multi-Objective BiogeographyBased Optimization for Virtual Machine Placement[C]//2015 15th IEEE/ACM International Symposium on Cluster,Cloud and Grid Computing.Shenzhen:IEEE,2015:687-696.
[15]YOU C,HUANG K,CHAE H,et al.Energy-Efficient Resource Allocation for Mobile-Edge Computation Offloading[J].IEEE Transactions on Wireless Communications,2017,16(3):1397-1411.
[16]GAI K,QIU M.Optimal resource allocation using reinforcement learning for IoT content-centric services [J].Applied Soft Computing,2018,70:12-21.
[17]KUMAR M,YADAV A K,KHATRI P,et al.Global host allocation policy for virtual machine in cloud computing[J].International Journal of Information Technology,2018,10(3):279-287.
[18]SANTRA S,MALI K.A new approach to survey on load balancing in VM in cloud computing:Using CloudSim[C]//International Conference on Computer,Communication and Control.IEEE,2016:1-5.
[19]DUONG T,CHU Y J,NGUYEN T,et al.Virtual MachinePlacement via Q-Learning with Function Approximation[C]//IEEE Global Communications Conference.San Diego:IEEE,2015:1-6.
[20]HABIB A,KHAN M I.Reinforcement learning based autonomic virtual machine management in clouds[C]//International Conference on Informatics,Electronics and Vision.Univ Dhaka:IEEE,2016:1083-1088.
[21]XU ZX,et al.Deep Reinforcement Learning with Sarsa and Q-Learning:A Hybrid Approach[J].IEICE Transactions on Information and Systems,2018,E101d(9):2315-2322.
[22]TENG L,BIN T,YUN A,et al.Parallel reinforcement learning:a framework and case study[J].IEEE/CAA Journal of Automatica Sinica,2018,5(4):827-835.
[23]NISHIYAMA R,YAMADA S.Reinforcement Learning withMultiple Actions[C]//Proceedings of the 3rd International Conference on Intelligent Technologies and Engineering Systems.New York:Springer2016:207-213.
[24]HOMEM T P D,PERICO D H,SANTOS P E,et al.Improving Reinforcement Learning Results with Qualitative Spatial Representation[C]//Brazilian Conference on Intelligent Systems.Brazil:IEEE,2017:151-156.
[25]DUAN Y,CHEN X,HOUTHOOFT R,et al.Benchmarkingdeep reinforcement learning for continuous control[C]//International Conference on International Conference on Machine Learning.New York:ACM,2016:1329-1338.
[26]LITTMAN M L.Reinforcement learning improves behaviourfrom evaluative feedback[J].Nature,2015,521(7553):445-451.
[27]THERRIEN A S,WOLPERT D M,BASTIAN A J.Effectivereinforcement learning following cerebellar damage requires a balance between exploration and motor noise[J].Brain,2016,139(1):101-114.
[28]CUTLER M,WALSH T J,HOW J P.Real-World Reinforcement Learning via Multifidelity Simulators[J].IEEE Transactions on Robotics,2017,31(3):655-671.
[29]LEONG Y C,RADULESCU A,DANIEL R,et al.Dynamic Interaction between Reinforcement Learning and Attention in Multidimensional Environments[J].Neuron,2017,93(2):451-463.
[30]KIM B G,ZHANG Y,SCHAAR M V D,et al.Dynamic Pricing and Energy Consumption Scheduling With Reinforcement Learning[J].IEEE Transactions on Smart Grid,2016,7(5):2187-2198.
[31]XIONG R,CAO J,YU Q.Reinforcement learning-based real-time power management for hybrid energy storage system in the plug-in hybrid electric vehicle[J].Applied Energy,2018,211:538-548.
[32] SAMBROOK T D,GOSLIN J.Principal Components Analysis of Reward Prediction Errors in a Reinforcement Learning Task[J].Neuroimage,2016,124(Pt A):276-286.
[33]CHEN H,LI X,ZHAO F.A Reinforcement Learning-BasedSleep Scheduling Algorithm for Desired Area Coverage in Solar-Powered Wireless Sensor Networks[J].IEEE Sensors Journal,2016,16(8):2763-2774.
[1] LIU Xing-guang, ZHOU Li, LIU Yan, ZHANG Xiao-ying, TAN Xiang, WEI Ji-bo. Construction and Distribution Method of REM Based on Edge Intelligence [J]. Computer Science, 2022, 49(9): 236-241.
[2] SHI Dian-xi, ZHAO Chen-ran, ZHANG Yao-wen, YANG Shao-wu, ZHANG Yong-jun. Adaptive Reward Method for End-to-End Cooperation Based on Multi-agent Reinforcement Learning [J]. Computer Science, 2022, 49(8): 247-256.
[3] YUAN Wei-lin, LUO Jun-ren, LU Li-na, CHEN Jia-xing, ZHANG Wan-peng, CHEN Jing. Methods in Adversarial Intelligent Game:A Holistic Comparative Analysis from Perspective of Game Theory and Reinforcement Learning [J]. Computer Science, 2022, 49(8): 191-204.
[4] YU Bin, LI Xue-hua, PAN Chun-yu, LI Na. Edge-Cloud Collaborative Resource Allocation Algorithm Based on Deep Reinforcement Learning [J]. Computer Science, 2022, 49(7): 248-253.
[5] LI Meng-fei, MAO Ying-chi, TU Zi-jian, WANG Xuan, XU Shu-fang. Server-reliability Task Offloading Strategy Based on Deep Deterministic Policy Gradient [J]. Computer Science, 2022, 49(7): 271-279.
[6] GUO Yu-xin, CHEN Xiu-hong. Automatic Summarization Model Combining BERT Word Embedding Representation and Topic Information Enhancement [J]. Computer Science, 2022, 49(6): 313-318.
[7] FAN Jing-yu, LIU Quan. Off-policy Maximum Entropy Deep Reinforcement Learning Algorithm Based on RandomlyWeighted Triple Q -Learning [J]. Computer Science, 2022, 49(6): 335-341.
[8] XIE Wan-cheng, LI Bin, DAI Yue-yue. PPO Based Task Offloading Scheme in Aerial Reconfigurable Intelligent Surface-assisted Edge Computing [J]. Computer Science, 2022, 49(6): 3-11.
[9] HONG Zhi-li, LAI Jun, CAO Lei, CHEN Xi-liang, XU Zhi-xiong. Study on Intelligent Recommendation Method of Dueling Network Reinforcement Learning Based on Regret Exploration [J]. Computer Science, 2022, 49(6): 149-157.
[10] ZHANG Jia-neng, LI Hui, WU Hao-lin, WANG Zhuang. Exploration and Exploitation Balanced Experience Replay [J]. Computer Science, 2022, 49(5): 179-185.
[11] LI Peng, YI Xiu-wen, QI De-kang, DUAN Zhe-wen, LI Tian-rui. Heating Strategy Optimization Method Based on Deep Learning [J]. Computer Science, 2022, 49(4): 263-268.
[12] OUYANG Zhuo, ZHOU Si-yuan, LYU Yong, TAN Guo-ping, ZHANG Yue, XIANG Liang-liang. DRL-based Vehicle Control Strategy for Signal-free Intersections [J]. Computer Science, 2022, 49(3): 46-51.
[13] ZHOU Qin, LUO Fei, DING Wei-chao, GU Chun-hua, ZHENG Shuai. Double Speedy Q-Learning Based on Successive Over Relaxation [J]. Computer Science, 2022, 49(3): 239-245.
[14] LI Su, SONG Bao-yan, LI Dong, WANG Jun-lu. Composite Blockchain Associated Event Tracing Method for Financial Activities [J]. Computer Science, 2022, 49(3): 346-353.
[15] GAO Shi-yao, CHEN Yan-li, XU Yu-lan. Expressive Attribute-based Searchable Encryption Scheme in Cloud Computing [J]. Computer Science, 2022, 49(3): 313-321.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!