Computer Science

Computer Science ›› 2026, Vol. 53 ›› Issue (8): 71-84.doi: 10.11896/jsjkx.250800093

• Database & Big Data & Data Science • Previous Articles     Next Articles

Adaptive Frequency Tuning Approach for Spark Clusters

HE Yulin1, XIAO Youqi1,2, YANG Zhenyu1,2, HUANG Zhexue1, CUI Laizhong2   

  1. 1 Guangdong Laboratory of Artificial Intelligence and Digital Economy(Shenzhen), Shenzhen, Guangdong 518107, China
    2 College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, Guangdong 518060, China
  • Received:2025-08-20 Revised:2025-12-03 Online:2026-08-15 Published:2026-08-17
  • About author:HE Yulin,born in 1982,Ph.D,research fellow,doctoral supervisor.His main research interests include novel theoretical models for big data,supporting technologies for big data-oriented software systems,machine learning algorithms for big data,complex big data application systems,etc.
    XIAO Youqi,born in 2001,postgra-duate.His main research interests include distributed computing for big data,Spark energy consumption optimization,and high-performance data mi-ning.
  • Supported by:
    Guangdong Basic and Applied Basic Research Foundation(2023B1515120020) and Science and Technology Major Project of Shenzhen(KJZD20230923114809020).

PDF (PC)

69

Abstract: Data center energy efficiency represents a critical challenge in green computing.Distributed big data frameworks such as Spark consistently exhibit high energy consumption due to their in-memory computation architecture.Current data centers face a structural imbalance between resource supply and dynamic demand:daily workload fluctuations frequently lead to hardware idling during low-activity periods,resulting in substantial resource wastage.Moreover,uneven load distribution across cluster nodes contributes to resource fragmentation and underutilization.Existing energy optimization methods for clusters often suffer from coarse control granularity,heavy reliance on historical data,and limited adaptability to heterogeneous workloads,leading to constrained effectiveness and high operational costs.To address these limitations,this paper introduces a deep reinforcement lear-ning-based approach for cluster energy efficiency optimization.The proposed method constructs an Actor-Critic agent(PPO-Agent) using the proximal policy optimization algorithm,achieves stable convergence through policy entropy regularization,and leverages dynamic voltage and frequency scaling(DVFS) commands to precisely align hardware resources with real-time job requirements.This enables more effective reduction of overall cluster energy consumption.Experimental evaluations under the HiBench benchmark on Spark-on-YARN clusters demonstrate that PPOAgent achieves energy savings of up to 9.7% and 12.9% compared to FAESS-DVFS and GRMSSY,respectively,while consistently meeting service level agreements.Furthermore,PPOAgent reduces execution time by up to 30.3% and 49.7%,respectively,and significantly improves task completion rates under high-load conditions.These results confirm that the proposed method successfully achieves synergistic optimization of energy efficiency and performance across diverse workloads and dynamic environments,demonstrating considerable practical value.

Key words: Big data computing, Apache Spark, Deep reinforcement learning, DVFS, Energy consumption modeling

CLC Number: 

  • TP302.1
[1] VAVILAPALLI V K,MURTHY A C,DOUGLAS C,et al.Apache hadoop yarn:Yet another resource negotiator[C]//Proceedings of the 4th Annual Symposium on Cloud Computing.2013:1-16.
[2] ZAHARIA M,XIN R S,WENDELL P,et al.Apache Spark:a unified engine for big data processing[J].Communications of the ACM,2016,59(11):56-65.
[3] TOSHNIWAL A,TANEJA S,SHUKLA A,et al.Storm@twitter[C]//Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data.2014:147-156.
[4] MASANET E,SHEHABI A,LEI N,et al.Recalibrating global data center energy-use estimates[J].Science,2020,367(6481):984-986.
[5] ZAHARIA M,CHOWDHURY M,DAS T,et al.Resilient distributed datasets:A {Fault-Tolerant} abstraction for {In-Memory} cluster computing[C]//9th USENIX Symposium on Networked Systems Design and Implementation(NSDI 12).2012:15-28.
[6] YOUSEFI M H N,GOUDARZI M.A task-based greedy sche-duling algorithm for minimizing energy of mapreduce jobs[J].Journal of Grid Computing,2018,16:535-551.
[7] DEAN J,GHEMAWAT S.MapReduce:simplified data proces-sing on large clusters[J].Communications of the ACM,2008,51(1):107-113.
[8] VEIGA J,EXPÓSITO R R,PARDO X C,et al.Performanceevaluation of big data frameworks for large-scale data analytics[C]//2016 IEEE International Conference on Big Data(Big Data).IEEE,2016:424-431.
[9] SHI W,LI H,GUAN J,et al.Energy-efficient scheduling algo-rithms based on task clustering in heterogeneous Spark clusters[J].Parallel Computing,2022,112:102947.
[10] DAYARATHNA M,WEN Y,FAN R.Data center energy consumption modeling:A survey[J].IEEE Communications Surveys & Tutorials,2015,18(1):732-794.
[11] GOIRI Í,LE K,NGUYEN T D,et al.Greenhadoop:leveraginggreen energy in data-processing frameworks[C]//Proceedings of the 7th ACM European Conference on Computer Systems.2012:57-70.
[12] YING Y,BIRKE R,WANG C,et al.Optimizing energy,locality and prI/Ority in a mapreduce cluster[C]//2015 IEEE International Conference on Autonomic Computing.IEEE,2015:21-30.
[13] ISLAM M T,WU H,KARUNASEKERA S,et al.SLA-based scheduling of Spark jobs in hybrid cloud computing environments[J].IEEE Transactions on Computers,2021,71(5):1117-1132.
[14] HANIF M,KIM E,HELAL S,et al.SLA-based adaptationschemes in distributed stream processing engines[J].Applied Sciences,2019,9(6):1045.
[15] SHABESTARI F,NAVIMIPOUR N J.An energy-aware re-source management strategy based on Spark and yarn in heterogeneous environments[J].IEEE Transactions on Green Communications and Networking,2023,8(2):635-644.
[16] MASHAYEKHY L,NEJAD M M,GROSU D,et al.Energy-aware scheduling of mapreduce jobs for big data Applications[J].IEEE Transactions on Parallel and Distributed Systems,2014,26(10):2720-2733.
[17] LI H,WANG H,FANG S,et al.An energy-aware scheduling algorithm for big data Applications in Spark[J].Cluster Computing,2020,23:593-609.
[18] MIRZA N M,ALI A,MUSA N S,et al.Enhancing Task Management in Apache Spark Through Energy-Efficient Data Segregation and Time-Based Scheduling[J].IEEE Access,2024,12:105080-105095.
[19] JALALI KHALIL ABADI Z,MANSOURI N,JAVIDI M M.Deep reinforcement learning-based scheduling in distributed systems:a critical review[J].Knowledge and Information Systems,2024,66(10):5709-5782.
[20] GU Y,LIU Z,DAI S,et al.Deep reinforcement learning for job scheduling and resource management in cloud computing:An algorithm-level review[J].arXiv:2501.01007,2025.
[21] GRINSZTAJN N,BEAUMONT O,JEANNOT E,et al.Readys:A reinforcement learning based strategy for heterogeneous dynamic scheduling[C]//2021 IEEE International Conference on Cluster Computing(CLUSTER).IEEE,2021:70-81.
[22] LIU N,LI Z,XU J,et al.A hierarchical framework of cloud resource allocation and power management using deep reinforcement learning[C]//2017 IEEE 37th International Conference on Distributed Computing Systems(ICDCS).IEEE,2017:372-382.
[23] LILHORE U K,SIMAIYA S,SHARMA Y K,et al.Cloud-edge hybrid deep learning framework for scalable IoT resource optimization[J].Journal of Cloud Computing,2025,14(1):5.
[24] LUO F,YUAN Y,DING W,et al.An improved particle swarm Optimization algorithm based on adaptive weight for task scheduling in cloud computing[C]//Proceedings of the 2nd International Conference on Computer Science and Application Engineering.2018:1-5.
[25] RJOUB G,BENTAHAR J,WAHAB O A,et al.Deep smartscheduling:A deep learning approach for automated big data scheduling over the cloud[C]//2019 7th International Confe-rence on Future Internet of Things and Cloud(FiCloud).IEEE,2019:189-196.
[26] LI H,LU L,SHI W,et al.Energy-aware scheduling for Spark job based on deep reinforcement learning in cloud[J].Computing,2023,105(8):1717-1743.
[27] BABAIYAN V,BUSHEHRIAN O.A deep-reinforcement-learning-based strategy selection approach for fault-tolerant offloading of delay-sensitive tasks in vehicular edge-cloud computing[J].The Journal of Supercomputing,2025,81(5):1-37.
[28] TANG Z,QI L,CHENG Z,et al.An energy-efficient taskscheduling algorithm in DVFS-enabled cloud environment[J].Journal of Grid Computing,2016,14:55-74.
[29] LUO L,WU W J,ZHANG F.Energy modeling based on cloud data center[J].Journal of Software,2014,25(7):1371-1387.
[30] LI H,WEI Y,XIONG Y,et al.A frequency-aware and energy-saving strategy based on DVFS for Spark[J].The Journal of Supercomputing,2021,77:11575-11596.
[31] LI S,ABDELZAHER T,YUAN M.Tapa:Temperature aware power allocation in data center with map-reduce[C]//2011 International Green Computing Conference and Workshops.IEEE,2011:1-8.
[32] IBRAHIM S,PHAN T D,CARPEN-AMARIE A,et al.Governing energy consumption in Hadoop through CPU frequency scaling:An analysis[J].Future Generation Computer Systems,2016,54:219-232.
[33] WU C M,CHANG R S,CHAN H Y.A green energy-efficient scheduling algorithm using the DVFS technique for cloud datacenters[J].Future Generation Computer Systems,2014,37:141-147.
[34] GE R,FENG X,FENG W,et al.Cpu miser:A performance-directed,run-time system for power-aware clusters[C]//2007 International Conference on Parallel Processing(ICPP 2007).IEEE,2007:18-18.
[35] MAROULIS S,ZACHEILAS N,KALOGERAKI V.A framework for efficient energy scheduling of Spark workloads[C]//2017 IEEE 37th International Conference on Distributed Computing Systems(ICDCS).IEEE,2017:2614-2615.
[36] SCHULMAN J,WOLSKI F,DHARIWAL P,et al.Proximalpolicy Optimization algorithms[J].arXiv:1707.06347,2017.
[37] HUANG S,HUANG J,DAI J,et al.The HiBench benchmark suite:Characterization of the MapReduce-based data analysis[C]//2010 IEEE 26th International Conference on Data Engineering Workshops(ICDEW 2010).IEEE,2010:41-51.
[38] ISMAIL L,MATERWALA H.Computing server power mo-deling in a data center:Survey,taxonomy,and performance eva-luation[J].ACM Computing Surveys,2020,53(3):1-34.
[39] SHI H K,WANG Z S,ZHANG S Z,et al.A survey on general CPU performance benchmark research[J].Acta Electronica Sinica,2023,51(1):246-256.
[40] HE Y L,WU D T,PHILIPPE F V,et al.Spark data balanced partitioning method based on priority filling strategy[J].Acta Electronica Sinica,2024,52(10):3322-3335.
[1] SHANG Kefeng, ZHANG Dan, ZHUAN Sunying, LI Dandan, LIU Yan, ZHU Kaige. Multi-party Inter-satellite Collaborative Computing Offloading Algorithm for Time-varying Topologies and Dynamic Heterogeneous Resources [J]. Computer Science, 2026, 53(7): 354-362.
[2] LIU Jiaqi, WANG Yujie, XIANG Guodu, YU Kui, CAO Fuyuan. Long-term Causal Effect Estimation Based on Deep Reinforcement Learning [J]. Computer Science, 2026, 53(4): 235-244.
[3] LI Fang, YUAN Baochun, SHEN Hang, WANG Tianjing, BAI Guangwei. Deep Reinforcement Learning-based Aircraft Task Offloading in Low Earth Orbit Satellite Networks [J]. Computer Science, 2026, 53(2): 406-415.
[4] WANG Haoyan, LI Chongshou, LI Tianrui. Reinforcement Learning Method for Solving Flexible Job Shop Scheduling Problem Based onDouble Layer Attention Network [J]. Computer Science, 2026, 53(1): 231-240.
[5] CHEN Jintao, LIN Bing, LIN Song, CHEN Jing, CHEN Xing. Dynamic Pricing and Energy Scheduling Strategy for Photovoltaic Storage Charging Stations Based on Multi-agent Deep Reinforcement Learning [J]. Computer Science, 2025, 52(9): 337-345.
[6] ZHANG Yongliang, LI Ziwen, XU Jiahao, JIANG Yuchen, CUI Ying. Congestion-aware and Cached Communication for Multi-agent Pathfinding [J]. Computer Science, 2025, 52(8): 317-325.
[7] HUO Dan, YU Fuping, SHEN Di, HAN Xueyan. Research on Multi-machine Conflict Resolution Based on Deep Reinforcement Learning [J]. Computer Science, 2025, 52(7): 271-278.
[8] WU Zongming, CAO Jijun, TANG Qiang. Online Parallel SDN Routing Optimization Algorithm Based on Deep Reinforcement Learning [J]. Computer Science, 2025, 52(6A): 240900018-9.
[9] ZHAO Xuejian, YE Hao, LI Hao, SUN Zhixin. Multi-AGV Path Planning Algorithm Based on Improved DDPG [J]. Computer Science, 2025, 52(6): 306-315.
[10] WANG Chenyuan, ZHANG Yanmei, YUAN Guan. Class Integration Test Order Generation Approach Fused with Deep Reinforcement Learning andGraph Convolutional Neural Network [J]. Computer Science, 2025, 52(6): 58-65.
[11] LI Yuanbo, HU Hongchao, YANG Xiaohan, GUO Wei, LIU Wenyan. Intrusion Tolerance Scheduling Algorithm for Microservice Workflow Based on Deep Reinforcement Learning [J]. Computer Science, 2025, 52(5): 375-383.
[12] ZHENG Longhai, XIAO Bohuai, YAO Zewei, CHEN Xing, MO Yuchang. Graph Reinforcement Learning Based Multi-edge Cooperative Load Balancing Method [J]. Computer Science, 2025, 52(3): 338-348.
[13] DU Likuan, LIU Chen, WANG Junlu, SONG Baoyan. Self-learning Star Chain Space Adaptive Allocation Method [J]. Computer Science, 2025, 52(3): 359-365.
[14] HUO Xingpeng, SHA Letian, LIU Jianwen, WU Shang, SU Ziyue. Windows Domain Penetration Testing Attack Path Generation Based on Deep Reinforcement Learning [J]. Computer Science, 2025, 52(3): 400-406.
[15] XU Donghong, LI Bin, QI Yong. Task Scheduling Strategy Based on Improved A2C Algorithm for Cloud Data Center [J]. Computer Science, 2025, 52(2): 310-322.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
Add:18# Honghu West Rd., Yubei ,Chongqing,China    Post Code: 401121
Tel: 023-63500828/023-67039612/023-67039623
E-mail: jsjkx12@163.com
Powered by Beijing Magtech Co., Ltd.