Abstract: MapReduce is a popular cloud computing model which has been applied in data-intensive fields,and Hadoop which is based on MapReduce has been successfully used in dealing with big data.However,when dealing with computation-intensive tasks,particularly iterative computation,frequent loading of Map and Reduce processes will lead to overload.Resilient distributed dataset has been implemented in Spark,and it is an in-memory clustering computing model which can overcome this shortcoming efficiently.In this paper,a parallel version of differential evolution based on RDD (resilient distributed datasets) model named SparkDE was proposed.In SparkDE,the whole population is divided into several islands which evolve on their own,and then each island is deployed into a partition of RDD.After evolution for predefined generation in each island,migration operator is used calculation between islands.A wide range of benchmark problems are adopted to conduct numerical experiments.Compared with MapReduce (MRDE) based DE and classical DE,the results show that SparkDE can achieve higher accuracy of solution and faster speed of computation.The speedup of SparkDE is obvious.Thus SparkDE can serve as the next generation of optimizer in cloud computing.

Key words: Parallel differential evolution,Island model,Resilient distributed datasets,Transformation operation,Action operation