Spark-based cooperative coevolution for large scale global optimization

The cooperative coevolution framework was introduced to address the shortcomings of metaheuristic algorithms in solving continuous large-scale global optimization problems. By dividing the problem into subcomponents which can be optimized separately, the framework can improve on both the solution’s quality as well as the computational speed by exposing a degree of parallelism. Distributed computing platforms, such as Apache Spark, have long been used to improve the speed of different algorithms in solving computational problems. This work proposes a distributed implementation of the cooperative coevolution framework for solving large-scale global optimization problems on the Apache Spark distributed computing platform. By using a formerly outlined distributed variant of the cooperative coevolution framework, features of the Spark platform are utilized to enhance the computational speed of the algorithm while maintaining comparable search quality to other works in the literature. To test for the proposed implementation’s improvement in computational speed, the CEC 2010 large-scale global optimization benchmark functions are used due to the diversity they offer in terms of complexity, separability and modality. Results of the proposed distributed implementation suggest that a speedup of up to ×3.36 is possible on large-scale global optimization benchmarks using the Apache Spark platform.