Powered Mean Ensemble Learning

Ensemble learning is widely recognized as an effective strategy for improving predictive performance by combining multiple machine learning models. In this study, we consider ensembles obtained as weighted averages of different models: a linear regression, a random forest and a neural network, and propose to optimise their weights by maximising a convex combination between the predictive R2 and the Rank Graduation accuracy, in a cross validation setting. We compare different types of averages: arithmetic, quadratic, geometric and harmonic. We illustrate our proposal by means of simulations and by the application to the Boston Housing Prices dataset. The optimised ensemble models achieve, under certain conditions, a lower mean squared error relative to the best single model.