Prediction-oriented model selection

Purpose

The prediction-oriented model selection criteria stem from information theory and have been introduced into the partial least squares structural equation modeling (PLS‐SEM) context by Sharma et al. (2019a,b). These model selection criteria help researchers to select the best predictive model from a pre-determined range of alternative model set-ups. They, thereby, allow researchers to fully exploit the predictive capabilities of PLS‐SEM.

Criteria

Sharma et al. (2019a,b) show by means of a Monte Carlo simulation that the in‐sample model selection criteria (e.g., BIC and GM) are useful substitutes for out‐of‐sample model selection criteria (e.g., RMSA and MAD).

SmartPLS 3.2.8 (and later) provides results of the following range of in-sample model selection criteria:

  • the Akaike's Information Criterion (AIC),
  • the corrected AIC (AIC_c),
  • the unbiased AIC (AIC_u),
  • the Bayesian information criterion (BIC),
  • the Hannan-Quinn Criterion (HQ), and
  • the corrected Hannan–Quinn Criterion (HQ_c).

SmartPLS does not include the Geweke–Meese criterion (GM), which is based on the model-complexity adjusted mean square error (MSE) from the saturated (full) model. In PLS-SEM, defining the saturated model is not always straightforward. Especially in advanced modeling situations (e.g. moderation and second-order models) expert judgement is needed to define the saturated model, while an automatic generation of a saturated model could produce strange results.

Note: You can download and run an Excel file to compute the model selection criteria by yourself, including the GM.

Interpretation

Sharma et al.’s (2019a,b) Monte Carlo simulations show that the BIC and GM are particularly suitable for model comparison tasks. These criteria tend to select the best model among a set of competing models. In addition, both criteria achieve a sound balance between theoretical consistency and high predictive power, even in the absence of a holdout sample. Researchers need to compare BIC and GM values for alternative model set-ups and select the model, which minimizes BIC and GM values. Hence, when using SmartPLS, one would select the model alternative with the lowest BIC outcome.

Links

References

  • Sharma, P. N., Shmueli, G., Sarstedt, M., Danks, N., and Ray, S. (2019a). Prediction-oriented Model Selection in Partial Least Squares Path Modeling, Decision Sciences, in press.

  • Sharma, P. N., Sarstedt, M., Shmueli, G., Kim, K.H, and Thiele, K. O. (2019b). PLS-Based Model Selection: The Role of Alternative Explanations in Information Systems Research, Journal of the Association for Information Systems, in press.

Link to More Literature