Aki Vehtari - Case studies

Cross-validation

• Uncertainty in Bayesian leave-one-out cross-validation based model comparison

• When cross-validation is not needed

  • Simple model we trust - betablockers

• When cross-validation is useful

  • We don’t trust the model - roaches
  • Complex model with posterior dependencies - collinear
  • A large number of models - diabetes
  • Avoiding double use of data in posterior predictive checking - Nabiximols

• Comparison of discrete and continuos observation model

  • Nabiximols

• Integrated PSIS-LOO for varying intercept models: roaches

• Cross-validation for hierarchical models: rats K-fold-CV

• LOO-R^2

• loo package vignettes

  • Bayesian Stacking and Pseudo-BMA weights using the loo package
  • Leave-one-out cross-validation for non-factorizable models
  • Approximate leave-future-out cross-validation for time series models
  • Using Leave-one-out cross-validation for large data
  • Avoiding model refits in leave-one-out cross-validation with moment matching

Projection predictive model selection – projpred

• mesquite collinearity
• bodyfat regression
• diabetes logistic regression
• candy random data vs original data
• winequality-red
• student projpred workflow
• See also projpred quick start vignette

Bayesian workflow

• Nabiximols: model checking and comparison, prior sensitivity analysis, model refinement
• Birthdays: iterative model building, use of fast approximate inference

Gaussian processes

• Birthdays: Workflow demo for building time series model from many Gaussian process components
• Motorcycle: Gaussian process demo with heteroscedastic noise model and Hilbert basis function and covariance matrix implementations

MCMC diagnostics and accuracy

• ESS comparison: Comparison of MCMC effective sample size (and MCSE) estimators
• Digits: How many digits to report and how many iterations to run
• Pareto-\(\hat{k}\) diagnostics:

Laplace approximation and Jacobian transformation

• Jacobian: Illustration of Laplace approximation and Jacobian adjustment in Stan.