Setup

Load packages

library(rstanarm)
library(loo)
library(ggplot2)
library(bayesplot)
theme_set(bayesplot::theme_default())
library(projpred)
SEED=170701694

1 Introduction

This notebook was inspired by Eric Novik’s slides “Deconstructing Stan Manual Part 1: Linear”. The idea is to demonstrate how easy it is to do good variable selection with rstanarm, loo, and projpred.

In this notebook we illustrate Bayesian inference for model selection, including PSIS-LOO (Vehtari, Gelman and Gabry, 2017) and projection predictive approach (Piironen and Vehtari, 2017a; Piironen, Paasiniemi and Vehtari, 2020; McLatchie et al., 2023) which makes decision theoretically justified inference after model selection..

2 Wine quality data

We use Wine quality data set from UCI Machine Learning repository

d <- read.delim("winequality-red.csv", sep = ";")
dim(d)
[1] 1599   12

Remove duplicated

d <- d[!duplicated(d), ] # remove the duplicates
(p <- ncol(d))
[1] 12
(n <- nrow(d))
[1] 1359
names(d)
 [1] "fixed.acidity"        "volatile.acidity"     "citric.acid"         
 [4] "residual.sugar"       "chlorides"            "free.sulfur.dioxide" 
 [7] "total.sulfur.dioxide" "density"              "pH"                  
[10] "sulphates"            "alcohol"              "quality"             
prednames <- names(d)[1:(p-1)]

We scale the covariates so that when looking at the marginal posteriors for the effects they are on the same scale.

ds <- scale(d)
winequality_red <- as.data.frame(ds)

3 Fit regression model

The rstanarm package provides stan_glm which accepts same arguments as glm, but makes full Bayesian inference using Stan (mc-stan.org). By default a weakly informative Gaussian prior is used for weights.

model_formula <- formula(paste("quality ~", paste(prednames, collapse = " + ")))
fitg <- stan_glm(model_formula, data = winequality_red, QR=TRUE,
                 seed=SEED, refresh=0)

Let’s look at the summary:

summary(fitg)

Model Info:
 function:     stan_glm
 family:       gaussian [identity]
 formula:      quality ~ fixed.acidity + volatile.acidity + citric.acid + residual.sugar + 
       chlorides + free.sulfur.dioxide + total.sulfur.dioxide + 
       density + pH + sulphates + alcohol
 algorithm:    sampling
 sample:       4000 (posterior sample size)
 priors:       see help('prior_summary')
 observations: 1359
 predictors:   12

Estimates:
                       mean   sd   10%   50%   90%
(Intercept)           0.0    0.0  0.0   0.0   0.0 
fixed.acidity         0.0    0.1 -0.1   0.0   0.1 
volatile.acidity     -0.2    0.0 -0.3  -0.2  -0.2 
citric.acid           0.0    0.0 -0.1   0.0   0.0 
residual.sugar        0.0    0.0  0.0   0.0   0.0 
chlorides            -0.1    0.0 -0.1  -0.1  -0.1 
free.sulfur.dioxide   0.0    0.0  0.0   0.0   0.1 
total.sulfur.dioxide -0.1    0.0 -0.2  -0.1  -0.1 
density               0.0    0.1 -0.1   0.0   0.0 
pH                   -0.1    0.0 -0.1  -0.1   0.0 
sulphates             0.2    0.0  0.2   0.2   0.2 
alcohol               0.4    0.0  0.3   0.4   0.4 
sigma                 0.8    0.0  0.8   0.8   0.8 

Fit Diagnostics:
           mean   sd   10%   50%   90%
mean_PPD 0.0    0.0  0.0   0.0   0.0  

The mean_ppd is the sample average posterior predictive distribution of the outcome variable (for details see help('summary.stanreg')).

MCMC diagnostics
                     mcse Rhat n_eff
(Intercept)          0.0  1.0  5009 
fixed.acidity        0.0  1.0  5523 
volatile.acidity     0.0  1.0  5615 
citric.acid          0.0  1.0  5884 
residual.sugar       0.0  1.0  4943 
chlorides            0.0  1.0  5330 
free.sulfur.dioxide  0.0  1.0  5237 
total.sulfur.dioxide 0.0  1.0  5496 
density              0.0  1.0  4811 
pH                   0.0  1.0  5149 
sulphates            0.0  1.0  5378 
alcohol              0.0  1.0  5213 
sigma                0.0  1.0  4864 
mean_PPD             0.0  1.0  4282 
log-posterior        0.1  1.0  1578 

For each parameter, mcse is Monte Carlo standard error, n_eff is a crude measure of effective sample size, and Rhat is the potential scale reduction factor on split chains (at convergence Rhat=1).

We didn’t get divergences, Rhat’s are less than 1.1 and n_eff’s are useful (see, e.g., RStan workflow).

mcmc_areas(as.matrix(fitg), pars=prednames, prob_outer = .95)

Several 95% posterior intervals are not overlapping 0, so maybe there is something useful here.

4 Cross-validation checking

In case of collinear variables it is possible that marginal posteriors overlap 0, but the covariates can still useful for prediction. With many variables it will be difficult to analyse joint posterior to see which variables are jointly relevant. We can easily test whether any of the covariates are useful by using cross-validation to compare to a null model,

fitg0 <- stan_glm(quality ~ 1, data = winequality_red, seed=SEED, refresh=0)

We use fast Pareto smoothed importance sampling leave-one-out cross-validation (Vehtari, Gelman and Gabry, 2017)

(loog <- loo(fitg))

Computed from 4000 by 1359 log-likelihood matrix

         Estimate   SE
elpd_loo  -1635.7 30.6
p_loo        16.6  1.6
looic      3271.4 61.2
------
Monte Carlo SE of elpd_loo is 0.1.

Pareto k diagnostic values:
                         Count Pct.    Min. n_eff
(-Inf, 0.5]   (good)     1358  99.9%   1600      
 (0.5, 0.7]   (ok)          1   0.1%   1270      
   (0.7, 1]   (bad)         0   0.0%   <NA>      
   (1, Inf)   (very bad)    0   0.0%   <NA>      

All Pareto k estimates are ok (k < 0.7).
See help('pareto-k-diagnostic') for details.
(loog0 <- loo(fitg0))

Computed from 4000 by 1359 log-likelihood matrix

         Estimate   SE
elpd_loo  -1929.9 28.3
p_loo         2.2  0.2
looic      3859.9 56.5
------
Monte Carlo SE of elpd_loo is 0.0.

All Pareto k estimates are good (k < 0.5).
See help('pareto-k-diagnostic') for details.
loo_compare(loog0, loog)
      elpd_diff se_diff
fitg     0.0       0.0 
fitg0 -294.3      22.9 

Based on cross-validation covariates together have a high predictive power. If we need just the predictions we can stop here, but if we want to learn more about the relevance of the covariates we can continue with variable selection.

5 Projection predictive variable selection

We make the projective predictive variable selection (Piironen and Vehtari, 2017a; Piironen, Paasiniemi and Vehtari, 2020) using projpred package. A fast PSIS-LOO (Vehtari, Gelman and Gabry, 2017) is used to choose the model size. As the number of observations is large compared to the number of covariates, we estimate the performance using LOO-CV only along the search path (validate_search=FALSE), as we may assume that the overfitting in search is negligible (see more about this in McLatchie et al. (2023)).

fitg_cv <- cv_varsel(fitg, method='forward', cv_method='loo', validate_search=FALSE)

We can now look at the estimated predictive performance of smaller models compared to the full model.

plot(fitg_cv, stats = c('elpd', 'rmse'), text_angle = 45)

Three or four variables seems to be needed to get the same performance as the full model. As the estimated predictive performance is not going much above the reference model performance, we know that the use of option validate_search=FALSE was safe (see more in McLatchie et al. (2023)).

We can get a loo-cv based recommendation for the model size to choose.

(nsel <- suggest_size(fitg_cv, alpha=0.1))
[1] 4
(vsel <- solution_terms(fitg_cv)[1:nsel])
[1] "alcohol"          "volatile.acidity" "sulphates"        "chlorides"       

projpred recommends to use four variables: alcohol, volatile.acidity, sulphates, and chlorides.

Next we form the projected posterior for the chosen model. This projected model can be used in the future to make predictions by using only the selected variables.

projg <- project(fitg_cv, nv = nsel, ns = 4000)

  |                                                                            
  |                                                                      |   0%
  |                                                                            
  |=                                                                     |   1%
  |                                                                            
  |=                                                                     |   2%
  |                                                                            
  |==                                                                    |   2%
  |                                                                            
  |==                                                                    |   3%
  |                                                                            
  |==                                                                    |   4%
  |                                                                            
  |===                                                                   |   4%
  |                                                                            
  |===                                                                   |   5%
  |                                                                            
  |====                                                                  |   5%
  |                                                                            
  |====                                                                  |   6%
  |                                                                            
  |=====                                                                 |   6%
  |                                                                            
  |=====                                                                 |   7%
  |                                                                            
  |=====                                                                 |   8%
  |                                                                            
  |======                                                                |   8%
  |                                                                            
  |======                                                                |   9%
  |                                                                            
  |=======                                                               |  10%
  |                                                                            
  |========                                                              |  11%
  |                                                                            
  |========                                                              |  12%
  |                                                                            
  |=========                                                             |  12%
  |                                                                            
  |=========                                                             |  13%
  |                                                                            
  |=========                                                             |  14%
  |                                                                            
  |==========                                                            |  14%
  |                                                                            
  |==========                                                            |  15%
  |                                                                            
  |===========                                                           |  15%
  |                                                                            
  |===========                                                           |  16%
  |                                                                            
  |============                                                          |  16%
  |                                                                            
  |============                                                          |  17%
  |                                                                            
  |============                                                          |  18%
  |                                                                            
  |=============                                                         |  18%
  |                                                                            
  |=============                                                         |  19%
  |                                                                            
  |==============                                                        |  20%
  |                                                                            
  |===============                                                       |  21%
  |                                                                            
  |===============                                                       |  22%
  |                                                                            
  |================                                                      |  22%
  |                                                                            
  |================                                                      |  23%
  |                                                                            
  |================                                                      |  24%
  |                                                                            
  |=================                                                     |  24%
  |                                                                            
  |=================                                                     |  25%
  |                                                                            
  |==================                                                    |  25%
  |                                                                            
  |==================                                                    |  26%
  |                                                                            
  |===================                                                   |  26%
  |                                                                            
  |===================                                                   |  27%
  |                                                                            
  |===================                                                   |  28%
  |                                                                            
  |====================                                                  |  28%
  |                                                                            
  |====================                                                  |  29%
  |                                                                            
  |=====================                                                 |  30%
  |                                                                            
  |======================                                                |  31%
  |                                                                            
  |======================                                                |  32%
  |                                                                            
  |=======================                                               |  32%
  |                                                                            
  |=======================                                               |  33%
  |                                                                            
  |=======================                                               |  34%
  |                                                                            
  |========================                                              |  34%
  |                                                                            
  |========================                                              |  35%
  |                                                                            
  |=========================                                             |  35%
  |                                                                            
  |=========================                                             |  36%
  |                                                                            
  |==========================                                            |  36%
  |                                                                            
  |==========================                                            |  37%
  |                                                                            
  |==========================                                            |  38%
  |                                                                            
  |===========================                                           |  38%
  |                                                                            
  |===========================                                           |  39%
  |                                                                            
  |============================                                          |  40%
  |                                                                            
  |=============================                                         |  41%
  |                                                                            
  |=============================                                         |  42%
  |                                                                            
  |==============================                                        |  42%
  |                                                                            
  |==============================                                        |  43%
  |                                                                            
  |==============================                                        |  44%
  |                                                                            
  |===============================                                       |  44%
  |                                                                            
  |===============================                                       |  45%
  |                                                                            
  |================================                                      |  45%
  |                                                                            
  |================================                                      |  46%
  |                                                                            
  |=================================                                     |  46%
  |                                                                            
  |=================================                                     |  47%
  |                                                                            
  |=================================                                     |  48%
  |                                                                            
  |==================================                                    |  48%
  |                                                                            
  |==================================                                    |  49%
  |                                                                            
  |===================================                                   |  50%
  |                                                                            
  |====================================                                  |  51%
  |                                                                            
  |====================================                                  |  52%
  |                                                                            
  |=====================================                                 |  52%
  |                                                                            
  |=====================================                                 |  53%
  |                                                                            
  |=====================================                                 |  54%
  |                                                                            
  |======================================                                |  54%
  |                                                                            
  |======================================                                |  55%
  |                                                                            
  |=======================================                               |  55%
  |                                                                            
  |=======================================                               |  56%
  |                                                                            
  |========================================                              |  56%
  |                                                                            
  |========================================                              |  57%
  |                                                                            
  |========================================                              |  58%
  |                                                                            
  |=========================================                             |  58%
  |                                                                            
  |=========================================                             |  59%
  |                                                                            
  |==========================================                            |  60%
  |                                                                            
  |===========================================                           |  61%
  |                                                                            
  |===========================================                           |  62%
  |                                                                            
  |============================================                          |  62%
  |                                                                            
  |============================================                          |  63%
  |                                                                            
  |============================================                          |  64%
  |                                                                            
  |=============================================                         |  64%
  |                                                                            
  |=============================================                         |  65%
  |                                                                            
  |==============================================                        |  65%
  |                                                                            
  |==============================================                        |  66%
  |                                                                            
  |===============================================                       |  66%
  |                                                                            
  |===============================================                       |  67%
  |                                                                            
  |===============================================                       |  68%
  |                                                                            
  |================================================                      |  68%
  |                                                                            
  |================================================                      |  69%
  |                                                                            
  |=================================================                     |  70%
  |                                                                            
  |==================================================                    |  71%
  |                                                                            
  |==================================================                    |  72%
  |                                                                            
  |===================================================                   |  72%
  |                                                                            
  |===================================================                   |  73%
  |                                                                            
  |===================================================                   |  74%
  |                                                                            
  |====================================================                  |  74%
  |                                                                            
  |====================================================                  |  75%
  |                                                                            
  |=====================================================                 |  75%
  |                                                                            
  |=====================================================                 |  76%
  |                                                                            
  |======================================================                |  76%
  |                                                                            
  |======================================================                |  77%
  |                                                                            
  |======================================================                |  78%
  |                                                                            
  |=======================================================               |  78%
  |                                                                            
  |=======================================================               |  79%
  |                                                                            
  |========================================================              |  80%
  |                                                                            
  |=========================================================             |  81%
  |                                                                            
  |=========================================================             |  82%
  |                                                                            
  |==========================================================            |  82%
  |                                                                            
  |==========================================================            |  83%
  |                                                                            
  |==========================================================            |  84%
  |                                                                            
  |===========================================================           |  84%
  |                                                                            
  |===========================================================           |  85%
  |                                                                            
  |============================================================          |  85%
  |                                                                            
  |============================================================          |  86%
  |                                                                            
  |=============================================================         |  86%
  |                                                                            
  |=============================================================         |  87%
  |                                                                            
  |=============================================================         |  88%
  |                                                                            
  |==============================================================        |  88%
  |                                                                            
  |==============================================================        |  89%
  |                                                                            
  |===============================================================       |  90%
  |                                                                            
  |================================================================      |  91%
  |                                                                            
  |================================================================      |  92%
  |                                                                            
  |=================================================================     |  92%
  |                                                                            
  |=================================================================     |  93%
  |                                                                            
  |=================================================================     |  94%
  |                                                                            
  |==================================================================    |  94%
  |                                                                            
  |==================================================================    |  95%
  |                                                                            
  |===================================================================   |  95%
  |                                                                            
  |===================================================================   |  96%
  |                                                                            
  |====================================================================  |  96%
  |                                                                            
  |====================================================================  |  97%
  |                                                                            
  |====================================================================  |  98%
  |                                                                            
  |===================================================================== |  98%
  |                                                                            
  |===================================================================== |  99%
  |                                                                            
  |======================================================================| 100%
round(colMeans(as.matrix(projg)), 1)
         (Intercept)              alcohol     volatile.acidity 
                 0.0                  0.4                 -0.3 
           sulphates            chlorides total.sulfur.dioxide 
                 0.2                 -0.1                 -0.1 
               sigma 
                 0.8 
round(posterior_interval(as.matrix(projg)), 1)
                       5%  95%
(Intercept)           0.0  0.0
alcohol               0.3  0.4
volatile.acidity     -0.3 -0.2
sulphates             0.2  0.2
chlorides            -0.1 -0.1
total.sulfur.dioxide -0.1  0.0
sigma                 0.8  0.8

The marginals of projected posteriors look like this.

mcmc_areas(as.matrix(projg), pars = vsel)

6 Alternative regularized horseshoe prior

We also test regularized horseshoe prior (Piironen and Vehtari, 2017b) which has more prior mass near 0.

p0 <- 5 # prior guess for the number of relevant variables
tau0 <- p0/(p-p0) * 1/sqrt(n)
hs_prior <- hs(df=1, global_df=1, global_scale=tau0)
fitrhs <- stan_glm(model_formula, data = winequality_red, prior=hs_prior,
                   seed=SEED, refresh=0)
mcmc_areas(as.matrix(fitrhs), pars=prednames, prob_outer = .95)

Many of the variables are shrunk more towards 0, but still based on these marginals it is not as easy to select the most useful variables as it is with projpred.

The posteriors with normal and regularized horseshoe priors are clearly different, but does this have an effect to the predictions? In case of collinearity prior may have a strong effect on posterior, but a weak effect on posterior predictions. We can use loo to compare

(loorhs <- loo(fitrhs))

Computed from 4000 by 1359 log-likelihood matrix

         Estimate   SE
elpd_loo  -1634.2 30.5
p_loo        13.8  1.3
looic      3268.3 61.0
------
Monte Carlo SE of elpd_loo is 0.1.

All Pareto k estimates are good (k < 0.5).
See help('pareto-k-diagnostic') for details.
loo_compare(loog, loorhs)
       elpd_diff se_diff
fitrhs  0.0       0.0   
fitg   -1.5       1.4   

There is no difference in predictive performance and thus we don’t need to repeat the projpred variable selection for the model with regularized horseshoe prior.


References

McLatchie, Y., Rögnvaldsson, S., Weber, F. and Vehtari, A. (2023) ‘Robust and efficient projection predictive inference’, arXiv preprint arXiv:2306.15581.

Piironen, J., Paasiniemi, M. and Vehtari, A. (2020) ‘Projective inference in high-dimensional problems: Prediction and feature selection’, Electronic Journal of Statistics, 14(1), pp. 2155–2197.

Piironen, J. and Vehtari, A. (2017a) ‘Comparison of Bayesian predictive methods for model selection’, Statistics and Computing, 27(3), pp. 711–735. doi: 10.1007/s11222-016-9649-y.

Piironen, J. and Vehtari, A. (2017b) ‘Sparsity information and regularization in the horseshoe and other shrinkage priors’, Electronic journal of Statistics, 11(2), pp. 5018–5051. doi: 10.1214/17-EJS1337SI.

Vehtari, A., Gelman, A. and Gabry, J. (2017) ‘Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC’, Statistics and Computing, 27(5), pp. 1413–1432. doi: 10.1007/s11222-016-9696-4.

Licenses

  • Code © 2017-2018, Aki Vehtari, licensed under BSD-3.
  • Text © 2017-2018, Aki Vehtari, licensed under CC-BY-NC 4.0.

Original Computing Environment

sessionInfo()
R version 4.2.2 Patched (2022-11-10 r83330)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 22.04.3 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.20.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=fi_FI.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=fi_FI.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=fi_FI.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=fi_FI.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] cmdstanr_0.6.1.9000  brms_2.20.1          projpred_2.7.0      
 [4] bayesplot_1.10.0     GGally_2.1.2         bridgesampling_1.1-2
 [7] ggridges_0.5.4       ggplot2_3.4.4        loo_2.6.0           
[10] rstanarm_2.26.1      Rcpp_1.0.11          tidyr_1.3.0         

loaded via a namespace (and not attached):
  [1] minqa_1.2.5          colorspace_2.1-0     ellipsis_0.3.2      
  [4] markdown_1.8         QuickJSR_1.0.5       base64enc_0.1-3     
  [7] farver_2.1.1         rstan_2.26.23        DT_0.29             
 [10] fansi_1.0.5          mvtnorm_1.2-3        codetools_0.2-19    
 [13] splines_4.2.2        cachem_1.0.8         knitr_1.43          
 [16] shinythemes_1.2.0    jsonlite_1.8.7       nloptr_2.0.3        
 [19] shiny_1.7.5          compiler_4.2.2       backports_1.4.1     
 [22] Matrix_1.5-1         fastmap_1.1.1        cli_3.6.1           
 [25] later_1.3.1          htmltools_0.5.6      prettyunits_1.1.1   
 [28] tools_4.2.2          igraph_1.5.1         coda_0.19-4         
 [31] gtable_0.3.4         glue_1.6.2           reshape2_1.4.4      
 [34] dplyr_1.1.3          posterior_1.4.1      jquerylib_0.1.4     
 [37] vctrs_0.6.4          nlme_3.1-162         crosstalk_1.2.0     
 [40] tensorA_0.36.2       xfun_0.40            stringr_1.5.0       
 [43] ps_1.7.5             lme4_1.1-34          mime_0.12           
 [46] miniUI_0.1.1.1       lifecycle_1.0.3      gtools_3.9.4        
 [49] MASS_7.3-58.2        zoo_1.8-12           scales_1.2.1        
 [52] colourpicker_1.3.0   hms_1.1.3            promises_1.2.1      
 [55] Brobdingnag_1.2-9    parallel_4.2.2       inline_0.3.19       
 [58] shinystan_2.6.0      RColorBrewer_1.1-3   yaml_2.3.7          
 [61] gridExtra_2.3        StanHeaders_2.26.28  sass_0.4.7          
 [64] reshape_0.8.9        stringi_1.7.12       highr_0.10          
 [67] dygraphs_1.1.1.6     checkmate_2.3.0      boot_1.3-28         
 [70] pkgbuild_1.4.2       rlang_1.1.1          pkgconfig_2.0.3     
 [73] matrixStats_1.0.0    distributional_0.3.2 evaluate_0.21       
 [76] lattice_0.20-45      purrr_1.0.2          rstantools_2.3.1.1  
 [79] htmlwidgets_1.6.2    labeling_0.4.3       processx_3.8.2      
 [82] tidyselect_1.2.0     plyr_1.8.8           magrittr_2.0.3      
 [85] R6_2.5.1             generics_0.1.3       pillar_1.9.0        
 [88] withr_2.5.1          xts_0.13.1           survival_3.4-0      
 [91] abind_1.4-5          tibble_3.2.1         crayon_1.5.2        
 [94] utf8_1.2.4           rmarkdown_2.24       progress_1.2.2      
 [97] grid_4.2.2           data.table_1.14.8    callr_3.7.3         
[100] threejs_0.3.3        digest_0.6.33        xtable_1.8-4        
[103] httpuv_1.6.11        RcppParallel_5.1.7   stats4_4.2.2        
[106] munsell_0.5.0        bslib_0.5.1          shinyjs_2.1.0       


LS0tCnRpdGxlOiAiQmF5ZXNpYW4gdmFyaWFibGUgc2VsZWN0aW9uIGZvciByZWQgd2luZSBxdWFsaXR5IHJhbmtpbmcgZGF0YSIKYXV0aG9yOiAiW0FraSBWZWh0YXJpXShodHRwczovL3VzZXJzLmFhbHRvLmZpL35hdmUvKSIKZGF0ZTogIkZpcnN0IHZlcnNpb24gMjAxOC0wMi0yNy4gTGFzdCBtb2RpZmllZCBgciBmb3JtYXQoU3lzLkRhdGUoKSlgLiIKb3V0cHV0OgogIGh0bWxfZG9jdW1lbnQ6CiAgICBmaWdfY2FwdGlvbjogeWVzCiAgICB0b2M6IFRSVUUKICAgIHRvY19kZXB0aDogMgogICAgbnVtYmVyX3NlY3Rpb25zOiBUUlVFCiAgICB0b2NfZmxvYXQ6CiAgICAgIHNtb290aF9zY3JvbGw6IEZBTFNFCiAgICBjb2RlX2Rvd25sb2FkOiB0cnVlCmJpYmxpb2dyYXBoeTogbW9kZWxzZWwuYmliCmNzbDogaGFydmFyZC1jaXRlLXRoZW0tcmlnaHQuY3NsCmxpbmstY2l0YXRpb25zOiB5ZXMKLS0tCgojIFNldHVwICB7LnVubnVtYmVyZWR9CgpgYGB7ciBzZXR1cCwgaW5jbHVkZT1GQUxTRX0Ka25pdHI6Om9wdHNfY2h1bmskc2V0KGNhY2hlPUZBTFNFLCBtZXNzYWdlPUZBTFNFLCBlcnJvcj1GQUxTRSwgd2FybmluZz1GQUxTRSwgY29tbWVudD1OQSwgb3V0LndpZHRoPSc5NSUnKQpgYGAKCioqTG9hZCBwYWNrYWdlcyoqCmBgYHtyfQpsaWJyYXJ5KHJzdGFuYXJtKQpsaWJyYXJ5KGxvbykKbGlicmFyeShnZ3Bsb3QyKQpsaWJyYXJ5KGJheWVzcGxvdCkKdGhlbWVfc2V0KGJheWVzcGxvdDo6dGhlbWVfZGVmYXVsdCgpKQpsaWJyYXJ5KHByb2pwcmVkKQpTRUVEPTE3MDcwMTY5NApgYGAKCiMgSW50cm9kdWN0aW9uCgpUaGlzIG5vdGVib29rIHdhcyBpbnNwaXJlZCBieSBFcmljIE5vdmlrJ3Mgc2xpZGVzICJEZWNvbnN0cnVjdGluZyBTdGFuIE1hbnVhbCBQYXJ0IDE6IExpbmVhciIuIFRoZSBpZGVhIGlzIHRvIGRlbW9uc3RyYXRlIGhvdyBlYXN5IGl0IGlzIHRvIGRvIGdvb2QgdmFyaWFibGUgc2VsZWN0aW9uIHdpdGggW2Byc3RhbmFybWBdKGh0dHBzOi8vY3Jhbi5yLXByb2plY3Qub3JnL3BhY2thZ2U9cnN0YW5hcm0pLCBbYGxvb2BdKGh0dHBzOi8vY3Jhbi5yLXByb2plY3Qub3JnL3BhY2thZ2U9bG9vKSwgYW5kIFtgcHJvanByZWRgXShodHRwczovL2NyYW4uci1wcm9qZWN0Lm9yZy9wYWNrYWdlPXByb2pwcmVkKS4KCkluIHRoaXMgbm90ZWJvb2sgd2UgaWxsdXN0cmF0ZSBCYXllc2lhbiBpbmZlcmVuY2UgZm9yIG1vZGVsIHNlbGVjdGlvbiwgaW5jbHVkaW5nIFBTSVMtTE9PIFtAVmVodGFyaStldGFsOlBTSVMtTE9POjIwMTddIGFuZCBwcm9qZWN0aW9uIHByZWRpY3RpdmUgYXBwcm9hY2ggW0BNY0xhdGNoaWUrZXRhbDoyMDIzOnByb2pwcmVkX3dvcmtmbG93OyBAUGlpcm9uZW4rZXRhbDpwcm9qcHJlZDoyMDIwOyBAUGlpcm9uZW4rVmVodGFyaToyMDE3YV0gd2hpY2ggbWFrZXMgZGVjaXNpb24gdGhlb3JldGljYWxseSBqdXN0aWZpZWQgaW5mZXJlbmNlIGFmdGVyIG1vZGVsIHNlbGVjdGlvbi4uCgojIFdpbmUgcXVhbGl0eSBkYXRhCgpXZSB1c2UgW1dpbmUgcXVhbGl0eSBkYXRhIHNldCBmcm9tIFVDSSBNYWNoaW5lIExlYXJuaW5nIHJlcG9zaXRvcnldKGh0dHBzOi8vYXJjaGl2ZS5pY3MudWNpLmVkdS9tbC9kYXRhc2V0cy93aW5lK3F1YWxpdHljYW5keSkKYGBge3J9CmQgPC0gcmVhZC5kZWxpbSgid2luZXF1YWxpdHktcmVkLmNzdiIsIHNlcCA9ICI7IikKZGltKGQpCmBgYApSZW1vdmUgZHVwbGljYXRlZApgYGB7cn0KZCA8LSBkWyFkdXBsaWNhdGVkKGQpLCBdICMgcmVtb3ZlIHRoZSBkdXBsaWNhdGVzCihwIDwtIG5jb2woZCkpCihuIDwtIG5yb3coZCkpCm5hbWVzKGQpCnByZWRuYW1lcyA8LSBuYW1lcyhkKVsxOihwLTEpXQpgYGAKCldlIHNjYWxlIHRoZSBjb3ZhcmlhdGVzIHNvIHRoYXQgd2hlbiBsb29raW5nIGF0IHRoZSBtYXJnaW5hbCBwb3N0ZXJpb3JzIGZvciB0aGUgZWZmZWN0cyB0aGV5IGFyZSBvbiB0aGUgc2FtZSBzY2FsZS4gCmBgYHtyfQpkcyA8LSBzY2FsZShkKQp3aW5lcXVhbGl0eV9yZWQgPC0gYXMuZGF0YS5mcmFtZShkcykKYGBgCgojIEZpdCByZWdyZXNzaW9uIG1vZGVsCgpUaGUgYHJzdGFuYXJtYCBwYWNrYWdlIHByb3ZpZGVzIGBzdGFuX2dsbWAgd2hpY2ggYWNjZXB0cyBzYW1lIGFyZ3VtZW50cyBhcyBgZ2xtYCwgYnV0IG1ha2VzIGZ1bGwgQmF5ZXNpYW4gaW5mZXJlbmNlIHVzaW5nIFN0YW4gKFttYy1zdGFuLm9yZ10oaHR0cHM6Ly9tYy1zdGFuLm9yZykpLiBCeSBkZWZhdWx0IGEgd2Vha2x5IGluZm9ybWF0aXZlIEdhdXNzaWFuIHByaW9yIGlzIHVzZWQgZm9yIHdlaWdodHMuCmBgYHtyLCByZXN1bHRzPSdoaWRlJ30KbW9kZWxfZm9ybXVsYSA8LSBmb3JtdWxhKHBhc3RlKCJxdWFsaXR5IH4iLCBwYXN0ZShwcmVkbmFtZXMsIGNvbGxhcHNlID0gIiArICIpKSkKZml0ZyA8LSBzdGFuX2dsbShtb2RlbF9mb3JtdWxhLCBkYXRhID0gd2luZXF1YWxpdHlfcmVkLCBRUj1UUlVFLAogICAgICAgICAgICAgICAgIHNlZWQ9U0VFRCwgcmVmcmVzaD0wKQpgYGAKTGV0J3MgbG9vayBhdCB0aGUgc3VtbWFyeToKYGBge3J9CnN1bW1hcnkoZml0ZykKYGBgCgpXZSBkaWRuJ3QgZ2V0IGRpdmVyZ2VuY2VzLCBSaGF0J3MgYXJlIGxlc3MgdGhhbiAxLjEgYW5kIG5fZWZmJ3MgYXJlIHVzZWZ1bCAoc2VlLCBlLmcuLCBbUlN0YW4gd29ya2Zsb3ddKGh0dHA6Ly9tYy1zdGFuLm9yZy91c2Vycy9kb2N1bWVudGF0aW9uL2Nhc2Utc3R1ZGllcy9yc3Rhbl93b3JrZmxvdy5odG1sKSkuCgpgYGB7cn0KbWNtY19hcmVhcyhhcy5tYXRyaXgoZml0ZyksIHBhcnM9cHJlZG5hbWVzLCBwcm9iX291dGVyID0gLjk1KQpgYGAKClNldmVyYWwgOTUlIHBvc3RlcmlvciBpbnRlcnZhbHMgYXJlIG5vdCBvdmVybGFwcGluZyAwLCBzbyBtYXliZSB0aGVyZSBpcyBzb21ldGhpbmcgdXNlZnVsIGhlcmUuCgojIENyb3NzLXZhbGlkYXRpb24gY2hlY2tpbmcKCkluIGNhc2Ugb2YgY29sbGluZWFyIHZhcmlhYmxlcyBpdCBpcyBwb3NzaWJsZSB0aGF0IG1hcmdpbmFsIHBvc3RlcmlvcnMgb3ZlcmxhcCAwLCBidXQgdGhlIGNvdmFyaWF0ZXMgY2FuIHN0aWxsIHVzZWZ1bCBmb3IgcHJlZGljdGlvbi4gV2l0aCBtYW55IHZhcmlhYmxlcyBpdCB3aWxsIGJlIGRpZmZpY3VsdCB0byBhbmFseXNlIGpvaW50IHBvc3RlcmlvciB0byBzZWUgd2hpY2ggdmFyaWFibGVzIGFyZSBqb2ludGx5IHJlbGV2YW50LiBXZSBjYW4gZWFzaWx5IHRlc3Qgd2hldGhlciBhbnkgb2YgdGhlIGNvdmFyaWF0ZXMgYXJlIHVzZWZ1bCBieSB1c2luZyBjcm9zcy12YWxpZGF0aW9uIHRvIGNvbXBhcmUgdG8gYSBudWxsIG1vZGVsLApgYGB7cn0KZml0ZzAgPC0gc3Rhbl9nbG0ocXVhbGl0eSB+IDEsIGRhdGEgPSB3aW5lcXVhbGl0eV9yZWQsIHNlZWQ9U0VFRCwgcmVmcmVzaD0wKQpgYGAKCldlIHVzZSBmYXN0IFBhcmV0byBzbW9vdGhlZCBpbXBvcnRhbmNlIHNhbXBsaW5nIGxlYXZlLW9uZS1vdXQgY3Jvc3MtdmFsaWRhdGlvbiBbQFZlaHRhcmkrZXRhbDpQU0lTLUxPTzoyMDE3XQpgYGB7cn0KKGxvb2cgPC0gbG9vKGZpdGcpKQoobG9vZzAgPC0gbG9vKGZpdGcwKSkKbG9vX2NvbXBhcmUobG9vZzAsIGxvb2cpCmBgYAoKQmFzZWQgb24gY3Jvc3MtdmFsaWRhdGlvbiBjb3ZhcmlhdGVzIHRvZ2V0aGVyIGhhdmUgYSBoaWdoIHByZWRpY3RpdmUgcG93ZXIuIElmIHdlIG5lZWQganVzdCB0aGUgcHJlZGljdGlvbnMgd2UgY2FuIHN0b3AgaGVyZSwgYnV0IGlmIHdlIHdhbnQgdG8gbGVhcm4gbW9yZSBhYm91dCB0aGUgcmVsZXZhbmNlIG9mIHRoZSBjb3ZhcmlhdGVzIHdlIGNhbiBjb250aW51ZSB3aXRoIHZhcmlhYmxlIHNlbGVjdGlvbi4KCiMgUHJvamVjdGlvbiBwcmVkaWN0aXZlIHZhcmlhYmxlIHNlbGVjdGlvbgoKV2UgbWFrZSB0aGUgcHJvamVjdGl2ZSBwcmVkaWN0aXZlIHZhcmlhYmxlIHNlbGVjdGlvbiBbQFBpaXJvbmVuK2V0YWw6cHJvanByZWQ6MjAyMDsgQFBpaXJvbmVuK1ZlaHRhcmk6MjAxN2FdIHVzaW5nIGBwcm9qcHJlZGAgcGFja2FnZS4gQSBmYXN0IFBTSVMtTE9PIFtAVmVodGFyaStldGFsOlBTSVMtTE9POjIwMTddIGlzIHVzZWQgdG8gY2hvb3NlIHRoZSBtb2RlbCBzaXplLiBBcyB0aGUgbnVtYmVyIG9mIG9ic2VydmF0aW9ucyBpcyBsYXJnZSBjb21wYXJlZCB0byB0aGUgbnVtYmVyIG9mIGNvdmFyaWF0ZXMsIHdlIGVzdGltYXRlIHRoZSBwZXJmb3JtYW5jZSB1c2luZyBMT08tQ1Ygb25seSBhbG9uZyB0aGUgc2VhcmNoIHBhdGggKGB2YWxpZGF0ZV9zZWFyY2g9RkFMU0VgKSwgYXMgd2UgbWF5IGFzc3VtZSB0aGF0IHRoZSBvdmVyZml0dGluZyBpbiBzZWFyY2ggaXMgbmVnbGlnaWJsZSAoc2VlIG1vcmUgYWJvdXQgdGhpcyBpbiBATWNMYXRjaGllK2V0YWw6MjAyMzpwcm9qcHJlZF93b3JrZmxvdykuCmBgYHtyLCByZXN1bHRzPSdoaWRlJ30KZml0Z19jdiA8LSBjdl92YXJzZWwoZml0ZywgbWV0aG9kPSdmb3J3YXJkJywgY3ZfbWV0aG9kPSdsb28nLCB2YWxpZGF0ZV9zZWFyY2g9RkFMU0UpCmBgYAoKV2UgY2FuIG5vdyBsb29rIGF0IHRoZSBlc3RpbWF0ZWQgcHJlZGljdGl2ZSBwZXJmb3JtYW5jZSBvZiBzbWFsbGVyIG1vZGVscyBjb21wYXJlZCB0byB0aGUgZnVsbCBtb2RlbC4KYGBge3J9CnBsb3QoZml0Z19jdiwgc3RhdHMgPSBjKCdlbHBkJywgJ3Jtc2UnKSwgdGV4dF9hbmdsZSA9IDQ1KQpgYGAKClRocmVlIG9yIGZvdXIgdmFyaWFibGVzIHNlZW1zIHRvIGJlIG5lZWRlZCB0byBnZXQgdGhlIHNhbWUgcGVyZm9ybWFuY2UgYXMgdGhlIGZ1bGwgbW9kZWwuIEFzIHRoZSBlc3RpbWF0ZWQgcHJlZGljdGl2ZSBwZXJmb3JtYW5jZSBpcyBub3QgZ29pbmcgbXVjaCBhYm92ZSB0aGUgcmVmZXJlbmNlIG1vZGVsIHBlcmZvcm1hbmNlLCB3ZSBrbm93IHRoYXQgdGhlIHVzZSBvZiBvcHRpb24gYHZhbGlkYXRlX3NlYXJjaD1GQUxTRWAgd2FzIHNhZmUgKHNlZSBtb3JlIGluIEBNY0xhdGNoaWUrZXRhbDoyMDIzOnByb2pwcmVkX3dvcmtmbG93KS4KCldlIGNhbiBnZXQgYSBsb28tY3YgYmFzZWQgcmVjb21tZW5kYXRpb24gZm9yIHRoZSBtb2RlbCBzaXplIHRvIGNob29zZS4KYGBge3J9Cihuc2VsIDwtIHN1Z2dlc3Rfc2l6ZShmaXRnX2N2LCBhbHBoYT0wLjEpKQoodnNlbCA8LSBzb2x1dGlvbl90ZXJtcyhmaXRnX2N2KVsxOm5zZWxdKQpgYGAKcHJvanByZWQgcmVjb21tZW5kcyB0byB1c2UgZm91ciB2YXJpYWJsZXM6IGFsY29ob2wsIHZvbGF0aWxlLmFjaWRpdHksIHN1bHBoYXRlcywgYW5kIGNobG9yaWRlcy4KCk5leHQgd2UgZm9ybSB0aGUgcHJvamVjdGVkIHBvc3RlcmlvciBmb3IgdGhlIGNob3NlbiBtb2RlbC4gVGhpcyBwcm9qZWN0ZWQgbW9kZWwgY2FuIGJlIHVzZWQgaW4gdGhlIGZ1dHVyZSB0byBtYWtlIHByZWRpY3Rpb25zIGJ5IHVzaW5nIG9ubHkgdGhlIHNlbGVjdGVkIHZhcmlhYmxlcy4KYGBge3J9CnByb2pnIDwtIHByb2plY3QoZml0Z19jdiwgbnYgPSBuc2VsLCBucyA9IDQwMDApCnJvdW5kKGNvbE1lYW5zKGFzLm1hdHJpeChwcm9qZykpLCAxKQpyb3VuZChwb3N0ZXJpb3JfaW50ZXJ2YWwoYXMubWF0cml4KHByb2pnKSksIDEpCmBgYAoKVGhlIG1hcmdpbmFscyBvZiBwcm9qZWN0ZWQgcG9zdGVyaW9ycyBsb29rIGxpa2UgdGhpcy4KYGBge3J9Cm1jbWNfYXJlYXMoYXMubWF0cml4KHByb2pnKSwgcGFycyA9IHZzZWwpCmBgYAoKIyBBbHRlcm5hdGl2ZSByZWd1bGFyaXplZCBob3JzZXNob2UgcHJpb3IKCldlIGFsc28gdGVzdCByZWd1bGFyaXplZCBob3JzZXNob2UgcHJpb3IgW0BQaWlyb25lbitWZWh0YXJpOlJIUzoyMDE3XSB3aGljaCBoYXMgbW9yZSBwcmlvciBtYXNzIG5lYXIgMC4KYGBge3IsIHJlc3VsdHM9J2hpZGUnfQpwMCA8LSA1ICMgcHJpb3IgZ3Vlc3MgZm9yIHRoZSBudW1iZXIgb2YgcmVsZXZhbnQgdmFyaWFibGVzCnRhdTAgPC0gcDAvKHAtcDApICogMS9zcXJ0KG4pCmhzX3ByaW9yIDwtIGhzKGRmPTEsIGdsb2JhbF9kZj0xLCBnbG9iYWxfc2NhbGU9dGF1MCkKZml0cmhzIDwtIHN0YW5fZ2xtKG1vZGVsX2Zvcm11bGEsIGRhdGEgPSB3aW5lcXVhbGl0eV9yZWQsIHByaW9yPWhzX3ByaW9yLAogICAgICAgICAgICAgICAgICAgc2VlZD1TRUVELCByZWZyZXNoPTApCmBgYAoKYGBge3J9Cm1jbWNfYXJlYXMoYXMubWF0cml4KGZpdHJocyksIHBhcnM9cHJlZG5hbWVzLCBwcm9iX291dGVyID0gLjk1KQpgYGAKCk1hbnkgb2YgdGhlIHZhcmlhYmxlcyBhcmUgc2hydW5rIG1vcmUgdG93YXJkcyAwLCBidXQgc3RpbGwgYmFzZWQgb24gdGhlc2UgbWFyZ2luYWxzIGl0IGlzIG5vdCBhcyBlYXN5IHRvIHNlbGVjdCB0aGUgbW9zdCB1c2VmdWwgdmFyaWFibGVzIGFzIGl0IGlzIHdpdGggcHJvanByZWQuCgpUaGUgcG9zdGVyaW9ycyB3aXRoIG5vcm1hbCBhbmQgcmVndWxhcml6ZWQgaG9yc2VzaG9lIHByaW9ycyBhcmUgY2xlYXJseSBkaWZmZXJlbnQsIGJ1dCBkb2VzIHRoaXMgaGF2ZSBhbiBlZmZlY3QgdG8gdGhlIHByZWRpY3Rpb25zPyBJbiBjYXNlIG9mIGNvbGxpbmVhcml0eSBwcmlvciBtYXkgaGF2ZSBhIHN0cm9uZyBlZmZlY3Qgb24gcG9zdGVyaW9yLCBidXQgYSB3ZWFrIGVmZmVjdCBvbiBwb3N0ZXJpb3IgcHJlZGljdGlvbnMuIFdlIGNhbiB1c2UgbG9vIHRvIGNvbXBhcmUKCmBgYHtyfQoobG9vcmhzIDwtIGxvbyhmaXRyaHMpKQpsb29fY29tcGFyZShsb29nLCBsb29yaHMpCmBgYApUaGVyZSBpcyBubyBkaWZmZXJlbmNlIGluIHByZWRpY3RpdmUgcGVyZm9ybWFuY2UgYW5kIHRodXMgd2UgZG9uJ3QgbmVlZCB0byByZXBlYXQgdGhlIHByb2pwcmVkIHZhcmlhYmxlIHNlbGVjdGlvbiBmb3IgdGhlIG1vZGVsIHdpdGggcmVndWxhcml6ZWQgaG9yc2VzaG9lIHByaW9yLgoKCjxiciAvPgoKIyBSZWZlcmVuY2VzIHsudW5udW1iZXJlZH0KCjxkaXYgaWQ9InJlZnMiPjwvZGl2PgoKIyBMaWNlbnNlcyB7LnVubnVtYmVyZWR9CgoqIENvZGUgJmNvcHk7IDIwMTctMjAxOCwgQWtpIFZlaHRhcmksIGxpY2Vuc2VkIHVuZGVyIEJTRC0zLgoqIFRleHQgJmNvcHk7IDIwMTctMjAxOCwgQWtpIFZlaHRhcmksIGxpY2Vuc2VkIHVuZGVyIENDLUJZLU5DIDQuMC4KCiMgT3JpZ2luYWwgQ29tcHV0aW5nIEVudmlyb25tZW50IHsudW5udW1iZXJlZH0KCmBgYHtyfQpzZXNzaW9uSW5mbygpCmBgYAoKPGJyIC8+Cg==