Validation and comparison

This vignette provides a lightweight, fully reproducible benchmark. It is designed for package documentation, not as a final simulation study for a journal article.

The benchmark includes:

  • 1D scenarios: S1 normal, S2 bimodal mixture, S3 heavy-tail (t with 4 d.f.)
  • 2D scenarios: S4 bimodal correlated mixture, S5 correlated Gaussian

Compared methods:

  • ASH_estimate()
  • LBFP_estimate()
  • GLBFP_estimate()
  • KDE (stats::density, MASS::kde2d)

Metrics:

  • ISE per run, MISE across repetitions
  • elapsed time per run
  • sensitivity to parameter changes (b, m)

For paper-grade evidence, increase the number of repetitions, document the simulation design, and report uncertainty around the summarized metrics.

Run benchmark suite

result <- run_validation_suite(reps = 2, grid_size_1d = 60, grid_size_2d = 16)
summary_tbl <- summarise_validation(result)
summary_tbl
#>    scenario dimension method    ise_mean elapsed_mean  ise_median
#> 1        S1         1    ASH 0.032815853       0.0025 0.032815853
#> 2        S1         1  GLBFP 0.021953737       0.0040 0.021953737
#> 3        S1         1    KDE 0.002658040       0.0005 0.002658040
#> 4        S1         1   LBFP 0.021953737       0.0040 0.021953737
#> 5        S2         1    ASH 0.020182782       0.0030 0.020182782
#> 6        S2         1  GLBFP 0.013839910       0.0040 0.013839910
#> 7        S2         1    KDE 0.005633544       0.0010 0.005633544
#> 8        S2         1   LBFP 0.013839910       0.0045 0.013839910
#> 9        S3         1    ASH 0.029677125       0.0020 0.029677125
#> 10       S3         1  GLBFP 0.021946772       0.0040 0.021946772
#> 11       S3         1    KDE 0.006869287       0.0005 0.006869287
#> 12       S3         1   LBFP 0.021946772       0.0040 0.021946772
#> 13       S4         2    ASH 0.038106086       0.0085 0.038106086
#> 14       S4         2  GLBFP 0.018294805       0.0235 0.018294805
#> 15       S4         2    KDE 0.003425851       0.0010 0.003425851
#> 16       S4         2   LBFP 0.018294805       0.0230 0.018294805
#> 17       S5         2    ASH 0.047296014       0.0080 0.047296014
#> 18       S5         2  GLBFP 0.023920100       0.0230 0.023920100
#> 19       S5         2    KDE 0.004657292       0.0010 0.004657292
#> 20       S5         2   LBFP 0.023920100       0.0230 0.023920100
#>    elapsed_median
#> 1          0.0025
#> 2          0.0040
#> 3          0.0005
#> 4          0.0040
#> 5          0.0030
#> 6          0.0040
#> 7          0.0010
#> 8          0.0045
#> 9          0.0020
#> 10         0.0040
#> 11         0.0005
#> 12         0.0040
#> 13         0.0085
#> 14         0.0235
#> 15         0.0010
#> 16         0.0230
#> 17         0.0080
#> 18         0.0230
#> 19         0.0010
#> 20         0.0230

MISE ranking by scenario

mi <- summary_tbl[order(summary_tbl$scenario, summary_tbl$ise_mean), ]
mi[, c("scenario", "dimension", "method", "ise_mean", "elapsed_mean")]
#>    scenario dimension method    ise_mean elapsed_mean
#> 3        S1         1    KDE 0.002658040       0.0005
#> 2        S1         1  GLBFP 0.021953737       0.0040
#> 4        S1         1   LBFP 0.021953737       0.0040
#> 1        S1         1    ASH 0.032815853       0.0025
#> 7        S2         1    KDE 0.005633544       0.0010
#> 6        S2         1  GLBFP 0.013839910       0.0040
#> 8        S2         1   LBFP 0.013839910       0.0045
#> 5        S2         1    ASH 0.020182782       0.0030
#> 11       S3         1    KDE 0.006869287       0.0005
#> 10       S3         1  GLBFP 0.021946772       0.0040
#> 12       S3         1   LBFP 0.021946772       0.0040
#> 9        S3         1    ASH 0.029677125       0.0020
#> 15       S4         2    KDE 0.003425851       0.0010
#> 14       S4         2  GLBFP 0.018294805       0.0235
#> 16       S4         2   LBFP 0.018294805       0.0230
#> 13       S4         2    ASH 0.038106086       0.0085
#> 19       S5         2    KDE 0.004657292       0.0010
#> 18       S5         2  GLBFP 0.023920100       0.0230
#> 20       S5         2   LBFP 0.023920100       0.0230
#> 17       S5         2    ASH 0.047296014       0.0080

Sensitivity analysis (b, m)

result$sensitivity
#>   b_scale m mean_density max_density
#> 1     0.8 1   0.01875666   0.1965201
#> 2     1.0 1   0.01847158   0.1951697
#> 3     1.2 1   0.01785758   0.1301115
#> 4     1.0 2   0.01783902   0.1514042

Notes

  • The benchmark is intentionally small to keep vignette runtime short.
  • Increase reps and grid sizes in inst/bench/run_validation.R for paper-grade tables.