Playing with Toys¶
As of v0.6.0, pyhf now supports toys! A lot of kinks have been discovered and worked out and we’re grateful to our ATLAS colleagues for beta-testing this in the meantime. We don’t believe that there may not be any more bugs, but we feel confident that we can release the current implementation.
Simple Model (again)¶
import pyhf
import numpy as np
model = pyhf.simplemodels.hepdata_like(
signal_data=[5.0, 10.0], bkg_data=[50.0, 60.0], bkg_uncerts=[5.0, 12.0]
)
Hypothesis Testing (revisited)¶
And like in the first exercise we did, let’s refresh what the hypothesis test looked like using \(\tilde{q}_\mu\):
CLs_obs, CLs_exp = pyhf.infer.hypotest(
1.0, # null hypothesis
[52.5, 65.0] + model.config.auxdata,
model,
test_stat="qtilde",
return_expected_set=True,
)
print(f" Observed CLs: {CLs_obs:.4f}")
for expected_value, n_sigma in zip(CLs_exp, np.arange(-2, 3)):
print(f"Expected CLs({n_sigma:2d} σ): {expected_value:.4f}")
Observed CLs: 0.4856
Expected CLs(-2 σ): 0.0830
Expected CLs(-1 σ): 0.1826
Expected CLs( 0 σ): 0.3702
Expected CLs( 1 σ): 0.6434
Expected CLs( 2 σ): 0.8853
So the question is, does the asymptotic approximation hold in this example? The standard assumption is that you have enough “statistics” (meaning enough events) to use the large-N approximation. So let’s use the toy-based calculator instead and compute the same values as above and see if they match in the asymptotic case (we certainly hope they mostly do here!)
CLs_obs, CLs_exp = pyhf.infer.hypotest(
1.0, # null hypothesis
[52.5, 65.0] + model.config.auxdata,
model,
test_stat="qtilde",
return_expected_set=True,
calctype="toybased",
ntoys=1000,
)
print(f" Observed CLs: {CLs_obs:.4f}")
for expected_value, n_sigma in zip(CLs_exp, np.arange(-2, 3)):
print(f"Expected CLs({n_sigma:2d} σ): {expected_value:.4f}")
Signal-like: 0%| | 0/1000 [00:00<?, ?toy/s]
Signal-like: 1%| | 7/1000 [00:00<00:16, 61.20toy/s]
Signal-like: 1%|▏ | 14/1000 [00:00<00:16, 60.52toy/s]
Signal-like: 2%|▏ | 21/1000 [00:00<00:16, 60.05toy/s]
Signal-like: 3%|▎ | 28/1000 [00:00<00:15, 61.86toy/s]
Signal-like: 4%|▎ | 35/1000 [00:00<00:15, 61.70toy/s]
Signal-like: 4%|▍ | 42/1000 [00:00<00:15, 62.25toy/s]
Signal-like: 5%|▍ | 49/1000 [00:00<00:15, 62.09toy/s]
Signal-like: 6%|▌ | 56/1000 [00:00<00:15, 61.86toy/s]
Signal-like: 6%|▋ | 63/1000 [00:01<00:15, 59.66toy/s]
Signal-like: 7%|▋ | 70/1000 [00:01<00:15, 60.17toy/s]
Signal-like: 8%|▊ | 77/1000 [00:01<00:15, 60.12toy/s]
Signal-like: 8%|▊ | 84/1000 [00:01<00:14, 61.29toy/s]
Signal-like: 9%|▉ | 91/1000 [00:01<00:15, 60.26toy/s]
Signal-like: 10%|▉ | 98/1000 [00:01<00:14, 61.81toy/s]
Signal-like: 10%|█ | 105/1000 [00:01<00:14, 62.31toy/s]
Signal-like: 11%|█ | 112/1000 [00:01<00:14, 60.83toy/s]
Signal-like: 12%|█▏ | 119/1000 [00:01<00:14, 59.57toy/s]
Signal-like: 12%|█▎ | 125/1000 [00:02<00:15, 58.32toy/s]
Signal-like: 13%|█▎ | 132/1000 [00:02<00:14, 58.67toy/s]
Signal-like: 14%|█▍ | 139/1000 [00:02<00:14, 59.72toy/s]
Signal-like: 14%|█▍ | 145/1000 [00:02<00:14, 59.01toy/s]
Signal-like: 15%|█▌ | 151/1000 [00:02<00:14, 59.17toy/s]
Signal-like: 16%|█▌ | 158/1000 [00:02<00:13, 60.27toy/s]
Signal-like: 16%|█▋ | 165/1000 [00:02<00:13, 60.29toy/s]
Signal-like: 17%|█▋ | 172/1000 [00:02<00:14, 58.57toy/s]
Signal-like: 18%|█▊ | 178/1000 [00:02<00:14, 57.73toy/s]
Signal-like: 18%|█▊ | 184/1000 [00:03<00:14, 57.39toy/s]
Signal-like: 19%|█▉ | 190/1000 [00:03<00:14, 57.70toy/s]
Signal-like: 20%|█▉ | 196/1000 [00:03<00:13, 57.95toy/s]
Signal-like: 20%|██ | 202/1000 [00:03<00:13, 57.01toy/s]
Signal-like: 21%|██ | 208/1000 [00:03<00:13, 57.26toy/s]
Signal-like: 22%|██▏ | 215/1000 [00:03<00:13, 58.34toy/s]
Signal-like: 22%|██▏ | 221/1000 [00:03<00:13, 58.42toy/s]
Signal-like: 23%|██▎ | 227/1000 [00:03<00:13, 58.70toy/s]
Signal-like: 23%|██▎ | 234/1000 [00:03<00:13, 58.55toy/s]
Signal-like: 24%|██▍ | 241/1000 [00:04<00:12, 58.97toy/s]
Signal-like: 25%|██▍ | 248/1000 [00:04<00:12, 59.69toy/s]
Signal-like: 25%|██▌ | 254/1000 [00:04<00:12, 59.33toy/s]
Signal-like: 26%|██▌ | 261/1000 [00:04<00:12, 60.21toy/s]
Signal-like: 27%|██▋ | 268/1000 [00:04<00:12, 60.53toy/s]
Signal-like: 28%|██▊ | 275/1000 [00:04<00:12, 60.11toy/s]
Signal-like: 28%|██▊ | 282/1000 [00:04<00:12, 58.94toy/s]
Signal-like: 29%|██▉ | 289/1000 [00:04<00:11, 59.94toy/s]
Signal-like: 30%|██▉ | 296/1000 [00:04<00:11, 59.77toy/s]
Signal-like: 30%|███ | 302/1000 [00:05<00:11, 58.91toy/s]
Signal-like: 31%|███ | 309/1000 [00:05<00:11, 59.79toy/s]
Signal-like: 32%|███▏ | 316/1000 [00:05<00:11, 60.47toy/s]
Signal-like: 32%|███▏ | 323/1000 [00:05<00:11, 60.61toy/s]
Signal-like: 33%|███▎ | 330/1000 [00:05<00:11, 57.63toy/s]
Signal-like: 34%|███▎ | 336/1000 [00:05<00:11, 57.18toy/s]
Signal-like: 34%|███▍ | 342/1000 [00:05<00:11, 57.19toy/s]
Signal-like: 35%|███▍ | 348/1000 [00:05<00:11, 56.89toy/s]
Signal-like: 35%|███▌ | 354/1000 [00:05<00:11, 57.52toy/s]
Signal-like: 36%|███▌ | 360/1000 [00:06<00:11, 57.21toy/s]
Signal-like: 37%|███▋ | 366/1000 [00:06<00:11, 57.61toy/s]
Signal-like: 37%|███▋ | 372/1000 [00:06<00:10, 57.99toy/s]
Signal-like: 38%|███▊ | 379/1000 [00:06<00:10, 58.66toy/s]
Signal-like: 39%|███▊ | 386/1000 [00:06<00:10, 59.93toy/s]
Signal-like: 39%|███▉ | 392/1000 [00:06<00:10, 59.35toy/s]
Signal-like: 40%|███▉ | 399/1000 [00:06<00:09, 60.20toy/s]
Signal-like: 41%|████ | 406/1000 [00:06<00:09, 59.55toy/s]
Signal-like: 41%|████▏ | 413/1000 [00:06<00:09, 59.71toy/s]
Signal-like: 42%|████▏ | 420/1000 [00:07<00:09, 60.77toy/s]
Signal-like: 43%|████▎ | 427/1000 [00:07<00:09, 60.08toy/s]
Signal-like: 43%|████▎ | 434/1000 [00:07<00:09, 59.93toy/s]
Signal-like: 44%|████▍ | 440/1000 [00:07<00:09, 59.19toy/s]
Signal-like: 45%|████▍ | 447/1000 [00:07<00:09, 60.36toy/s]
Signal-like: 45%|████▌ | 454/1000 [00:07<00:09, 59.41toy/s]
Signal-like: 46%|████▌ | 461/1000 [00:07<00:08, 60.78toy/s]
Signal-like: 47%|████▋ | 468/1000 [00:07<00:08, 60.77toy/s]
Signal-like: 48%|████▊ | 475/1000 [00:07<00:08, 60.34toy/s]
Signal-like: 48%|████▊ | 482/1000 [00:08<00:08, 60.67toy/s]
Signal-like: 49%|████▉ | 489/1000 [00:08<00:08, 60.54toy/s]
Signal-like: 50%|████▉ | 496/1000 [00:08<00:08, 60.47toy/s]
Signal-like: 50%|█████ | 503/1000 [00:08<00:08, 61.39toy/s]
Signal-like: 51%|█████ | 510/1000 [00:08<00:08, 61.23toy/s]
Signal-like: 52%|█████▏ | 517/1000 [00:08<00:07, 60.40toy/s]
Signal-like: 52%|█████▏ | 524/1000 [00:08<00:07, 61.66toy/s]
Signal-like: 53%|█████▎ | 531/1000 [00:08<00:07, 60.86toy/s]
Signal-like: 54%|█████▍ | 538/1000 [00:09<00:07, 61.90toy/s]
Signal-like: 55%|█████▍ | 545/1000 [00:09<00:07, 60.72toy/s]
Signal-like: 55%|█████▌ | 552/1000 [00:09<00:07, 59.38toy/s]
Signal-like: 56%|█████▌ | 558/1000 [00:09<00:07, 58.11toy/s]
Signal-like: 56%|█████▋ | 565/1000 [00:09<00:07, 59.33toy/s]
Signal-like: 57%|█████▋ | 572/1000 [00:09<00:07, 60.71toy/s]
Signal-like: 58%|█████▊ | 579/1000 [00:09<00:06, 61.35toy/s]
Signal-like: 59%|█████▊ | 586/1000 [00:09<00:06, 61.12toy/s]
Signal-like: 59%|█████▉ | 593/1000 [00:09<00:06, 61.17toy/s]
Signal-like: 60%|██████ | 600/1000 [00:10<00:06, 59.45toy/s]
Signal-like: 61%|██████ | 606/1000 [00:10<00:06, 59.52toy/s]
Signal-like: 61%|██████▏ | 613/1000 [00:10<00:06, 59.71toy/s]
Signal-like: 62%|██████▏ | 620/1000 [00:10<00:06, 60.79toy/s]
Signal-like: 63%|██████▎ | 627/1000 [00:10<00:06, 61.01toy/s]
Signal-like: 63%|██████▎ | 634/1000 [00:10<00:06, 60.70toy/s]
Signal-like: 64%|██████▍ | 641/1000 [00:10<00:05, 60.64toy/s]
Signal-like: 65%|██████▍ | 648/1000 [00:10<00:05, 60.88toy/s]
Signal-like: 66%|██████▌ | 655/1000 [00:10<00:05, 60.20toy/s]
Signal-like: 66%|██████▌ | 662/1000 [00:11<00:05, 59.64toy/s]
Signal-like: 67%|██████▋ | 669/1000 [00:11<00:05, 60.13toy/s]
Signal-like: 68%|██████▊ | 676/1000 [00:11<00:05, 60.95toy/s]
Signal-like: 68%|██████▊ | 683/1000 [00:11<00:05, 60.83toy/s]
Signal-like: 69%|██████▉ | 690/1000 [00:11<00:05, 61.39toy/s]
Signal-like: 70%|██████▉ | 697/1000 [00:11<00:04, 61.99toy/s]
Signal-like: 70%|███████ | 704/1000 [00:11<00:04, 61.38toy/s]
Signal-like: 71%|███████ | 711/1000 [00:11<00:04, 60.87toy/s]
Signal-like: 72%|███████▏ | 718/1000 [00:11<00:04, 60.63toy/s]
Signal-like: 72%|███████▎ | 725/1000 [00:12<00:04, 59.41toy/s]
Signal-like: 73%|███████▎ | 732/1000 [00:12<00:04, 60.06toy/s]
Signal-like: 74%|███████▍ | 739/1000 [00:12<00:04, 59.16toy/s]
Signal-like: 75%|███████▍ | 746/1000 [00:12<00:04, 59.91toy/s]
Signal-like: 75%|███████▌ | 753/1000 [00:12<00:04, 59.80toy/s]
Signal-like: 76%|███████▌ | 760/1000 [00:12<00:03, 61.03toy/s]
Signal-like: 77%|███████▋ | 767/1000 [00:12<00:03, 60.55toy/s]
Signal-like: 77%|███████▋ | 774/1000 [00:12<00:03, 59.60toy/s]
Signal-like: 78%|███████▊ | 780/1000 [00:13<00:03, 58.78toy/s]
Signal-like: 79%|███████▊ | 786/1000 [00:13<00:03, 57.98toy/s]
Signal-like: 79%|███████▉ | 792/1000 [00:13<00:03, 58.18toy/s]
Signal-like: 80%|███████▉ | 798/1000 [00:13<00:03, 55.71toy/s]
Signal-like: 80%|████████ | 804/1000 [00:13<00:03, 56.59toy/s]
Signal-like: 81%|████████ | 811/1000 [00:13<00:03, 58.74toy/s]
Signal-like: 82%|████████▏ | 818/1000 [00:13<00:03, 59.77toy/s]
Signal-like: 82%|████████▎ | 825/1000 [00:13<00:02, 60.44toy/s]
Signal-like: 83%|████████▎ | 832/1000 [00:13<00:02, 58.70toy/s]
Signal-like: 84%|████████▍ | 839/1000 [00:14<00:02, 59.58toy/s]
Signal-like: 84%|████████▍ | 845/1000 [00:14<00:02, 57.72toy/s]
Signal-like: 85%|████████▌ | 851/1000 [00:14<00:02, 56.53toy/s]
Signal-like: 86%|████████▌ | 857/1000 [00:14<00:02, 57.29toy/s]
Signal-like: 86%|████████▋ | 864/1000 [00:14<00:02, 58.24toy/s]
Signal-like: 87%|████████▋ | 871/1000 [00:14<00:02, 59.59toy/s]
Signal-like: 88%|████████▊ | 878/1000 [00:14<00:01, 61.26toy/s]
Signal-like: 88%|████████▊ | 885/1000 [00:14<00:01, 61.02toy/s]
Signal-like: 89%|████████▉ | 892/1000 [00:14<00:01, 61.34toy/s]
Signal-like: 90%|████████▉ | 899/1000 [00:15<00:01, 62.07toy/s]
Signal-like: 91%|█████████ | 906/1000 [00:15<00:01, 60.31toy/s]
Signal-like: 91%|█████████▏| 913/1000 [00:15<00:01, 59.59toy/s]
Signal-like: 92%|█████████▏| 920/1000 [00:15<00:01, 59.99toy/s]
Signal-like: 93%|█████████▎| 927/1000 [00:15<00:01, 61.51toy/s]
Signal-like: 93%|█████████▎| 934/1000 [00:15<00:01, 62.15toy/s]
Signal-like: 94%|█████████▍| 941/1000 [00:15<00:00, 61.28toy/s]
Signal-like: 95%|█████████▍| 948/1000 [00:15<00:00, 61.17toy/s]
Signal-like: 96%|█████████▌| 955/1000 [00:15<00:00, 61.60toy/s]
Signal-like: 96%|█████████▌| 962/1000 [00:16<00:00, 59.84toy/s]
Signal-like: 97%|█████████▋| 969/1000 [00:16<00:00, 60.00toy/s]
Signal-like: 98%|█████████▊| 976/1000 [00:16<00:00, 60.91toy/s]
Signal-like: 98%|█████████▊| 983/1000 [00:16<00:00, 60.84toy/s]
Signal-like: 99%|█████████▉| 990/1000 [00:16<00:00, 61.84toy/s]
Signal-like: 100%|█████████▉| 997/1000 [00:16<00:00, 61.09toy/s]
Background-like: 0%| | 0/1000 [00:00<?, ?toy/s]
Background-like: 1%| | 6/1000 [00:00<00:16, 59.50toy/s]
Background-like: 1%| | 12/1000 [00:00<00:17, 56.75toy/s]
Background-like: 2%|▏ | 18/1000 [00:00<00:17, 57.42toy/s]
Background-like: 2%|▏ | 24/1000 [00:00<00:16, 58.26toy/s]
Background-like: 3%|▎ | 31/1000 [00:00<00:16, 58.93toy/s]
Background-like: 4%|▍ | 38/1000 [00:00<00:16, 59.47toy/s]
Background-like: 4%|▍ | 45/1000 [00:00<00:15, 60.58toy/s]
Background-like: 5%|▌ | 52/1000 [00:00<00:15, 60.63toy/s]
Background-like: 6%|▌ | 59/1000 [00:00<00:15, 60.69toy/s]
Background-like: 7%|▋ | 66/1000 [00:01<00:15, 60.73toy/s]
Background-like: 7%|▋ | 73/1000 [00:01<00:15, 61.58toy/s]
Background-like: 8%|▊ | 80/1000 [00:01<00:15, 61.18toy/s]
Background-like: 9%|▊ | 87/1000 [00:01<00:14, 61.23toy/s]
Background-like: 9%|▉ | 94/1000 [00:01<00:14, 61.43toy/s]
Background-like: 10%|█ | 101/1000 [00:01<00:14, 61.06toy/s]
Background-like: 11%|█ | 108/1000 [00:01<00:14, 59.51toy/s]
Background-like: 12%|█▏ | 115/1000 [00:01<00:14, 60.95toy/s]
Background-like: 12%|█▏ | 122/1000 [00:02<00:14, 59.71toy/s]
Background-like: 13%|█▎ | 128/1000 [00:02<00:14, 59.10toy/s]
Background-like: 14%|█▎ | 135/1000 [00:02<00:14, 60.70toy/s]
Background-like: 14%|█▍ | 142/1000 [00:02<00:14, 60.13toy/s]
Background-like: 15%|█▍ | 149/1000 [00:02<00:14, 60.05toy/s]
Background-like: 16%|█▌ | 156/1000 [00:02<00:14, 59.25toy/s]
Background-like: 16%|█▌ | 162/1000 [00:02<00:14, 58.81toy/s]
Background-like: 17%|█▋ | 168/1000 [00:02<00:14, 57.81toy/s]
Background-like: 17%|█▋ | 174/1000 [00:02<00:14, 57.46toy/s]
Background-like: 18%|█▊ | 181/1000 [00:03<00:13, 58.63toy/s]
Background-like: 19%|█▉ | 188/1000 [00:03<00:13, 58.33toy/s]
Background-like: 19%|█▉ | 194/1000 [00:03<00:14, 57.15toy/s]
Background-like: 20%|██ | 200/1000 [00:03<00:13, 57.23toy/s]
Background-like: 21%|██ | 206/1000 [00:03<00:13, 57.28toy/s]
Background-like: 21%|██ | 212/1000 [00:03<00:13, 56.39toy/s]
Background-like: 22%|██▏ | 218/1000 [00:03<00:13, 56.85toy/s]
Background-like: 22%|██▏ | 224/1000 [00:03<00:13, 57.06toy/s]
Background-like: 23%|██▎ | 230/1000 [00:03<00:13, 56.09toy/s]
Background-like: 24%|██▎ | 236/1000 [00:04<00:13, 56.10toy/s]
Background-like: 24%|██▍ | 242/1000 [00:04<00:13, 55.03toy/s]
Background-like: 25%|██▍ | 248/1000 [00:04<00:13, 55.95toy/s]
Background-like: 25%|██▌ | 254/1000 [00:04<00:13, 56.29toy/s]
Background-like: 26%|██▌ | 260/1000 [00:04<00:13, 55.75toy/s]
Background-like: 27%|██▋ | 266/1000 [00:04<00:13, 54.77toy/s]
Background-like: 27%|██▋ | 272/1000 [00:04<00:13, 54.23toy/s]
Background-like: 28%|██▊ | 278/1000 [00:04<00:13, 54.98toy/s]
Background-like: 28%|██▊ | 284/1000 [00:04<00:12, 56.34toy/s]
Background-like: 29%|██▉ | 290/1000 [00:04<00:12, 56.38toy/s]
Background-like: 30%|██▉ | 297/1000 [00:05<00:12, 57.81toy/s]
Background-like: 30%|███ | 303/1000 [00:05<00:12, 57.78toy/s]
Background-like: 31%|███ | 310/1000 [00:05<00:11, 59.30toy/s]
Background-like: 32%|███▏ | 316/1000 [00:05<00:11, 58.60toy/s]
Background-like: 32%|███▏ | 323/1000 [00:05<00:11, 59.50toy/s]
Background-like: 33%|███▎ | 329/1000 [00:05<00:11, 58.29toy/s]
Background-like: 34%|███▎ | 335/1000 [00:05<00:11, 58.11toy/s]
Background-like: 34%|███▍ | 341/1000 [00:05<00:11, 56.24toy/s]
Background-like: 35%|███▍ | 347/1000 [00:05<00:11, 56.04toy/s]
Background-like: 35%|███▌ | 353/1000 [00:06<00:11, 56.14toy/s]
Background-like: 36%|███▌ | 359/1000 [00:06<00:11, 55.75toy/s]
Background-like: 37%|███▋ | 366/1000 [00:06<00:10, 57.73toy/s]
Background-like: 37%|███▋ | 373/1000 [00:06<00:10, 58.79toy/s]
Background-like: 38%|███▊ | 379/1000 [00:06<00:10, 58.56toy/s]
Background-like: 39%|███▊ | 386/1000 [00:06<00:10, 59.86toy/s]
Background-like: 39%|███▉ | 392/1000 [00:06<00:10, 58.14toy/s]
Background-like: 40%|███▉ | 398/1000 [00:06<00:10, 58.12toy/s]
Background-like: 40%|████ | 405/1000 [00:06<00:10, 58.98toy/s]
Background-like: 41%|████ | 411/1000 [00:07<00:10, 57.72toy/s]
Background-like: 42%|████▏ | 417/1000 [00:07<00:10, 57.67toy/s]
Background-like: 42%|████▏ | 423/1000 [00:07<00:10, 57.58toy/s]
Background-like: 43%|████▎ | 429/1000 [00:07<00:10, 56.89toy/s]
Background-like: 44%|████▎ | 435/1000 [00:07<00:10, 56.47toy/s]
Background-like: 44%|████▍ | 441/1000 [00:07<00:09, 56.69toy/s]
Background-like: 45%|████▍ | 447/1000 [00:07<00:09, 56.28toy/s]
Background-like: 45%|████▌ | 453/1000 [00:07<00:09, 56.00toy/s]
Background-like: 46%|████▌ | 459/1000 [00:07<00:09, 56.99toy/s]
Background-like: 46%|████▋ | 465/1000 [00:08<00:09, 56.90toy/s]
Background-like: 47%|████▋ | 471/1000 [00:08<00:09, 57.60toy/s]
Background-like: 48%|████▊ | 478/1000 [00:08<00:08, 58.65toy/s]
Background-like: 48%|████▊ | 484/1000 [00:08<00:08, 57.75toy/s]
Background-like: 49%|████▉ | 490/1000 [00:08<00:09, 55.39toy/s]
Background-like: 50%|████▉ | 496/1000 [00:08<00:09, 55.70toy/s]
Background-like: 50%|█████ | 502/1000 [00:08<00:08, 55.44toy/s]
Background-like: 51%|█████ | 508/1000 [00:08<00:08, 54.86toy/s]
Background-like: 51%|█████▏ | 514/1000 [00:08<00:08, 54.81toy/s]
Background-like: 52%|█████▏ | 520/1000 [00:09<00:08, 55.32toy/s]
Background-like: 53%|█████▎ | 526/1000 [00:09<00:08, 55.17toy/s]
Background-like: 53%|█████▎ | 533/1000 [00:09<00:08, 56.70toy/s]
Background-like: 54%|█████▍ | 539/1000 [00:09<00:08, 55.66toy/s]
Background-like: 55%|█████▍ | 545/1000 [00:09<00:08, 54.11toy/s]
Background-like: 55%|█████▌ | 551/1000 [00:09<00:08, 53.18toy/s]
Background-like: 56%|█████▌ | 557/1000 [00:09<00:08, 53.36toy/s]
Background-like: 56%|█████▋ | 563/1000 [00:09<00:08, 54.21toy/s]
Background-like: 57%|█████▋ | 569/1000 [00:09<00:07, 54.17toy/s]
Background-like: 57%|█████▊ | 575/1000 [00:10<00:07, 54.15toy/s]
Background-like: 58%|█████▊ | 581/1000 [00:10<00:07, 54.86toy/s]
Background-like: 59%|█████▊ | 587/1000 [00:10<00:07, 55.22toy/s]
Background-like: 59%|█████▉ | 594/1000 [00:10<00:07, 57.08toy/s]
Background-like: 60%|██████ | 600/1000 [00:10<00:07, 55.56toy/s]
Background-like: 61%|██████ | 606/1000 [00:10<00:07, 55.80toy/s]
Background-like: 61%|██████ | 612/1000 [00:10<00:06, 55.57toy/s]
Background-like: 62%|██████▏ | 619/1000 [00:10<00:06, 56.84toy/s]
Background-like: 63%|██████▎ | 626/1000 [00:10<00:06, 58.22toy/s]
Background-like: 63%|██████▎ | 632/1000 [00:11<00:06, 57.90toy/s]
Background-like: 64%|██████▍ | 638/1000 [00:11<00:06, 56.87toy/s]
Background-like: 64%|██████▍ | 644/1000 [00:11<00:06, 56.24toy/s]
Background-like: 65%|██████▌ | 650/1000 [00:11<00:06, 56.97toy/s]
Background-like: 66%|██████▌ | 656/1000 [00:11<00:06, 56.86toy/s]
Background-like: 66%|██████▌ | 662/1000 [00:11<00:05, 56.44toy/s]
Background-like: 67%|██████▋ | 669/1000 [00:11<00:05, 58.41toy/s]
Background-like: 68%|██████▊ | 675/1000 [00:11<00:05, 58.24toy/s]
Background-like: 68%|██████▊ | 681/1000 [00:11<00:05, 58.28toy/s]
Background-like: 69%|██████▊ | 687/1000 [00:11<00:05, 57.01toy/s]
Background-like: 69%|██████▉ | 694/1000 [00:12<00:05, 57.90toy/s]
Background-like: 70%|███████ | 700/1000 [00:12<00:05, 57.34toy/s]
Background-like: 71%|███████ | 707/1000 [00:12<00:04, 58.69toy/s]
Background-like: 71%|███████▏ | 714/1000 [00:12<00:04, 59.88toy/s]
Background-like: 72%|███████▏ | 720/1000 [00:12<00:04, 59.29toy/s]
Background-like: 73%|███████▎ | 726/1000 [00:12<00:04, 58.52toy/s]
Background-like: 73%|███████▎ | 732/1000 [00:12<00:04, 57.83toy/s]
Background-like: 74%|███████▍ | 738/1000 [00:12<00:04, 57.38toy/s]
Background-like: 74%|███████▍ | 744/1000 [00:12<00:04, 56.91toy/s]
Background-like: 75%|███████▌ | 750/1000 [00:13<00:04, 57.42toy/s]
Background-like: 76%|███████▌ | 757/1000 [00:13<00:04, 58.31toy/s]
Background-like: 76%|███████▋ | 763/1000 [00:13<00:04, 58.50toy/s]
Background-like: 77%|███████▋ | 769/1000 [00:13<00:04, 55.57toy/s]
Background-like: 78%|███████▊ | 775/1000 [00:13<00:04, 54.63toy/s]
Background-like: 78%|███████▊ | 781/1000 [00:13<00:03, 55.76toy/s]
Background-like: 79%|███████▉ | 788/1000 [00:13<00:03, 57.42toy/s]
Background-like: 79%|███████▉ | 794/1000 [00:13<00:03, 57.43toy/s]
Background-like: 80%|████████ | 800/1000 [00:13<00:03, 57.09toy/s]
Background-like: 81%|████████ | 807/1000 [00:14<00:03, 59.96toy/s]
Background-like: 81%|████████▏ | 814/1000 [00:14<00:03, 60.99toy/s]
Background-like: 82%|████████▏ | 821/1000 [00:14<00:02, 61.12toy/s]
Background-like: 83%|████████▎ | 828/1000 [00:14<00:02, 60.45toy/s]
Background-like: 84%|████████▎ | 835/1000 [00:14<00:02, 61.74toy/s]
Background-like: 84%|████████▍ | 842/1000 [00:14<00:02, 59.66toy/s]
Background-like: 85%|████████▍ | 849/1000 [00:14<00:02, 59.86toy/s]
Background-like: 86%|████████▌ | 856/1000 [00:14<00:02, 58.36toy/s]
Background-like: 86%|████████▌ | 862/1000 [00:14<00:02, 57.56toy/s]
Background-like: 87%|████████▋ | 868/1000 [00:15<00:02, 57.96toy/s]
Background-like: 88%|████████▊ | 875/1000 [00:15<00:02, 59.58toy/s]
Background-like: 88%|████████▊ | 881/1000 [00:15<00:02, 59.36toy/s]
Background-like: 89%|████████▊ | 887/1000 [00:15<00:01, 58.33toy/s]
Background-like: 89%|████████▉ | 893/1000 [00:15<00:01, 57.29toy/s]
Background-like: 90%|████████▉ | 899/1000 [00:15<00:01, 57.13toy/s]
Background-like: 90%|█████████ | 905/1000 [00:15<00:01, 53.45toy/s]
Background-like: 91%|█████████ | 912/1000 [00:15<00:01, 56.35toy/s]
Background-like: 92%|█████████▏| 918/1000 [00:15<00:01, 57.31toy/s]
Background-like: 92%|█████████▏| 924/1000 [00:16<00:01, 55.63toy/s]
Background-like: 93%|█████████▎| 930/1000 [00:16<00:01, 55.54toy/s]
Background-like: 94%|█████████▎| 937/1000 [00:16<00:01, 57.33toy/s]
Background-like: 94%|█████████▍| 943/1000 [00:16<00:00, 57.68toy/s]
Background-like: 95%|█████████▍| 949/1000 [00:16<00:00, 57.07toy/s]
Background-like: 96%|█████████▌| 955/1000 [00:16<00:00, 57.56toy/s]
Background-like: 96%|█████████▌| 962/1000 [00:16<00:00, 58.30toy/s]
Background-like: 97%|█████████▋| 968/1000 [00:16<00:00, 58.41toy/s]
Background-like: 97%|█████████▋| 974/1000 [00:16<00:00, 57.60toy/s]
Background-like: 98%|█████████▊| 980/1000 [00:17<00:00, 56.12toy/s]
Background-like: 99%|█████████▊| 986/1000 [00:17<00:00, 57.09toy/s]
Background-like: 99%|█████████▉| 992/1000 [00:17<00:00, 57.71toy/s]
Background-like: 100%|█████████▉| 998/1000 [00:17<00:00, 55.76toy/s]
Observed CLs: 0.4878
Expected CLs(-2 σ): 0.0273
Expected CLs(-1 σ): 0.1432
Expected CLs( 0 σ): 0.3778
Expected CLs( 1 σ): 1.0000
Expected CLs( 2 σ): 1.0000
You’ll notice that this time, a progress bar pops up! This is running the fits for each of these toys. We’re hard at work to find new ways to improve the performance of this, but evaluating 50+ toys/s is not very bad for the initial implementation.
Overall, this is not so bad given that we’re only running 1000 toys. What does this look like in a case where we have lower statistics?
Hypothesis Testing (low stats)¶
model_low = pyhf.simplemodels.hepdata_like(
signal_data=[0.5, 1.0], bkg_data=[5.0, 6.0], bkg_uncerts=[0.5, 1.2]
)
In the asymptotics case:
CLs_obs, CLs_exp = pyhf.infer.hypotest(
1.0, # null hypothesis
[5.25, 6.5] + model.config.auxdata,
model,
test_stat="qtilde",
return_expected_set=True,
calctype="asymptotics",
)
print(f" Observed CLs: {CLs_obs:.4f}")
for expected_value, n_sigma in zip(CLs_exp, np.arange(-2, 3)):
print(f"Expected CLs({n_sigma:2d} σ): {expected_value:.4f}")
And now throwing 2000 toys:
CLs_obs, CLs_exp = pyhf.infer.hypotest(
1.0, # null hypothesis
[5.25, 6.5] + model.config.auxdata,
model,
test_stat="qtilde",
return_expected_set=True,
calctype="toybased",
ntoys=1000,
)
print(f" Observed CLs: {CLs_obs:.4f}")
for expected_value, n_sigma in zip(CLs_exp, np.arange(-2, 3)):
print(f"Expected CLs({n_sigma:2d} σ): {expected_value:.4f}")
And as you can see in the case of lower statistics, the asymptotic approximation starts failing!