Merge branch 'luca_updates'

Author: pglpm

vignettes/figure/hist10-1.png

@@ -373,9 +373,9 @@ which produces an object `learnt10` (just a string, in this case) which can be u
 
 ## Examples of analysis
 
-### Estimating the relative frequencies of the three species
+### Estimating the relative frequencies of the three penguin species
 
-Consider our first research question **Q1**: what's the overall statistical occurrence of the three species in the whole population? The relative frequencies of the three species in the whole population are unknown to us, so we cannot give a simple answer such as "Adélie: 0.10, Chinstrap: 0.34, Gentoo: 0.56".
+Consider our first research question **Q1**: "what's the overall statistical occurrence of the three species in the whole population?" The relative frequencies of the three species in the whole population are unknown to us, so we cannot give a simple answer such as "Adélie: 0.10, Chinstrap: 0.34, Gentoo: 0.56".
 
 Bayesian nonparametrics and ***inferno*** give a first answer to this question in the form of:
 
@@ -413,7 +413,10 @@ The answer to our question is now contained in the `Fspecies10` object (we chose
 plot(Fspecies10)
 ```
 
-<img src="figure/vis10-1.png" width="100%" />
+<div class="figure">
+<img src="figure/vis10-1.png" alt="**Estimates and uncertainty of relative frequencies of the three penguin species**" width="100%" />
+<p class="caption">**Estimates and uncertainty of relative frequencies of the three penguin species**</p>
+</div>
 
 The x-axis of this plot shows the three possible values of the `species` variate. The y-axis reports fractions which may be read as *frequencies* or *probabilities*.
 
@@ -424,135 +427,95 @@ The three estimated frequencies are Adélie: 0.32, Chinstrap: 0.31, Gentoo: 0
 
 ``` r
 Fspecies10$values
-##       X
-## Y          [,1]
-##   [1,] 0.315265
-##   [2,] 0.310161
-##   [3,] 0.374574
+##            X
+## Y               [,1]
+##   Adelie    0.315265
+##   Chinstrap 0.310161
+##   Gentoo    0.374574
 ```
 
-What about the uncertainty of these estimates? In Bayesian theory, a compact way of expressing the uncertainty in a quantity is by means of a **credibility interval** with a given probability. For instance, if we say that a particular quantity has a 80%-credibility interval equal to $(0.41, 0.69)$, what we mean is that there's an 80% probability that the true value of that quantity is between $0.41$ and $0.69$ -- pretty straightforward! (*Please be careful not to confuse the Bayesian credibility interval with a "confidence interval"*: the latter has a much more involved and less straightforward meaning.)
+What about the uncertainty of these estimates?
 
-The `Pr()` function by default calculates two credibility intervals for each frequency estimate: a 50% one, and an 89% one. In our present inference, the 50%-credibility intervals are shown in the plot above as the darker grey band, and the 89%-credibility interval as the lighter grey band. Note that this interval contains the 50% one.
+### Uncertainty of the estimates: credibility intervals and probabilities
 
-For instance, the plot indicates that there's a 50% probability that the relative frequency of all Adélie penguins is roughly between 0.25 and 0.4; and an 89% probability that their relative frequency is roughly between 0.3 and 0.45.
+In Bayesian theory, a compact way of expressing the uncertainty in a quantity is by means of a **credibility interval** with a given probability. For instance, if we say that a particular quantity has a 80%-credibility interval equal to $(0.41, 0.69)$, what we mean is that there's an 80% probability that the true value of that quantity is between $0.41$ and $0.69$ -- pretty straightforward! (*Please be careful not to confuse the Bayesian credibility interval with a "confidence interval"*: the latter has a much more involved and less straightforward meaning.)
 
-For the Gentoo species, there's a 50% probability that its relative frequency is roughly between 0.3 and 0.45; and an 89% probability that their relative frequency is roughly between 0.2 and 0.6.
+The `Pr()` function by default calculates two credibility intervals for each frequency estimate: a 50% one, and an 89% one. In our present inference, the 50%-credibility intervals are shown in the plot above as the darker grey band, and the 89%-credibility intervals as the lighter grey band. Note that these intervals contain the 50% ones.
 
-The credibility intervals can be read from the `quantiles` element of the `Fspecies10` object:
+For instance, the plot indicates that there's a 50% probability that the relative frequency of all `Adelie` penguins is roughly between 0.25 and 0.4; and an 89% probability that their relative frequency is roughly between 0.3 and 0.45.
 
+For the `Gentoo` species, there's a 50% probability that its relative frequency is roughly between 0.3 and 0.45; and an 89% probability that their relative frequency is roughly between 0.2 and 0.6.
 
-``` r
-Fspecies10$quantiles
-## , ,  = 5.5%
-## 
-##       X
-## Y          [,1]
-##   [1,] 0.134878
-##   [2,] 0.132087
-##   [3,] 0.180867
-## 
-## , ,  = 25%
-## 
-##       X
-## Y          [,1]
-##   [1,] 0.227634
-##   [2,] 0.219339
-##   [3,] 0.284369
-## 
-## , ,  = 75%
-## 
-##       X
-## Y          [,1]
-##   [1,] 0.395804
-##   [2,] 0.391340
-##   [3,] 0.457973
-## 
-## , ,  = 94.5%
-## 
-##       X
-## Y          [,1]
-##   [1,] 0.525858
-##   [2,] 0.521080
-##   [3,] 0.587441
-```
+We can Actually do more: we can plot the probabilities of all possible frequencies. To understand this idea, let's ask: what is the relative frequency of Adélie penguins in the whole population? Possible values could be anything between 0 and 1. But some of these values may be more probable than others. If we look at our 10 samples, we see that 3 out of 10 are `Adelie`. So a relative frequency around 0.3 is a little more probable, although there's still a lot of uncertainty because this is just a small sample.
 
-As the name tells, this element actually report the *quantiles* of the probability. For example, look at the `= 25%` output: among other things it says `Adelie 0.227634`. This means that there's a 25% probability that the frequency of Adélie penguins is below $0.227634$, and thus a 75% probability that it's above $0.227634$.
+The `Pr()` function has actually calculated, in an approximate way, the probabilities for all possible frequencies of each species. We can plot the probabilities of the frequencies of the `Adelie` species as follows:
 
 
 ``` r
+# # select only Adelie
 onlyAdelie <- subset(Fspecies10, list(species = 'Adelie'))
-hist(onlyAdelie, legend = 'topright')
-```
 
-<img src="figure/hist10-1.png" width="100%" />
+hist(onlyAdelie, xlim = c(0, 1), legend = 'topright', col = 2)
+```
 
+<div class="figure">
+<img src="figure/hist10adelie-1.png" alt="**Probability distribution for the frequency of Adelie penguins**" width="100%" />
+<p class="caption">**Probability distribution for the frequency of Adelie penguins**</p>
+</div>
 
-by giving the probabilities of all possible relative-frequency distributions.
+With the `subset()` function we first extract from the `Fspecies10` object only the information pertaining to the `Adelie` species, saving it into the `onlyAdelie` object. Then the `hist()` function plots the probability distribution for all the frequency values. We see that frequencies between 0.2 and 0.4 have highest probability. But the probability distribution is quite wide; even a frequency of 0.6 has some probability. The vertical line indicates the mean of this probability distribution, which we take as our frequency estimate.
 
+The width of this probability distribution reflects our large uncertainty, which in turn comes from the fact that we have only seen a small sample of penguins. We shall see that this uncertainty decreases and we gather more samples.
 
+\
+The credibility intervals are calculated from the probability distribution above. They can be quickly obtained from the `quantiles` element of the `Fspecies10` object. For instance, we can read the 89%-credibility intervals as follows:
 
 
 ``` r
-prQ1 <- Pr(
-    Y = data.frame(species = c('Adelie', 'Chinstrap', 'Gentoo')),
-    X = NULL,
-    learnt = learnt10,
-    parallel = parallel
-)
-## 
-## Registered doParallelSNOW with 8 workers
-## 
-## Closing connections to cores.
-
-plot(prQ1, ylim = c(0, 1))
+Fspecies10$quantiles[, , c('5.5%', '94.5%')]
+##            
+## Y               5.5%    94.5%
+##   Adelie    0.134878 0.525858
+##   Chinstrap 0.132087 0.521080
+##   Gentoo    0.180867 0.587441
 ```
 
-<div class="figure">
-<img src="figure/freqs10-1.png" alt="plot of chunk freqs10" width="100%" />
-<p class="caption">plot of chunk freqs10</p>
-</div>
+<!-- As the name tells, this element actually report the *quantiles* of the probability. For example, look at the `= 25%` output: among other things it says `Adelie 0.227634`. This means that there's a 25% probability that the frequency of Adélie penguins is below $0.227634$, and thus a 75% probability that it's above $0.227634$.
+ -->
 
-``` r
+### A preliminary report
 
-hist(prQ1, xlim = c(0, 1))
-```
+We can now give a preliminary answer to our question **Q1**, "what's the overall statistical occurrence of the three species in the whole population?".
 
-<div class="figure">
-<img src="figure/freqs10-2.png" alt="plot of chunk freqs10" width="100%" />
-<p class="caption">plot of chunk freqs10</p>
-</div>
+A summary answer could be as follows:
 
+> From a sample of 10 penguins, the inference about the relative frequencies of the three species in the full population is as follows:
+> 
+> Adélie
+> : rel. frequency between 0.143 and 0.53, with 89% probability
+> 
+> Chinstrap
+> : rel. frequency between 0.13 and 0.52, with 89% probability
+> 
+> Gentoo
+> : rel. frequency between 0.18 and 0.59, with 89% probability
+
+But we can also give a fuller answer by displaying the probability distributions for all three frequencies:
 
-``` r
-prQ1 <- Pr(
-    Y = data.frame(species = c('Adelie', 'Chinstrap', 'Gentoo')),
-    X = NULL,
-    learnt = 'learnt_all.rds',
-    parallel = parallel
-)
-## 
-## Registered doParallelSNOW with 8 workers
-## 
-## Closing connections to cores.
 
-plot(prQ1, ylim = c(0, 1))
+``` r
+# # select only Adelie
+hist(Fspecies10, legend = 'topright', col = 2:4)
 ```
 
 <div class="figure">
-<img src="figure/freqsall-1.png" alt="plot of chunk freqsall" width="100%" />
-<p class="caption">plot of chunk freqsall</p>
+<img src="figure/hist10-1.png" alt="**Probability distribution for the rel. requencies of the three species**" width="100%" />
+<p class="caption">**Probability distribution for the rel. requencies of the three species**</p>
 </div>
 
-``` r
+It's important to keep in mind that the three frequencies must add up to 1, so if our future estimates of some of them increase, then the estimate of some others must decrease.
 
-hist(prQ1, xlim = c(0, 1))
-```
 
-<div class="figure">
-<img src="figure/freqsall-2.png" alt="plot of chunk freqsall" width="100%" />
-<p class="caption">plot of chunk freqsall</p>
-</div>
 
 
 
@@ -845,13 +808,13 @@ dataset <- penguins
 Here are the values for two subjects:
 
 
-|    |species |island | bill_len| bill_dep| flipper_len| body_mass|sex    | year|
-|:---|:-------|:------|--------:|--------:|-----------:|---------:|:------|----:|
-|173 |Gentoo  |Biscoe |     50.2|     14.3|         218|      5700|male   | 2007|
-|210 |Gentoo  |Biscoe |     45.5|     15.0|         220|      5000|male   | 2008|
-|45  |Adelie  |Dream  |     37.0|     16.9|         185|      3000|female | 2007|
-|52  |Adelie  |Biscoe |     40.1|     18.9|         188|      4300|male   | 2008|
-|247 |Gentoo  |Biscoe |     44.5|     14.7|         214|      4850|female | 2009|
+|    |species |island    | bill_len| bill_dep| flipper_len| body_mass|sex    | year|
+|:---|:-------|:---------|--------:|--------:|-----------:|---------:|:------|----:|
+|30  |Adelie  |Biscoe    |     40.5|     18.9|         180|      3950|male   | 2007|
+|81  |Adelie  |Torgersen |     34.6|     17.2|         189|      3200|female | 2008|
+|108 |Adelie  |Biscoe    |     38.2|     20.0|         190|      3900|male   | 2009|
+|28  |Adelie  |Biscoe    |     40.5|     17.9|         187|      3200|female | 2007|
+|107 |Adelie  |Biscoe    |     38.6|     17.2|         199|      3750|female | 2009|