From 88151b3fc8a1ad1b675b70a6b696601daf822a13 Mon Sep 17 00:00:00 2001 From: "Felix.S" Date: Thu, 27 Aug 2026 09:48:46 +0200 Subject: [PATCH] Some cosmetic changes --- .gitignore | 4 ++- README.md | 2 +- _quarto.yml | 2 +- index.qmd | 2 +- tutorial-pages/basic-principles.qmd | 2 +- tutorial-pages/check-alpha.qmd | 28 ++++++++++--------- tutorial-pages/check-power.qmd | 26 ++++++++--------- tutorial-pages/number-of-simulations-nrep.qmd | 4 +-- tutorial-pages/purpose.qmd | 2 +- tutorial-pages/sample-size-n.qmd | 16 +++++------ tutorial-pages/seed.qmd | 2 +- .../simulate-for-preregistration.qmd | 2 +- 12 files changed, 48 insertions(+), 44 deletions(-) diff --git a/.gitignore b/.gitignore index dea1398..1c6958e 100644 --- a/.gitignore +++ b/.gitignore @@ -63,4 +63,6 @@ _site/ **/*.quarto_ipynb # ignore posi assistant settings -.posit/assistant \ No newline at end of file +.posit/assistant + +*.code-workspace \ No newline at end of file diff --git a/README.md b/README.md index cd2fb5c..144fbcc 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,7 @@ It is necessary that you work through the sections of the tutorial in order. Ple * [Sample size `n`](./tutorial-pages/sample-size-n.qmd) – How many values should you generate within a simulation? * [Number of repetitions `nrep`](./tutorial-pages/number-of-simulations-nrep.qmd) – How many repeats of a simulation should you run? * [DRY rule](./tutorial-pages/dry-rule.qmd) – How to write your own functions? -* [Simulate to check alpha](./tutorial-pages/check-alpha.qmd) – Write your first simulation and check the rate of false-positive findings. +* [Simulate to check $\alpha$](./tutorial-pages/check-alpha.qmd) – Write your first simulation and check the rate of false-positive findings. * [Simulate to check power](./tutorial-pages/check-power.qmd) – Simulate data to perform a power analysis. * [Simulate to prepare a preregistration](./tutorial-pages/simulate-for-preregistration.qmd) – Simulate data to test statistical analyses before preregistering them. * [General structure](./tutorial-pages/general-structure.qmd) – What is the general structure of a simulation? diff --git a/_quarto.yml b/_quarto.yml index 63c29e8..64c26c6 100644 --- a/_quarto.yml +++ b/_quarto.yml @@ -60,7 +60,7 @@ website: - href: tutorial-pages/dry-rule.qmd text: DRY rule - href: tutorial-pages/check-alpha.qmd - text: Simulate to check alpha + text: Simulate to check $\alpha$ - href: tutorial-pages/check-power.qmd text: Simulate to check power - href: tutorial-pages/simulate-for-preregistration.qmd diff --git a/index.qmd b/index.qmd index b5b97df..f2b6ae6 100644 --- a/index.qmd +++ b/index.qmd @@ -42,7 +42,7 @@ It is necessary that you work through the sections of the tutorial in order. Ple * [Sample size `n`](./tutorial-pages/sample-size-n.qmd) – How many values should you generate within a simulation? * [Number of repetitions `nrep`](./tutorial-pages/number-of-simulations-nrep.qmd) – How many repeats of a simulation should you run? * [DRY rule](./tutorial-pages/dry-rule.qmd) – How to write your own functions? -* [Simulate to check alpha](./tutorial-pages/check-alpha.qmd) – Write your first simulation and check the rate of false-positive findings. +* [Simulate to check $\alpha$](./tutorial-pages/check-alpha.qmd) – Write your first simulation and check the rate of false-positive findings. * [Simulate to check power](./tutorial-pages/check-power.qmd) – Simulate data to perform a power analysis. * [Simulate to prepare a preregistration](./tutorial-pages/simulate-for-preregistration.qmd) – Simulate data to test statistical analyses before preregistering them. * [General structure](./tutorial-pages/general-structure.qmd) – What is the general structure of a simulation? diff --git a/tutorial-pages/basic-principles.qmd b/tutorial-pages/basic-principles.qmd index 7f7fc3d..051ef26 100644 --- a/tutorial-pages/basic-principles.qmd +++ b/tutorial-pages/basic-principles.qmd @@ -3,7 +3,7 @@ aliases: [tutorial_pages/basic-principles.html] title: "Basic principles" --- -Basically, a simulation consists of: +Basically, a simulation in the context of this tutorial consists of: **1) Generating `n` random numbers from a known distribution.** **2) Repeating this `nrep` times.** diff --git a/tutorial-pages/check-alpha.qmd b/tutorial-pages/check-alpha.qmd index 129cc32..4d6afa6 100644 --- a/tutorial-pages/check-alpha.qmd +++ b/tutorial-pages/check-alpha.qmd @@ -1,20 +1,21 @@ --- aliases: [tutorial_pages/check-alpha.html] -title: "Using simulations to check alpha" +title: "Using simulations to check $\alpha$" --- -In most quantitative sciences, we accept a type I error rate of 0.05, which is often called the `alpha` or significance level. This value tells us the probability of rejecting the null hypothesis (i.e. of finding an effect) given that the null hypothesis is true. +In most quantitative sciences, we accept a type I error rate of 0.05, which is often called $\alpha$ or significance level. This value tells us the probability of rejecting the null hypothesis (i.e., of assuming an effect) given that the null hypothesis is true. -In other words, if there is no true effect (e.g. no difference between two groups), we would expect our null hypothesis of no effect to be rejected (incorrectly) (`alpha` * 100)% of the time. +In other words, if there is no true effect (e.g., no difference between two groups), we would expect our null hypothesis of no effect to be incorrectly rejected ($\alpha$ * 100)% of the time. -If you draw from the same normal distribution twice, will the mean of the two samples differ significantly in 5% of the cases? +If you draw from the same normal distribution twice, there is no difference in their *population* means. +Can we reproduce in our simulations that the means of the two *samples* differ significantly in 5% of the cases? *** **YOUR TURN:** 1. Figure out how to do a *t*-test in R. -2. Generate two vectors of 10 values drawn from N(0,1) and compare them with a *t*-test. +2. Generate two vectors of 10 values drawn from $N(0,1)$ and compare them with a *t*-test. 3. Figure out how to extract the *p*-value from that object (explore your R object with the functions `str()` or `names()`). 4. Write a function `simT()` that generates two vectors of `n` values drawn from a standard normal distribution (N(0,1)), compares them with a *t*-test, and returns the *p*-value. 5. Test your function by calling it for `n = 50`. @@ -23,15 +24,16 @@ If you draw from the same normal distribution twice, will the mean of the two sa *** -***p*-values of *t*-tests comparing means from 20 or 100 repetitions simulating N(0,1) with n = 10:** +***p*-values of *t*-tests comparing means from 20 or 100 repetitions simulating $N(0,1)$ with n = 10:**

-In the first case, where `nrep = 20`, we expect 1 out of the 20 tests to be significant (5%). In my case, I did! How many did you get? -In the second case, where `nrep = 100`, we expect 5 out of the 100 tests to be significant. In my case, I got 6. How many did you get? -Are those deviations meaningful? Are they significant? +In the first case, where `nrep = 20`, we expect 1 out of the 20 tests to be significant (5%). With my seed, I did! How many did you get? +In the second case, where `nrep = 100`, we expect 5 out of the 100 tests to be significant. With my seed, I got 6. How many did you get? + +Are those deviations meaningful? Do they significantly differ from the nominal level of 5%? *** @@ -41,12 +43,12 @@ Are those deviations meaningful? Are they significant? *** -***p*-values of *t*-tests comparing means from 1000 repetitions simulating N(0,1) with n=10 or n=100:** +***p*-values of *t*-tests comparing means from 1000 repetitions simulating $N(0,1)$ with n=10 or n=100:**

-In both cases, we expect 50 out of the 1000 tests to be significant by chance (i.e. with a *p*-value under 0.05). In my simulation repetitions, I get 40 and 45 false positive results for `n = 10` and `n = 100`, respectively. How many did you get? +In both cases, we expect 50 out of the 1000 tests to be significant by chance (i.e. with a *p*-value under 0.05). In my repeated simulations, I get 40 and 45 false positive results for `n = 10` and `n = 100`, respectively. How many did you get? These proportions are not significantly different from 5%. @@ -60,9 +62,9 @@ prop.test(45, 1000, p = 0.05, alternative = "two.sided", correct = TRUE) It is important to note that, although `alpha = 0.05` is commonly used, this is an arbitrary choice and you should consider what is an appropriate type 1 error rate for your particular investigation. -Although it isn't necessary to check that a statistical analysis as simple as a *t*-test does not yield more than 5% false-positive results, in situations where the structure of the data is complex and analysed with more advanced models (e.g. when explanatory variables are mathematically linked to each other or are combined in a mixed-effect model), this may allow to compare different modelling approaches and select one that does not produce more than 5% false-positive results. +Although it isn't necessary to check that a statistical analysis as simple as a *t*-test does not yield more than 5% false-positive results, in situations where the structure of the data is complex and analysed with more advanced models (e.g., when explanatory variables are mathematically linked to each other, are combined in a mixed-effect model, or your data deviates from assumptions of the statistical model), this may allow to compare different modelling approaches and select one that does not produce more than 5% false-positive results. -Such complex example, where simulation is the only viable approach to construct a statistical model that does not lead to spurious effects, can be found in this paper: +Such a complex example, where simulation is the only viable approach to construct a statistical model that does not lead to spurious effects, can be found in this paper: * Ihle, M., Pick, J. L., Winney, I. S., Nakagawa, S., & Burke, T. (2019). Measuring Up to Reality: Null Models and Analysis Simulations to Study Parental Coordination Over Provisioning Offspring. *Frontiers in Ecology and Evolution*, *7*, 142. https://doi.org/10.3389/fevo.2019.00142 diff --git a/tutorial-pages/check-power.qmd b/tutorial-pages/check-power.qmd index 0a56461..83f248a 100644 --- a/tutorial-pages/check-power.qmd +++ b/tutorial-pages/check-power.qmd @@ -3,34 +3,32 @@ aliases: [tutorial_pages/check-power.html] title: "Checking power through simulations" --- -The power of a statistical test tells us the probability that the test correctly rejects the null hypothesis. In other words, if we only examine true effects, the power is the proportion of tests that will (correctly) reject the null hypothesis. Often, the power is set to 80%, though, as with `alpha = 0.05`, this is an arbitrary choice. +The power of a statistical test tells us the probability that the test correctly rejects the null hypothesis. In other words, if we only examine true effects, the power is the proportion of tests that will (correctly) reject the null hypothesis. Often, the power is set to 80%, though, as with $\alpha = .05$, this is an arbitrary choice. -Generally, we want to do power analysis before collecting data, to work out the sample size we need to detect some effect. If we are calculating a required sample size, the power analysis can also be called a sample size calculation. +Generally, we want to do a power analysis before collecting data, to work out the sample size we need to detect some effect with a sufficient probability. If we are calculating a required sample size, the analysis is called "a priori power analyses", or simply "sample size calculation". Taking the example of a *t*-test, we need to understand a few parameters: * `n`, the sample size. * `delta`, the difference in means that you want to be able to detect. Deciding what this value should be is tricky. You might rely on estimates from the literature (though bear in mind they are likely to be inflated), or you can use a minimally important difference, which is the threshold below which you do not consider a difference interesting enough to be worth detecting. In a clinical trial, for example, this might be the smallest difference that a patient would care about. * `sd`, the standard deviation. Usually, this needs to be estimated from the literature or pilot studies. -* `sig.level`, the alpha, as discussed previously. +* `sig.level`, the $\alpha$, as discussed previously. * `power`, the power as defined above. -You can calculate any one of these parameters, given all of the others. We usually want to specify, `delta`, `sd`, `sig.level` and `power` and calculate the required sample size. - -We can calculate the required sample size for a *t*-test using: +You can calculate any one of these parameters, given all of the others. In an a-priori power analysis, we need to specify `delta`, `sd`, `sig.level` and `power` to calculate the required sample size: ```r power.t.test(n = NULL, delta = 0.5, sd = 1, sig.level = 0.05, power = 0.8) ``` -Notice that `n = NULL`, so this parameter is calculated. +Notice that `n = NULL`, so this parameter is calculated and returned by the function. The sample size `n` we need, given this set of parameters, is 64 per group. -Just as we can check the alpha of our test by sampling from the same distribution (i.e. simulating data without an effect), we can check the power by sampling from different distributions (i.e. simulating data with an effect). +Just as we can check the $\alpha$ of our test by sampling from the same distribution (i.e., simulating data without an effect), we can check the power by sampling from different distributions (i.e., simulating data with an effect). -If we sample values from two normal distributions with different means (e.g. N(0,1) and N(0.5,1)), what is the minimum sample size we need to detect a significant difference in means with a *t*-test 80% of the time? +If we sample values from two normal distributions with different means (e.g. $N(0,1)$ and $N(0.5,1)$), what is the minimum sample size we need to detect a significant difference in means with a *t*-test 80% of the time? *** @@ -38,21 +36,23 @@ If we sample values from two normal distributions with different means (e.g. N(0 1. Use your simulation skills to work out the power through simulation. Write a function that does the following: - i. Draws `n` values from a random normal distribution with `mean1` and another `n` values from a normal distribution with `mean2`. - ii. Compares the means of these two samples with a *t*-test and extracts the *p*-value. + i. Draw `n` values from a random normal distribution with `mean1` and another `n` values from a normal distribution with `mean2`. + ii. Compare the means of these two samples with a *t*-test and extract the *p*-value. 2. Repeat the function 1000 times using the parameters used in the power calculation above (that used the `power.t.test()` function). 3. Calculate the proportion of *p*-values that are smaller than 0.05. *** -***p*-values of *t*-tests comparing means from 1000 repetitions simulating N(0,1) and N(0.5,1) with n = 64:** +***p*-values of *t*-tests comparing means from 1000 repetitions simulating $N(0,1)$ and $N(0.5,1)$ with n = 64:**

-The proportion of correctly rejected null hypotheses in the simulation is close to 0.8, which is what we would expect. +The proportion of correctly rejected null hypotheses in the simulation is close to 0.8, which is what we would expect: + + Using simulations for power analysis is not really necessary for simple examples like a *t*-test, though it is useful to check your understanding. diff --git a/tutorial-pages/number-of-simulations-nrep.qmd b/tutorial-pages/number-of-simulations-nrep.qmd index 73b17dd..5fd10d8 100644 --- a/tutorial-pages/number-of-simulations-nrep.qmd +++ b/tutorial-pages/number-of-simulations-nrep.qmd @@ -5,14 +5,14 @@ title: "Number of repetitions `nrep`" Sampling theory applies to the number of repetitions `nrep` (also referred to as the number of *replications*) just as much as it does to the sample size `n` within a simulation. -**Means and SDs from 24 repetitions simulating N(0,1) with n = 10:** +**Means and SDs from 24 repetitions simulating $N(0,1)$ with n = 10:**

Now, let's do the same with a number of repetitions `nrep` of 1000. -**Means and SDs from 1000 repetitions simulating N(0,1) with n = 10:** +**Means and SDs from 1000 repetitions simulating $N(0,1)$ with n = 10:**

diff --git a/tutorial-pages/purpose.qmd b/tutorial-pages/purpose.qmd index 743136b..82361fc 100644 --- a/tutorial-pages/purpose.qmd +++ b/tutorial-pages/purpose.qmd @@ -22,7 +22,7 @@ You can use computer simulations to:
* **Prepare a pre-analysis plan.** - * *To be confident about the (confirmatory) statistical analyses you may wish to commit to before data collection (e.g. through a preregistration or registered report), practising the analyses on a simulated dataset is very helpful! If you are still unsure about the most appropriate statistical test to apply to your data, providing a simulated dataset to a statistician or mentor will allow them to provide concrete suggestions! The code containing the analyses of simulated data can be submitted along with your preregistration or registered report for reviewers to exactly understand what analyses you intend to perform. Once you get your real data, you may simply plug them into this code and get the results of your confirmatory analyses immediately!* + * *To be confident about the (confirmatory) statistical analyses you may wish to commit to before data collection (e.g. through a preregistration or registered report), practising the analyses on a simulated dataset is very helpful! If you are still unsure about the most appropriate statistical test to apply to your data, providing a simulated dataset to a statistician or mentor will allow them to provide concrete suggestions. The code containing the analyses of simulated data can be submitted along with your preregistration or registered report for reviewers to exactly understand what analyses you intend to perform. Once you get your real data, you may simply plug them into this code and get the results of your confirmatory analyses immediately.* *** diff --git a/tutorial-pages/sample-size-n.qmd b/tutorial-pages/sample-size-n.qmd index 6396272..800ae17 100644 --- a/tutorial-pages/sample-size-n.qmd +++ b/tutorial-pages/sample-size-n.qmd @@ -4,38 +4,38 @@ title: "Sample size `n`" How many values should you generate within a simulation? Let's explore. -If I draw 10 data points from a normal distribution with a mean of 0 and a standard deviation of 1 (i.e N(0,1)), after setting the seed to 10 (for no specific reason), here is the distribution of the values I get: +If I draw 10 data points from a normal distribution with a mean of 0 and a standard deviation of 1 (i.e., $N(0,1)$), after setting the seed to the arbitrary value of 10, here is the distribution of the values I get: -**1 repetition simulating N(0,1) with n = 10:** +**1 repetition simulating $N(0,1)$ with n = 10:**
-If I repeat this simulation 24 times, here are the distributions of the 10 values pseudo-randomly sampled from N(0,1): +If I repeat this simulation 24 times, here are the distributions of the 10 values pseudo-randomly sampled from $N(0,1)$: -**24 repetitions simulating N(0,1) with n = 10:** +**24 repetitions simulating $N(0,1)$ with n = 10:**

-Note that because we are drawing from N(0,1), we expect the mean of the values drawn (mean(x), blue lines) to be very close to 0, i.e. the mean of the normal distribution we sample from (red dashed lines). +Note that because we are drawing from $N(0,1)$, we expect the mean of the values drawn (blue lines) to be very close to 0, i.e. the true mean of the normal distribution we sampled from (red dashed lines).
How are the means and standard deviations of the 24 repetitions distributed? -**Distributions of the means and SDs from 24 repetitions simulating N(0,1) with n = 10:** +**Distributions of the means and SDs from 24 repetitions simulating $N(0,1)$ with n = 10:**

Now, let's do the same with a sample size `n` of 1000. -**24 repetitions simulating the same distribution, i.e. N(0,1), with n = 1000:** +**24 repetitions simulating the same distribution, i.e. $N(0,1)$, with n = 1000:**

-**Distributions of the means and SDs from 24 repetitions simulating N(0,1) with n = 1000:** +**Distributions of the means and SDs from 24 repetitions simulating $N(0,1)$ with n = 1000:**

diff --git a/tutorial-pages/seed.qmd b/tutorial-pages/seed.qmd index 04f7dcb..4aded79 100644 --- a/tutorial-pages/seed.qmd +++ b/tutorial-pages/seed.qmd @@ -11,7 +11,7 @@ A pseudorandom number generator's number sequence is completely determined by it Thus, if a pseudorandom number generator is reinitialized with the same seed, it will produce the same sequence of numbers. You can set the seed (with any arbitrary number) at the beginning of a script, and, if commands drawing random numbers are run in the exact same order, they will provide the same output in subsequent runs. -This is useful for sharing code and reproduce simulations, as well as for debugging code. +This is useful for sharing code and for reproducing simulations, as well as for debugging code. *** diff --git a/tutorial-pages/simulate-for-preregistration.qmd b/tutorial-pages/simulate-for-preregistration.qmd index a814c6f..6f1a470 100644 --- a/tutorial-pages/simulate-for-preregistration.qmd +++ b/tutorial-pages/simulate-for-preregistration.qmd @@ -10,7 +10,7 @@ For example, let's say you plan to collect an observational dataset and want to A typical protocol might state that the data will be analysed by logistic regression, adjusting for confounders. But which confounders exactly? How will each variable be coded? What type of logistic regression will you use? -Making a dataset that has the variables you expect your real dataset will have allows you to exactly state (in **code** rather than potentially ambiguous words) what you will do. +Preparing a simulated dataset that has the same variables which you expect in your real dataset allows you to exactly state (in **code** rather than potentially ambiguous words) what you will do. This is really useful if you want input, e.g. from a statistician -- they can look at your code and more clearly see what you are trying to do. It has the added benefit of forcing you to really think about what your dataset will look like! I've found this to be very useful in the past.