Step 3.2 argues for the inclusion of a random sample. The primary argument for a random sample is to ensure coverage for Web Content that may not be present in the structured sample. There’s language that makes it clear that this is optional, which is good. But I also feel as though the idea of a random sample is, at best, useful in settings where the goal is academic or where the only end goal is conformance measurement. This tends to ignore the fact that a vast majority of audit work is not academic in nature and not primarily aimed at measuring conformance. The overwhelming volume of auditing work is done to inform and define future work to improve the tested system. The cost and time involved is very high. As a result, the inclusion of random samples is unnecessary scope creep.
Another argument against random samples is that the Note also acknowledges that random sampling won’t be statistically significant. I think this further demonstrates that maybe the random sampling idea is one that should be removed. If the number of random samples is not statistically significant, then it also doesn't serve as a better basis for any conformance claim than the structured sample does.
In short: It seems as though the concept of random sampling exists as a band-aid for a poorly structured sample set. However, the content that exists in Step 3 is already very solid and the concept of random sampling should go away.
Step 3.2 argues for the inclusion of a random sample. The primary argument for a random sample is to ensure coverage for Web Content that may not be present in the structured sample. There’s language that makes it clear that this is optional, which is good. But I also feel as though the idea of a random sample is, at best, useful in settings where the goal is academic or where the only end goal is conformance measurement. This tends to ignore the fact that a vast majority of audit work is not academic in nature and not primarily aimed at measuring conformance. The overwhelming volume of auditing work is done to inform and define future work to improve the tested system. The cost and time involved is very high. As a result, the inclusion of random samples is unnecessary scope creep.
Another argument against random samples is that the Note also acknowledges that random sampling won’t be statistically significant. I think this further demonstrates that maybe the random sampling idea is one that should be removed. If the number of random samples is not statistically significant, then it also doesn't serve as a better basis for any conformance claim than the structured sample does.
In short: It seems as though the concept of random sampling exists as a band-aid for a poorly structured sample set. However, the content that exists in Step 3 is already very solid and the concept of random sampling should go away.