Statistical Tests View

The Statistical Tests View offers tools for statistical analysis.

Normality Check

Check for normality of the data is done using ALGLIB's Jarque-Bera-test [4]. It requires a sample size of at least 5 samples. The test checks whether sample data is normally distributed (H0) or not (H1). The table shows the calculated p-values. If a p-value is below the significance level α (default: 0.05), the chance that the data is normaly distributed is very unlikely:

p ≤ α: reject H0, data is not normally distributed
p > α: accept H0, data is very likely normally distributed

A green check is only given if all groups are normally distributed. This test should be given special attention if the t-test is used afterwards as the t-test requires normally distributed sample data (except for large sample sizes, see central limit theorem [5]).

Statistical Testing beween Groups

The statistical tests view provides 3 different tests: 
•	T-test
•	Mann-Whitney-U-test
•	Kruskal-Wallis-test

The first 2 tests are provided by the ALGLIB package (see ALGLIB documentation [0, 1]) while the Kruskal-Wallis test is a C# port of R's kruskal.test() function [2].

The p-values calculated by all of the 3 tests can be interpreted as follows: The null hypothesis H0 states that the distributions of the compared groups are equal while H1 states the opposite (the samples are from populations with different distributions). 
All tests, similar to the Jarque-Bera-test, calculate a p-value. If this value is below the significance level α (default: 0.05), the null hypothesis is rejected, which therefore means that the groups do not have the same distribution:
 
p ≤ α: reject H0, groups don't come from the same distribution
p > α: accept H0, groups have equal distribution

Statistical Testing of all Groups

Statistical hypothesis testing between all groups is performed using the Kruskal-Wallis-test [3]. A green check is given if any of the groups is different. Note that if H0 gets rejected, this means that there is at least one group that has another underlying distribution as at least one other group. The test provides no information about which group(s) are different. Therefore, if H0 gets rejected, pairwise tests should be performed to answer that question. 

Pairwise Statistical Tests

For pairwise comparision the Mann-Whitney-U-test as well as the t-test is used. They also require group sizes of at least 5 samples. As mentioned before, the t-test requires the groups to be normally distributed (though it does not require the groups to have equal variance; this is called a two-sample unpooled test) while the Mann-Whitney-U-test is a non-parametric test that can be used for samples with unknown distribution.
Besides the p-values, also corrected p-values are calculated using the Bonferroni-Holm correction [7] (implemented as described at [8]) to control the familywise error rate. 
Additionally, the effect size is calculated using Cohen's d and Hedges' g. The effect size describes the strength of a phenomen [6]. Higher values mean that there is a stronger effect. Statistical hypothesis tests may, especially with high sample sizes, lead to significant results even though there may not be a relevant difference. In such a case the effect size is small and therefore gives a hint that the practical relevance may not be given. On the other hand these values are also often used for making considerations about the required sample size: Lower d's and g's may mean that a larger sample size is required.

References:
[0]  http://www.alglib.net/hypothesistesting/studentttest.php
[1] http://www.alglib.net/hypothesistesting/mannwhitneyu.php
[2] https://stat.ethz.ch/R-manual/R-devel/library/stats/html/kruskal.test.html
[3] http://en.wikipedia.org/wiki/Kruskal%E2%80%93Wallis_one-way_analysis_of_variance
[4] http://www.alglib.net/hypothesistesting/jarqueberatest.php
[5] http://en.wikipedia.org/wiki/Central_limit_theorem
[6] http://en.wikipedia.org/wiki/Effect_size
[7] http://en.wikipedia.org/wiki/Holm%E2%80%93Bonferroni_method
[8] http://www.mathworks.com/matlabcentral/fileexchange/28303-bonferroni-holm-correction-for-multiple-comparisons