heterogeneity tests

StatistiXL specifies a test for heterogeneity before combining a number of data sets into a single pool for analysis. If the suggested heterogeneity test is passed (meaning the data is homogeneous), StatistiXL says the data sets can then be aggregated into a single pool for analysis.

The American Institutes for Research (AIR) has stated that a separate test for homogeneity (or heterogeneity) is not required when the direct probability is calculated for each of the data sets and then combined in their Multiple Events Exact Probability (MEEP).

StatXact specifies a two step process: (1) run their HO test to see if the data sets are homogeneous; if that test is passed, then (2) run their CI test to obtain the exact probability for the mulitple pools.

The single pools probability transformed into a standard deviation produces higher standard deviations often compared to the multiple pools standard deviations of AIR or StatXact.

Is there some authority to help decide which is the best alternative?

Dick Biddle

Comments

  • statistiXL provides a procedure for deciding whether it is valid to pool data sets, for contingency table and goodness of fit tests. This procedure is follows Zar (1999), based on the fact that the sum of chi-square values is also a chi-square value. Other approaches such as those of AIR/MEEP and StatXact need to be evaluated on the merits provided by the respective publishers.

    There is no authority that decides which is the best alternative, so it is up to the user of statistical packages to make their own judgement about whether a specific procedure is necessarily appropriate, and to accordingly treat any statistical conclusions warily if there is any doubt or alternative approaches. There are many other examples of various statistical approaches (e.g. the various multiple comparison tests in ANOVA) where the user needs to take responsibility for deciding which statistical approach to take.

    In writing statistiXL, we have tried to be conservative in our approach and provide a thorough help file and clearly documenting the sources of our procedures so that these can be accessed and scrutinised by statistiXL users. In statistiXL, we offer alternative approaches where appropriate (e.g. multiple comparison tests in ANOVA), and in some instances we do not provide the code for certain tests but describe one possible approach (e.g. chi-square heterogeneity testing; and Bartlett's test of homogeneity for ANOVA).

    Philip Withers
    phil@statistixl.com
Sign In or Register to comment.