For a fuller discussion of software for multiphase experiments see Brien (2017c, Section 5).

Designing multitiered experiments

Because multitiered experiments are often constructed using standard designs, the tools available for obtaining standard designs can be used. Thus, as Brien (2017c, Section 5.1) suggests, one might (i) deploy a straightforward standard design, like a randomized complete-block design; (ii) manually construct a design, perhaps using design keys; (iii) consult a catalogue of standard designs; or (iv) search for a design using computer algorithms.

Three packages for the R statistical computing environment that can play a role in obtaining the layout for a multitiered experiment are: dae and od.

The package dae has two functions that are particularly applicable: designRandomize, for randomizing systematic layouts such as when straightforward standard designs are to be employed, and designAnatomy, for producing the decomposition table that exhibits the anatomy of the design. For examples that use these functions see a two-phase sensory experiment and a two-phase wheat experiment. Scripts for other examples are available in the Supplementary materials for Brien (2017c).

On the other hand, od can be used to obtain an optimal multiphase design. For a two-phase design, this involves two steps. Firstly, obtain the first-phase design, using any of the methods above, including generating it with od. Secondly, based on a mixed model for the two-phase experiment (e.g. the prior allocation model), use od to obtain the allocation of the first-phase units to the second-phase units that are optimal for specified values of the variance parameters in the mixed model for the experiment.

Another possibility is to use PROC OPTEX in SAS to obtain an optimal multiphase design.

Analysis of multitiered experiments

Two methods that can be used to analyse the data are mixed-model estimation and analysis of variance.

Mixed-model estimation

Mixed-model estimation, based on Residual or Restricted Maximum Likelihood (REML) is widely available. It is particularly applicable to nonorthogonal experiments and those with pseudofactors, because combined estimates will be obtained and the specification of pseudofactors can usually be avoided. It is strongly advocated that the mixed model is allocation-based.

However, it has the disadvantage that, when random terms are totally confounded by fixed effects, they need to be removed for the fitting algorithms to work. In addition, when recipient random terms from the first allocation are exhaustively confounded by recipient random terms from the second allocation, then some must be omitted. Consequently, the model supplied to a procedure may not be the full allocation-based mixed model for the data, but a model of convenience. Also, convergence problems may be experienced when some random terms are near zero or have low degrees of freedom. In addition, low degrees of freedom cause difficulties because of the asymptotic approximations on which the analysis is based. (See Bailey and Brien (2016, section 11.3).

As an example, take the meatloaf tasting experiments. The initial allocation model for this experiment is:

Rosemary + Irradiation + Rosemary^Irradiation | Blocks + Blocks^Plots + Sessions + Sessions^Panellists + Sessions^Time-orders + Sessions^Panellists^Time-orders.

However, the skeleton analysis-of-variance table for this example shows that Blocks and Sessions are confounded so that the variance matrix for the model will be singular. To be able to fit a mode, on of Blocks and Sessions will have to be omitted.

Analysis of variance

The AMTIER procedure (Brien and Payne, 2006) is purpose-built to perform the analysis of variance of multitiered experiments that can be specified with three formulae and are structure balanced; it uses the algorithm described by Brien and Payne (1999). If a response variable is not supplied, it generates a skeleton ANOVA table.

To perform an analysis using standard analysis of variance or regression software, it will be necessary to determine the set of sources that will produce the set of sums of squares that correctly partitions the total sums of squares . That is, the derivation of the analysis of variance table will have to be done manually and the sums of squares added to it once they have been obtained using a package. In practice this is only feasible for orthogonal designs

For example, three analysis formula were used in deriving the decomposition tables for the meatloaf tasting experiment illustrating the proposed procedure. However, although there are problems as discussed below, this analysis could be obtained with the following single structure formula:

Sessions / (Panellist + Time-orders) + Rosemary * Irradiation * Blocks

This formula will provide all the sums of squares that are required in the ANOVA table, corresponding to the skeleton analysis-of-variance table for this example. The interactions of Rosemary and Irradiation with Blocks will provide the Residual sums of squares once they have been summed.