Example
The following table contains an experimental design whereby respondents were shown 10 questions, each of which contains 4 options, with the experiment designed to assess a total of 8 alternatives (e.g., 8 different products). In the 6th question, for example, the respondent is to be asked which they like most and which they like least of alternatives D, A, H and c.
Options
The inputs used to generate the MaxDiff design are shown below.
Number of alternatives The number of alternatives in the experiment. For example, if you are doing a study investigating preferences for 10 brands, then 10 is the number of alternatives.
Add alternative labels Optional labels for the alternatives specified in the first column of the data editor.
Alternatives per question For example, if you have a study of 10 brands, and in each question you show five brands, asking the respondent to choose the one of the five that they like the most and the one that they like the least, then Alternatives per question = 5.
Questions The number of MaxDiff questions to show to respondents. Sawtooth Software suggests that a rough guideline is:
Number of questions >= 3 * Number of alternatives / Alternatives per question
Versions The number of versions of the experimental design (defaults to 1). Where this is set to 2 or more, subsequent versions (after the first version) are obtained by permuting the columns of the binary design while maintaining pairwise balance.
Repeats The number of times that the algorithm seeks to find a solution. The higher the number, the greater the chance that the best possible solution is found. For most problems, this makes little difference (i.e., a marginally sub-optimal experimental design will tend not to have any meaningful consequence on the conclusions drawn from the analyses).
Random order of alternatives Randomize the order of the alternatives within each question. If not checked, alternatives for each question are shown in the order of the labels (or numeric order if no labels are provided).
Detailed outputs If selected, various technical outputs shown below are provided (otherwise, the table above is shown).
Detailed Outputs
The first output is a table showing the design as binary variables. For more than one version this is permuted for each version, hence only the first version is shown.
$binary.design
Version Question 1 2 3 4 5 6 7 8
[1,] 1 1 1 0 0 1 1 0 1 0
[2,] 1 2 0 0 1 0 1 1 0 1
[3,] 1 3 0 0 1 1 0 1 1 0
[4,] 1 4 0 1 1 1 1 0 0 0
[5,] 1 5 0 1 0 0 1 0 1 1
[6,] 1 6 1 0 1 1 0 0 0 1
[7,] 1 7 1 1 0 0 1 1 0 0
[8,] 1 8 1 0 0 0 0 1 1 1
[9,] 1 9 1 1 1 0 0 0 1 0
[10,] 1 10 0 1 0 1 0 1 0 1
The next output is the design described above except without labels (if supplied).
The frequencies indicate the number of times each alternative appears in the experiment. In this case, each person sees each alternative five times. Where this number varies substantially, it can be useful to increase the Number of repeats. Again, this and pairwise frequencies below are shown for the first version.
$frequencies 1 2 3 4 5 6 7 8 5 5 5 5 5 5 5 5
The pairwise frequencies show the number of times that each alternative appears with each other alternative. Note that the main diagonal is a the same as the frequencies. Where there is substantial variation on the off-diagonal entries, it can be useful to increase the Number of repeats.
$pairwise.frequencies
Alternative
Alternative 1 2 3 4 5 6 7 8
1 5 2 2 2 2 2 3 2
2 2 5 2 2 3 2 2 2
3 2 2 5 3 2 2 2 2
4 2 2 3 5 2 2 2 2
5 2 3 2 2 5 2 2 2
6 2 2 2 2 2 5 2 3
7 3 2 2 2 2 2 5 2
8 2 2 2 2 2 3 2 5
The final output is a Correlation Matrix showing the correlations between the columns in the binary design for the first version. Ideally, the off-diagonal entries should be close to 0, and should not contain any large values (e.g., values more than 0.5 or less than -0.5). Where there are large values, you can reduce the size of these by increasing the Number of Questions. Where there is variation, it can be useful to increase the Number of repeats. Note that with incomplete block designs it is not possible for the correlations of the off-diagonal entries to be 0.
$binary.correlations
Alternative
Alternative 1 2 3 4 5 6 7 8
1 1.0 -0.2 -0.2 -0.2 -0.2 -0.2 0.2 -0.2
2 -0.2 1.0 -0.2 -0.2 0.2 -0.2 -0.2 -0.2
3 -0.2 -0.2 1.0 0.2 -0.2 -0.2 -0.2 -0.2
4 -0.2 -0.2 0.2 1.0 -0.2 -0.2 -0.2 -0.2
5 -0.2 0.2 -0.2 -0.2 1.0 -0.2 -0.2 -0.2
6 -0.2 -0.2 -0.2 -0.2 -0.2 1.0 -0.2 0.2
7 0.2 -0.2 -0.2 -0.2 -0.2 -0.2 1.0 -0.2
8 -0.2 -0.2 -0.2 -0.2 -0.2 0.2 -0.2 1.0
For more than one version, the following additional outputs are shown:
- versions.binary.correlations This shows the correlations of the design across all the respondents. The more versions, the less variation that will exist.
- versions.pairwise.frequencies This has the same interpretation as above, except that the analysis is across versions. The more versions, the less variation among the off-diagonal cells.
The following outputs are on a scale between 0 and 1 where 1 means perfect balance, i.e. each alternative or pair of alternatives is shown the same number of times. 0 means the worst possible balance, i.e. each question shows the same alternatives.
- mean.version.balance The average balance (between 0 and 1) of alternatives of all versions.
- mean.version.pairwise.balance The average pairwise balance of pairs of alternatives of all versions.
- across.version.balance The balance of alternatives for the whole design across all versions.
- across.version.pairwise.balance The pairwise balance of alternatives for the whole design across all versions.
Finally the original labelled design is shown.
Technical Details
Single version designs are created with the optBlock algorithm from the AlgDesign R package which optimizes the D-criterion of the design. Multiple-version designs are created by permuting the columns of the single version binary.design. A collection of 20 * Versions such permutations are computed, from which versions that maintain the overall pairwise balance of the multi-version design are greedily selected.
More information
See How to Create a MaxDiff Experimental Design in Q for a guide on using this feature.
See Also
See MaxDiff Software or What is MaxDiff? for an overview of key MaxDiff concepts and resources.
Next
Displayr: How to Create a MaxDiff Experimental Design
Q: How to Create a MaxDiff Experimental Design in Q