Anchored MaxDiff experiments are a more exotic version of the standard MaxDiff questions with additional questions designed to work out the absolute importance of the attributes. For more details on this analysis, please read through our Anchored MaxDiff article in our Data Story Guide. The two types of anchored MaxDiff covered in this article are:
However, you should be sure to understand Technical details before following the methods and Interpreting the results. You can review the example below using the QPack here.
Technical details
This type of MaxDiff also is not done through an R-based output, but rather our built-in tables, which use a multinomial logit model to estimate coefficients.
Data Setup
Specific data setup depends on what type of anchored MaxDiff is being performed (see Methods below). Data should be in the legacy MaxDiff format, where there is a variable for each individual alternative for each question coded to indicate relative preferences and which alternative was shown to the respondent in that question. Then to apply the anchor, an extra anchor variable is created for each question/set of alternatives.
The logic of legacy MaxDiff analysis
In order to understand how to set up MaxDiff experiments in the software it is necessary to first understand how the software interprets MaxDiff data. The software interprets each MaxDiff question as a partial ranking. Thus, if a respondent was given a question showing options A, B, C, and F, and chose B as most preferred and F as least preferred, the software interprets this as the following ranking: B > A = C > F. Note that this is a different model to that used by Sawtooth Software.
As discussed in Legacy MaxDiff Specifications, when each of the choice questions is set up: 1 is used for the most preferred item, a -1 for the least preferred item, 0 for the other items that were shown but not chosen, and NaN for the items not shown. Thus the variables representing the alternatives, where B > A = C > D are encoded as follows for one of the tasks for a respondent:
A B C D E F 0 1 0 NaN NaN -1
Remember, the alternatives not shown in the task are coded as NaN. Note that when analyzing this data, the software only looks at the relative ordering of the values (-1, 0, 1) and any other values could be used instead, provided that they imply the same ordering.
Method - Anchored MaxDiff with ratings
Prior to explaining how to use ratings to anchor the MaxDiff it is useful to first understand how ratings data can be combined with the MaxDiff experiment without anchoring. Again the situation where a MaxDiff task reveals that B > A = C > F. Consider a rating question where the six alternatives are rated, respectively, 9, 10, 7, 7, 7, 7, and 3. Thus, the ratings imply that: B > A > C = D = E > F, and this information can be incorporated as if just another question in the MaxDiff experiment:
A B C D E F 9 10 7 7 7 3
Anchoring is achieved by using the scale points - this is seen in the example above. We can use some or all of the scale points as anchors. From an interpretation perspective, it is usually most straightforward to choose a specific point as the anchor value. For example, consider the case above where we decide to use a rating of 7 as the anchor point. We create a new alternative for the analysis which we will call Passive (7), and add a Passive (7) anchor variable to each set of alternatives plus our rating variable set. This is easiest done by inserting a new JavaScript - Numeric variable with a single value (or NaN) in the Expression. Then you can duplicate and modify it for the other variable sets.
The setup of the ratings data is then straightforward. It is just the actual rating variable set, but with an additional item containing the benchmark value:
A B C D E F Passive (7) 9 10 7 7 7 3 7
Below is a preview of the NPS likelihood to recommend variables we are using in the Data Sets tree, along with the Passive (7) variable that anchors those values:
The standard MaxDiff variables only focus on relativities and are set up in the basic way described in Technical details above. Since these questions do not have a follow-up question or scaled assumption on which to anchor them, the Passive (7)anchor variable for each set of MaxDiff alternatives is given a missing value (NaN). For example, where a MaxDiff question reveals that B > A > C = D = E > F, we include this new anchoring alternative, but it is assigned a value of NaN as nothing is learned about its relative appeal from this task.
A B C D E F Passive (7) 0 1 0 NaN NaN -1 NaN
You would have a Passive (7) anchor variable with NaN value for each group of alternatives/questions in your MaxDiff. Below is a screenshot of how that would look like in the Data Sets tree:
After confirming, all variables are named the same across variable sets and they are in a consistent order, you will then highlight all of them in the Data Sets tree and Combine them into a set with a Structure > Ranking. You'll see a preview here with the NPS variables on top and all the MaxDiff variables beneath:
When you drag the variable set to the page, it will create a table of the Probability %, which you can change to Coefficient, and you will need to drag your anchor row to the top to make its coefficient 0:
See Interpreting the results below.
Method - Anchored MaxDiff with dual-response
Dual-response MaxDiff data should be recoded according to the respondent's response to each task's follow-up dual-response question. In the software, the anchoring is accommodated by introducing a new alternative for each choice question that anchors the user's ratings to a passive or neutral score. In the case of the dual response, we will call this new alternative Zero. The Zero alternative is always assigned a value of 0. This is easiest done by inserting a new JavaScript - Numeric variable with a 0 in the Expression. The value assigned to the other alternatives is then relative to this zero value.
Ultimately, new variables need to be created that combine the MaxDiff data with the dual-response anchoring data for each MaxDiff task. In the example where the MaxDiff has 6 alternatives (A-F), plus the dual-response, with 4 tasks in the MaxDiff, 28 new variables will need to be created (6 alternatives x 4 tasks + 4 dual-response = 28).
Coding the data for the analysis
Consider again the situation where the respondent has been faced with a choice of A, B, C, and F and has chosen B as best and F as worst, which leads to B > A = C > F.
Example coding when respondent says all are important
Where all of the items are important this implies that all are ranked higher than the Zero option:
A B C D E F Zero 2 3 2 NaN NaN 1 0
Example coding when respondent says some are important
Where some of the items are important this implies that the most preferred item must be more important than Zero, the least preferred item must be less preferred than Zero, but we do not know the relative preference of the remaining items relative to Zero, and thus this is coded as:
A B C D E F Zero 0 1 0 NaN NaN -1 0
Note that although this coding implies that A = C = Zero, the underlying algorithm does not explicitly assume these things are equal. Rather, it simply treats this particular set of data as not providing any evidence about the relative ordering of these alternatives.
Example coding when respondent says none are important
Where none of the items are important, this implies all items have a score less than Zero.
A B C D E F Zero -2 -1 -2 NaN NaN -3 0
Worked examples for recoding using best/worst variables
Many MaxDiff data formats store the responses as one best and one worst variable per MaxDiff task along with the response to the dual-response question. New variables must be manually created per the logic above in the software or created externally and added into the data set by merging data files.
When some are important
Given the MaxDiff design above with 6 alternatives. Respondent 1's first task has alternatives A, B, C, F. The best choice for Respondent 1, task 1 is B (alternative 2), and the worst choice is F (alternative 6), therefore values for best and worst are 2 and 6, respectively. A 2 for Task1_Dual means some are important:
Task1_Best Task1_Worst Task1_Dual
2 6 2
The new variables created from combining the MaxDiff responses with the anchor data using the logic outlined above looks like below where the best (alternative 2) is greater than 0, the worst (alternative 7) as less than 0, and the other alternatives that are not selected are 0.:
task1alt1 task1alt2 task1alt3 task1alt4 task1alt5 task1alt6 task1zero
0 1 0 NaN Nan -1 0
When all are important
This example is the same as when some are important, only this time, the Task1_Dual is "1", indicating that all alternatives are important:
Task1_best Task1_Worst Task1_Dual
2 6 1
The new variables created from combining the MaxDiff responses with the anchor data using the logic outlined above look like this:
task1alt1 task1alt2 task1alt3 task1alt4 task1alt5 task1alt6 task1zero
2 3 2 NaN Nan 1 0
Alternative 2 (B) is assigned 3 because it is the best, alternatives 1 (A) and 3 (C) are assigned 2 because they were seen in the task, not as preferred as alternative 2 but better than the worst alternative which is alternative 7 (F), which is assigned 1. Since they are all preferred (Task1_Dual = 1), they are all greater than 0. The Zero alternative must be included for each task and is always assigned 0. Any alternatives that did not appear in the task are assigned NaN.
When none are important
Finally, if the Task1_Dual value is 3, meaning none of the alternatives are important:
Task1_best Task1_Worst Task1_Dual
2 6 3
Then all of the values fall below zero, with the most important closest to zero, and the least important furthest from zero:
task1alt1 task1alt2 task1alt3 task1alt4 task1alt5 task1alt6 task1zero -2 -1 -2 NaN Nan -3 0
Analyzing the data
Once you've created all of your recoded and zero variables for each task, you will Combine them into a variable set with a Structure > Ranking. When you drag the variable set to the page, it will create a table of the Probability %, which you can change to Coefficient, and you will need to drag your anchor row Zero to the top to make its coefficient 0. See Interpreting the results.
Interpreting the results
The table below is the output of an Anchored MaxDiff experiment anchored using ratings. The ratings used were a likelihood to recommend scale. The anchor value chosen was 7. This row will need to be moved to the top of the table to apply the anchor and set it as the benchmark (so its coefficient is 0). The table is crossed by gender, and significance tests indicated by the color-coding are showing relative performance.
If we wish to test whether the difference between Apple by gender is significant we need to remove all the other brands from the analysis. You can do this by selecting those rows on the table, right-click and Delete. (To add the other rows back you can right-click on the table and Reset > Table Categories.) This is shown on the left below for Apple. The table on the right only contains the data for Intel; it shows that Intel is significantly more preferred, in an absolute sense, amongst men than women.
To see how the brands perform in an absolute sense for a particular gender, we need to use gender as a filter rather than as the Columns (the software interprets genders in columns as meaning you are wanting to compare the genders). The table output below shows the table with a Filter for women and with a Planned Test of Statistical Significance explicitly comparing Apple with the benchmark value of 0, which reveals that the absolute performance of Apple amongst women is significantly below the benchmark rating of 7 out of 10.