This page contains links to MaxDiff resources for Q and Displayr. The main sections of the page lists the key resources and concepts. The bottom of the page contains additional resources. Topics include:
- What is MaxDiff?
- MaxDiff Analyses Available
- Saving analysis results
- Relationship of MaxDiff to other techniques
What is MaxDiff?
The Data Story Guide has a quick overview as well as a more detailed webinar and supplemental ebook for MaxDiff. The Displayr blog has lots of helpful articles discussing specific approaches and techniques. The Learn More About MaxDiff article in particular is a great place to start to learn about how the analysis works and what techniques to consider in your analysis.
Depending on what data you have and how you'd like to analyze the MaxDiff experiment, our standard MaxDiff features available are in:
- Displayr: Anything > Advanced Analysis > MaxDiff
- Q: Create > Marketing > MaxDiff
They include items to create the experimental design, analyze the responses, run diagnostics, and present results. Most modules pertain to our standard, R-based algorithms Multinomial Logit, Latent Class Analysis, Varying Coefficients, and Hierarchical Bayes (the current-day gold standard).
There are other methods of analyzing MaxDiff data and ways of handling exotic MaxDiff designs that can be done in the software as well, such as:
- Counting Analysis - the most rudimentary way of analyzing the data.
- Anchored MaxDiff (definition here) using best/worst selections - see general steps on How to Setup a MaxDiff Experiment as a Ranking.
- Our technical Anchored MaxDiff article describes how to setup and analyze MaxDiff data stored as Ratings (i.e. 1-10) and also how to incorporate a dual-response option to Anchored MaxDiff.
- Other exotic techniques.
Analyzing MaxDiff using standard R-based modules
These modules support Multinomial Logit, Latent Class Analysis, Varying Coefficients, and Hierarchical Bayes (the current-day gold standard) algorithms. You will need 2 bits of information to analyze your data using these methods:
1. Experimental design
Experimental designs are needed so the software knows what alternatives were shown to respondents. If you are embarking on a new MaxDiff experiment, you can create your own experimental design in the software, see Advanced MaxDiff Experimental Designs for an overview of the different types of designs, and How to Create a MaxDiff Experimental Design in Q and How to Create a MaxDiff Experimental Design in Displayr. Once the design has been created it should be checked. This is discussed in How to Check an Experimental Design.
If you've already fielded a MaxDiff experiment and want to analyze it in the software, you can get the design in the software by:
- Copying over the experimental design output from a Q project or Displayr document where it was originally created.
- Add a design file (usually in .csv or .xlsx format) exported from the survey software as a new data set.
- Use the Paste or Enter Table feature and paste in the design exported from the survey software into an item in the report.
Format of experimental design
No matter how you get your design into the software, it will always need to have a consistent format.
- Column 1 should be called Version, and it should tell us which version of the experiment each row corresponds to. If you have a single version of the experiment, then this column should simply contain the number '1' in each row. If there are multiple versions, then this column should contain integers starting at 1 and ranging up to the total number of versions you have included. These should be in blocks, so that all rows for version number 1 appear together, followed by all rows for version number 2, and so on.
- Column 2 should be called Task or Question, and it should contain a number that tells us which task within the experiment each row corresponds to. If there are 6 tasks shown to each respondent, then this column should contain numbers 1 to 6, in order. These should be repeated for each task, so that the rows for version number 2 begin again at 1.
- The remaining columns are for the alternatives shown. If there are 5 alternatives shown to each respondent in each task, then you should have 5 columns here. The numbers in these columns indicate which of the total set of alternatives are shown in each alternative of each task. The names of these columns will not affect the analysis.
- There should be no hidden rows and no extraneous rows or columns.
We also have a couple built-in automations to format MaxDiff data from Alchemer or Qualtrics into the needed format:
- In Displayr: Using Qualtrics and Using Alchemer.
- In Q: Using Qualtrics and Using Alchemer
Example of an experimental design in Excel
You can download example files from the Q Help Center. The following screenshot shows a file where there are at least 2 versions, 13 tasks (i.e., questions) per version, 4 alternatives per task, and 13 alternatives.
Example experimental design in Displayr or Q
The following is an example of an experimental design which was pasted into a table in the software. It has 2 versions, 6 tasks/questions in each, and 5 alternatives for each task:
2. A respondent data file
The respondent data file is the actual responses from respondents from the MaxDiff questions. Different analyses methods require different data file setups. Most of the time, the approach described for standard analyses is appropriate (if this is your first study, start with this).
Standard analyses
If performing a standard R-based analysis (denoted by the R icon) from the MaxDiff sub-menus in Displayr and Q, the data file needs to be set up as follows:
- It needs to be a data file with good quality metadata (e.g., SPSS .SAV, Triple-S, MDD, or, a data file set up in R with factors).
- One variable needs to indicate the version (if versions have been used in the experimental design). This variable would simply contain a number for each person. The numbers should begin at '1' and range up to the number of versions that are present in the design.
- Each of the 'best' or 'most preferred' questions and each of the 'worst' or 'least preferred' questions needs to be represented by its own variable, where the variable needs to be either Categorical or Pick One (Q), Nominal (SPSS, Displayr), or a factor (R), with labels containing the wordings of the alternative that were selected. Ideally, the Value Attributes should be consistent across these variables. For example, if 'Price' is the third alternative in the experimental design, then each variable containing the respondents choices should have a value of 3 if 'Price' is chosen, regardless of what other alternatives were available in that question (i.e., 'Price' should not be stored as a 3 in one variable and a 4 in another). When selecting these variables in the Object Inspector for the analysis, they should be selected in the same order as the design (e.g. the variable for the first task should appear first in the selection, followed by the variable for the second task, and so on).
- Note, the newer Latent Class and Hierarchical Bayes modules for MaxDiff can handle the best and worst variables coded by the option number of the question (i.e. Price was the second alternative shown in the question and will be coded as 2 if chosen). However, you need to be extra sure that your variables are formatted as numeric per Marketing - MaxDiff - Hierarchical Bayes.
You can download example files from the Q Wiki.
Analyzing more advanced MaxDiff designs such as Anchored MaxDiff
More exotic analyses, such as anchored MaxDiff, require the MaxDiff experiment to be set up as a variable set with a Ranking structure. When using a Ranking set in a built-in table, a multinomial logit model is used to estimate the coefficients (average utilities) automatically. The Legacy MaxDiff Case Study describes how to interpret results. See Setting Up a MaxDiff Experiment as a Ranking for more information on how data should be structured when using Best/Worst selections. If using MaxDiff data stored as Ratings (i.e. 1-10) or using a dual-response option, you should refer to our technical Anchored MaxDiff article.
Analyzing using R
More exotic modeling can be performed on MaxDiff in an R Calculation. Examples include:
- Analyzing MaxDiff Using Standard Logit Models Using R
- Analyzing MaxDiff Using the Rank-Ordered Logit Model With Ties Using R
Analyzing using more exotic approaches in the software
- The general-purpose Latent Class Analysis tool in Q and Displayr can be used can be used to analyze MaxDiff data. In general, there is little point in doing this, as it is easier to use the standard analyses (which also use latent class analysis, but are designed specifically for MaxDiff data). However, if the desire is to form segments using multiple types of data (e.g., MaxDiff and ratings scale data), this can be done using Latent Class Analysis.
- Mixed-Mode Trees can be created using a MaxDiff experiment as the outcome, see Mixed-Mode Tree Analysis
Saving analysis results
If you have used one of the standard analyses in Displayr or Q, you can extract variables by selecting your MaxDiff analysis, and selecting one of the following options in the Save Variables sub-menu:
- Class Membership (when performing Latent Class Analysis)
- Class Membership Probabilities (when performing Latent Class Analysis)
- Preference Shares
- Sawtooth-Style Preference Shares (K Alternatives)
- Zero-Centered Utilities
- Individual-level Coefficients
- Proportion of Correct Predictions
- Root Likelihood (RLH)
Ranking structured analyses
See the MaxDiff Case Study in Q for more information about computing respondent-level information using MaxDiff.
Exotic analyses in R
Often the key variables can be extracted using fitted or predicted functions, but there is no widely-recognized standard, so consulting the documentation of any package that is used is recommended.
Relationship of MaxDiff to other techniques
MaxDiff can be viewed obtaining Incomplete Rankings and Partial Rankings. If a person is shown a list of Coke, Pepsi, Diet Coke and Coke Zero and indicates that Coke Zero is best and Coke is worst then the ranking obtained is: Coke Zero > Diet Coke = Pepsi > Coke.
Although the "max" and "diff" in "MaxDiff" are short for "Maximum Difference", MaxDiff as practiced in survey research is unrelated to Maximum Difference Scaling in psychophysical experiments (there is a conceptual relationship, but both the statistical models and the software developed for one are not readily adaptable to the other).[1]
MaxDiff can be viewed as a form of discrete choice experiment. Additionally, standard discrete choice models, such as the various generalizations of Multinomial Logit can be use to analyze MaxDiff experiments, although the resulting parameter estimates are biased.
Other resources
Information of a more technical information can be found by searching our blog and on the Sawtooth Software website.
References
- Kenneth Knoblauch, Laurence T. Maloney. (2008) “Kenneth Knoblauch, Laurence T. Maloney”. Journal of Statistical Software, Vol. 25, Issue 2, Mar 2008.