For this case study, you can download the QPack here. This file contains data from a completely randomized single factor in which four samples of respondents were each shown a different concept for a new yogurt and were asked:
Q4. How likely would you be to buy this product? I would definitely buy it I would probably buy it I am not sure whether I would buy it or not I would probably not buy it I would definitely not buy it
Q4 is a Nominal variable set. Another Nominal variable set in the project called Concept indicates which concept was seen by which respondent.
Method - Analyzing via crosstab
The most straightforward way to analyze such data is as a crosstab. The table below shows this crosstab, revealing that Concept D was most popular, with more people saying they would “definitely” buy it and many more people saying they would “probably” buy it.
Method - Analyzing via a numeric variable
A second way to analyze the question is to change Q4 to Structure > Numeric variable set, assigning some values to the different categories. The table below shows such an analysis, where “definitely” has been given a Value of 100, “probably” 50, etc. You can think of these numbers as a guess at the probability that the person would buy.
This analysis also shows that Concept D is the most appealing concept and is significantly more appealing than the average of the other concepts. Similarly, it shows that Concept A is least appealing, although we can see from the z-Statistic that Concept A is only marginally less appealing than Concept B and C.
Q automatically applies the False Discovery Rate Correction to control Type 1 error when determining which cells in the table to indicate as being statistically significant (by color-coding or arrows, depending on the options set for the Table Styles in Project Options). However, you will need to turn this on in Displayr through the object inspector > Properties > Significance > Advanced > Exception Tests > Multiple Comparison Correct > False Discovery Rate Correction (this is already done in the QPack from above). This means that a coefficient that is only just significant at the 0.05 level (e.g., with a z-statistic of more than 1.96 or less than -1.96) will often not be shown as significant. For example, we can see in this example that Concept C has a p-value of 0.0469, but is not shaded as being statistically significant. By setting the Multiple Comparisons Correction to None, these tests can be conducted in the more common way.
This experiment is considered to have the following properties:
- It has a single task. That is, the respondent was only presented with one set of alternatives (where in this instance, the set contained a single alternative).
- It has a single alternative. That is, the task contains a single alternative. In other words, each respondent was only shown one thing, where the thing is called an alternative.
- It has only a single factor. That is, there is only a single attribute (or dimension) which describes the differences between the alternatives. In this case, the attribute contains four attribute levels (i.e., the four concepts).
- The dependent variable (also known as response variable and outcome measure) is numeric.
Method - Analyzing via an experiment
As with all other types of questions, an Experiment is set up in the Data Sets pane by selecting relevant variables, right-clicking, and selecting Combine. The data for this completely randomized single-factor experiment is set up in two variables: a Numeric variable containing the ratings and a Nominal variable containing the attribute:
Because the dependent variable, Q4, is Numeric, the conditional density is assumed to be normally distributed (this is the same assumption as linear regression and ANOVA). The other studies illustrate other types of dependent variables.
The attribute, which has the name Concept, is contained in the second variable. By pressing the Values button to view the variable’s Value Attributes we see that this variable (and thus the attribute) contains four levels:
The table below shows the results obtained by creating a drag-and-drop table of the Single attribute split cell experiment variable set, and adding the columns statistics through the STATISTICS > Cells dropdown.
Although at first glance these results appear different to those shown above in the table that contains averages, they are equivalent. The average rating for each concept can be reconstructed by adding their Coefficient to the Intercept (for example, Concept A’s score is 37.1 + 0.0 and Concept B’s score is 37.1 + 1.6). Note that the first concept has been assigned a coefficient of 0 and all other concepts’ coefficients are indexed relative to this 0; the automatic significance tests compare each attribute level (i.e., the concepts in this instance) to the base level. Whichever level is shown first in an attribute is the base level; you can move these around by drag and drop. Similarly, you can drag and drop categories to merge them.
Detailed diagnostics
Detailed diagnostics can be obtained in Q (not available in Displayr yet) by selecting all the coefficients and pressing . The first section of the diagnostics shows the sample size and any filters and weights:
Total sample Unweighted base n = 1609
The software automatically conducts a likelihood ratio chi-square test, which essentially checks that the data is not completely random; if the null hypothesis is not rejected, it almost certainly means that there is some error in the way the data has been set up.
Likelihood Ratio Chi-Square Test Chi-Square = 38.872 Degrees of Freedom = 4 Fox, John (1997): Applied Regression Analysis, Linear Models, And Related Methods, Sage. p < 0.000001 Null hypothesis: In the population, all the coefficients are equal to 0. At the 0.05 level of significance, the null hypothesis is rejected.
The Data Summary shows key aspects of the design of the experiment. First, it shows the number of tasks per respondent. In this example, each respondent completed only a single task. Second, it shows the number of alternatives per task. Third, it shows the total number of unique alternatives in the experiment; in this example, there were four unique alternatives (i.e., the first concept), which equated to 0.0025 per task (i.e., 4 unique alternatives divided by 1 * 1,609, where 1 is the number of tasks per respondent and 1,609 the number of respondents). Fourth, it shows the number of unique tasks, which in this instance is also 4. Lastly, it shows the number of unique response patterns, where a response pattern is defined as a task together with a response to the task. In this example, there were five possible answers per task, and four unique tasks. Therefore there were 20 possible unique response patterns. In the data all 20 of these patterns were present, leading to 20 unique response patterns.
Data summary 1 task(s) per respondent 1 alternative(s) per task 4 unique alternatives; 0.00248601615910503 per task 4 unique tasks; 0.00248601615910503 per task 20 unique response patterns
The Invalid task report identifies any instances where missing or invalid data has caused the exclusion of data from the analysis. In general, this section of the report should appear blank (as in this example), unless different respondents have been asked to complete different numbers of tasks.
Invalid task report Respondent row number (task number)
The Model results present various statistics for the model as a whole and the model’s parameters.
Model results
Dependent Question Type: Number (numeric)
log-likelihood -7,692.330
Null log-likelihood -7,711.766
Parameters 4.000
Effective n 1,609.000
p .000
R-Squared (McFadden) .003
Adjusted R-Squared (McFadden) .002
R-Squared .024
AIC 15,392.661
AIC3 15,396.661
BIC 15,414.194
CAIC 15,414.197
BIC(case) 15,414.194
CAIC(case) 15,414.197
D-Error .000
Parameters
Coef S.E. t-Stat p-Value Beta VIF
Intercept 37.06467662 1.44036180 25.73289336 .00000100 NaN NaN
Concept B 1.63311208 2.03071348 .80420606 .42139708 .02431752 1.50338735
Concept C .90524820 2.04080451 .44357418 .65741026 .01339062 1.49842769
Concept D 11.18968747 2.03824873 5.48985376 .00000005 .16579729 1.49968616