This item performs the initial phases of text analysis, and it is required as the input to other text analysis options. The processing is designed to work with English-language inputs. See this blog post for an example and description of the process.
This item works by splitting up all of the text from the input text and then processing each word separately before combining the words from each entry back together again. Processing of the words includes basic spell-checking, stemming, removal of words, replacement of specific words, and combination of words into phrases.
The item allows you to view the frequencies of the words that remain in the processed text or the final text responses that have been produced by the processing.
If an input was not blank before processing but processing removes all words, the message <NO_WORDS_REMAIN_AFTER_PROCESSING> is displayed.
The output generated by this function will look like below. The colors represent unique words and are intended to draw your attention to where they appear. The colors have no meaning beyond this.
Options
- Text variable The variable containing the text responses that you want to process.
- Correct spelling Whether or not you want the setup to include basic English-language spell-checking.
- Stemming Whether or not you want the setup to include basic stemming of English-language words. Stemming refers to replacing a word with its root word, or with the most common word in the text that corresponds to the same root. This includes removing plurals and past tense. For example, the words tests, tested, and test, all share a common word stem and will be replaced with whichever of the three words is most common in the text variable.
- Replace synonyms Whether words with the same meaning (in English) will be grouped. They will be replaced with the variant that occurs most frequently in the data set. Words with ambiguous meanings (e.g. 'fast', 'book') will not be replaced. This operation occurs after spelling corrections and stemming.
- Replace these words/phrases Specify any words or phrases that should be replaced. For example, if you want to replace the word ease with the word easy wherever it appears, and the phrase one half with the phrase fifty percent wherever it appears, then use the syntax: ease:easy, one half:fifty percent.
- Remove these words/phrases Any words or phrases that you want to ignore from the analysis. English-language stopwords (like at, of, the, etc.) will be removed automatically. If you want to remove additional words or phrases, type them separated by commas.
- Phrases Specify any combinations of words that should not be split up when they occur together. For example, if your text contains lots of responses that say the product is easy to use and you want to keep this as a unit in your analysis rather than splitting it up as the words "easy", and "to", and "use" then you can enter this phrase here. Additional phrases should be separated by commas. Any phrases specified in Replace these words/phrases and Remove these words/phrases are automatically treated as phrases throughout the text without being specified here.
- Maximum n for n-gram identification This value determines the largest number of words to be considered as possible n-grams (contiguous chunks of n words). For example, a value of 5 will search the text for pentagrams or smaller; 3 will search for trigrams, bigrams, unigrams, etc.
- Minimum frequency as a percentage of cases Whether to specify the minimum frequency as a percentage of cases or as a fixed number.
- Percentage (%) of cases for minimum frequency The percentage of cases to use as the minimum frequency. Words or phrases must appear in least at this percentage of cases to be included in the analysis.
- Minimum frequency Words or phrases must appear at least this many times in your text to be included in the analysis.
- Count frequency by number of respondents If checked, multiple occurrences of a word or phrase by the same respondent will only be counted once in the category's frequency. Otherwise, each occurrence will be counted separately.
- Sort alphabetically When displaying the Word Frequencies output, display words and phrases in alphabetical order instead of in order of frequency.
- Page of raw text to show The page number to show, e.g., if the page number is 3 and there are 1000 rows per page, then rows 2001 to 3000 will be shown.
- Rows of raw text per page The number of rows of raw text to show on a page.
Save Variable(s)
- Maximum number of unique categories to save Maximum number of unique categories that are to be saved. If the number of unique categories exceeds this limit, then the most popular categories are chosen with ties broken alphabetically.
- Maximum number of categories per case to save Maximum number of categories per case to be saved. If this control is set to a value n, but the maximum number of categories for a case is m > n, then only the first n categories are saved.
- Save categories Adds variables to the data set containing the categories. Where there are multiple input variables, multiple sets of variables are added for each.
- Save first category Adds a variable to the data set containing the first category mentioned. Where there are multiple input categories, the first category of each will be saved as a separate variable.
- Save sentiment scores Adds a variable that assigns scores to quantify how positive or negative each text response is.
- Save tidied text Adds a variable to the data set that contains the tidied text.
Other Analyses
This item is used as an input in the following analyses:
- Text Analysis - Advanced - Term Document Matrix creates a matrix whose columns correspond to the words identified in the text. This can be used as an input to other models.
- Text Analysis - Advanced - Predictive Tree creates a predictive tree that shows how the presence of particular words in the text predicts the values of an outcome variable.
- How to Calculate Sentiment Scores for Open-Ended Responses creates a new variable in your data that measures how many negative and positive words occur in each text response.
- Text Analysis - Save Variables - Tidied Text creates a new variable in your data that contains the processed text. This can be used as an input to coding or a word cloud.
Extract Word Frequencies
To extract the word frequencies table from the output, take the following steps:
- In Q, go to Create > R Output, or in Displayr from the toolbar, click Calculation
> Custom Code.
- In the R code editor (in Q) or the Code
panel (in Displayr), paste in the code below, remembering to replace text.analysis.setup with the reference name of your text analysis setup object:
data.frame("word" = text.analysis.setup$final.tokens, "count" = as.numeric(text.analysis.setup$final.counts))Click Calculate if the calculation does not automatically update.
More Information
A standard list of stopwords is removed automatically using this function. If you'd like to keep stopwords in your text, you can modify the underlying Data > Attributes > Code to do this. You will need to add the remove.stopwords argument to the arguments list and set it equal to FALSE -- like in the picture below: