Open Surveys and their analysis
1. Open Surveys
The purpose of Open Surveys is to help understand and improve the effectiveness of an organisational change or some aspect of organisational performance based on respondent comments.
Open Surveys and their analysis are based on, and developed from, respondents’ thoughts and feelings expressed in their own words.
Open Surveys are very different to what are termed Statistical Surveys for, in a sense, they work in reverse. That is to say:
- In an Open Survey respondents answer a few questions in detail by expressing themselves in their own words. Responses are grouped into categories, and from these categories broader patterns are built showing how comments are linked. Such patterns are, or can be fitted together, into a model covering and representing the collective views of respondents. In comparison;
- Statistical Surveys use a “forced” response approach to questions that are structured around a pre-agreed model. These models have factors (categories) that are used as the basis to develop questions around an area of interest. Respondents have little to no opportunity to explain their answers. The answers provide a means of statistically (quantitatively) confirming responses against a survey’s underlying model’s factors (categories).
As Statistical Survey’s generally do not explore questions in detail or depth, this can affect how valid they are where issues are more complex and need respondents to explain reasons for believing something, as is available through Open Surveys.
Open Surveys are also termed Open Question Surveys and use respondent comments that are sometimes described as Narrative data; Text data; or Qualitative data.
2. Advantages and disadvantages of Open Surveys
Advantages:
- Allows respondents to answer in their own words.
- Allows for “richness” or “depth” of explanation in responses.
- Can identify and explore issues not currently fully recognised or understood.
- Can identify options for further action.
Disadvantages:
- Limited by the writing skills of respondents.
- Analysis involves a more complex interpretation.
- Stronger role of interpretation in analysing responses.
3. Designing Open Survey Questions
Questions used in Open Surveys usually use a word such as how, what, when, where, and why, to allow respondents to express their thoughts and feelings such as their attitudes, opinions, understanding, likes, dislikes, suggestions, and ideas.
Open (sometimes referred to as unstructured) questions are ones in which possible answers are not suggested in the design of the survey questions, and where respondents answer in the way they see the world.
Open ended questions are framed to encourage self explanation of thoughts and feelings in a sentence, paragraph, or more.
Open ended questions include some direction on which the feedback is sought, such as:
- Please tell us what changes you would like to see?
- Please tell us what in your view we do well?
- Where could we improve?
- Why do you use our services?
4. Analysing Open Survey Responses
4.1 Open Survey and Analysis Framework
The purpose of an Open Survey design and analysis of responses is to provide a framework that:
a) Accurately and as thoroughly as possible captures the intent of respondent comments; and
b) Analyses these comments so that they construct, step by step, an understanding of issues raised in a way that is both meaningful and relevant: That “fit and work” [1].
An explanation of how respondent comments are analysed, developed, and tested is provided in 4.2 Steps in Analysing Comments.
4.2 Steps in Analysing Comments
Step 1. Read through all comments to get a feeling for the responses, and themes that emerge from this overview.
Step 2 Develop Categories
2.1 Create categories. Identify categories (sometimes referred to as labels) from the different themes that emerge from comments. When a response is allocated to a category this is based on what is actually said and also any underlying meaning that might be recognised.
2.2 Test to improve Category Analysis. Test to see if there are alternative categories that have a better fit.
2.3 Decide if there are sub categories. Think about what the categories are about. Once comments have been categorised look again at the responses in each category to identify what is being expressed by different viewpoints. Are these consistent with the category, and could they also identify sub categories that are of value in the analysis?
2.4 Quantify Categories. Assign comments to at least one category (or sub category) and group them so the quantity of responses in each category can be more easily counted.
Refer to Diagram 1. Developing Categories from Respondent Comments.
Step 3 Link Categories to develop Patterns and Model(s)
Once comments have been studied and categorised (and sub categorised), the next step is to see how these categories link to form patterns and models:
- Do some categories link in some way, and how do unrelated others link to form patterns?
- Do these patterns, together, form to represent one or more models?
In developing patterns check if there are exceptions to the rule that require the patterns to be broadened to include other categories, or changed into other patterns, or cast doubt over a pattern as a “rule”.
Refer to Diagram 2. Developing Patterns from Categories.
In developing models these may be either:
a) Drawn from the patterns of analysis;
b) Brought to the analysis from a recognised published model that links to categories and their patterns. For example a recognised model relevant to the type of survey carried out such as a Hospitality, Health Service, Service Delivery, Management, Environmental Management, or Community Model.
Either approach to developing a model offers advantages, such as an internally developed model can assist in gaining commitment from those involved in its development whereas the use of an external model offers validity through its credibility as a recognised benchmark.
Whichever approach is used the selected model should be as inclusive as possible of all respondent comments.
Refer to Diagram 3. Organisational Alignment Model based on fit to Patterns.
Step 4. Write up the analysis: Once all comments are categorised and analysed, and the patterns they form identified along with any model that fits those patterns, then the next step is to write up in a summary.
The analysis can include:
- An explanation of a model and the patterns that pull the analysis together.
- Categories and sub categories identified (at the very least the more significant categories).
- The number of comments covered by a category. This can be in either broad quantitative terms covering more than one category, or specific to each category.
- An explanation about the categories and sub categories along with supporting (non identifiable) quotations drawn from respondent comments.
- Recommendations based on the analysis. Recommendations for improvements can come directly from comments where there are several similar responses or from a single, different, comment.
The survey design and analysis should demonstrate validity:
- Face Validity:
- How survey findings make sense in terms of credibility, relevance, and usefulness to respondents and those who decide to use survey recommendations;
- Use respondents own words in providing credibility to survey recommendations.
- Construct Validity: The extent to which the survey design and analysis minimise error and misinterpretation and is consistent in “fitting” with respondent comments. That is, where comments fit with constructs (or concepts) in terms of developing criteria; patterns; and models.
Notation.
[1] In references below, see Concept Mapping as an alternative Approach for the Analysis of Open-ended Survey Responses, the term Concept Mapping is used. The term Construct Mapping is used here instead of the term Concept Mapping so that there can be a better and more logical alignment to Construct Validity. In this respect an Open Survey analysis could be viewed as a form of Construct Mapping in that comments are organised (i.e. constructed) first into categories; these categories are then organised (constructed) into patterns; and where these patterns may then be organised (constructed) into a model. When constructing such a map the outcomes (whether this relates to the construction of categories, patterns, or a model) should readily, not forcibly, “fit” with respondent comments and also “work” in the sense that categories, patterns, and any model selected can be seen to provide meaningful and relevant conclusions. “Fit” in this sense relates to Construct Validity, and “work” to Face Validity.
References include:
Outline explanations of Open Survey Analysis Frameworks
Analysing open-ended questions.Website: http://intelligentmeasurement.wordpress.com/2007/12/18/analyzing-open-ended-questions/ Downloaded 8/8/12
Analysing Qualitative Data. Website: http://learningstore.uwex.edu/assets/pdfs/g3658-12.pdf Downloaded 8/8/12
A Brief Guide to the Analysis of Open-Ended Survey Questions. Website: http://cms.cerritos.edu/uploads/ResearchandPlanning/Brief_Guide_to_Open-Ended_Survey_Questions.pdf Downloaded 8/8/12
Concept Mapping Explained
Concept Mapping as an alternative Approach for the Analysis of Open-ended Survey Responses Website: http://www.socialresearchmethods.net/research/Concept%20Mapping%20as%20an%20Alternative%20Approach%20for%20the%20Analysis%20of%20Open-Ended%20Survey%20Responses.pdf Downloaded 8/8/12
Comprehensive References on Open Survey Design and Analysis
Qualitative Data Analysis. Ian Dey. Website: http://www.drapuig.info/files/Qualitative_data_analysis.pdf Downloaded 8/8/12
Patton, M.Q. (1990). Qualitative evaluation and research methods (2nd ed). Newbury Park, California: SAGE Publications Inc.
Organisational Alignment Model in Diagram 3 adapted from Figure 3.9 in:
Addison, R., Haig. C., & Kearney. L. (2009). Performance architecture: The art and science of improving organizations. San Francisco, California: Pfeiffer.
Like this:
Like Loading...
