Toolkit - methods
Learn about questionnaires, interviews, focus groups and peer reviews
Questionnaires
Questionnaires offer an opportunity to gather basic data from groups of people. They are good for measuring responses to specific questions and they enable researchers to make comparisons across groups. They can also help you collect data over time with the same cohort and across different cohorts in order to help ascertain how attitudes and perceptions have changed.
Strengths
- allow you to survey a large group of people relatively quickly
- good at collecting factual information
- can be anonymous
- can be offered on paper or online
- tend to be less time intensive than other methods in terms of administering, completing and analysing
- a useful tool for making comparisons across cohorts and year groups
- can collect quantitative or qualitative data
- quantitative data can be systematically analysed
- online surveys packages incorporate tools for analysis
- you can gather contact details for follow up interviews or focus groups.
Challenges
- completion rates are often linked to length of time a questionnaire takes and the demands it makes
- additional factors such as tone of voice, facial expressions can not be taken into account
- it is harder to gather responses to more complex questions
- developing clear, unbiased questions can be challenging and may require training.
Things to consider
- be clear about what you are evaluating
- determine what types of questions are most relevant to your study: closed – T/F; multi-choice; Likert style or open – with free text answers
- pilot the questionnaire to test the clarity and validity of the questions.
When to use
Questionnaires are useful to gather baseline data, take a snapshot of participants' views midway in a course, understand attitudes towards satisfaction at the end of a course or as a means of rating the relative value of components of a CPD programme in a longitudinal study. Additionally, questionnaires can be used as one element of a mixed methods study, and may offer findings, which can be explored in more depth using tools such as interviews or focus groups.
Ethical issues
Finally, it is important that respondents understand their rights and the researcher’s responsibilities in relation to storage of data, anonymity and withdrawing from answering a questionnaire at any point.
Questionnaire resources
1. Graham Gibbs lectures
A series of three video lectures with creative commons licenses. Taken together, they offer quite a comprehensive guide, pitched at an audience that has little experience of using questionnaires:
- Video one: Question types and piloting. This also addresses the point of when a questionnaire is an appropriate method.
- Video two: Questionnaire layout and question wording
- Video three: Ratings and scales
Creative Commons license: CC-BY-NC-SA
2. Sheffield Learning and Teaching Resource
This University of Sheffield online resource contains a summary of advantages and disadvantages of using questionnaires, some advice about the process and a link to some sample evaluation questions.
3. University of Wisconsin – Cooperative Extension 2009
This online resource provides an excellent introduction and many helpful examples of how to develop an end-of-session evaluation questionnaire for teaching-related CPD.
Further resources
Interviews
Interviews offer a way to explore questions in more depth and they tend to yield qualitative data that are potentially richer and more complex than that are collected through questionnaires. They are also useful to follow up findings from questionnaires.
Things to consider
What format of interview would best suit your evaluation? Interview formats may be:
- structured – fixed set of questions asked to all interviewees
- semi-structured – set of questions and key themes for exploration, but questions vary between interviews
- unstructured – no predetermined questions; interviewee is encouraged to talk around a set of issues.
How will you keep a record of the interview? You may wish to use a voice recorder so that you can concentrate on asking questions and listening to the responses. If you are using a recording device, ensure that you have consent from your interviewee.
How do you intend to analyse the data? You might wish to transcribe in full and use a grounded approach (that involves coding of the data) as a means of interpretation. This is labour-intensive but may help you make comparisons across groups over time. Or you may prefer to draw out broad themes that you observe in the data.
Strengths
- topics can be probed more fully
- interviews are appropriate for complex, nuanced discussions
- questions can be tailored to suit the context
- the interviewee’s views are brought to the fore and can shape the direction and flow of the conversation
- topics that are unanticipated by the interviewer can be accommodated and explored
- a fine-grained perspective of the question under consideration can be obtained.
Challenges
- interviews can be time-consuming to organise, administer and analyse
- anonymity is harder to organise with interviews
- interviews are not practical for large groups; sampling would be required.
- analysis of interview data can be challenging and less straightforward than survey data.
- they may require transcribing or extensive note making
- participants may be less willing to be interviewed than to complete a survey
- it can require some practice to guard against ‘leading’ an interviewee towards a particular response.
When to use
Interviewing might be a suitable technique to use if you are following a set of individuals in a longitudinal study as they develop their practice after completing a CPD course. So, you could arrange to interview them at regular intervals, with a view to building up over time an understanding of their journey as teachers. The data could be reported as a case study.
Ethical issues
Finally, it is important that respondents understand their rights and the researcher’s responsibilities in relation to storage of data, anonymity, responsible reporting of data and withdrawing from the interview process at any point. If the interview is being recorded, a specific consent form will need to be provided.
Interview resources
Djebarni, R., Burnett, S. ; Richards, B. (2014) Research Methods for Business students, Managers and Entrepreneurs. University of South Wales.
This multimodal e-book is an excellent resource; it is good for interviewing techniques but also covers questionnaires and focus groups. It is freely available, CC-licensed, but has been designed to work with iPads and Macs.
License: Attribution-NonCommercial-ShareAlike 4.0 International
What makes a good interview?
In this 18-minute video, Graham Gibbs comments on examples of good and bad interview technique and runs through what he considers to be the ten characteristics of a good interviewer. Some people might feel they know most of this but the replay of an interview with commentary is quite engaging.
Further resources
Focus groups
Focus groups are effectively group interviews and they can be an efficient way of exploring a set of issues in depth with a cohort. Focus groups are also a useful way of gathering a range of views and offer an opportunity to hear points and counterpoints.
Things to consider
Having two people to run the focus groups works well - with one person facilitating the discussion and the other taking notes. It is particularly helpful to make a recording of the session.
Ensure that members of the group feel comfortable with each other before the session officially begins. The establishment of a sense of trust, an environment conducive to discussion and a relaxed atmosphere should enable a free flowing conversation.
Strengths
- focus groups offer a good opportunity for participants to reflect upon and share experiences
- participants can stimulate observations or recollections that might not have surfaced in either surveys or interviews
- the experience of thinking collaboratively about a topic can be a rewarding activity for groups of peers. Participants may feel they have gained new perspectives or ideas from the session
- the data generated from a focus group has the potential to be rich and multi-voiced.
Challenges
- arranging focus groups can be more time-intensive than surveys or interviews
- good facilitation skills are needed to keep discussions focused and ensure everyone is contributing
- dominant, persuasive speakers can steer the discussion in a way that may not be representative of the group
- it is difficult to take notes in a focus group, so you may wish to record the session (with consent of participants)
- participants need to feel secure in the presence of peers to express their views. A trusting, confidential environment should be established.
When to use
Focus groups would be a good way of getting feedback about a programme midway through and at the end. It could be a good way of working with institutional stakeholders to canvas their views about the impact of CPD programme and it would be a useful way of gathering longitudinal data from participants.
Ethical issues
Finally, it is important that respondents understand their rights and the researcher’s responsibilities in relation to storage of data, anonymity, responsible reporting of data and withdrawing from the focus group at any point. If the focus group is being recorded, a specific consent form will need to be provided. The facilitator should emphasize the importance of confidentiality within the group to establish a sense of trust for all contributors.
Focus groups resources
1. Using focus groups in evaluation and research- helpful, clear and practical. Addresses evaluation directly and offers useful advice for how to handle data if using for a research project.
Smith, M. K. (2011). ‘Using focus groups in evaluation and research’, the informal education homepage. [http://infed.org/mobi/using-focus-groups-in-evaluation-and-research/. Retrieved: 9 March 2-15].
2. Facilitating focus groups - this is an excellent CC-licensed blog post describing the process of setting up and running a focus group. Jo Alcock is a librarian at Birmingham City University.
This work by Jo Alcock is licensed under a Creative Commons Attribution-Non-Commercial-Share Alike 2.0 UK: England & Wales License.
3. How to run a focus group- general advice on how to run a focus group by the Big Local charity partnership.
CC licensed: 'All our work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 2.0 UK: England and Wales License, unless it says otherwise. See more at: http://localtrust.org.uk/library/how-to-guides/focus-group#sthash.3tlX8C79.dpuf'.
Karen Vinall © University of Leeds 2014. This work is made available for reuse under the terms of the Creative Commons Attribution-NonCommercial- ShareAlike 4.0 International Licence.