A. Specificities of qualitative research methods

First, qualitative research encompasses all forms of field research performed with qualitative data. “Qualitative” refers to data in non­numeric form, such as words and narratives. There are different sources for qualitative data, such as observations, document analysis, interviews, pictures or video’s, etc. Each of these data-gathering techniques has its particular strengths and weaknesses that have to be reflected upon when choosing for a qualitative research technique. In the social sciences, the use of qualitative data is also closely related to different paradigms trying to develop insight in social reality. Elaboration on these paradigms is however outside the scope of this process note [1].

Second, the aim of qualitative research is developing a “thick description[2]” and “grounded or in-depth understanding” of the focus of inquiry. The benefits of well developed qualitative data-collection are precisely richness of data and deeper insight into the problem studied. They do not only target to describe but help also to get more meaningful explanations on a phenomenon. They are also useful in generating hypotheses (Sofaer, 1999). Types of research questions typically answered by qualitative research are “What is going on? What are the dimensions of the concept? What variations exist? Why is this happening?” ^{(Huston,1998)}. Qualitative research techniques are primarily used to trace “meanings that people give to social phenomena” and “interaction processes”, including the interpretation of these interactions ^{(Pope, 1995)}. “They allow people to speak in their own voice, rather than conforming to categories and terms imposed on them by others.” (Sofaer, 1999, p. 1105). This kind of research is also appropriate to investigate social phenomena related to health (Huston,1998).

Third, one of the key strengths of qualitative research is that it studies people in their natural settings rather than in artificial or experimental ones. Since health related experiences and beliefs are closely linked to daily life situations it is less meaningful to research them in an artificial context such as an experiment. Therefore data is collected by interacting with people in their own language and observing them in their own territory (Kirk, 1986) or a place of their own choice. This is also referred to as naturalism. Therefore the term naturalistic methods is sometimes used to denote some, but not all, qualitative research ^{(Pope, 2006)}. Also this characteristic is not always relevant to the use of QRM at the KCE. For example focus group interviews are usually not performed in the natural setting of the participants, but rather in the setting of a meeting room.

A fourth feature of qualitative research in health care is that it often employs several different qualitative methods to answer one and the same research question ^{(Pope, 2006)}. This relates partly to what is called triangulation (see here).

Finally, qualitative research is always iterative starting with assumptions, hypotheses, mind sets or general theories which change and develop throughout the successive steps of the research process. It is desirable to make these initial assumptions explicit at the beginning of the process and document the acquired new insights or knowledge at each step.

 

---

[1]           For those interested we refer to Denzin and Lincoln, 2008 a^, Denzin and Lincoln, 2008 b, Bourgeault et al., 2012 or in Dutch, Mortelmans, 2009

[2]           A “thick description” of a human practice or behavior include not only the focus of the study, but its context as well, such it becomes meaningful to an outsider. The term was introduced in the social science literature by the anthropologist C. Geertz in his essay in 1973

B. Qualitative versus quantitative approaches

Although it is meaningful to do qualitative research in itself, qualitative research is often defined by reference to quantitative research. Often it is assumed that because qualitative research does not seek to quantify or enumerate, it does not ‘measure’. Qualitative research generally deals with words or discourses rather than numbers, and measurement in qualitative research is usually concerned with taxonomies or classifications. “Qualitative research answers questions such as, ‘what is X, and how does X vary in different circumstances, and why’, rather than ‘how big is X or how many X’s are there?”^{(Pope, 2006,} p3).

By emphasizing the differences the qualitative and quantitative approach are presented as opposites. However, qualitative and quantitative approaches are complementary and are often integrated in one and the same research project. For example in mixed methods research the strengths of quantitative and qualitative research are combined for the purpose of obtaining a richer and deeper understanding ^{(Zang, 2012)}. Also qualitative data could be analyzed in a quantitative way by for example counting the occurrence of certain words.

Often health services researchers draw on multiple sources of data and multiple strategies of inquiry in order to explore the complex processes, structures and outcomes of health care. It is common that quantitative and qualitative methods answer different questions to provide a well-integrated picture of the situation under study ^{(Patton, 1999)}. Especially in the field of health services research qualitative and quantitative methods are increasingly being used together in mixed method approaches. The ways QRMs could be used combined or not, are:

 Qualitative research only:

• To know the variation in experiences related to health or illness.
• To build typologies regarding health services use, patient attitudes, health beliefs, etc.
• Qualitative preliminarly to quantitative:
• To explore new area, new concepts, new behaviour, etc.^{(Pope, 1995)} before to start with measurement.
• To build quantitative data collection tools (questionnaires): using appropriate wording ^{(Pope, 1995)}, variables to submit, to develop reliable and valid survey instruments ^{(Sofaer, 1999)}, etc.
• To pre-test survey instruments(Sofaer, 1999).
• In supplement to quantitative work:
• As a part of a triangulation process that consist in confronting results coming from several data sources ^{(Pope, 1995)}.
• To reach a different level of knowledge ^{(Pope, 1995)}: “If we focus research only on what we already know how to quantify, indeed only on that which can ultimately be reliably quantified, we risk ignoring factors that are more significant in explaining important realities and relationships.” (Sofaer, 1999, p. 1102).
• In complement to quantitative work by exploring complex phenomena or areas that are not reachable with quantitative approaches ^{(Pope, 1995).}
• Sofaer (Sofaer, 1999) provides us the insight that in many cases, inquiry can move from being unstructured, largely qualitative in nature, to being structured and largely quantitative in nature. This is how she describes the continuum: “(…) there is uncertainty not only about answers, but about what the right questions might be; about how they should be framed to get meaningful answers; and about where and to whom questions should be addressed. As understanding increases, some of the right questions emerge, but uncertainty remains about whether all of the right questions have been identified. Further along, confidence grows that almost all of the important questions have been identified and perhaps framed in more specific terms, but uncertainty still exists about the range of possible answers to those questions. Eventually, a high level of certainty is reached about the range of almost all of the possible answers.” (p. 1103).
• In sum, over time investigations related to a certain area, start with qualitative research to explore the field, find the right questions, prepare for more focused questions and discover theories and hypotheses. Next, quantitative research is in place to test hypotheses and finally, qualitative research can be used to deepen the findings or to search for explanations quantitative research techniques cannot provide.

3.1.1 General principles

Attachment	Size
fig2_interviewing_people.png	91 KB
fig3_conceptualisation.png	37.37 KB

3.1.2 Individual interviews

3.1.2.1        What are individual semi-structured interviews?

Interviews are used in many contexts (journalism, human resource managers, etc.) and for many purposes (entertainment, recruitment of personnel, etc.), hence scientific data collection is only one very specific application, which should not be confused with other applications. The interview is easily trivialized as it is common practice in the media landscape which surrounds us. Fontana and Frey even speak about “the interview society” according to Atkinson and Silverman. Practicing health professionals routinely interview patients during their clinical work, and they may wonder whether simply talking to people constitutes a legitimate form of research ^{(DiCicco-Bloom et al,2006)}. In qualitative research, however, interviewing is a well established research technique and two types can be distinguished: semi-structured and unstructured. Structured interviews are out of scope here, because they consist of administering structured questionnaires producing quantitative data.

Unstructured interviews are more or less equivalent to guided conversations⁽DiCicco-Bloom et al,2006). Originally they were part of ethnographers’ field work, consisting of participant observation and interviewing key informants on an ongoing basis to elicit information about the meaning of observed behaviors, interactions, or artifacts (DiCicco-Bloom et al,2006). There is no list of questions, nor an interview guide, the questions asked are based on the responses of the interviewee, as in the natural flow of a conversation (Britten, 1995).

Semi-structured interviews are often the sole data source in a qualitative research project. A set of predetermined open-ended questions is used to guide the interview, but other questions emerging from the dialogue can be added ^{(Britten, 1995)}. Also the iterative nature of the research process in which preliminary data analysis coincides with data collection, results in altering questions as the research process proceeds. Even so, questions that are not effective in eliciting the necessary information can be dropped or replaced by new ones ^{(Britten, 1995)}.

Essentially an interview consists of someone who asks questions (interviewer), someone who answers these questions (interviewee) and the registration of those answers in some way (Mortelmans, 2009).

The interview as qualitative research method differentiates from other forms of interviewing used in varied domains. Mortelmans pays attention to four characteristics:

• Flexibility; with flexibility internal and external flexibility is meant: external refers to the iterative use of interviewing and data analysis. Structure and content of the subsequent interview may be changed in function of the analysis of the previous one. Internal flexibility points to the fact that the sequence of the prepared interview questions and themes should stands in function of the interviewee in order to guard the natural flow of the conversation.
• The interviewee leads so to speak the conversation. The interviewer only guards the scope of the conversation and makes sure that all the topics are covered.
• Non-directiveness; the interviewee steers the interview and the interviewer only makes sure that the conversation does not stray too far by means of non-directive interview techniques.
• Direct face-to-face contact is important to built trust and get in-depth information, but this depends on the topic and should be considered case by case.

3.1.2.2        When to use individual semi-structured interviews?

Individual semi-structured interviews are useful to:

• Collect data on individuals’ personal histories, perspectives, and experiences, particularly when sensitive topics are being explored ^{(Mack, 2005)}.
• Elicit a vivid picture of the participant’s perspective (Mack, 2005).
• Provide context to other data, offering a more complete picture ^{(Boyce et al, 2006)}
• Learn about the perspectives of individuals, as opposed to, for example, group norms of a community, for which focus groups are more appropriate (Mack, 2005).
• Get people to talk about their personal feelings, opinions, and experiences (Mack, 2005).
• Gain insight into how people interpret and order the world on the research topic ⁽Mack, 2005).
• Address sensitive topics that people might be reluctant to discuss in a group setting (Mack, 2005).
• Elicit information from key informants ^{(Sofaer, 1999)}.
• Examine people’s experiences, attitudes and beliefs ^{(Huston et al, 1998)}.

3.1.2.3        Strengths and weaknesses of the method

Strengths:

• They provide much more detailed information than what is available through other data collection methods, such as surveys (Boyce et al, 2006).
• Questions can be prepared ahead of time. This allows the interviewer to be prepared and appear competent during the interview (Cohen, 2008).
• Semi-structured interviews also allow informants the freedom to express their views in their own terms (Cohen, 2008).
• Semi-structured interviews can provide reliable, comparable qualitative data (Cohen, 2008).

Weaknesses:

• Interviews can be time-intensive because of the time it takes to recruit participants, conduct interviews, transcribe them, and analyse the results. In planning your data collection effort, care must be taken to include time for transcription and analysis of this detailed data (Boyce et al, 2006).
• Interviewers must be appropriately trained in interviewing techniques. To provide the most detailed and rich data from an interviewee, the interviewer must make that person comfortable and appear interested in what they are saying. They must also be sure to use effective interview techniques, such as avoiding yes/no and leading questions, using appropriate body language, and keeping their personal opinions in check ⁽Boyce et al, 2006)
• Data from individual semi-structured interviews are not generalizable in a statistical way, but they are theoretically transferrable, because small samples are chosen and no random sampling methods are used. Individual semi-structured interviews however, provide valuable information, particularly when supplementing other methods of data collection. It should be noted that the general rule on sample size for interviews is that when the same stories, themes, issues, and topics are emerging from the interviewees, then a sufficient sample size has been reached (Boyce et al, 2006).

3.1.2.4        How to plan the research design?

See  “How to plan the research design?”

3.1.2.5        Modalities of data collection

Individual semi-structured interviews are usually conducted face-to-face and involve one interviewer and one participant. Phone conversations and interviews with more than one participant also qualify as semi-structured interviews, but, in this chapter, we focus on individual, face-to-face interviews (Mack, 2005).

3.1.2.6        Data collection tools

The data collection tools to carry out interviews are topic lists, questionnaires and field notes. Topic lists and questionnaires are described here.

Researchers use field notes to record observations and fragments of speech. Field notes should be written up as soon as possible after the events to which they refer. If possible, short “aide-mémoire” or pocket dictaphones may be used in fieldwork settings, to facilitate later expansion of the notes into proper fieldnotes (Bloor et al, 2006). In the chapter on observational techniques field notes are addressed in more detail (here).

3.1.2.7        Sampling

For general issues on sampling, see  “Sampling issues in qualitative research: who and how many?”.

3.1.2.8        Human resources necessary

In the ideal scenario researchers plan, organize, carry out and transcribe the interviews themselves, to be completely immersed in the data, but in practice the interviews are often carried out by subcontractors and the transcriptions are often done by professional typists.

3.1.2.9        Practical aspects

Preparations for the interview see “How to run the data collection” .

Physical organisation of an interview. Take the following rules into account:

1. Interviewee and interviewer should not sit opposite each other, but rather at an angle of 90° or less.
2. The interview should take place in a quiet place where the interviewee feels at ease.
3. Avoid the presence of third parties.

3.1.2.10      Analysis and reporting of findings

See "How to prepare data for analysis",  “How to analyse?” and  “How to report qualitative research findings?” .

3.1.2.11      Examples of KCE reports using the method

• Home monitoring of infants in prevention of sudden infant death syndrome ^{(Eyssen et al, 2006)}
• Making general practice attractive: encouraging GP attraction and retention ^{(Lorant et al, 2008)}
• Osteopathy and chiropractic: state of affairs in Belgium ^{(De Gendt et al, 2010)}
• Acupuncture: state of affairs in Belgium (De Gendt et al, 2011)
• Homeopathy: state of affairs in Belgium (De Gendt et al, 2011)
• Burnout among general practitioners: prevention and management ^{(Jonckheer et al, 2011)}
• Evaluation of a fixed personal fee on the use of emergency services ^{(Gourbin et al, 2005)}

---

[1]           We propose a example of a ‘standard introductive text’ in appendix.

3.1.3 Focus groups

3.1.3.1        What are focus groups ?

A focus group is a particular technique in qualitative research. In order to do a focus group interview a group of individuals is gathered in function of their specific profile or characteristics to explore a limited number of “focused questions” (Sofaer,1999). Groups are generally homogenous on a or several criteria relevant to the focus of the discussion.

“In essence, a focus group is a small (usually 6-12 people) group brought together to discuss a particular issue (..) under the direction of a facilitator who has a list of topics to discuss” (Green and Thorogood, 2009, p. 111).

Focus groups are group semi-structured interviews used for the purpose of collecting information focused on a specific subject or area of concern, for exploration and discovery, in-depth understanding of a problem as it is experienced in context, to assess needs, preferences, attitudes and interests related (in the context of KCE research) to health and health care issues.

 

It differs from individual semi-structured interviews, as the interaction component is used to bring out insights and understandings in ways which questionnaire items or individual questions may not be able to do. The interaction between the moderator and the group, as well as the interaction between group members, may result in more in-depth information, and to elicit differing perspectives related to carefully designed questions. Focus groups are thus not to be considered as a pragmatic time saving substitute for individual semi-structured interviews (e.g. if for any reason the planning does not allow for individual interviews), as the methodological groundings of both techniques differ.

A focus group is not synonymous to ‘group interview’: For a focus group, people are recruited specifically to participate in a research protocol, using a certain method. It is a group interview in the sense that it gathers data simultaneously from different participants (Green and Thorogood, 2009) However it differs from a group interview in the importance that is attached to the interaction among participants. Participants might change their perspective during the focus group interview because of this interaction. In a group interview the interaction between participants is limited, and occurs mainly between interviewer and interviewees.

Figure 4 – Interaction patterns in a group interview versus focus group interview

 

 

Depending on sampling strategy and aims, group interviews can take several forms, e.g. consensus panel, focus group, natural group or community interview (Coreil 2005 cited by Green and Thorogood, 2009).

Focus groups can be used as a single research strategy, as well as in combination with other methods in a multi-method research strategy.

3.1.3.2        Specific questions suitable for the method

The principal feature of focus group interviews is interaction between participants. Kitzinger (2006, p. 22) highlights that this particularity could be used to:

• “Highlight the respondents’ attitudes, priorities, language and framework of understanding.
• Encourage participants to generate and explore their own questions, and to develop their own analysis of common experiences.
• Encourage a variety of communication from participants – tapping into a wide range and different forms of discourse.
• Help to identify group norms/cultural values.
• Provide insight into the operation of group social processes in the articulation of knowledge (e.g. through the examination of what information is sensitive within the group.
• Encourage open conversation about embarrassing subject and to permit the expression of criticism.
• Facilitate the expression of ideas and experiences that might be left underdeveloped in an interview, and to illuminate the research patient’s perspectives through the debate with the group.”
• Allow topics which participants have given little thought in advance to emerge from the discussion (Barbour, 2010).

3.1.3.3        Strengths and weaknesses of the method

The benefits from focus groups highlighted are:

• Interaction between participants (Green and Thorogood, 2009)
• Ability to produce a large amount of data on a topic in a short time (Cohen et al, 2008)
• Access to topics that might be otherwise unobservable (Cohen et al, 2008)
• Access to explore sensitive topics, such as dissatifaction with a service: it can be easier for an interviewee if negative ideas are reported as coming from a group than from one single person (Green and Thorogood, 2009)
• Ability to insure that data directly targets researcher's topic (Cohen et al, 2008)
• Access to comparisons that focus group participants make between their experiences. This can be very valuable and provide access to consensus/diversity of experiences on a topic (Cohen et al, 2008)

The limitations of focus groups are related to the limitations of group interviews:

• Inappropriate to uncover marginal or deviant opinions (Green and Thorogood, 2009)
• Importance of social norms: participants are influencing each other, creating a certain kind of implicit norm (Baribeau, 2010), or consensus.
• Otherwise, group dynamics may contribute to cristallization of opinions.
• Not easy to organize: several selected people have to be gathered in the same place during a couple of hours .

3.1.3.4        How to plan the research design?

Since focus group interviews are a collective data collection technique requiring direct person-to-person contact (several people have to come together at the same moment and in the same place) a careful planning of all activities and related tasks is necessary.

3.1.3.5        Modalities of data collection

The data collection by focus group could vary according to (Cohen et al, 2008):

• The level of standardization of the questions
• The number of focus groups
• The number of participants in each groups
• The level of implication of the moderator

3.1.3.6        Data collection tools

During the preparation of the focus group interviews a set of topics or questions is developed and takes the form of a topic list or questionnaire. For the general principles, see here

A focus group interview is in most cases a structured group process structured by means of an agenda to keep the group focused and on track. A focus-group should be experienced as free-flowing and relatively unstructured, but in reality, the moderator must follow a pre-planned script of specific issues and set goals for the type of information to be gathered. An introduction of up to 15 minutes should be carefully planned, as well as a good opening question. In order to keep the time schedule, as several people are going to participate and answer to the questions, it is important to foresee a maximum duration for each question.

The use of a well designed guide is helpful to compare information from one group to another as it is expected to have more than one focus group for a given topic.

3.1.3.7        Sampling

For general issues on sampling, see  “Sampling issues in qualitative research: who and how many?”

Identification of units of analysis

The starting point for selecting participants for focus groups is to identify the unit of analysis. Is the unit of analysis “individuals for their personal opinions/experience/expertise”, or is it “individuals because they represent organizational perspectives”? It has a major impact on the people invited to the focus group interview and therefore it should be clearly described.

The sample of focus groups will consist of groups of people, instead of individuals. People who are invited to take part need to have an interest in the subject.

Composition of the groups

Ideally groups have to be internally homogenous on criteria relevant to the topic but externally heterogeneous between groups. Homogeneity in the group capitalizes on people’s shared experiences (Kitzinger, 2006).

It is best to select people who do not know one another, but have similar relationships with the topic being investigated (although it could in practice be difficult for particular topics). Selecting participants who are similar may help them to share ideas more freely and develop an in-depth analysis of a topic (homogeneous groups).

Sometimes, heterogeneous groups can be used after the primary analysis of homogeneous focus groups has started. Heterogeneous groups are used to “confront” diverging opinions. In general terms, heterogeneous groups are composed of representatives of all relevant stakeholders.
In this case, the researcher has to pay attention to potential power differences or inequalities between participants. This may prevent some people from talking freely during the discussion and by consequence prevent the collection of rich data (Kitzinger, 2006).

In the Belgian context, focus group interviews can be carried out with French-speaking or Dutch-speaking and even German-speaking, participants. It is advisable to conduct unilingual groups: it is easier and richer for facilitators and participants. For heterogeneous groups, like stakeholders samples, it could be difficult to separate people in groups according to their mother tongue. In this particular case, it is important that participants express themselves in their mother tongue and to be sure that every participant understands the other language. The moderator has to be thus perfectly bilingual.

Number of participants per group

A group of six to twelve people is sufficient for a focus group. The ideal size for a focus group is eight to ten respondents. In general, the smaller the group, the more manageable it is. From experience, a group of 6‑8 participants allows enough time for discussion and is easier to manage. Where the purpose is to generate in-depth expression from participants, a smaller group size may be preferable in combination with carrying out more focus groups to attain saturation.

In order to make sure that a group counts enough participants, it is advisable to recruit 25% more people than required (Green and Thorogood, 2009). If too few participants turn up, one should foresee an additional focus group to substitute for the low attendance.

Number of groups

The number of focus group interviews needed depends on the aims and available resources . It is almost impossible to give clear standardized guidelines on the number of focus groups needed.

It is methodologically important for both approaches to conduct at least two focus groups by ‘type of people’. Using only one focus group to arrive at conclusions is risky since the opinions expressed may have had more to do with the group dynamics (i.e. persuasive skills of one or two members) than a true sampling of the opinions of the population that the group represents. Even the preset number of two focus groups is generally too limited to make in-depth analyses, especially if the topics discussed are rather “broad” or general (see also paragraph analysis on continuous comparative method). Having two homogeneous groups that provide different results suggests that more information is necessary (data saturation is not reached). One rule of thumb is to conduct focus groups until they no longer provide any new information on the topic discussed.

3.1.3.8        Human resources necessary

Three people (from the research team) could chair the focus group interview:

1. The moderator (also called ‘facilitator’) plays a crucial role in the success of a focus group interview and can have a major impact on the outcomes of the data collection. He should lay down some ‘rules’, explain the duration of the focus group interview, plan a break in between, make everybody welcome before hand, do the paperwork (e.g. informed consent) before actually starting the interview. Before the opening question, is it important to ask everybody to introduce themselves briefly. He has “to establish a relaxed atmosphere, enable participants to tell their stories, and listen actively” (Green and Thorogood, 2009, p 126.). Facilitating or moderating focus group interviews requires particular competencies: interpersonal skills (including non-verbal communication skills) are needed as well as a non-biased attitude towards the issues discussed. A focus group moderator should be able to keep the discussion on track and make sure every participant is heard. He/she has to be able to summarize what has been said, to structure the discussion. However he/she should not take position, avoid to make quick assumptions or conclusions, avoid to develop answers for the participants or give advice. Focus groups are intended to make in-depth studies of the perceptions, attitude and opinions of the participants, not of the research team (or moderator). The moderator makes it socially acceptable for participants to have another point of view. If participants get off track or get ahead of the issue being discussed the moderator must pull the group back together. He/she does not need to be an expert in the domain of the research.The moderator needs to use “probing techniques” when necessary: probing is essentially a means of further investigating a topic that has already been introduced. Probing can be used to clarify, to obtain more detail and to assure completeness. For this purpose, see also here. In the particular case of focus group interviews, the moderator could use disagreements in the group to force participants to develop and elucidate their point of view. An experienced interviewer could decide whether or not to follow the lead of the interview or to return to the sequence of the interview guide1 In the particular case of bilingual groups, the moderator has to master both languages.
2. The note-taker will take notes during the discussion while the moderator is introducing questions. The note-taker could sit next to the moderator. Nevertheless, pay attention that if he/she is typewriting on a laptop directly, the sound of the typing on the keyboard is not disturbing. Moderator and note-taker can take turns in asking questions and taking notes (this requires a well functioning team that clearly understands its roles and can adapt to the situation). It should be discussed and reported whether different or the same persons facilitate the respective focus group interviews.
3. The observer is a third facilitator who could be useful to observe the focus group participants (non-verbal language) and to help the moderator in identifying not very talkative participants and in keeping time.

As focus group have to be transcribed afterwards. It is also useful to engage the services of an audio typist.

3.1.3.9        Running of data collection

For general principles see  “How to run the data collection?”.

In the case of focus groups, once the group of respondents is gathered for the discussion, the moderator should give a brief introduction to set everybody at ease[1]. More concretely, the moderator should:

• Explain the purpose of the discussion, how the information collected will be used and reported.
• Introduce note-taker and observer who will remain in the room during the discussion.
• Explain that the discussion is for scientific purposes and that information will solely be used with the context of the research.
• Ensure participants that the rules of confidentiality apply to everyone in the room, including the note-takers, observers.
• Explain how names will be used (real names or pseudonyms).
• Explain the group rules (speak one at a time, avoid interrupting or monopolizing, etc.).
• If the discussion is to be tape-or video-recorded, obtain permission from the respondents first, and explain how the tapes will be used, stored and eventually destroyed. – Tip to increase the quality of the recording: use 2 recorders, preferably stereo recording, one at each side of the table: it is useful to understand everybody and prevent the loss of data in case of disfunctioning of the recorder.

The Moderator will then begin the focus group interview by asking an ‘icebreaker question’ to facilitate the discussion in the group. Afterwards, he/she will come to the focus of the discussion.

Immediately after the focus group a debriefing has to be foreseen with the moderators/facilitators. The debriefing part is an essential step for the analysis. The debriefing exercise is best supported by a template of dimensions, upon which the moderator/facilitator team needs to comment (example in Appendix).

The facilitators should review the notes taken during the focus group and have a first assessment of clarity and understanding.

They should discuss, compare and record observations or impressions about the group not readily apparent from the notes.

Discuss and record any insights or ideas emerging during the interviews while they are still fresh in the mind.

3.1.3.10      Practical aspects

Preparations for the interview

See also part “How to run the data collection? ”

Location & timing

• The location where the focus groups will be held should be carefully selected.
• Accessibility and transport issues (and mobility needs of participants) should be considered.
• Avoid noisy areas where it will be difficult for participants and the moderator to hear each other.
• The setting should be comfortable, non-threatening for the respondents. Refreshments should be provided.
• The focus group table can be organized before hand and this allows the researcher to place name tags in the way he wants.
• Seating should be arranged to encourage participation and interaction, preferably in a circle, with or without name tags. It can be discussed whether tables are needed. Moderators/facilitators (and note takers) should be integrated as much as possible within the discussion setting.
• The timing of the focus group interview need to be acceptable for all potential respondents in order to avoid selective “non-response” as much as possible (take into account the socio-demographic profiles of the targeted participants such as working times, daily activities, family life, etc.).

Duration

The length of the focus group should be between 1 and 3 hours.

Allow sufficient time at the beginning to welcome participants, give them an introduction and let them introduce themselves. This part should not take excessive time (about 10 minutes).

Material

Data are collected through different sources: audio or video-taping can be considered. When focus group interviews are recorded, the equipment should be of good quality and easy to use (check batteries and microphone). For larger groups, it may be necessary to use two tape recorders or multi-channel equipment, strategically placed to maximize the probability of recording contributions from all participants.

“Field notes” are an essential part during data collection. They capture all of the essential “non-verbal” information during the focus group interview.

Information has to be collected in an unbiased manner (avoid to filter out information as pre-interpreting it as unimportant, especially in the first focus groups).

The context of statements made during focus groups should be documented (important for giving meaning to the statements in the phase of analysis).

Try to capture nonverbal behavior of group participants (nonverbal reactions of other participants after a participant statement may indicate consensus or disagreement).

3.1.3.11      Analysis and reporting of findings

For issues on analysis, see  “How to analyse the data?”.

In the particular case of focus groups, separate analyses have to be performed on data gathered “within-focus group” and continuously compared “between focus group”. This is also an iterative process.

It is important that statements be understood in the context which they were made. Nonverbal communication observed during the interview can also be very informative.

For reporting, see part  “How to report qualitative research findings”

Note that findings are reported by focus group as unit of analysis and not by person.

3.1.3.12      Quality criteria

See section part  “How to evaluate qualitative research?”

Vermeire et al propose a checklist specific to critical appraise the quality of focus groups in health care research articles in primary healthcare (Vermeire et al, 2002).

3.1.3.13      Examples of KCE reports using the method

• Evaluation of the Belgian reference reimbursement system (LePolain et al, 2010)
• Evidence-based content of the written information provided by the pharmaceutical industry to the general practitioner (Van Linden et al, 2007)
• Quality development in general practice in Belgium: status quo or quo vadis ? (Remmen et al, 2008)
• Mental health care reforms: evaluation research of ‘therapeutic projects’ (Schmitz et al, 2010)
• Emergency psychiatric care for children and adolescents (Deboutte et al, 2010)

3.2.1 What is (naturalistic) observation?

Observing is more than looking around, it is actively registering information along a number of dimensions, namely places (physical place or setting), persons (the actors involved) and activities (a series of acts) ⁸³. Observing means having attention for (1) the detail of the observation, (2) visual as well as auditory information, (3) the time dimension, (4) the interaction between people, and (5) making links with mental categories (Mortelmans, 2009).

Observing includes roughly three steps:

1. A descriptive step; the researcher enters the research setting and gets a general overview of the social setting.
2. A focused step; more focused observations are a step closer to the research question. The aim is to search for relationships or connections between several elements in his research question, for example X is a characteristic of Y, or X is the result of Y. More concrete, suppose a researcher wants to study the way emergency care is organized in Belgium, he would do some descriptive observations in the emergency department of hospitals to get an idea of the general structures and processes characteristic for emergency care. In a next step he turns to his research question which is about how cost-effectiveness of emergency care could be attained. Hence the focus of his observation will relate to all possible costs and which could be avoided.
3. Selective step ⁸³;. In this last phase, after the researcher may have analysed his data (field notes), he may have identified a lack of information of one specific category of costs, e.g. cleaning and housekeeping costs, and may therefore decide to do extra observations in function of this specific aspect.

3.2.2 When to use observations?

• To collect data on naturally occurring behaviors in their usual contexts⁵⁴. Observation also captures the whole social setting in which people function by recording the context in which they live⁸⁴.
• Unstructured observation illustrates the whole picture, captures context/process and informs about the influence of the physical environment⁸⁴.
• To check whether what people say they do is the same as what they actually do⁸⁴. Both what people perceive that they do and what they actually do are however valid in their own right and just represent different perspectives on the data⁸⁴.
• Observation is also an ongoing dynamic activity that is more likely than interviews to provide evidence for processes, things that are continually moving and evolving⁸⁴.
• To study the working of organisations and peoples’ roles and functioning within organisations²⁰.
• To uncover behaviours or routines of which the observed themselves are not aware of²⁰. What the researcher considers an important finding may belong to the self-evident nature of daily life from the participants’ point of view.
• To understand data collected through other methods (e.g. interviews) and also to design the right questions for those methods⁵⁴. 

3.2.3 What are the strengths and weaknesses of observations?

3.2.3.1        Strengths

A number of strenghts have already been described under “When to use observations?”. We could add that:

• Observation has the advantage of capturing data in more natural circumstances⁸⁴.
• The Hawthorne effect[1] is an obvious drawback but once the initial stages of entering the field are past most professionals are too busy to maintain behaviour that is radically different from normal⁸⁴

3.2.3.2        Weaknesses

• It can be very difficult to get access to the setting.: An observer is often experienced as a threat, especially if the setting is not asking for the research to take place. Observation (and especially participant observation) might lead to knowledge of informal procedures or rules, which people do not want to be uncovered. Also the researcher can be experienced or perceived as a barrier for the normal daily routine in the setting10. In direct observation, the researcher does not participate in the setting, hence is known as a stranger and gets only access to the public or formal layer of the social reality. He does not become an insider and will miss inside information because he is too distant from the actors he is observing10. “Access, then, is not a straightforward process of speaking to the person in charge and obtaining the approval of the ethics committee. It usually involves considerable time and effort and a constant endeavour to strive for ‘cultural acceptability’ with the gatekeepers and participants in research sites” (p. 310)84.
• Once inside the setting there is the problem of avoiding “going native”: This means “becoming so immersed in the group culture that the research agenda is lost or that it becomes extremely difficult or emotionally draining to exit the field and conclude the data collection” (p. 183)20.
• Observational data, are more than interview data, subject to interpretation by the researcher. Observers have a great degree of freedom and autonomy regarding what they choose to observe and how they filter the information84.
• Observations are time-consuming and hard work at every possible hour of the day.
• An observer can get emotionally involved in what he observes, and by consequence lose his neutrality.
• It is impossible to write down everything that is important while observing (and participating). The researcher must rely on his memory and have the discipline to write down and expand the field notes soon and as completely as possible54.

---

[1]           The Hawthorne effect is the process where human subjects of an experiment change their behavior, simply because they are being studied http://www.experiment-resources.com/hawthorne-effect.html.

3.2.4 How to plan the research design?

Often observations are carried out at the beginning of the data collection phase, but the method can also be used later on during the research process to address questions suggested by data collected though other methods ^{(Mack, 2005)}. Before starting the observations, the researcher should try to find out as much as possible about the site where he will be observing.

At the KCE, site visits are common to allow the researchers to become familiar with the research topic and setting. This is often combined with interviews or less formalized talks to key persons on the site. After a number of site visits the scope of the research project is determined and precise research questions are formulated.

3.2.5 Modalities of data collection

3.2.5.1        Participant versus direct observation

The role to adopt during observation and the extent to which participants are fully informed are somewhat intertwined⁸⁴. Typically researchers refer to Gold’s typology of research roles⁸⁵:

• The complete observer, who maintains some distance, does not interact and whose role is concealed;
• The observer as participant, who undertakes intermittent observation alongside interviewing, but whose role is known;
• The participant as observer, who undertakes prolonged observation, is involved in all the central activities of the organization and whose role is known;
• The complete participant, who interacts within the social situation, but again whose role is concealed.

Mack et al.⁵⁴ describe observing as remaining an “outsider” and simply observing and documenting events or behaviors being studied, while participating is taking part in the activity while also documenting it. Pure observing, without participating is a situations that in fact seldom occurs, because once you are present, you are visible, you influence the activities around you, you participate in some degree. There are two reasons for this participation, or to better understand the local perspective, or in order not to call attention to yourself⁵⁴.

3.2.5.2        Structured versus unstructured observation

• Structured observations are associated with the positivist paradigm and aim at recording physical and verbal behavior by means of a list of predetermined behaviours⁸⁴.
• Unstructured observations are not ‘unstructured’ in the sense of unsystematic or messy, “instead, observers using unstructured methods usually enter ‘the field’ with no predetermined notions as to the discrete behaviours that they might observe. They may have some ideas as to what to observe, but these may change over time as they gather data and gain experience in the particular setting. Moreover, in unstructured observation the researcher may adopt a number of roles from complete participant to complete observer, whereas in structured observation the intention is always to ‘stand apart’ from that which is being observed” (p307)⁸⁴.

3.2.5.3        Overt versus covert observation

Covert observation corresponds to two roles in Gold’s typology⁸⁵, i.e. complete observer and complete participant (see above). Most authors agree that covert observation is only legitimate in very specific circumstances and should be avoided. Mack et al. ⁵⁴ formulate the following ethical guideline regarding observations: “When conducting participant observation, you should be discreet enough about who you are and what you are doing that you do not disrupt normal activity, yet open enough that the people you observe and interact with do not feel that your presence compromises their privacy.”(p. 16) As with all qualitative research methods, researchers must also protect the identities of the people they observe or with whom they interact, even if informally. “Maintaining confidentiality means ensuring that particual individuals can never be linked to the data they provide”⁵⁴. 

3.2.6 Data collection tools

3.2.6.1        Checklists

Before you enter the setting and start observing, it might be a good idea to have some questions in mind. It may be helpful to carry a checklist in your pocket to help you remember what you are meant to observe⁵⁴.

3.2.6.2        Fieldnotes

“Fieldnotes are used by researchers to record observations and fragments of remembered speech. Although researchers may use other means of recording (such as video) and other form s of data (such as interview transcripts), fieldnotes remain one of the primary analytic materials used in ethnography.” (p. 82) ³⁵.

Depending on the research questions, the researcher is interested in other aspects of social reality. Mulhalls’ schema⁸⁴ includes the following types of field notes, each covering an aspect of social reality:

• Structural and organizational features – what the actual buildings and environment look like and how they are used
• People – how they behave, interact, dress, move.
• The daily process of activities.
• Special events – in a hospital ward this might be the consultant’s round or the multidisciplinary team meeting.
• Dialogue.
• An everyday diary of events as they occur chronologically – both in the field and before entering the field.
• A personal/reflective diary – this includes both my thoughts about going into the field and being there, and reflections on my own life experiences that might influence the way in which I filter what I observe.

It is particularly important to detail any contradictory or negative cases. Unusual things often reveal most about the setting or situation²⁰.

Documenting observations consists of the following steps^{54, 86}:

• Quick notes during the observation.,
• Once the researcher left the setting, he expands his notes into fieldnotes. This means he reads them through and adds other things he can remember, but has not yet written down. Note taking in the setting is not self-evident and it is impossible to write down everything you see. Therefore good note taking should trigger the memory by means of key words, symbols, drawings, etc.
• After expansion, the researcher “translates” his shorthand into sentences., and
• Together with the translation phase, a descriptive narrative can be composed. The researcher writes down a description of what happened and what he has learned about the setting. In this step the researcher should distinguish between describing what happened and interpreting.

The researcher should be well aware of the difference between describing what he observes versus interpreting what he observed. It should be avoided to report interpretations rather than an objective account of the observations⁵⁴. For example, an interpretive description of a patient could be “he was in terrible pain”. An objective description would be “he was screaming and his face turned pale while grimacing”. “To interpret is to impose your own judgment on what you see” (Mack, 2005⁵⁴, p23). The danger is that interpretations can turn out to be wrong. Therefore the researcher should ask her/himself “what is my evidence for this claim?”⁵⁴. One way of separating descriptions and interpretations is by separating them visually on paper or screen.

3.2.6.3        Draw a map of the setting or settings you observe.

Maps might support your memory and are a tool to reconstruct interactions and movements of people in a room.

3.2.6.4        Audio or video

Audio or video recordings of observations are generally not permissible unless all ethical requirements are fulfilled and informed consent has been obtained.

3.2.7 Sampling

As outlined in the general principles of the chapter on interviewing, sampling in qualitative research is seldom statistically based. Also samples of settings or groups to observe are purposive.

Specifically for observation the sampling units are places, locations, and blocks of time, but usually not individuals. The aim is to select ‘information-rich’ cases, but in practice site selection is often a pragmatic decision based on existing networks and accessibility. Ideally however, sites are chosen because they typify some larger population of sites (such as clinics) or perhaps because they are exceptional in some way. Observation methods may be used across multiple sites and one could select the ones representing a range of typical settings (Green et al, 2009). 

3.2.8 Human resources necessary

Observations can be the work of one researcher, a pair of researcher, or a whole team. Which arrangement is most appropriate depends on the research questions and the features of the setting. Also members of a team can disperse to different locations individually, or in pairs or groups, in order to construct a more complete picture of the issues being studied.

One of the advantages of team work is that field notes can be compared and that team members can question each other about assertions being made. “Taking another perspective on validity Graneheim et al. (2001) used multiple data collectors with different perspectives (insider or outsider) to observe the same situation. This may not accord with the idea that every researcher may produce a unique account of a situation that is valid in its own right. But with extensive mutual reflection, as undertaken by Graneheim and colleagues, these combined observations may have consensual validity. However, from a practical standpoint few projects are afforded the luxury of multiple data collectors.” (Mulhall, 2003⁸⁴, p. 309).

3.2.9 Practical aspects

• Try to be “invisible” as an observator. Adapt to the setting in which you will do the observations, in terms of dress code, the way of behaving, and what is expected from you by the other actors in the setting.
• Start with short observations to explore the field and to get yourself used to your role as observer.
• First you should get an idea of “the normal” way of life in a setting, before you are able to identify unusual or abnormal situations.
• Circumstances may make it difficult or unacceptable to make fieldnotes, hence the researcher has to write down his observations afterwards. This can lead to a memory bias.
• Field notes should not contain interpretations, but merely descriptions.
• There is also the practical problem of how, especially in large and busy social settings, like an emergency department, to inform and obtain consent from everyone who might ‘enter’ the field of observation⁸⁴.
• Note that once inside the setting it might be difficult to get out again: Ending the fieldwork should not happen abruptly. The researcher must take time to “ease out”. In the ‘easing out’ phase the researcher is more and more absent from the setting. This means more time to analyse the data. When present in the setting, the researcher can confront his preliminary analysis with new observations in the setting¹⁰. In the literature the advice is to keep in contact with the setting until the final report is written⁸⁷.

3.2.10 Analysis

Field notes contain a lot of detail and are highly descriptive. In order to find explanations or answers to the research questions, the researcher should develop categories and test them against hypotheses, and refine them. This is an iterative process that starts during the data collection phase. 

3.2.11 Reporting of findings

As with other qualitative research methods it is important that evidence from the data is presented to support the conclusions of the researcher, by means of examples or quotations. The main principles have already been mentioned in (see  “How to report qualitative research findings”).

3.2.12 Quality criteria

The quality of observational studies depends largely on the quality of the descriptions of data collection and analysis provided by the researcher. Details about how the research was conducted are crucial and should be well documented. For example, how much time was spent in the field, how typical were the events recorded, description of the attempts to verify the observations made, etc.

The general criteria to assess the quality of qualitative research are described here and also apply to observational methods.

3.2.13 Examples of KCE reports using the method

So far no observational studies have been carried out at the KCE.

3.3.1 Description of the method

The Delphi method (named so because of the Delphi Oracle) was initiated by the RAND corporation, a nonprofit institution that helps improve policy and decision making through research and analysis[a]. The original definition given in the 50^s was that it “entails a group of experts who anonymously reply to questionnaires and subsequently receive feedback in the form of a statistical representation of the "group response," after which the process repeats itself. The goal is to reduce the range of responses and arrive at something closer to expert consensus.”⁸⁹ Today, the method has evolved and Delphi surveys could aim at different goals or have several designs[b]. It could be define more as “a method for structuring a group communication process” and not as a method to produce consensus⁹⁰. The method could also be defined as a systematic collection and aggregation tool of informed judgment from a group of experts on specific questions and issues” (Hasson, 2011⁹¹, p. 1696).

Delphi surveys are used in several domains (politics, psychology, agriculture, etc.) and could vary in different ways. Several types of Delphi often used in health research (non exhaustive) are presented in Table 10.

 

Table 10 – Types of Delphi designs

Design Type	Aim	Target panellists	Administration	Number of rounds	Round 1 design
Classical	To elicit opinion and gain consensus	Experts selected based on aims of research	Traditionally postal	Employs three or more rounds[3]	Open qualitative first round, to allow panelists to record responses
Modified	Aim varies according to project design, from predicting future events to achieving consensus	Experts selected based on aims of research	Varies, postal, online, etc.	May employ fewer than 3 rounds	Panelists provided with pre-selected items, drawn from various sources, within which they are asked to consider their responses
Decision	To structure decision-making and create the future in reality rather than predicting it	Decision makers, selected according to hierarchical position and level of expertise	Varies	Varies	Can adopt similar process to classical Delphi
Policy	To generate opposing views on policy and potential resolutions.	Policy makers selected to obtain divergent opinions	Can adopt a number of formats including bringing participants together in a group meeting	Varies : It theoretically needs 5 rounds but could be done in 3 or 4 rounds:	Can adopt similar process to classical Delphi or 1- preformulating the obvious issues by the research team; 2- seeding the list with an initial range of options but allowing for the respondents to add to the lists92, 93
Real time/consensus conference	To elicit opinion and gain consensus on real time	Experts selected based on aims of research	Use of computer technology that panelists use in the same room to achieve consensus in real time rather than post or via Internet94	Varies	Can adopt similar process

Adapted from Hasson, 2011⁹¹, p. 1697 and Keeney, 2011⁹⁵

---

[a]           http://www.rand.org

[b]           See the special issue 78 of the review ‘Technological Forecasting & Social change” (2011) available at http://www.journals.elsevier.com/technological-forecasting-and-social-ch....

[3]           Note that the number of rounds should ideally be based on the saturation of the responses and is difficult to fix in advance

3.3.2 Specific questions suitable for the method

The following questions could be answered by using a consensus reaching method such as the Delphi panel:

• To help the decision making process.
• When personal contact is not necessary⁹⁶.
• To choose the most appropriate method or tool (e.g. data collection technique, scales, questionnaires, etc.).
• To identify the best choice of treatment (when no other evidence is available or to complete it).
• To identify the form of a programme.
• To clarify professional roles⁹⁷.
• To develop clinical guidelines⁹⁸.

3.3.3 Strengths and weaknesses of the method

3.3.3.1        Strengths

• Lower production cost⁹⁹.
• Relatively rapid results⁹⁹.
• Participant can express their opinion anonymously⁹⁶, without external (perceived) pressure while the process allows to catch the view of the entire group⁹⁶.

• Avoid domination by individuals or professional interests97;

3.3.3.2        Weaknesses

• Success depends on the qualities of the participants.
• Reliability increases with the number of participants (and the number of rounds). In addition, it is difficult to keep everybody in successive rounds96.
• Coordination is difficult96.
• The existence of a consensus does not necessary mean that it reflects an appropriate or “correct” answer97.

3.3.4 How to plan the research design?

A Delphi survey takes several weeks, even if the number of participants is small.

It has to be planned in the beginning of the project or, if the necessity to conduct such a study appears late in the course of the project, it is important to realize that the whole process takes several weeks, depending on the number of rounds needed. The next figure illustrates the whole process and the time needed.

Figure 5 – The Delphi process

 Adapted from Slocum et al.⁹³

3.3.5 Modalities of data collection

Delphi could be administrated ‘paper-and-pencil’ by mail or e-mail.

Online Delphi’s are more and more carried out. Software is available to support the data collection and the analysis (Delphi_Survey_Web (DSW)¹⁰⁰, Mesydel©¹⁰¹)

The number of rounds is not necessarily defined a priori (often because of budgetary, time or human resources limitations): data collection must stop when the saturation or the consensus is reached.

3.3.6 Data collection tools

The Delphi method uses iterative (e-)mailed questionnaires in successive rounds. Because there is no interaction between the respondent and the researcher, the formulation of the questions has to be clear, and definitions should be given where necessary.

The questionnaire of the first round encompasses open-ended questions, to identify items to include in the second round.

Next rounds could be exclusively qualitative or composed of closed questions with scales (from totally agree to totally disagree, i.e. from 1 to 9), or combining both qualitative and quantitative questions. They present a synthesis of the results issued from the previous round.

In the case of closed questions, agreement is usually summarized by using the median and consensus assessed by presenting interquartile ranges for continuous numerical scales⁹⁷. Graphical presentations of the results are welcomed.

In KCE reports the questionnaires used in each round are presented in appendices.

3.3.7 Sampling

Participants have to be carefully chosen because of their expertise, experience or knowledge in the field of the research question. In addition, the variety of positions in the field or opinions regarding the subject, should be covered. In that way, lay people could be added to increase the variety of viewpoints¹⁰².

They could be identified through publically available bibliographic information¹⁰². Snowballing recruitment could be useful to secure easy agreement to panelist invitation and strengthen panelist retention¹⁰².

There is no practical limit to the number of participants in a Delphi survey⁸⁹. 

3.3.8 Human resources necessary

The administrator of the survey develops the questionnaires, identifies, mobilizes and recruits participants, analyses findings and reports them. He/she is responsible for keeping a low attrition rate and insure the coherence between the different steps of the method.

Administrative support could be needed to (e-)mail the questionnaires and manage reminders and answers.

3.3.9 Practical aspects

• It is important to clearly explain the goal of the questionnaire and the way it will be analysed. The redaction of the invitation/introduction letter is thus crucial. “Stressing the practical policy application of the Delphi yield to experts panelists to aid their retention” (Rowe, 2011¹⁰², p. 1489).
• The research team should have managers skills to follow up the returned questionnaires and mailing.
• The utilization of online tools could be very useful as well for the research team (rapid results) as for the participants.
• While anonymity in the process of the Delphi is required, “using social rewards for recognition in participation, such as subsequently publishing panel membership listings” (Rowe, 2001¹⁰², p. 1489) could improve panelists recruitment and retention.

3.3.10 Analysis

Each step of the Delphi requires a specific analysis.

In a classical Delphi, open-ended questions from round 1 should be content analysed ‘in order to group statements generated by the experts panel into similar areas’⁹⁵.

Round that uses closed questions should be statistically analysed. Summary statistics are used to decide whether or not consensus is reached. The level of the consensus has to be defined in advance (i.e. 70% of agreement).

There is no agreement on the threshold indicating a consensus, nor how to choose this threshold⁹⁵. Each researcher has to reflect on it, case by case.

The proposals that have reached consensus should be eliminated from the next round.

3.3.11 Reporting of findings

Intermediary results are reported directly in the successive questionnaires.

All the consensus and dissensus items are listed and discussed at the end of the process. 

3.3.12 Quality criteria

It seems that no consensus exists with regards to the standard of methodological rigor to apply. And that “no definitive evidence exists which demonstrates the reliability or validity of the technique” (Keeney, 2011⁹⁵, p. 104). This is partly due to the variety of the Delphi surveys and the constant evolutions in this field⁹¹.

We have not identified any checklists to assess the quality of a Delphi survey.

However, the following aspects of the survey could be assessed (adapted from Jillson¹⁰³ and Hasson⁹¹):

• Applicability of the method to the specific research problem
• The quality of the composition of the Delphi panel. Participants have to be carefully chosen in function of their expertise and position in the group.
• Design and administration of the questionnaire
• Feedback

A Delphi survey should be reviewed in terms of reliability, validity and trustworthiness to judge its worth⁹¹.

3.1.13 Examples of KCE reports using the method

• Impact of academic detailing on primary care physicians¹⁰⁴
• Burnout among general practitioners: prevention and management⁷²
• Methods for including public preference values in reimbursement decision making processes for health interventions. Exploration of the feasibility of different models in Belgium (ongoing project, publication foreseen end 2012)

3.1.14 Basis references

For practical tips see the report of the King Baudouin Foundation available in French, Dutch and English⁹³

4.3.1 Generic methods for analyzing

Globally, a generic method for analyzing is used in many situations: How to analyze the data? To get the meaning of the data? It encompasses the technical and intellectual operations and manipulations helping the researcher to catch the meanings.

• Technical operations for analyzing are processes, operations and management of the data such as transcriptions, cutting of the text, putting it in tables, etc.
• Intellectual operations for analyzing consist of the transposition of terms in other terms, intuitive groupings, confrontation, induction …

Classically, 3 generic methods of analysis are used in qualitative health (care) research, each of them using specific tools

• The phenomenological examination of the empirical data, aiming to report the authentic comprehension of the material
• The thematic analysis, more specifically this is the creation and the refinement of categories to give a global picture of the material
• The analysis using conceptualising categories, aiming at the creation and the refinement of categories to go further than the description, to reach conceptualization of

4.3.2 Specific traditions

Specific traditions are embedded in the generic methods used in health(care) research we described. We give an example for each of them:

4.1.1.1     Phenomenology

Phenomenology focuses on “how human beings make sense of experience and transform experience into consciousness, both individually and as shared meaning” (Patton 2015, p.115). Phenomenology is about understanding the nature or meaning of everyday life. In-depth interviews with people who have directly experienced the phenomenon of interest, is the most used data collection technique. Phenomenology in qualitative research goes back to a philosophical tradition that was first applied to social science by E. H. Husserl to study people’s daily experiences.

Phenomenology will not be developed into detail, because it is less relevant to KCE projects.

4.1.1.2     Framework analysis

Framework analysis has been developed specifically for applied or policy relevant qualitative research, and is a deductive research strategy. In a framework analysis the objectives of the investigation are set in advance. The thematic framework for the content analysis is identified before the research or the qualitative research part in the project sets off.

The decision on using frameworks when analyzing data is closely related to the question for what purpose the qualitative material will be used in the overall research strategy. “Frameworks” are generally deducted from hypotheses of theoretical frameworks: e.g. if the aim of a focus group is trying to get a picture of stakeholders interests and potential conflicting perspectives on a health care issue, and the focus group tries to grasp how stakeholders develop power plays or influence strategies to set agenda’s, a conceptual framework on decision-making processes and power play will serve as a useful tool to orient data-collection and data-analysis.

Applying framework analysis concretely means that the themes emerging from the data are placed in the framework defined a priori. The framework is systematically applied to all the data. Although an analytical framework can be very useful, it is not suited, if the aim is to discover new ideas, since a framework or grid could be blinding (Paillé and Mucchielli 2011).

For the specificity of the analysis of data according to this method see Framework analysis

4.1.1.3     Grounded theory

Grounded theory was developed by Glaser and Strauss in the late 1960s as a methodology for extracting meaning from qualitative data. Typically, the researcher does not start from a preconceived theory, but allows the theory to emerge from the data (Durant-Law 2005). Hence grounded theory is an inductive rather than a deductive methodology. Emergence is also a key assumption in grounded theory: data, information and knowledge are seen as emergent phenomena that are actively constructed. They can only have meaning when positioned in time, space and culture (Durant-Law 2005).

The power of grounded theory lies in the depth of the analysis. Grounded theory explains rather than describes and aims at a deep understanding of phenomena (Durant-Law 2005). Key to grounded theory is the emphasis on theory as the final output of research. Other approaches may stop at the level of description or interpretation of the data (e.g. thematic analysis).

Grounded theory is a complete method, a way of conceptualizing a qualitative research project.

For the specificity of the analysis of data according to this method see Data analysis in the Grounded Theory

4.3.3 Inductive versus deductive approaches

The approach chosen depends largely on the design and the aims of the research. Some designs and/or research questions require an inductive, others a deductive approach. Inductive means that themes emerge from the data, while deductive implies a pre-existing theory or framework which is applied to the data. Qualitative data analysis tends to be inductive, which means that the researcher identifies categories in the data, without predefined hypotheses. However, this is not always the case. A qualitative research analysis can also be top down, with predefined categories to which the data are coded, for example a priori concepts can be adopted from the literature or a relevant field. Framework analysis can be used this way.

 

The next table shows how the different methods, approaches and types of coding  relate to each other.

Generic methods, specific methods/ traditions, approaches and type of coding for qualitative analysis

Generic methods	Specific methods / traditions	Approaches	Type of coding
Phenomenological examination of the empirical data	Phenomenology	Inductive	Statements
Thematic analysis	Descriptive analysis Framework analysis	Mainly deductive Mainly deductive	Themes
Analysis using conceptualizing categories	Grounded Theory	Mainly inductive Mainly deductive	Conceptualizing categories

Step 0: Preparing the data for analysis

Independent of the methodological approach, a qualitative analysis always starts with the preparation of the gathered data. Ideally, to enable accurate data analysis the recorded information is transcribed. A transcript is the full length literal text of the interview. It often produces a lot of written text.

Good quality transcribing is not simply transferring words from the tape to the page. The wording is only part of the message. A lot of additional information is to be found in the way people speak. Tone and inflection, timing of reactions are important indicators too. With experienced observers and note-takers, a thematic analysis of the notes taken during the interviews could be used as a basis for analysis of the “non-verbal” communication.

Transcribing is time consuming and costly. The research team should consider in advance the question "who should do the transcribing”? Resources may be needed to pay an audio typist, a strategy usually more cost effective than a researcher. Be aware that “typists” are often unfamiliar with the terminology or language used in the interviews or focus groups which can lead to mistakes and/or prolong the transcribing time.

It may not be essential to transcribe every interview or focus group. It is possible to use a technique known as tape and notebook analysis, which means taking notes from a playback of the tape recorded interview and triangulating them with the notes taken by the observers and note-takers. However, bias can occur if inexperienced qualitative researchers attempt tape and notebook analysis. It is certainly preferable to produce full transcripts of the first few interviews. Once the researcher becomes familiar with the key messages emerging from the data tape analysis may be possible. Transcripts are especially valuable when several researchers work with the same data.

Step 1: Familiarization

Researchers immerse themselves in the data (interview transcripts and/or field notes), mostly by reading through the transcripts, gaining an overview of the substantive content and identifying topics of interest (Spencer et al, 2014). Doing this, they get familiar with the data.

Step 2: Coding the data - Construction of initial categories

By reading and re-reading the data in order to develop a profound knowledge of the data, an initial set of labels is identified. This step is very laborious (especially with large amounts of data). Pieces of text are coded, i.e. given a label or a name. Generally, in the qualitative analysis literature, “ data coding” refers to this data management. However data coding refers to different levels of analysis.

Here are some commonly used terms (Paillé and Muchielli, 2011):

Label:

Labeling a text or part of a text is the identification of the topic of the extract, not what is said about it. “What is the extract about?” The labels allow to make a first classification of the documents/ extracts. They are useful in a first quick reading of the corpus.

Example: “Familial difficulties”

Code:

The code is the numerical/truncated form of the label. This tool is not very useful in qualitative data analysis.

Example: “Fam.Diff.”

Theme:

The theme goes further than the label. It requires a more attentive lecture.

 “What is the topic more precisely?”

Example: “Difficulties to care for children”

Statement:

Statements are short extracts, short syntheses of the content of the extract. “What is the key message of what is said?”, “What is told?”           
The statement is more precise than the theme because it resumes, reformulates or synthetizes the extract. They are mainly used in phenomenology.

Example: The respondent tells that she has financial difficulties because she has to spend time and money to take care of her children.

Conceptualizing category:

Conceptualizing categories are the substantive designations of phenomena occurring in the extract of the analyzed corpus. Hence, this approaches theory construction.

Example: “Parental overload”

 

These types of coding terms are generally more specific to certain types of qualitative data analysis methods (Paillé and Muchielli, 2011).

By coding qualitative data, meanings are isolated in function of answering the research question. One piece of text may belong to more than one category or label. Hence there is likely to be overlap between categories. Major attention should be paid to “rival explanations” or interpretations about the data.

 

For further detailed information on coding qualitative data:
Saldaña J. The coding manual for qualitative researchers. 2nd edition ed. London: Sage Publications; 2013.

Step 3: Refine and regroup categories

In a third step the categories are further refined and reduced by being grouped together. “While reading through extracts of the data that have been labelled in a particular way, the researchers assesses the coherence of the data to see whether they are indeed ‘about the same thing’ and whether labels need to be amended and reapplied to the data” (Spencer et al. 2014a), p. 282).

Word processors or software for qualitative data analysis [LAK1] will prove to be very helpful at this stage.

---

 [LAK1]Add crosslink vers section process book existante

Step 4: Constant comparison

During the analysis the researcher might (as a third step) constantly compare the constructed categories with new data, and the new categories with already analyzed data. This results in a kind of inductive cycle of constant comparison to fine tune categories and concepts arising from the data. NB: In the particular case of focus groups, separate analyses have to be performed on data gathered “within-focus group” and continuously compared “between focus group”. This is also an iterative process.

(Step 5): New data collection

New data collection could also be necessary to verify new point of views or insights emerging from the analysis.

Before moving to the more interpretive stage of analysis, the researchers may decide to write a description for each subtheme in the study (Spencer et al., 2014).

Step 6: Abstraction and interpretation

“Taking each theme in turn, the researcher reviews all the relevant data extracts or summaries, mapping the range and diversity of views and experiences, identifying constituent elements and underlying dimensions, and proposing key themes or concepts that underpin them. The process of categorization typically involves moving from surface features of the data to more analytic properties. Researchers may proceed through several iterations, comparing and combining the data at higher levels of abstraction to create more analytic concepts or themes, each of which may be divided into a set of categories. Where appropriate, categories may be further refined and combined into more abstract classes. Dey (1993) uses the term ‘splitting’ and ‘slicing’ to describe the way ideas are broken down and then recombined at a higher level – whereas splitting gives greater precision and detail, slicing achieves greater integration and scope. In this way, more descriptive themes used at the data management stage may well undergo a major transformation to form part of a new, more abstract categorical or classificatory system” (Spencer et al., 2014, p. 285). At this stage typologies can be created.

Step 7: Description of the findings and reporting

Findings can be presented in a number of ways, there is no specific format to follow.

When writing up findings issued from interviews or texts qualitative researchers often use quotes. Quotes are useful in order to (Corden and Roy 2006):

• Illustrate the themes emerging from the analysis.
• Provide evidence for interpretations, comparable to the use of tables of statistical data appearing in reports based on quantitative findings.
• Strengthen credibility of the findings (despites critics argue that researchers can always find at least one quote to support any point they might with to make).
• Deepen understanding. The actual words of a respondent could sometimes be a better representation of the depth of feeling.
• Enable voice to research participants. This enables participants to speak for themselves and is especially relevant in a participatory paradigm.
• Enhance readability by providing some vividness and sometimes humour: Braking up long passages of text by inserting spoken words, could help to keep the reader focused, but there could be a danger in moving too far towards a journalistic approach.

Ideally, quotes are anonymous and are accompanied by a pseudonym or description of the respondents. For example, in a research about normal birth, this could be: (Midwife, 36 years). There are however exceptions the rule of anonymity, e.g. stakeholder interviews, in which the identity of the respondent is important for the interpretation of the findings. In that case the respondent should self-evidently be informed and his agreement is needed in order to proceed.

Also in terms of lay out quotations should be different from the rest of the text, for example by using indents, italic fond or quotation marks. Quotes are used to strengthen the argument, but should be used sparingly and in function of the findings. Try to choose citations in a way that all respondents are represented. Be aware that readers might give more weight to themes illustrated with a quotation.

When the research is conducted in another language than the language of the report in which the findings are presented, quotes are most often translated. “As translation is also an interpretive act, meaning may get lost in the translation process (van Nes et al.), p. 313)”. It is recommended to stay in the original language as long and as much as possible and delay the use of translations to the stage of writing up the findings (van Nes et al.).

KCE practice is to translate quotes only for publications in international scientific journals, but not for KCE reports. Although KCE reports are written in English, inserted quotes are in Dutch or French to stay close to the original meaning. The authors should pay attention to the readability of the text and make sure that the text without quotes is comprehensive to English speaking readers.

So far, this general a-theoretic procedure reflects what in the literature is called the general inductive approach for analyzing qualitative data. It does not aim at the construction of theories, but the mere description of emerging themes. It provides a simple, straightforward approach for deriving findings in the context of focused research questions without having to learn an underlying philosophy or technical language associated with other qualitative analysis approaches (Thomas, 2006).

4.5.1 An analysis with (predefined) themes: a deductive approach

Adapted from Paillé and Muchielli , 2011.

 

The thematic analysis is a process to reduce data. It is not a deep analysis, but rather to describe the topic(s) appearing in the corpus. “Thematization” is a preliminary step in all types of analysis of qualitative data. It consists of transposing the corpus into a number of themes issued from the analyzed content and according to the problematic.

A first step is the location, i.e. the listing of all the themes pertinent for the research question. The second step is to document it: identify the importance of specific themes, repetitions, crosschecks, what goes together, what goes opposite…

4.5.2 Framework analysis

Adapted from Spencer L, Ritchie J, O'connor W, G. M, Ormston R. Analysis in practice. In: Ritchie J, Lewis J, McNaughton Nicholls C, Ormston R, editors. Qualitative research practice. London: Natcen, Sage; 2014. p. 295-345.

In the framework analysis data will be sifted, charted and sorted in accordance with key issues and themes (Srivastava et al. 2009). The analytical journey using this approach could be simply described as:

1. Familiarization
2. Constructing the initial framework
3. Indexing
4. Charting
5. Abstraction and interpretation

The familiarization is the same as explained previously [add crossref]. In this approach, it is the occasion to identify topics or issues of interest, recurrent across the data and relevant for the research question, taking thus into account the aims of the study and the subjects contained in the topic guide.

The construction of an initial thematic framework can begin once the list of topics has been reviewed. This step aims to organize the data. The analyst will identify underlying ideas or themes related to particular items. (s)He will use these to group and sort the items according to different levels of generality, building a hierarchical arrangement of themes and subthemes. It results in a sort of table of content of what could be found in the corpus. These themes or issues “may have arisen from a priori themes (…) however it is at this stage that the researcher must allow the data to dictate the themes and issues”. “Although the researcher may have a set of a priori issues, it is important to maintain an open mind and not force the data to fit the a priori issues. However since the research was designed around a priori issues it is most likely that these issues will guide the thematic framework. Ritchie and Spencer stress that the thematic framework is only tentative and there are further chances of refining it at subsequent stages of analysis (1994).”  (Srivastava et al. 2009, p.76).

The next step consists of indexing the data, i.e. labelling sections of the corpus according to the thematic framework. This could be done by annotation in the margin of the transcript.

The fourth stage consist of charting: the indexed data are arranged in charts of themes. One chart is built for each theme. Subthemes are headings of the columns while each row represent an interview, transcript or unit of analysis. The content of each cell is a summary of the section of the corpus related to the subtheme.

To write useful summaries, “the general principle should be to include enough details and context so that the analyst is not required to go back to the transcribed data to understand the point being made, but not include so much that the matrices become full of undigested material (…)”. (Spencer et al. 2014b, p 309)

Spencer et al identified 3 requirements essential in order to retain the essence of the original material (Spencer et al. 2014b, p 309).

1. Key terms phrases or expressions should be taken as much as possible from the participant’s own language;
2. Interpretation should be kept to a minimum at this stage;
3. Material should not be dismissed as irrelevant just because its inclusion is not immediately clear.

The last step is the mapping and interpretation. Spencer et al. advice to take the time to do this, have a break, read through the management of the data, etc.

In this phase, concept, categories could be developed. Linkage between them could be described and explanations and patterns could be raised. This could even be performed by a theorizing deduction. The category is issued of a theoretical preexisting referent. The categories exist because a former analysis of the problematic has already been carried out. (Paillé and Muchielli. 2011). In the framework analysis, the main categorical analysis grid is preexisting. This could be because the research object is already well studied, because of the research is commissioned by an institution or because the research is spread through different teams in different locations (Paillé and Muchielli. 2011).

Nivivo [add cross ref] could be very helpful in the management of the data and creation of the matrix when using the Framework approach.

4.5.3 An analysis with conceptualizing categories: an inductive approach

Adapted from Paillé and Muchielli , 2011.

 

The analysis by conceptualizing categories allows a more in depth analysis. It is more than only the identification of themes, without a link between the annotation of the corpus and the conceptualizing of the data. It is more than a synthesis of the material. It includes an intention to analyze, to reach the meaning and use then a type of annotation reflecting the comprehension made by the analyst.

5.1.1 What is it about?

Checkland and Poulter defined SSM as follows:

“SSM is an organized way of tackling perceived problematical (social) situations. It is action oriented. It organizes thinking about such situations so that action to bring about improvement can be taken” (Checkland and Poulter 2010), p. xv).

SSM uses system ideas developed within hard systems thinking in problem solving. SSM is an approach which in a systematic way tries to establish and structure a debate concerning actions for improving the problem situation (Simonsen, 1994, http://www.jespersimonsen.dk/Downloads/SSM-IntroductionJS.pdf)(Simonsen 1994). Soft systems approaches diverge from hard systems approaches in explicitly integrating the assumption that an objective representation of reality does not exist. Our perspective is always directed and filtered by our world view. We always have only a partial picture of reality (See illustration).

Illustration: The blind men and the matter of the elephant (reproduced from Meadows, D., 2008, p. 7)

Beyond Ghor, there was a city. All its inhabitants were blind. A king with his entourage arrived nearby; he brought his army and camped in the desert. He had a mighty elephant, which he used to increase the people’s awe.

The populace became anxious to see the elephant, and some sightless from among this blind community ran like fools to find it.

As they did not even know the form or shape of the elephant, they groped sightlessly, gathering information by touching some part of it.

Each thought that he knew something, because he could feel a part…

The man whose hand had reached an ear… said: “It is a large, rough thing, wide and broad, like a rug.”

And the one who had felt the trunk said: “I have the real fact about it. It is like a straight and hollow pipe, awful and destructive.”

The one who had felt its feet and legs said: “It is mighty and firm, like a pillar.”

Each had felt one part out of many. Each had perceived it wrongly…

This ancient Sufi story was told to teach a simple lesson but one that we often ignore: The behavior of a system cannot be known just by knowing the elements of which the system is made.

 

Soft system methodology tries to align the partial pictures to be able to take coordinated action. This radical constructivist perspective includes that social systems do not exist as such, but are always informed by intentionality. Identifying this intentionality is at the core of SSM (Vandenbroeck 2015).

In short, SSM can be characterised by the following points:

• In contrast to the approaches described above (grounded theory and framework analysis), SSM is an action-oriented approach, which means that its purpose is to enable actions to improve (Checkland, 2000, research paper)(Checkland 2000). The change sought can be structural change, process change or changes of attitude, or all three at once (Checkland, 2000, research paper)(Checkland 2000).
• SSM is used “to make sense of complex situations” (Checkland, 2000, research paper)(Checkland 2000).
• SSM is flexible. Any approach able to deal with the changing complexity of real life needs to be flexible, because every situation involving human beings is unique. SSM offers a set of principles which can be adopted and adapted for use in any real situation in which people want to take action to improve it. SSM is not a clear sequence of steps. (Checkland, 2000, research paper)(Checkland 2000)
• SSM is a learning cycle, which goes from finding out about a problematical situation to defining/taking action to improve it. (Checkland, 2000, research paper)(Checkland 2000)

Checkland (Checkland and Poulter 2010) emphasized that SSM is not a technique in the sense of a recipe, nor a method, but a methodology. That means it is a set of principles which can be adapted for use in a way which suits the specific nature of each situation in which it is used. The set of principles can be adopted or adapted for use in any real situation in which people are intending to take action to improve it.

 

5.1.2 The SSM learning cycle

Adapted from Checkland and Poulter (2010)

 

The SSM process takes the form of a cycle. It is a cycle of learning which goes from finding out about a problematical situation to defining/taking action to improve it. The steps in the learning cycle are described below (see also Figure X).

 

Figure X: The SSM’s learning cycle

 

Real system

Soft  system world

2) Formulate root definitions

3) Build activity models

1) Find out about the problematical situation

4) Use the models to question the real world situation

5) Define actions to improve the situation

 

5.2.1 The iceberg model

The iceberg is a metaphor associated with systems thinking (Senge 1990). Systems thinking approaches problems by asking how various elements within a system influence one another. The visible world around us is represented by the top of the iceberg, but this is only a “manifestation of patterns and structures that are below the water surface, hence cannot be observed directly” (Vandenbroeck 2015). What happens under water is what creates the icebergs behavior at its top. The iceberg represents a hierarchy of levels of understanding with observable events at the top and mental models at the bottom.

 

• Observable events

The guiding question to find out about events is: “What just happened?”. The response is the events resulting from system behaviour or repeating patterns of cause and effect at the lower layer of the iceberg.

• Patterns/trends

Below the events level, patterns and trends become visible, by asking “What trends have there been over time?”. Similar events have been taking place over time.

• Underlying structures

At the structure level we could ask: “What is causing the pattern we are observing?” or “What are the relationships between the parts?”. Structures might consist of physical things (like buildings, roads, etc.), organisations (e.g. schools), policies (e.g. laws) or rituals (e.g. habits).

• Mental models

“Mental models are the images, assumptions, and stories which we carry in our minds of ourselves, other people, institutions, and every aspect of the world. Like a pane of glass framing and subtly distorting our vision, mental models determine what we see” (Senge et al. 1994). Also “Differences between mental models explain why two people can observe the same event and describe it differently” (Senge et al. 1994). In qualitative research we encounter mental models often in (mis)beliefs, expectations, values and attitudes.

We are unaware of our mental models or those of others, until we diliberately look for them. By means of qualitative research, and especially in combination with a systems thinking or grounded theory approach, we can bring mental models to the surface and explore them. Once we identified them we can try to re-form mental models or create new ones that serve us better in the world. Soft systems methodology (but also for example imagineering) can help us doing this. Mental models are the deepest layer of the iceberg, which is suggesting that they are difficult to reach and unresponsive to change. However, if mental models can be changed they offer the highest leverage for change (e.g. within an organisation or system) (Senge et al., 1994).

 

“The lower level of the iceberg gives context and meaning to the higher level” (Vandenbroeck 2015). For every event you can work your way down the iceberg through the patterns, underlying systems and mental models. It can also be useful to move up and down between levels as you think more about the event. The iceberg should help to broaden your perspective. Each layer offers opportunities to “enter” the system. New leverage points, these are points at which to intervene in a system to systematically transform it, may become apparent.

 

5.2.2 Reinforcing and balancing feedback loops

Adapted from Senge, P. et al. 1994, the fifth discipline field book, p. 113-120

 

In a feedback loop every element is both ‘cause’ and ‘effect’. For every variable you can trace links that represent influence on another element. This way cycles are revealed that repeat themselves. Figure X presents an example with increasing numbers of patients increasing waiting times in a clinic, and increasing waiting times leading to decreasing numbers of patients, leading to decreasing waiting times again, and so on.

Figure X: Example of a feedback loop

 

There are basically two building blocks of all systems representations:

• Reinforcing loops: generate growth and collapse, in which the growth or collapse continues at an ever-increasing rate. A small change builds on itself, resulting in big changes after some time. There can be any number of elements in a reinforcing loop, all propelling each others’ growth. Reinforcing loop situations generally “snowball” into highly amplified growth or decline. Somewhere sometime the reinforcing loop will run up against at least one balancing mechanism that limits it. The letter R is used to mark a reinforcing loop.
• Balancing loops: generate stability. Balancing processes generate the forces of resistance, which eventually limit growth. Balancing loops are found in situations which seem to be self-correcting and self-regulating. The letter B is used to mark a balancing loop.

In addition to feedback loops also time needs to be taken into account. Both in reinforcing and balancing loops delays may occur. Delays are the points where the link takes a particularly long time to play out. Delays can have enormous influence in a system, frequently accentuating the impact of other forces. When unacknowledged delays occur, people tent to react impatiently, usually redoubling their efforts to get what they want. This results in unnecessarily violent oscillations. One of the purposes of drawing systems diagrams is to flag the delays which you might otherwise miss. 

 

5.2.3 System Archetypes

Adapted from Senge, P. et al. 1994, the fifth discipline field book, p. 165-172

 

Archetypes are accessible tools with which credible and consistent hypotheses can be constructed. Kim and Lannon (Kim and Lannon 1997) rightly point out that they can be used in at least for different ways:

• As “lenses”: it is not about which archetype is “right”, but rather about what unique insights each archetype offers.
• As structural pattern templates: archetypes can help focus a group’s attention on the heart of an issue. After a group has drawn a causal loop diagram of the problem at hand, they can stand back and compare their diagram with the pattern of an archetype.
• As dynamic scripts (or theories): each archetype offers prescriptions for effective action. Once we recognize a specific archetype at work, we can use the theory of that archetype to expel a particular problem and work toward an intervention.
• As tools for predicting behavior: systems archetypes can help us identify predetermined outcomes of a particular situation.

To find out which archetype applies, a good strategy is to look at your situation through the lens of several different archetypes. Two or three may fit together, each highlighting a different aspect.

You can start by drawing just a simple balancing or reinforcing loop. Then add more elements, one link at a time. About each element ask what is causing changes in this element, and also what is the effect when this variable changes.

In what follows, three archetypes are presented. However many more archetypes are described in

• Senge, P. et al. 1994, The fifth discipline field book, p. 125 – 150.
• Meadows, D. 2008, Thinking in Systems. p. 110 – 141.

 

5.2.3.1     The “fixes that backfire” archetype

The central theme of this archetype is that almost any decision carries long-term and short-term consequences, and the two are often diametrically opposed. A problem symptom cries out for resolution. A solution is quickly implemented (the fix) which alleviates the symptom (balancing loop), but the unintended consequences of the fix (reinforcing loop) actually worsen the performance or condition which we are attempting to correct.

Example: child abuse is underreported to authorities. In the US they made reporting mandatory. However, child protection services were not reinforced, hence were overwhelmed by the number of reports, and could only investigate a small part of all reports. By consequence they got the reputation of being untrustworthy. In response, people decided not to report (although mandatory) and tried to find solutions themselves or did not do anything. Number of reports decreased again, hence the problem of underdetection was reinforced.

 

Figure X: System dynamics model for “Fixes that backfire” – example

 

 

5.2.3.2     The “Limits to growth” archetype

We never grow without limits. In every aspect of life, patterns of growth and limits come together. In this archetype the growth process is usually shown as a virtuous reinforcing loop. The limiting process is usually shown as a balancing loop, which reacts to imbalances imposed on it by the growth loop. The balancing loop is also driven to move toward its target – a limit or constraint on the whole system, difficult to see because it is so far removed from the growth process.

By pushing hard to overcome the constraints, we make the effects of those constraints even worse than they otherwise would be. Typically, there has been an acceleration of growth and performance, usually the result of hard work, but the growth mysteriously leveled off. A natural reaction is to increase efforts that worked so well before. However, the harder you push, the harder the system seems to push back. Some source of resistance prevents further improvements. Instead of the expected growth, performance remains in equilibrium or completely crashes.

The limiting force may be within the organization, within ourselves or it might be external (e.g. a saturated market).

Example: Quality improvements within an organization often start with the quick wins. This may lead to significant gains in the quality of services or processes. But as the easy changes (known as the low hanging fruit) are completed, the level of improvement plateaus. The next wave of improvements are more complex and tougher to make. The lack of organization-wide support may become a limiting factor.

 

Figure X: System dynamics model for “limits to growth” - example

 

 

5.2.3.3     The “Shifting the burden” archetype

A ‘shifting the burden’ situation (like a ‘fixes that backfire’ situation) usually begins with a problem symptom that prompts someone to solve it. The solution(s) relieve(s) the problem symptom quickly. However the solutions divert the attention away from the fundamental source of the problem.

The ‘shifting the burden’ model has two balancing loops, each representing a different type of fix for the problem symptom:

• The upper loop is a symptomatic quick fix
• The bottom loop represents measures which take longer (note the delay) and are often more difficult, but ultimately address the real problem.

In many ‘shifting the burden’ situations there are additional reinforcing loops. Like the “unintended consequences” loop in ‘fixes that backfire’, these loops represent unintended consequences that make the problem worse.

Example: Many cases of child abuse remain undetected (= problem symptom). An attempt to fix this underdetection could be to increase detection skills of general practitioners and pediatricians. However, if physicians detect more cases of child abuse, they often rely on child protection services for support, advice or to report the case. This means more work for the already overburdened protection services. They cannot manage the overwhelming demands of physicians and restrict uptake criteria or respond with ‘you are doing fine’. Physicians get discouraged and feel let down. As trying to handle cases of child abuse is very time and energy consuming, physicians go back to their former management of bruised children. A more fundamental solution would be to invest in the capacity of child protection services. This way physicians could get the support they need in the detection of child abuse and reported cases get the specialized care they need.

 

Figure X: System dynamics model for “shifting the burden” – example of the detection of child abuse

 

5.2.3.4     Links to other archetypes

• https://thesystemsthinker.com/a-pocket-guide-to-using-the-archetypes/
• http://www.systems-thinking.org/arch/arch.htm
• http://blog.iseesystems.com/tag/archetypes/

 

5.2.4 Seven steps to break through organizational gridlock

Adapted from Senge, P. et al. 1994, the fifth discipline field book, p. 165-172.

 

Gridlock results when people behave as if they are independent, each pulling in a different direction.

Step 1: Identify the original problem symptom

Look back over a period of time and identify a class of symptoms that have been recurring.

Step 2: Map all quick fixes

Try to map out all the fixes that have been used to tackle the identified problem. The objective is to identify a set of balancing loops that appear to be keeping the problems under control.

Step 3: Identify undesirable impacts

Actions taken by one group almost always affect others in the organization (e.g. if each team’s solution causes a problem for the other team). Identify a reinforcing process that locks the players into a patterned response.

Step 4: Identify fundamental solutions

Having identified the undesirable effects of your quick fix, you need to find a solution that will more fundamentally address the problem. You will need to look at the situation from everyone’s perspective to achieve a fundamental solution.

Step 5: Map additive side effects of quick fixes

There are usually side effects of the quick fixes that steadily undermine the viability of the fundamental solution. This leads to a reinforcing spiral of dependency.

Step 6: Find interconnections between to fundamental loops

Finding links between the interaction effects and the fundamental solution. The interaction effects create spiraling resentment, which leads to an increasing unwillingness to communicate with the other team, resulting in an ‘us’ versus ‘them’ mentality.

Step 7: Identify high leverage actions

If you are able to get a bird’s eye-view, you can see the larger grid. The process of mapping out a gridlocked situation can be a high leverage action and be a starting point for communication across walls.

You know you found a high leverage intervention when you can see the long-term pattern of behavior shift qualitatively in a system, for example if stagnation gives way to growth or if oscillations dampen. This kind of breakthrough happens most readily when you can make alterations in the structure you’ve mapped out. You either add new desirable loops or break linkages that produce undesirable impacts.

• Adding a loop: translates into designing and implementing a new process, monitoring information in a new way, or establishing new policies.
• Breaking a link: eliminating or weakening undesirable consequences of your actions or ceasing strategies which are counterproductive in the long run.

When you add loops or break links, it’s critical to try to make such mental models explicit, because the reasons underlying peoples’ actions are fundamental to the system’s structure.

 

5.5.1.1. Categorical outcomes

As we explained before, it is good to discuss the clinical decision threshold and try to determine them on beforehand. However, one must be careful not to apply this in a too mechanical way, as you need also to take into account the downsides of the intervention, such as side effect or morbidity induced by the intervention, on which in a lot of cases you have only information after collecting the evidence. If it is not possible to determine a threshold, then GRADE suggests to use a RRR or RRI greater than 25%. This 25% threshold is a good and pragmatic starting point, but other factors should be taken into account as well, such as the absolute risk reduction.

5.5.1.2. Continuous outcomes

If the 95%CI upper or lower CI crosses the minimally important difference (MID), either for benefit of harm.

Note: if the MID is not known or the use of different outcome measures required calculation of an effect size, we suggest downgrading if the upper or lower CI crosses an effect size of 0.5 in either direction.

Effect size (ES):

A generic term for the estimate of effect of treatment for a study. Sometimes, the term is used to refer to the standardized mean difference, the difference between two estimated means divided by an estimate of the standard deviation.

To facilitate the understanding we suggest an interpretation of the effect size offered by Cohen, also known as ‘Cohen’s d’[1]. According to this interpretation, an effect size or SMD of around:

• 0.2 is considered a small effect;


• 0.5 is considered a moderate effect;


• 0.8 or higher is considered a large effect.

There are also other methods to standardise effect sizes, such as Glass ∆ or hedges d, depending on the choice of the estimator of the standard deviation.

---

[1] (Cohen J. Statistical Power Analysis for the Behavioral Sciences. 2nd ed; 1988).