A group of experts was formed, consisting of healthcare professionals, patient organisations, health insurers and researchers. This group was consulted for decisions regarding the creation and adaptation of the questionnaire.
Patients with all types of malignant tumours were included, as long as they received treatment in a Dutch hospital or specialized cancer treatment centre. We chose the following inclusion criteria for both the focus group discussions and the questionnaires. Participants were at least 18 years old and the last treatment occurred no longer than two years before the focus group discussion or conduct of the survey. This study was performed in agreement with the declaration of Helsinki. Approval by a medical ethics committee was not required. All participants in this study gave written informed consent for the use of the data provided by them. Data from focus group discussions and questionnaires were analyzed anonymously.
Sampling
The chairmen of the cancer patient organisations affiliated to the Netherlands Federation of Cancer patient organisations (NFK) invited all their active members, ever diagnosed with cancer, through letters or e-mails to participate in the focus group discussions.
The claims data of a Dutch healthcare insurance company with national coverage, were used to draw a random selection of 1,489 patients, ever diagnosed with cancer, who received cancer care in any hospital in the Netherlands, or in a specialized cancer center in the last two year for the experience questionnaire. Patients were selected using diagnosis-related groups codes (DRG codes). It concerned patients with the following types of cancer: lung, breast, colorectal, prostate, haematological, gynaecological and skin. Basal cell cancer patients were excluded (on their DRG code), since these patients are not always told they have cancer. Receiving a questionnaire concerning cancer would cause unnecessary distress. Patients in all phases of their treatment were included, receiving surgical treatment, chemotherapy, radiotherapy, immune therapy or hormonal treatment, with the exception of patients in a palliative phase, who were excluded through their DRG code. A second random sample of 600 patients using the same criteria was drawn for the importance questionnaire.
Construction of the questionnaire
Three focus group discussions were held with seven, nine and nine cancer patients in November and December of 2009. Each focus group discussion was chaired by a researcher from the Centre for Consumer Experience in Healthcare (Centrum Klantervaring Zorg, CKZ) and Netherlands Institute for Health Services Research (NIVEL) and a policy officer from the NFK. Discussions were audio-taped with permission of the participants. Each participant was asked to write down three positive and three negative experiences with the hospital care surrounding their referral, diagnosis, treatment or aftercare. Subjects were then discussed in the group. Two researchers independently analyzed transcripts from the focus group discussions for the presence of quality aspects using descriptive thematic analysis.
Relevant items mentioned more than once were included in the questionnaire, along with additional important items from the following sources. These included a list of quality aspects for cancer care from the patient’s perspective created by the NFK [22] and a list of general quality criteria from the patient perspective in healthcare constructed by the Netherlands patient consumer federation (NPCF) [23]. We also included quality aspects from three questionnaires. The first was the EORTC-IN-PATSAT32 questionnaire from the European Organisation for Research and Treatment of Cancer (EORTC). This questionnaire is used to assess in-patient satisfaction with cancer care [24]. The second questionnaire was the CQI breast care [10] developed by the NIVEL and CKZ to assess patients’ experiences with breast care. The third was the CQI hospital care [25], developed by the NIVEL and CKZ, that measures the quality of hospital care.
We developed an ‘experience’ questionnaire that measures the experiences of patients and an ‘importance’ questionnaire to assess the importance patients attach to each quality aspect. Both questionnaires contained a number of questions regarding patient characteristics. All other questions in the experience questionnaire had one of the following response categories ‘never-sometimes-usually-always’, ‘no not at all-somewhat-largely-yes completely’, ‘none-some-most-all’, or one through ten for grades. Responses, with the exception of grades, were converted into a scale of one to four, where the highest score was the most positive answer. The importance questionnaire included all experience questions from the experience questionnaire, with the response categories ‘not important-somewhat important-quite important-very important’. These responses were also converted into a scale of one to four, four being very important.
Pre-testing
All patients who took part in the focus group discussions received the experience questionnaire with instructions to comment on the clarity of the questions. They were asked to judge how long it took them to fill out the questionnaire, to give comments on unclear wording, to record if questions were clear and if all important aspects were covered in the questionnaire. The questionnaire was filled out by 20 cancer patients. Unclear or incorrect questions were altered before data collection.
Data collection
We based our data collection on the Dillman method[26]. In April 2010 (week 0) an invitation letter with a link to the online ‘experience’ questionnaire was sent to 1,498 cancer patients. An invitation with a link to the ‘importance’ questionnaire was sent to 600 additional patients. In week 1, a note was sent thanking respondents and reminding non-responders to a link to the questionnaire. In week 4 another reminder was sent to the non-responders along with a paper version of the questionnaire. Finally, in week 6, a note was sent, thanking respondents and reminding non-responders to the questionnaire through a link to the website.
Analyses
Data were analyzed following the CKZ manual on the development of CQI questionnaires [27]. In short, the data entry of ten percent of the paper questionnaires was checked for errors. Up to one percent was allowed. Subsequently, a histogram was made for all questions to identify values that are outside the scope of the answer categories. Double entries (on patient identification number), and non-responders (with and without statement of a reason) were removed from the dataset. Questionnaires that were not filled out by the patient the questionnaire was sent to, were removed. Finally, questionnaires where less than 50 percent of mandatory questions were answered (<41) were removed. All questions were considered mandatory with the exception of questions following a ‘skip-question’, a question where the answer given may redirect the respondent to a certain question, thereby skipping questions that are irrelevant. Representativeness of respondents was checked by comparing our data on age, gender and tumour type with numbers from the Dutch National cancer registration (NKR) [28].
Importance questionnaire
For the importance questions we calculated the percentage of responses in the highest category (very important). We regarded the top 10 questions to be the most important questions in the questionnaire and the bottom 10 questions to be eligible for removal from the questionnaire.
Experience questionnaire
For the experience questionnaire we performed analyses on item-level to identify extremely skewed items (as a rule of thumb: more than 90 percent of responses in the most positive or the most negative answer category) and questions with a high non-response (as a rule of thumb: more than 5 percent). Subsequently, we performed inter-item analyses to identify questions with a large overlap in response. If two questions had a Pearson’s correlation coefficient exceeding 0.70 and overlap concerning content, one of the questions was considered potentially redundant.
Prior to carrying out factor analyses on the experience questionnaire, we removed extremely skewed items and redundant questions. All remaining 73 questions with a 1 to 4 answer-scale were included in the factor analysis. We performed principal component analysis with oblique rotation in order to ascertain the underlying structure of the questionnaire while taking into account a certain minimal amount of overlap between factors due to the fact that people who respond positively to certain questions may also respond positively to other questions. Questions with a factor load of more than 0.3 for a certain scale were included in the scale where they had the highest factor load.
We performed reliability analyses on the formed scales. The reliability of each scale was measured with Cronbach’s α. A scale with an α greater than 0.70 was considered reliable, a scale with an α between 0.60 and 0.70 was considered to be moderately reliable and was not removed from the questionnaire. However, a moderately reliable scale needs to be re-evaluated in future measurements. The correlation of a question to all questions in its scale (item-test correlation (ITC)) was required to be greater than 0.40 for inclusion in the scale. We also registered those questions for which removal of the question from the scale would lead to an increase in the reliability of the scale (Cronbach’s α).
Subsequently, we calculated average scale scores with 95% confidence intervals (95% CIs) for all the scales in the experience questionnaire and compared the averages for the five most frequently reported tumour types using ANOVA.
We calculated improvement scores in order to identify quality aspects where patients had negative experiences while regarding the aspect as important. These improvement scores combine data from the experience questionnaire and the importance questionnaire. Scores are defined by the mean score of a question (importance questionnaire) times the percentage of people with a negative experience (experience questionnaire) times 100 [10, 16].
We discussed all results with the group of experts. The final decision for removal of questions from a scale and from the questionnaire was based on the results from the psychometric analyses and on arguments from the group of experts with respect to the content of the questions.
For international use, the final questionnaire was translated, reviewed by a panel of experts and back-translated according to the WHO guidelines [29].