Item generation and development of a draft questionnaire
A literature review was undertaken to identify relevant issues and terminology as well as items and scales within existing instruments that could be used to measure aspects of cancer care coordination [8–13]. The literature review was used to develop a series of open-ended questions that were used in a qualitative study to explore issues in care coordination specific to oncology. Focus groups and semi-structured interviews with 24 patients and carers and 29 clinicians in metropolitan, regional and rural areas of New South Wales (NSW) were undertaken to investigate stakeholders' views of the most important components of cancer care coordination and to identify potential questionnaire items. Full details of this qualitative study are reported elsewhere  but in brief, eight components of care were identified as being crucial for effective cancer care coordination, namely, organisation of patient care, access to and navigation through the healthcare system, the allocation of a "key contact" person, recognition and understanding of medical team roles, effective communication and cooperation amongst the multidisciplinary team and other health service providers, delivery of services in a complementary and timely manner, needs assessment and sufficient and timely information for the patient.
The results of the literature review and this qualitative work were used to identify existing items and to generate new items that addressed these eight components of cancer care coordination as well as the concepts espoused in the AHRQ definition . To generate new items, the study team developed statements that addressed the concept in question and sought input from clinicians and other researchers about clarity and wording.
Forty items that related to concepts considered important by a broad range of stakeholders in the qualitative phase and that addressed the theoretical components of cancer care coordination were selected for inclusion in a draft questionnaire. Items were worded both in the positive and the negative with bolding of words used to highlight differences between similar statements. To investigate the most reliable format for response options, two formats were tested. Eighteen items were phrased as statements to which respondents were asked to indicate their level of agreement, using a five-point Likert scale ('Strongly agree', 'Agree', 'Neutral', 'Disagree', 'Strongly disagree'). The remaining 22 items asked about patients' experiences of care in the previous three months, again using a five-point Likert scale ('Never', 'Rarely', 'Sometimes', 'Frequently', 'Always'). A time frame of three months was chosen on clinical grounds to provide a sufficient time window for patients to have received multidisciplinary cancer care. The items with the 'agreement' format were included in random order, followed by the items using the 'experience' format, again in random order. The response option headings were repeated at intervals down the page to break up the lines of text and tick boxes so as to improve the ease of completing the questionnaire. In addition, the questionnaire included two global assessment questions in which respondents were asked to rate firstly, the coordination of their care and secondly, the overall quality of the care they had received, on a scale from one ('Very poor') to ten ('Excellent'). The draft questionnaire was reviewed by clinicians and researchers to assess comprehensiveness of items (face validity) and clarity of wording.
The draft questionnaire was then tested in two separate samples of patients.
A purposive sample was recruited from six centres (two in Sydney, four in regional New South Wales (NSW)) to provide patients with a range of cancer types, treatment modalities and geographical location. Eligible patients were in follow up for any cancer that had been treated between three and twelve months previously. This time-frame was considered optimal as patients would have experienced the full range of care-co-ordination through the treatment phase of their illness. Patients were considered ineligible if they had insufficient English skills or were cognitively impaired such that they could not complete the questionnaire or were receiving end of life care.
Patients were asked to read and sign a consent form, complete the questionnaire and return these items to the research team in a reply paid envelope. In addition, patients completed items assessing demographic and clinical information, including age; sex; country of birth; marital, education and occupational status; cancer type, year of diagnosis and treatment modalities. To assess test-retest reliability, on receipt of their completed questionnaire, patients in the first three month period of recruitment were mailed a second, identical copy of the questionnaire to complete two weeks later.
This sample comprised patients with a newly diagnosed colorectal cancer who were participating in an ongoing randomised trial. Patients treated at 22 public and private hospitals in metropolitan and regional centres in NSW were recruited at the time of initial surgical treatment and asked to complete self-administered questionnaires at baseline, one, three and six months. The data for the present study are from the 3-month assessment which included the draft questionnaire about cancer care coordination. Demographic and clinical information was collected at the time of enrolment into the trial.
Characteristics of participants were summarized. For the subsample of Sample 1 who completed the questionnaire twice, test-retest reliability (repeatability) of individual items was assessed by calculating weighted Kappa statistics with 95% confidence intervals (CIs). Items with kappa values of less than 0.40, representing 'fair' or 'poor' agreement, were eliminated from further analyses.
Using the combined dataset (n = 686), frequency distributions for each item were examined. Items with more than 5% missing data and those with limited response distributions (70% or more respondents gave the same response) were eliminated .
Investigation of factor structure
Study data were then randomly split into two equal sized sub-samples. Exploratory factor analysis using squared multiple correlations as prior communality estimates was conducted in each sub-sample separately to assess the consistency of the factor pattern. The principal factor method was used to extract the factors, followed by a promax rotation [17, 18]. The number of meaningful factors was determined on the basis of examination of the scree plot, assessment of the proportion of variance accounted for and interpretability of the factors. Factors that explained at least 5% of the common variance were retained. For interpretation of the rotated factor pattern, an item was said to load onto a particular factor if the factor loading was greater than 0.40 for that factor, and was less than 0.40 for the other factors.
Development of subscales
The factors were used to develop subscales within the questionnaire. First, the scoring for items worded in the negative were reversed, so that a higher score indicated better care coordination for all items. Items that loaded onto each factor were summed to create factor-based scales. To assess whether any individual items reduced the internal consistency of the total score or individual subscales, item-total correlations were calculated. These statistics provide a measure of the correlation between an item and the sum of the remaining items in the scale, with low values (less than 0.2)  indicating that an item is not measuring the same construct as other items. Cronbach's alpha was calculated with each item removed in turn. Where Cronbach's alpha was substantially improved by removal of the item, this item was eliminated from the scale and Cronbach's alpha for the remaining items was recalculated . Values of Cronbach's alpha between 0.7 and 0.9 were considered optimal . Correlations between variables were assessed to determine whether any were highly correlated (r > 0.70) suggesting redundancy.
The distribution of subscale scores and the total score were assessed with descriptive statistics and the proportions of respondents with the highest ('ceiling') and lowest ('floor') scores were calculated. Spearman's rank correlation coefficient was calculated for the total score and each of the subscales firstly with the global cancer care coordination item and secondly with the global quality of care item. All statistical analyses were undertaken using SAS statistical software .
A sample of five times the number of questionnaire items is considered the minimum for factor analysis . As the questionnaire contained 40 items, we needed a minimum of 200 patients in each split sample for this analysis. A minimum sample size of 50 is recommended for assessment of test-retest reliability .
The study was approved by the Sydney South West Area Health Service Ethics Review Committee (RPAH zone).