Because of its voluntary nature, this study found the EFFECT database to be incomplete and somewhat biased, both in terms of the hospitals choosing to participate and the patients being registered by participating centers. More precisely, low-volume and Walloon-region centers were less likely to participate in EFFECT. Furthermore, participating hospitals were less likely to include patients with a less favorable risk profile, with missing data for several clinical-pathological risk factors, that did not undergo curative surgery, and that were not discussed in a multidisciplinary tumor board. Finally, clinical practice patterns were found to be different for participating and non-participating institutions.
The observed participation bias could potentially be explained by the following two mechanisms. First, despite our efforts to inform all hospitals about EFFECT, low-volume and Walloon-region centers might have been informed to a lesser extent. Second, particularly low-volume centers might not have disposed of the resources necessary to participate (e.g., time, funding, personnel and technical support). Furthermore, the observed registration bias could potentially be explained by the following three mechanisms. First, in some to many of the participating institutions, EFFECT registration might have been performed by the healthcare team itself, which might have preferred to particularly include patients that they curatively treated. Second, as many aspects of the patient’s treatment scheme were known at the time of first registration, this information might have biased one’s decision whether to include the patient. For instance, when standard of care was offered but refused, one could have decided not to include the patient. Third, EFFECT registration might have been more time-consuming and labor intensive for certain cases. At this point, these mechanisms are merely theoretical and therefore require further investigation.
PROCARE is a quality of care initiative that was performed in Belgium in the context of rectal cancer and was also relying on hospitals to voluntarily register healthcare data [16]. A study by Jegou et al. found the PROCARE database to be incomplete and biased in a highly similar way as EFFECT. More precisely, they also found that low-volume, Walloon-region and non-university centers were less likely to participate. Furthermore, participating centers were less likely to include patients with a less favorable risk profile and who did not undergo surgical resection. This way, the PROCARE database was found to cover 37% of all Belgian rectal cancer patients. These were registered by 72% of centers involved, which included 56% of their cases [17]. Furthermore, a similar underreporting of hospitals and cases has also been described by other clinical audit programs relying on voluntary participation [24,25,26,27].
In line with the facilitators and barriers of clinical audit as previously described [28, 29], two survey-based studies by Cornish et al. and Voeten et al. recently found that hospitals and healthcare providers generally think clinical audit programs to be a powerful and relevant tool for improving clinical practice and patient outcomes. However, lack of resources (e.g., technical support, time, personnel and funding) was found to be one of the major reasons for non-participation [30, 31]. Our results reflect these findings, as most hospitals and healthcare teams had a positive attitude towards EFFECT. However, many might not have disposed of the resources necessary to participate, particularly low-volume centers.
Conflicting results have been reported by studies comparing the performance of hospitals and healthcare providers that do participate voluntarily in clinical audit with the performance of those that do not [24, 26, 32, 33]. Similarly, although differences were found in the clinical practice of centers participating and not participating in EFFECT, whether this reflects real differences in quality of care warrants further investigation.
Altogether, for the purpose of measuring and improving the quality of cancer care, these findings highlight the feasibility of voluntarily collecting detailed information on the real-world clinical care offered to the patient, from diagnosis to follow-up. Compared to the use of routinely available administrative data, the major advantage of this approach is that it enables a more detailed and meaningful assessment of clinical practice [15]. Nevertheless, in contrast to administrative databases that are highly complete and free of bias, the major disadvantage of this approach is that such clinical databases are at risk of being incomplete and biased, both in terms of the hospitals choosing to participate and the patients being registered by the participating institutions. As a result, hospitals that would arguably benefit most from quality improvement (i.e., low-volume hospitals) tend not to participate [34,35,36,37]. Furthermore, assessing the clinical practice of participating hospitals may be complicated substantially by the bias that tends to be present in their registration of patients. Consequently, to enable meaningful interpretation and feedback, this bias should always be characterized and taken into account. Furthermore, for clinical audit programs to promote quality improvement on the national level, measures should be taken to prevent such selection bias as much as possible, as this requires coverage of all hospitals and patients involved.
Based on the aforementioned mechanisms that could be driving the observed selection bias, we present a couple of methods to potentially reduce the risk of bias in the registration of data, as this would further enhance the potential of clinical audit programs to promote quality improvement. We first suggest to make participation in clinical audit less resource intensive, so that centers and healthcare providers with less resources may also be able to participate. This could potentially be done by making the data extraction and registration process more automated or by giving technical and/or financial support to participating institutions [28,29,30,31]. Second, we suggest to ensure that all centers and healthcare teams involved are sufficiently informed about the project. This could possibly be achieved by presenting in person the rationale and importance of the project, which should preferably be done by a colleague renowned in the field [14]. Third, we suggest patient registration to be performed by someone independent from the healthcare team, preferably a data manager specifically trained in cancer registration. Fourth, we suggest the patient to be registered at time of diagnosis, not when many aspects of the treatment scheme are already known. Finally, we suggest rewarding institutions and healthcare teams for their active participation in clinical audit, on the condition that their participation is of sufficient quality (i.e., when a high enough proportion of patients are registered without selection bias). This could potentially be achieved by some sort of accreditation. However, these suggestions are merely theoretical and therefore require further investigation.
The work presented has a couple of limitations that are mainly associated with the databases used. First, although the BCR database has an excellent coverage of nearly all incident cancer cases in Belgium, its data on WHO score, combined stage and differentiation grade was missing for a substantial number of patients. Second, although IMA data was pivotal for this study, it had some major limitations: (a) miscoding or misuse of nomenclature might have occurred; (b) nomenclature was often vague and unspecific, which made detailed analyses and interpretation of data difficult; and (c) the number of patients that underwent a certain medical procedure may have been under- or overestimated due to the impossibility to unambiguously link nomenclature to one specific indication, or to the fact that not all procedures are reimbursed (e.g., when performed in the context of a clinical trial). Different measures were taken to tackle these limitations. For example, cases with missing data were included in the analyses as a separate category within the respective variable, and patients with less reliable IMA data were excluded.
At the same time, these national population-based databases are the major strength of our study: as they are highly complete covering all corpus uteri cancer cases and institutions involved, they allowed us to accurately assess the completeness and potential selection bias of the EFFECT database.
Future studies should focus on unraveling the underlying mechanisms that are driving the selection bias observed in clinical audit programs, as well as on effective ways to counteract these mechanisms. This knowledge could then be applied to further enhance the potential of clinical audit programs to promote quality improvement in healthcare on the national level.
