Skip to main content

Developing a real-time electronic symptom monitoring system for patients after discharge following cancer-related surgery



Patients undergoing major cancer surgery frequently require post-acute care for complications and adverse effects. Enhanced recovery after surgery programmes mean that patients are increasingly discharged home earlier. Symptom/complication detection post-discharge is sub-optimal. Systematic patient monitoring post-discharge following surgery may be optimally achieved through routine electronic patient-reported outcome (ePRO) data capture. ePRO systems that employ clinical algorithms to guide management of patients and automatically alert clinicians of clinically-concerning symptoms can improve patient outcomes and decrease hospital admissions. ePRO systems that provide individually-tailored self-management advice and integrate live ePRO data into electronic health records (EHR) may also advance personalised health and patient-centred care. This study aims to develop a hospital EHR-integrated ePRO system to improve detection and management of complications post-discharge following cancer-related surgery.


The ePRO system was developed in two phases: (1) Development of a web-based ePRO symptom-report from validated European Organisation for Research and Treatment of Cancer (EORTC) questionnaires, clinical opinion and patient interviews, followed by hospital EHR integration; (2) Development of clinical algorithms triggering symptom severity-dependent patient advice and clinician alerts from: (i) prospectively-collected patient-completed ePRO symptom-report data; (ii) stakeholder meetings; (iii) patient interviews. Patient advice was developed from: (i) clinician-patient telephone consultations and patient interviews; (ii) review of hospital patient information leaflets (PIL) and patient support websites.


Phase 1, including interviews with 18 patients, identified 35 symptom-report items. In phase 2, 130/300 (43%) screened patients were eligible. 61 (47%) consented to participate and 59 (97%) provided 444 complete self-reports. Stakeholder meetings (9 clinicians, 1 patient/public representative) and patient interviews (n = 66) refined advice/alert accuracy. 15 telephone consultations, 7 patient interviews and review of 28 PILs and 3 patient support websites identified 4 themes to inform self-management advice. Comparisons between ePRO symptom-report data, telephone consultations and clinical events/outcomes (n = 27 patients) further refined clinical algorithms.


A hospital EHR-integrated ePRO system that alerts clinicians and provides patient self-management advice has been developed to improve the detection and management of problems and complications after discharge following surgery. An ongoing pilot study will inform a multicentre randomised trial to evaluate the effectiveness of the ePRO system compared to usual care.

Peer Review reports


Surgery may cure or alleviate cancer but is associated with significant complications and adverse effects (AEs). Up to 30% of patients undergoing major abdominal surgery for oesophageal cancer experience acute or long-term complications, such as infection or the need for further intervention [1], and as many as 45% of patients who have undergone major cancer surgery require post-acute care, such as hospital readmission or community care [2,3,4,5]. With more widespread implementation of enhanced recovery after surgery (ERAS) programmes, patients are discharged home increasingly earlier. Detection of complications once at home, however, requires patients to distinguish between symptoms that are typical of recovery (such as expected levels of pain in the first few days) and those that are clinically concerning (such as a change in or worsening of pain). Uncertainty about the significance of symptoms and concerns about self-monitoring and self-care can be worrying and burdensome for patients, their family and caregivers [6], and can delay patients receiving treatment [7,8,9]. Late detection and treatment of complications can lead to poor patient outcomes [10], impaired quality of life and increased emergency department admissions, with up to 25% of patients being readmitted to hospital within 30 days of major cancer surgery [11, 12].

Prompt identification of complications and AEs after patients are discharged from hospital following surgery for cancer is critical to improve patient safety, outcomes and experiences and enable healthcare professionals (HCP) to plan appropriate care [13,14,15,16,17]. Telephone calls from healthcare providers have been shown to effectively reduce symptoms in cancer patients [18] but allocating HCP time to make patient calls can be cost prohibitive [19]. Instead, systematic monitoring may be optimally achieved through the routine capture of electronic patient-reported outcome data (ePRO), their automatic scoring and the real-time communication of this data to clinicians after hospital discharge. A systematic review of 24 controlled trials concluded that the routine use of PRO measures (PROMs) enhances consultations and may offer improved symptom control and patient satisfaction [20]. However, only two of the 11 electronic systems included were delivered in the patients’ home setting.

Specific features of ePRO systems may provide added benefit. Firstly, the use of clinical algorithms to evaluate patient-reported symptom data and alert healthcare providers when severity reaches a pre-determined threshold can help improve patient outcomes [15, 19]. Basch et al. [15] conducted a large randomised controlled trial (RCT) comparing a web-based Symptom Tracking and Reporting (STAR) ePRO system versus usual care in 766 patients receiving chemotherapy for metastatic solid tumours in the United States, in which severe or worsening symptoms triggered an email alert to a clinical nurse. Findings indicated that the routine and real-time reporting of PROs during cancer surveillance can reduce hospital readmissions and improve patients’ survival and health-related quality of life (HRQL) measured using the EuroQol EQ-5D Index [15]. This study was, however, conducted in a single tertiary referral centre, limiting its generalisability and HRQL was not assessed at follow-up for nearly a third of patients due to attrition resulting from death or discontinuation of treatment. Furthermore, use of a generic PROM measuring only broad aspects of health did not provide insight into the extent to which specific symptoms were improved by symptom reporting.

Patient safety and care may be further enhanced if ePRO are embedded within systems with the functionality to provide patients with individually-tailored advice to support at-home symptom self-management [21]. In a multicentre placebo-controlled RCT of 173 patients conducted in the Netherlands, van der Meij et al. showed how an eHealth intervention that included personalised advice promoted a faster return to normal activities compared to usual care for patients undergoing surgery for benign conditions [22]. Provision of tailored advice is, however, not a common feature of ePRO systems, with many generating PROM data only for clinician review [20, 23]. Finally, making routinely collected ePRO data available to clinicians through its integration into patients’ electronic health records (EHR) has the potential to advance precision health and patient-centred care [24] by enabling ePRO data to be considered alongside other clinical data to better plan appropriate care, to inform patient-clinician encounters and to improve shared clinical decision-making [25]. Integration of ePRO data into patients’ EHR is, however, rare, likely because it is challenging, requiring resources, familiarisation and synchronisation with local information technology (IT) systems, adherence to data security regulations and user training [20, 25].

Several of the ePRO systems developed to date are intended for the oncology setting [26,27,28]. Most, however, are targeted to patients undergoing chemotherapy, including the STAR system for patients with breast, genitourinary, gynaecological or lung cancer in the US [26] and the ASyMS system for patients with lung, breast and colorectal cancer in the UK [27]. Comparably, few ePRO systems have been developed specifically for patients undergoing surgery for cancer. SIS.NET, for example, is a web-based system developed in the US to improve follow-up care in patients undergoing surgery, radiotherapy, chemotherapy or experimental therapies for breast cancer [28]. Patients randomised to SIS.NET completed regular online health questionnaires with remote review by a nurse practitioner and attended 3 clinic appointments over 18 months. Depending on patient responses, the questionnaire generated automated referrals to support resources at a local cancer centre. Findings from 102 patients showed that the system improved the quality and efficiency of follow-up care, with SIS.NET patients reporting more new or changed symptoms than patients receiving standard care. The symptom questionnaire reports were not, however, integrated within routine hospital EHR. The exact number of patients who received surgery is, also, unknown. To the best of our knowledge, an ePRO system for patients undergoing surgery for cancer with the functionality to integrate with hospital EHR and apply clinical algorithms to alert clinicians and provide individually-tailored self-management advice to guide patient management is lacking [19, 20, 29]. Here, we describe the development of a hospital EHR-integrated ePRO system to improve the detection of complications and AEs after discharge following cancer-related surgery, illustrated in the first instance in the example context of cancer-related major abdominal surgery.


ePRO platform

The platform used to develop the ePRO system was adapted from the model developed for the eRAPID (Electronic patient self-Reporting of Adverse-events: Patient Information and aDvice) study [30]. The eRAPID IT platform developed at the University of Leeds as an example of a ‘hybrid’ system involving a separate PRO facility linking with an existing EHR. The IT elements, developed in a previous eRAPID study [30] include a patient website (with secure login function), web-based symptom-report questionnaire software (QTool) and a web application interface for secure transfer of the patient data to EHR and viewing of the symptom-report by clinicians. Algorithms can be programmed into the self-report scoring (in QTool), allowing severity-specific tailored self-management advice to be provided to patients and email notifications sent to allocated clinicians. The system was initially developed for patients undergoing systemic therapy and radiotherapy [30,31,32].

The surgical ePRO system was developed between February 2014 and August 2017 in two phases (Figs. 1 and 2), including: (1) Development of the symptom-report for the ePRO system along with necessary IT integration into hospital EHR, and; (2) Development of clinical algorithms to trigger symptom severity-dependent patient advice and clinician alerts based on ePRO self-report responses. Input from key stakeholders, including patients and clinicians, was sought in both phases, as described below.

Fig. 1
figure 1

Phase 1: Symptom-report development and ePRO system integration into hospital electronic health records

Fig. 2
figure 2

Phase 2: Development and testing of clinical algorithms to guide patient management by symptom severity

Phase 1: symptom-report development and ePRO system integration into hospital EHR (Fig. 1)

Step 1: selection of relevant ePRO symptom-report items

Items for inclusion in the ePRO symptom-report questionnaire were identified from a scoping review of existing established, validated European Organisation for Research and Treatment of Cancer (EORTC) questionnaires. EORTC items were selected for their relevance to symptoms and complications experienced by patients after cancer-related major abdominal surgery, including oesophageal, gastric and hepato-pancreato biliary (upper gastrointestinal or UGI) cancer. In this context, UGI is used as a fully inclusive term, to include hepato-pancreato biliary cancer. EORTC items were also selected based on the familiarity of the measures to the clinical and study teams and their established routine use in international trials of PROMs to improve patient outcomes and care [20]. Cognitive interviews with patients with experience of undergoing major UGI surgery were performed to evaluate the acceptability and suitability of the scope of selected items. Between November 2013 and May 2014, inpatient and outpatient lists at University Hospitals Bristol NHS Foundation Trust (UHBT) were screened by a research nurse to identify potentially eligible participants aged 18 or over who had undergone cancer-related major abdominal surgery, including surgery for oesophageal, gastric or hepato-pancreato biliary cancer. Eligible patients were purposively sampled to ensure the inclusion of a wide range of patients. The researcher explained the study and provided the patient with a participant information sheet. The patient was given time to read through the information and ask questions before confirming whether they wished to participate in the study and providing written informed consent. For those patients who expressed an interest in participating, the researcher scheduled a convenient time to conduct the interview either by telephone or face-to-face at the hospital or in the patient’s home. Cognitive interviews were conducted by a researcher to determine participant’s comprehension of the self-report items. Verbal probes were used to investigate participants’ thought processes while responding to items, and participants were encouraged to indicate any trouble understanding or answering items. This work was conducted in iterative cycles of sampling and data analyses to refine the selection of items.

Step 2: patient care pathway analysis

Analysis of the care pathway for patients undergoing major UGI surgery at a single UK hospital trust (University Hospitals Bristol NHS Foundation Trust, UHBT) was undertaken by study researchers (EOCF, KA) and the cancer nurse specialist team (JB, TR) to determine relevant clinician points of contact and appropriate timepoints for ePRO self-report completion by patients during their recovery post-discharge following surgery.

Step 3: integration of the ePRO system into hospital electronic health records

To facilitate the integration of the ePRO system into the existing hospital EHR system at UHBT (Medway, SystemC), several meetings were held between the study team, the University of Leeds IT team and the UHBT IT team between March 2015 and January 2016. Analysis of IT system performance and integration (e.g. downtime) was undertaken to monitor any issues related to integration.

Phase 2: development and testing of clinical algorithms to guide patient management by symptom severity (Fig. 2)

Phase 2 of the study was conducted in three steps.

Step 1: development of clinical algorithms

Informed by previous research undertaken to develop the eRAPID systems in Leeds and Manchester [32]; [31], it was decided a priori that clinical algorithms would stratify patients into three levels of symptom severity based on their completed ePRO symptom-reports, each triggering a different ‘level’ of action within the ePRO system (Table 1). In addition, so that patients reporting multiple symptoms were not over-burdened with self-management advice, advice was provided for a maximum of the six most clinically concerning symptoms per ePRO self-report completion. To achieve this, four clinicians responsible for the clinical care of patients undergoing major UGI surgery ranked symptoms measured by the ePRO self-report in order of clinical importance. A similar approach has been used in the eRAPID study [30].

Table 1 Guided management of patients by symptom severity and ePRO system actions

Clinical algorithms to guide severity-specific tailored patient self-management advice based on patient-reported symptoms were developed using four data sources (Fig. 2). Between 15/04/2016 and 15/08/2017, all consecutive patients being discharged from UHBT following cancer-related major abdominal surgery were screened for study eligibility by a cancer nurse specialist at the point of discharge (e.g. immediately preceding or on the day of discharge). Patients were considered eligible to enter the study if they met all the following inclusion criteria: (i) undergone cancer-related major abdominal surgery (including surgery for oesophageal, gastric or hepato-pancreato biliary cancer); (ii) ready for hospital discharge to their home; (iii) have access to personal computer/tablet and internet from home; (iii) sufficient capacity and understanding of English; (iv) aged 18 or over; (v) able to adhere to the study follow-up schedule. Eligible patients were provided with a participant information leaflet explaining the purpose and details of the study when approaching the point of discharge (e.g. on the inpatient ward prior to discharge) and given the opportunity to ask questions by the research nurse. Participants indicating that they wished to participate were asked by the nurse to provide written informed consent and baseline demographic and clinical details were recorded. On the day of discharge, participants were given unique login details and information about accessing the ePRO system via the website on a study information card. A demonstration of the ePRO system was also provided by the research nurse.

Following discharge, participants were requested to prospectively complete the ePRO self-report at discharge (baseline), twice in the first week (day 2 and day 7 approximately) and weekly thereafter for 8 weeks. One email and/or text message reminder with a direct link to the ePRO website was sent to participants on the days they were due to complete the ePRO self-report. However, patients could choose to log in and complete the self-report at any time during this period that they felt unwell or wanted advice. Participants were also asked to participate in a weekly telephone interview with a study researcher/research nurse, in which they were asked to reflect on the nature and severity of any symptoms they had experienced during the previous week. In addition, two stakeholder meetings with a patient and public representative and clinicians responsible for the clinical care of patients undergoing major UGI surgery were conducted. Finally, end of study interviews with a subset of approximately 10% of study participants were conducted approximately 10 weeks post-discharge. Purposive sampling was used to obtain a sample of participants with a diverse range of post-operative experiences and who had used the ePRO system to varying extents. Targeted verbatim transcription of interview audio-recordings was undertaken [33], whereby only those interviews containing data relevant to the research questions were transcribed for analysis (i.e. if a participant’s completion of the ePRO self-report had generated advice or feedback).

Data from the completed patient-completed ePRO self-reports, weekly patient interviews, stakeholder meetings and end of study patient interviews were considered together. Data were prospectively analysed in an iterative, cyclical manner as data collection and analyses proceeded to develop and refine the symptom severity thresholds that would subsequently inform the development of the clinical algorithms to trigger ePRO system actions. The purpose of this was to identify and agree upon symptoms and symptom severities associated with a ‘typical’ recovery following surgery and those that would be considered clinically concerning and to identify symptoms (and associated timepoints) for which advice or reassurance would have been beneficial. These data were used to further refine the selection of items included in the developmental version of the ePRO self-report questionnaire (described above).

Step 2: development of patient self-management advice

Patient self-management advice was developed using three data sources. Data were collated from, firstly, audio-recordings of end-of-study interviews (described above) and, secondly, telephone-based clinical consultations undertaken in the first week post-discharge (typically on or around days 2 and 7 post-discharge) between a UHBT cancer nurse specialist (CNS) and participants as part of usual care. These consultations were audio-recorded to explore any advice and/or reassurance offered by the CNS to patients. Targeted verbatim transcription of audio-recordings was undertaken, and data analysed to identify themes relating to post-operative advice and reassurance and consider appropriate phrasing and terminology to inform the development of the ePRO system self-management advice. Between September 2016 and January 2017, a scoping search of online patient information/advice and literature provided by or recommended by the UK National Health Service (NHS) Trusts in England (e.g. on NHS Trust websites or NHS Trust recommended websites such as Macmillan Cancer Support) was undertaken. Search terms specific to patient information and advice (e.g. ‘patient information leaflet’) and relevant surgical procedures (e.g. ‘oesophagectomy’) were used. Information/literature was excluded if it did not relate to relevant procedures or was not from an NHS-approved source. Relevant themes and terminology were extracted from included information/literature using a content analysis approach and used, alongside data from the telephone consultations (described above), to draft the ePRO system patient self-management advice, in accordance with the three pre-determined symptom severity levels described above (Table 1). The draft advice was iteratively refined through discussion within the study group and with clinicians involved in the stakeholder meetings until discussions confirmed that no further iterations were required.

Step 3: testing and refinement of clinical algorithms

Data previously generated from a subset of participants’ earlier in Phase 2 of the study (described above) who reported clinically-significant symptoms were reviewed and used to test and refine the clinical algorithms. Participants included in the subset were those who had reported symptoms of a severity that would trigger actions by the ePRO system (i.e. a Level 1, 2 or 3 action). Actions triggered by the ePRO system and the content of any patient self-management advice provided as part of those actions were compared, firstly, with the patient advice provided by the CNS or study research nurse during the routine telephone consultations and weekly telephone interviews and, secondly, with any subsequent clinical events or outcomes of participants (e.g. such as re-intervention, re-admission to hospital, visit to GP or primary healthcare providers). The latter were identified from hospital readmission alerts, hospital EHR, and patients’ reports of accessing healthcare services reported during weekly follow-up telephone interviews. Any discrepancies between actions triggered by the ePRO system and advice provided by the nurses were discussed within the study team. Where considered necessary, the clinical algorithms were refined and tested further in an iterative, cyclical manner until no further refinements were required.


Sociodemographic and clinical details of participants taking part in Phase 1 and 2 of the study are shown in Table 2.

Table 2 Baseline demographic and clinical characteristics of participants

Phase 1: symptom report development and ePRO system integration into hospital EHR

Step 1: selection of relevant self-report items

Ninety-five items from 7 validated EORTC questionnaires were identified. Cognitive interviews with patients (n = 18, 16 men, mean age 66 – Table 2) refined the long list of items and confirmed the acceptability and feasibility of items. The final short list of 30 items to be included in the developmental version of the ePRO symptom-report questionnaire were taken from the following cancer-specific health-related quality of life EORTC questionnaires: C30 (core cancer quality of life module), OG25 (oesophago-gastric cancer module), OES18 (oesophageal cancer module), LMC21 (colorectal liver metastases module), and HCC18 (hepatocellular carcinoma module). Consideration by clinical study team members (JB, TR) resulted in 5 additional items not already included in EORTC questionnaires being added (e.g. wound problems), resulting in an ePRO symptom-report comprising 35 items (Table 3).

Table 3 Final ePRO self-report items

Step 2: patient care pathway analysis

Analysis of patient care pathways identified that the most appropriate time points for ePRO completion were twice in the first week then weekly for 8 weeks post-discharge. The most appropriate clinical contact for patients experiencing potential AEs was identified as the cancer nurse specialist team.

Step 3: integration of the ePRO system into hospital electronic health records

Multiple meetings were held with the study and IT team at UHBT between February 2014 and January 2016 to integrate the ePRO system into the UHBT hospital EHR system (Medway). The integrated ePRO system was launched in April 2016. Between April 2016 and August 2017 there were four instances of downtime/integration loss totalling 145 days. This was due to an EHR software update resulting in a loss of EHR integration and clinicians being unable to access ePRO results through the EHR. A temporary workaround solution was developed to allow researchers and clinicians to access ePRO results independently of EHR during periods of downtime. This workaround required clinicians to access the ‘administrator view’ of the live ePRO system hosted on a secure server until an additional software update was issued to resolve the issue.

Phase 2: development and testing of clinical algorithms to guide patient management by symptom severity

Step 1: development of clinical algorithms

Of 300 patients screened, 130 (43%) patients were considered eligible and invited to participate, and 61 (47%) patients consented (see Fig. 3). Of these, 59 (97%) participants (34 men, mean age 61 years) accessed the ePRO system a total of 459 times, resulting in 444 completed self-reports. Two (3%) participants (1 man, mean age 72 years) did not complete the self-report at least once and were excluded from subsequent analyses. Analysis of weekly follow-up interviews with these 59 participants and additional end-of-study interviews with 7 participants (4 men, mean age 58 years, described above) indicated a need for additional sub-items and branching logic to make items more relevant to patients’ symptoms and recovery. Specifically, 9 sub-items were added to allow the ePRO self-report to distinguish between ‘typical’ symptoms (e.g. shortness of breath after physical activity) and symptoms indicative of potential AEs (e.g. shortness of breath at rest). Weekly telephone interviews (n = 59 participants) also indicated that, for some items, patients tended to report symptoms that were already being appropriately managed or had resolved. To overcome this retrospective reporting of managed or resolved symptoms, an additional sub-item was added to 8 items. This sub-item asked patients if the reported symptom was a current issue, and the response incorporated into the clinical algorithm to trigger the appropriate level of action by the ePRO system in order to guide patient management appropriately.

Fig. 3
figure 3

Phase 2 recruitment flow diagram. a including: n = 31 patients not undergoing planned procedure; n = 39 patients with no definitive cancer diagnosis (prior to ethics amendment to enable inclusion of patients with no definitive diagnosis); n = 5 patients missed due to administration errors; n = 1 patient who was under 18. b including: n = 35 patients not having home access to a PC/internet; n = 10 patients discharged home unexpectedly early or not discharged to home; n = 3 patients not fluent in English; n = 2 patients unable to comply with follow up

Two stakeholder meetings (1 patient representative, 6 nurses, 2 dieticians and 1 surgeon from 4 hospital trusts) indicated that, to account for natural improvements in symptoms during recovery, length of time since hospital discharge should be considered alongside symptom severity in the development of the clinical algorithms. For example, a high level of pain is expected at discharge but would be concerning and potentially indicative of an AE if experienced 6 weeks later. The clinical algorithms were refined to incorporate time since hospital discharge accordingly so that threshold scores for relevant symptoms were altered after 3 weeks post discharge to reflect the changes in symptoms experienced during typical recovery. Items included in the final ePRO self-report questionnaire are detailed in Table 3. Details of all changes made to the ePRO self-report during its development are summarised in Table 4.

Table 4 Summary of refinements made to ePRO symptom-report and clinical algorithms during ePRO system development

Step 2: development of patient self-management advice

Data from 15 routine care telephone consultations conducted by a CNS with 8 participants (5 men, mean age 62 years) in the first week post-discharge were analysed. Four main themes relating to post-operative advice and reassurance sought by participants and/or provided by the CNS were identified, including pain, other physical symptoms, diet and nutrition and managing recovery (Table 5). Findings from the end of study patient interviews (n = 7, 4 men, mean age 58 years) confirmed the acceptability and relevance of self-management advice and reassurance themes. These themes then formed the basis for developing the content of the self-management advice.

Table 5 Summary of advice and reassurance themes and subthemes identified from telephone-based clinical consultations during recovery

A total of 28 PILs from 16 NHS Trusts and 3 cancer support charities (Macmillan Cancer Support, Oesophageal Patients Association, Oxfordshire Oesophageal and Stomach Organisation) were sourced. Using methods of content analysis, these data were combined to produce first drafts of patient self-management advice. These first drafts were reviewed and revised according to input from stakeholder meetings (6 nurses, 2 dieticians, 1 surgeon, 1 patient representative), resulting in the development of self-management advice for 22 symptoms. It was recognised through these iterative processes that some symptom advice differed depending on type of surgical procedure; this was made clear in the advice content to ensure patients followed the advice relevant to their procedure.

Step 3: testing and refinement of clinical algorithms

Data previously generated from 27 participants’ (18 men, mean age 63 years) earlier in Phase 2 of the study (described above) who reported clinically-significant symptoms were reviewed and used to test and refine the clinical algorithms. Comparisons were made between sets of data (ePRO system actions and advice, clinician telephone advice and clinical events/outcomes) from these 27 patients, and any discrepancies identified and discussed with the study team. Over an 8-month period, refinements to the ePRO self-report questionnaire and clinical algorithms were made (Table 3), including adjustments to the symptom severity thresholds triggering ePRO system actions. Examples of final algorithm thresholds for Level 1, 2 and 3 actions triggered by the ePRO system are shown in Table 6.

Table 6 Examples of final algorithm thresholds for Level 1, Level 2 and Level 3 actions


An electronic system for the routine capture of PRO data has been developed to improve the detection and management of complications after hospital discharge following surgery. Unique features of the ePRO system include its full integration into hospital EHR and the function to apply clinical algorithms based on patient-reported symptom-severity to guide the management of patients, and automatically and instantly alert clinicians of atypical symptoms. The ePRO system also accounts for patient-level characteristics to provide individually-tailored self-management advice necessary to provide the wide range of support required by different patients during their recovery at home [34], with patients able to use the system at any time they feel unwell or want advice. In addition, the system is web-based for easy and convenient access by patients using a home computer or portable devices. Full evaluation is now needed to examine its merits compared to standard care.

Until now, few ePRO systems have been developed specifically to support patients after discharge from hospital following surgery. Of 24 controlled trials identified in a 2014 systematic review examining the effects of including PROMs in routine clinical practice [20], only two were relevant to surgical populations [19, 29]. Only one of these was intended for post-discharge follow-up (19). In this study, Cleeland et al. [19] described the use of an ePRO system using interactive voice response technology (as opposed to a web-based system) for triggering clinician alerts to reduce post-operative symptom severity after lung cancer surgery. It is noted that this system did not provide patient self-management advice, was not integrated within patients’ EHR and lacked patient input in the development of alert thresholds [19]. Similarly, Andikyan et al. [26] described evaluation of the feasibility of a web-based ePRO (STAR) system for assessing patient recovery up to 6 weeks after gynaecologic cancer surgery. The STAR questionnaire comprised items from the patient-adaptation of the National Cancer Institute (NCI) Common Terminology Criteria for Adverse Events (CTCAE) and EORTC core questionnaire (QLQ-C30) but was not integrated within hospital EHR and symptom reports were provided to clinicians only at the time of post-operative visits. Embedding ePRO data capture within EHR may be time-consuming and costly but previous research shows that a lack of integration into standard systems is a key barrier to clinicians’ uptake of electronic health systems [35].

This study has several strengths. Multiple data sources, including analyses of interviews and patient-clinician consultations, have directly informed the selection and refinement of ePRO self-report items, content of patient self-management advice and development of the clinical algorithms. The ePRO system has also been developed in close collaboration with multiple key stakeholder groups such as patients, patient representatives, nurses, dieticians and surgeons. This approach has ensured that the ePRO system actions are relevant and meaningful to both patients and clinicians. In addition, the ePRO system uses established, validated measures to assess patients’ HRQL, with the EORTC modules the most frequently used PROMs in oncology studies.

This study does, however, have some limitations. While patient participants with a broad range of sociodemographic characteristics were included, the study was conducted in a single centre in the example context of cancer-related major abdominal surgery and it is possible that the characteristics of the sample do not typify those of the wider population of patients undergoing surgery. There is also a difference in the characteristics of patients who participated in phase 1 and phase 2 of the study, with more patients undergoing oesophago-gastric surgery in phase 1 compared to more undergoing hepatobiliary surgery in phase 2. Further work to develop and test the system in a wider setting and in a broader sample of patients is necessary to evaluate its application as a useful adjunct to patient post-operative care more generally. Depending on the surgical context, the use of alternative disease-specific patient-reported questionnaires may be considered to adapt the ePRO system self-report and algorithms to other patient groups. Integration of the ePRO system with the hospital EPR was lost for a total of 145 days over the 17-month study period due to an unsupported EHR software update. While it was possible to implement a temporary workaround, this incident highlights the potential impact IT issues can have on the widespread use of ePRO systems within EHR and indicates that successful use in routine clinical care would require full integration into hospital EHR and ongoing maintenance by hospital IT departments.

Surgery is associated with significant complications and AEs, many of which occur after patients are discharged home and require further clinical intervention. Self-management of symptoms in the absence of adequate medical knowledge to distinguish between expected and concerning symptoms can be worrying and burdensome for patients and delay necessary treatment [7,8,9]. An ePRO system with the functionality to apply clinical algorithms to automatically alert clinicians in real-time of atypical symptoms has the potential to improve symptom control, patient safety and outcomes and decrease emergency admissions, with the additional potential of healthcare cost savings. Findings from a randomised evaluation of the STAR ePRO system by Basch et al. showed that the routine and real-time reporting of PROs significantly reduces hospital readmissions and improves patients’ survival and HRQL in cancer patients compared to usual care, though this single centre trial had several methodological weaknesses and limited generalisability [15]. Another RCT concluded that automated symptom monitoring via an interactive voice response system and clinician alerts reduced symptom severity in 100 patients during the month after lung cancer surgery compared to automated monitoring plus usual care alone [19]. Symptoms were assessed using a validated though generic cancer PROM. The study was also small and conducted in a specialist tertiary care centre, limiting its generalisability. It is also unclear how patients were selected for inclusion in the study. Furthermore, neither of these systems provided feedback or advice to patients for symptom self-management or were integrated in routine EHR.


Embedding prospectively-collected ePRO data into routine clinical practice has the potential to bring wider benefits to patients and healthcare systems through standardising practice [13], streamlining and enhancing clinical consultations [20, 25] and optimising personalised and patient-centred care [24]. Data collected may also be valuable to informing NHS policy and the development of future treatments and services. Nevertheless, standardised and routine ePRO data capture in patients discharged from hospital after surgery is lacking. This study describes the development of a real-time electronic symptom monitoring system for patients after discharge following cancer-related surgery. A multicentre prospective pilot study is ongoing to fully evaluate the usability and acceptability of the ePRO system, including data completeness, profiles of patient-reported symptoms and actions triggered, and patients’ and clinicians’ experiences of its usefulness. This pilot work will inform a future RCT that will compare the effectiveness and cost-effectiveness of the ePRO system versus usual care for improving the detection of symptoms, complications and patient outcomes after hospital discharge following major elective surgery.



Adverse effect


Electronic health record(s)


Electronic patient reported outcome


Electronic patient self-Reporting of Adverse-events: Patient Information and aDvice


Enhanced recovery after surgery


Healthcare professional


Health-related quality of life


Patient-reported outcome


Patient-reported outcome measure


Upper gastrointestinal


  1. Audit NO-GC: An audit of the care received by people with Oesophago-gastric Cancer in England and Wales 2017 Annual Report In London: National Oesophago-Gastric Cancer Audit; 2017.

  2. Balentine CJ, Naik AD, Robinson CN, Petersen NJ, Chen GJ, Berger DH, Anaya DA. Association of high-volume hospitals with greater likelihood of discharge to home following colorectal surgery. JAMA Surg. 2014;149(3):244–51.

    Article  Google Scholar 

  3. Li LT, Barden GM, Balentine CJ, Orcutt ST, Naik AD, Artinyan A, Sansgiry S, Albo D, Berger DH, Anaya DA. Postoperative transitional care needs in the elderly: an outcome of recovery associated with worse long-term survival. Ann Surg. 2015;261(4):695–701.

    Article  Google Scholar 

  4. Lightner AM, Glasgow RE, Jordan TH, Krassner AD, Way LW, Mulvihill SJ, Kirkwood KS. Pancreatic resection in the elderly. J Am Coll Surg. 2004;198(5):697–706.

    Article  Google Scholar 

  5. Sacks GD, Lawson EH, Dawes AJ, Gibbons MM, Zingmond DS, Ko CY. Which patients require more care after hospital discharge? An analysis of post-acute care use among elderly patients undergoing elective surgery. J Am Coll Surg. 2015;220(6):1113–1121 e1112.

    Article  Google Scholar 

  6. Sanger PC, Hartzler A, Han SM, Armstrong CA, Stewart MR, Lordon RJ, Lober WB, Evans HL. Patient perspectives on post-discharge surgical site infections: towards a patient-centered mobile health solution. PLoS One. 2014;9(12):e114016.

    Article  Google Scholar 

  7. Gillis C, Gill M, Marlett N, MacKean G, GermAnn K, Gilmour L, Nelson G, Wasylak T, Nguyen S, Araujo E, et al. Patients as partners in enhanced recovery after surgery: a qualitative patient-led study. BMJ Open. 2017;7(6):e017002.

    Article  Google Scholar 

  8. Laugsand EA, Sprangers MA, Bjordal K, Skorpen F, Kaasa S, Klepstad P. Health care providers underestimate symptom intensities of cancer patients: a multicenter European study. Health Qual Life Outcomes. 2010;8:104.

    Article  Google Scholar 

  9. Atkinson TM, Li Y, Coffey CW, Sit L, Shaw M, Lavene D, Bennett AV, Fruscione M, Rogak L, Hay J, et al. Reliability of adverse symptom event reporting by clinicians. Qual Life Res. 2012;21(7):1159–64.

    Article  Google Scholar 

  10. Khuri SF, Henderson WG, DePalma RG, Mosca C, Healey NA, Kumbhani DJ. Participants in the VANSQIP: determinants of long-term survival after major surgery and the adverse effect of postoperative complications. Ann Surg. 2005;242(3):326–41 discussion 341-323.

    PubMed  PubMed Central  Google Scholar 

  11. Gawlas I, Sethi M, Winner M, Epelboym I, Lee JL, Schrope BA, Chabot JA, Allendorf JD. Readmission after pancreatic resection is not an appropriate measure of quality. Ann Surg Oncol. 2013;20(6):1781–7.

    Article  Google Scholar 

  12. Rochefort MM, Tomlinson JS. Unexpected readmissions after major cancer surgery: an evaluation of readmissions as a quality-of-care indicator. Surg Oncol Clin N Am. 2012;21(3):397–405 viii.

    Article  Google Scholar 

  13. Trotti A, Colevas AD, Setser A, Basch E. Patient-reported outcomes and the evolution of adverse event reporting in oncology. J Clin Oncol. 2007;25(32):5121–7.

    Article  Google Scholar 

  14. Velikova G, Booth L, Smith AB, Brown PM, Lynch P, Brown JM, Selby PJ. Measuring quality of life in routine oncology practice improves communication and patient well-being: a randomized controlled trial. J Clin Oncol. 2004;22(4):714–24.

    Article  Google Scholar 

  15. Basch E, Bennett A, Pietanza MC. Use of patient-reported outcomes to improve the predictive accuracy of clinician-reported adverse events. JNCI: Journal of the National Cancer Institute. 2011;103(24):1808–10.

    Article  Google Scholar 

  16. Black N. Patient reported outcome measures could help transform healthcare. BMJ. 2013;346:f167.

    Article  Google Scholar 

  17. Johansen MA, Berntsen GK, Schuster T, Henriksen E, Horsch A. Electronic symptom reporting between patient and provider for improved health care service quality: a systematic review of randomized controlled trials. Part 2: methodological quality and effects. J Med Internet Res. 2012;14(5):e126.

    Article  Google Scholar 

  18. Kornblith AB, Dowell JM, Herndon JE 2nd, Engelman BJ, Bauer-Wu S, Small EJ, Morrison VA, Atkins J, Cohen HJ, Holland JC. Telephone monitoring of distress in patients aged 65 years or older with advanced stage cancer: a cancer and leukemia group B study. Cancer. 2006;107(11):2706–14.

    Article  Google Scholar 

  19. Cleeland CS, Wang XS, Shi Q, Mendoza TR, Wright SL, Berry MD, Malveaux D, Shah PK, Gning I, Hofstetter WL, et al. Automated symptom alerts reduce postoperative symptom severity after cancer surgery: a randomized controlled clinical trial. J Clin Oncol. 2011;29(8):994–1000.

    Article  Google Scholar 

  20. Kotronoulas G, Kearney N, Maguire R, Harrow A, Di Domenico D, Croy S, MacGillivray S. What is the value of the routine use of patient-reported outcome measures toward improvement of patient outcomes, processes of care, and health service outcomes in cancer care? A systematic review of controlled trials. J Clin Oncol. 2014;32(14):1480–501.

    Article  Google Scholar 

  21. Kreuter MW, Wray RJ. Tailored and targeted health communication: strategies for enhancing information relevance. Am J Health Behav. 2003;27(Suppl 3):S227–32.

    Article  Google Scholar 

  22. van der Meij E, Anema JR, Leclercq WKG, Bongers MY, Consten ECJ, Koops SES, van de Ven PM, Terwee CB, van Dongen JM, Schaafsma FG, et al. Personalised perioperative care by e-health after intermediate-grade abdominal surgery: a multicentre, single-blind, randomised, placebo-controlled trial. Lancet. 2018;392(10141):51–9.

    Article  Google Scholar 

  23. Warrington L, Absolom K, Conner M, Kellar I, Clayton B, Ayres M, Velikova G. Electronic Systems for Patients to Report and Manage Side Effects of Cancer Treatment: Systematic Review. J Med Internet Res. 2019;21(1):e10875.

    Article  Google Scholar 

  24. Glasgow RE, Kwan BM, Matlock DD. Realizing the full potential of precision health: the need to include patient-reported health behavior, mental health, social determinants, and patient preferences data. J Clin Transl Sci. 2018;2(3):183–5.

    Article  Google Scholar 

  25. Bennett AV, Jensen RE, Basch E. Electronic patient-reported outcome systems in oncology clinical practice. CA Cancer J Clin. 2012;62(5):337–47.

    Article  Google Scholar 

  26. Andikyan V, Rezk Y, Einstein MH, Gualtiere G, Leitao MM Jr, Sonoda Y, Abu-Rustum NR, Barakat RR, Basch EM, Chi DS. A prospective study of the feasibility and acceptability of a web-based, electronic patient-reported outcome system in assessing patient recovery after major gynecologic cancer surgery. Gynecol Oncol. 2012;127(2):273–7.

    Article  Google Scholar 

  27. Moradian S, Krzyzanowska MK, Maguire R, Morita PP, Kukreti V, Avery J, Liu G, Cafazzo J, Howell D. Usability evaluation of a Mobile phone-based system for remote monitoring and Management of Chemotherapy-Related Side Effects in Cancer patients: mixed-methods study. JMIR Cancer. 2018;4(2):e10932.

    Article  Google Scholar 

  28. Wheelock AE, Bock MA, Martin EL, Hwang J, Ernest ML, Rugo HS, Esserman LJ, Melisko ME. SIS.NET: a randomized controlled trial evaluating a web-based system for symptom management after treatment of breast cancer. Cancer. 2015;121(6):893–9.

    Article  Google Scholar 

  29. Klinkhammer-Schalke M, Koller M, Steinger B, Ehret C, Ernst B, Wyatt JC, Hofstadter F, Lorenz W, Regensburg Qo LSG. Direct improvement of quality of life using a tailored quality of life diagnosis and therapy pathway: randomised trial in 200 women with breast cancer. Br J Cancer. 2012;106(5):826–38.

    Article  CAS  Google Scholar 

  30. Holch P, Warrington L, Bamforth LCA, Keding A, Ziegler LE, Absolom K, Hector C, Harley C, Johnson O, Hall G, et al. Development of an integrated electronic platform for patient self-report and management of adverse events during cancer treatment. Ann Oncol. 2017;28(9):2305–11.

    Article  CAS  Google Scholar 

  31. Absolom K, Holch P, Warrington L, Samy F, Hulme C, Hewison J, Morris C, Bamforth L, Conner M, Brown J, et al. Electronic patient self-reporting of adverse-events: patient information and aDvice (eRAPID): a randomised controlled trial in systemic cancer treatment. BMC Cancer. 2017;17(1):318.

    Article  Google Scholar 

  32. Holch P, Pini S, Henry AM, Davidson S, Routledge J, Brown J, Absolom K, Gilbert A, Franks K, Hulme C, et al. eRAPID electronic patient self-reporting of adverse-events: patient information and aDvice: a pilot study protocol in pelvic radiotherapy. Pilot Feasibility Stud. 2018;4:110.

    Article  Google Scholar 

  33. Tashakkori A, Teddlie C: Sage handbook of mixed methods in social & behavioral research: sage; 2010.

  34. Ruland CM, Maffei RM, Borosund E, Krahn A, Andersen T, Grimsbo GH. Evaluation of different features of an eHealth application for personalized illness management support: cancer patients' use and appraisal of usefulness. Int J Med Inform. 2013;82(7):593–603.

    Article  Google Scholar 

  35. Bowens FM, Frye PA, Jones WA. Health information technology: integration of clinical workflow into meaningful use of electronic health records. Perspect Health Inf Manag. 2010;7:1d.

    PubMed  PubMed Central  Google Scholar 

Download references


The authors would like to thank all the patients who took part in this study, and members of the Surgical Research Nurse team at University Hospitals Bristol NHS Foundation Trust for their assistance in recruiting patients to the study.


This report is independent research funded by the National Institute for Health Research (Programme Grants for Applied Health research, Towards safer delivery and monitoring of cancer treatments. Electronic patient self-Reporting of Adverse -events: Patient Information and aDvice (eRAPID) RP-PG-0611-20008). The views expressed in this publication are those of the author(s) and not necessarily those of the NHS, the National Institute for Health Research or the Department of Health and Social Care. This work was undertaken with the support of the Medical Research Council ConDuCT-II (Collaboration and innovation for Difficult and Complex randomised controlled Trials In Invasive procedures) Hub for Trials Methodology Research (MR/K025643/1) ( and Royal College of Surgeons of England Bristol Surgical Trials Centre. This study was supported by the National Institute for Health Research (NIHR) Biomedical Research Centre (BRC) at the University Hospitals Bristol NHS Foundation Trust and the University of Bristol. The views expressed are those of the authors and not necessarily those of the UK National Health Service, National Institute for Health Research, or Department of Health. JMB holds an NIHR Senior Investigator award.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request. Full interview transcripts are not available to protect participant anonymity.

Author information

Authors and Affiliations



JB and GV conceived the study. KAb, JB, LB and GV designed the study; RC and LB oversaw IT components and integration with hospital systems; TR and EOCF mapped patient usual care pathways; EOCF conducted day-to-day management of phase 1 of the study; HR conducted day-to-day management of phase 2 of the study; KNLA, JB and GV oversaw the study; RH, EOCF, AP, HR and TR collected data; KNLA, EOCF, AP and HR carried out data analyses; KNLA, JB, EOCF, AP and HR interpreted data and drafted the manuscript; all authors critically reviewed and approved the final version of the manuscript.

Corresponding author

Correspondence to Kerry N. L. Avery.

Ethics declarations

Ethics approval and consent to participate

The research protocol for Phase 1 of the study was reviewed and approved by Yorkshire & The Humber Leeds East Research Ethics Committee (REC reference 12/YH/0531). The research protocol for Phase 2 of the study was reviewed and approved by London Surrey Borders Research Ethics Committee (15/LO/2017). All patients provided written informed consent prior to participation, including for audio-recording of interviews and telephone consultations.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Avery, K.N.L., Richards, H.S., Portal, A. et al. Developing a real-time electronic symptom monitoring system for patients after discharge following cancer-related surgery. BMC Cancer 19, 463 (2019).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: