Written by Trudy Flynn, Patient Partner and Tianna Magel, Project and Research Analyst at CIHR-IMHA
Edited by Hetty Mulhall
A recent review found that one of the key reasons researchers participate in patient engagement is to do “better research” (Pratte et al., 2023). Yet there is still insufficient awareness of patient engagement in academia which means that the research community is still falling short when it comes to meaningfully integrating patient perspectives. Less than half (36.5%) of the health researchers surveyed in a Canadian study agreed that patients and the public are being meaningfully engaged in research (Crockett et al., 2019). Begging the question: How can we increase awareness on the importance of meaningfully engaging patient partners in research?
Patient Engagement in research within Canada is not a new phenomenon, and has been defined by the Canadian Institutes of Health Research (CIHR) as patients being “actively engaged in governance, priority setting, developing the research questions, and even performing parts of the research itself” (Canadian Institutes of Health Research, 2019). The value of patient engagement is also not inconsequential and has demonstrated several benefits, including improved clinical and research relevance, continued engagement in care, and facilitation of knowledge translation initiatives (Domecq et al., 2014; Forsythe et al., 2019; McVey et al., 2023; Richards et al., 2024).
To effectively move the dial on patient engagement best practices, and in turn, do “better research”, increased educational initiatives are needed to raise awareness on the value of involving patients, not as participants (e.g., someone signing up to try a new treatment), but as members of a research team.
To address this issue, in collaboration with patient partners and CIHR institutes, we developed virtual and in-person 101 workshops on patient engagement in research. Therefore, the purpose of this blog is threefold: 1) to share our learnings from the workshop and peer review so that other organizations might use these approaches, 2) to flag the need for further education on patient engagement in research, and 3) to highlight the importance of patient partners as co-creators and co-facilitators of the workshop and peer review materials.
This is a long blog post, so we have broken it up into sections to make it easier to read. We have displayed each section in its own collapsible text box with a heading, to make it easier to keep track of where you are reading.
To gauge researcher interest in the workshop and the value they would extract from engagement in the session, we developed an online form or ‘expression of interest’. This form asked applicants to detail their previous patient partner engagement experiences, how participation in the workshop would strengthen their research skills, and how they intended to amplify their learnings from the workshop.
Following submission, each application went through a peer review process where it was carefully reviewed and ranked by several peer reviewers. Our peer review process was co-designed by patient partners, as with all steps in this project, and was unique in that it was primarily patient partner led and driven. This is contrary to typical peer review panels which are often made up of individuals from more traditional academic roles. Indeed, our peer review team was composed of a diverse panel of six patient partners from across Canada and three members from CIHR’s Institute of Musculoskeletal Health and Arthritis (IMHA) and Institute of Genetics (IG).
For the actual peer review itself, two patient partners and a member of CIHR were randomly assigned to review and rank written responses from each application using a 5-point evaluation rubric. Where possible, our rubric was intended to support an objective and unbiased assessment of applicant written responses. In addition to the rubric, several supplementary materials were developed to support peer reviewers with the evaluation process, including a detailed text explanation of rubric scores as well as a video tutorial, as it can be beneficial to provide information in a variety of formats. Using these materials, patient partner reviewers then scored the written responses of their assigned applicants. Final reviewer scores (i.e., two patient partner scores and one CIHR score) for each applicant were combined to create an average score out of 5. A score of 3.0 was determined to be the cut-off for admittance to the workshop, and applicants who did not receive an average score above this were sent to full panel review.
Following peer review, reviewers attended a panel meeting to discuss low scoring applicants and whether their application merited admission to the patient engagement workshop. This discussion was primarily patient partner led, with members of CIHR adding additional comments when needed. It is important to highlight that the high attendance and engagement observed in the workshop may be attributed to the mandatory expression of interest and peer review process. Arguably, applicants felt they had earned their place at the workshop rather than having been automatically accepted so we may have seen greater commitment levels as a result.
The workshop itself was designed in collaboration with IMHA’s Patient Engagement Specialist, Dr. Dawn Richards. This was a 2-hour interactive session offered twice on two consecutive days in Halifax, Nova Scotia, in both hybrid and in-person only formats. Our hybrid session was offered on the first day as a means of facilitating accessible attendance for those who could not attend in person. Additionally, our in-person attendance was also located in an accessible building and classroom, and attendee masking was required.
Each session of the workshop opened with patient partner introductions and the sharing of personal stories and experiences within a research team, which served as a means to situate attendees to the unique expertise and roles of patient partners in research, and key things they should know about patient engagement. Patient partner stories were then followed by breakout group discussions which were comprised of 4-5 attendees and a patient partner facilitator. With the help of patient partner facilitators, and using basic science and clinical case studies, breakout groups were designed to facilitate the discussion of patient engagement throughout the entire research cycle. Attendees used the small group opportunity with patient partners to ask questions and gain further clarity about patient engagement within research. This was especially helpful for attendees in areas of research in which patient engagement is not highly utilized yet, and included questions about where to find and engage patient partners in their respective areas of research. Indeed, some groups utilized the case studies as a framework for discussion, whereas several others had more of a free-flowing Q and A session, choosing to not stick with the case studies. Both approaches were supported by the workshop facilitators.
Following the case study discussion, one member from each breakout group was asked to briefly summarize what their group had discussed. The workshop then closed with a discussion of several attendee questions as well as the sharing of accessible resources that researchers and trainees could access. As is often the case with hybrid workshops, we experienced technical difficulties with the virtual group session. Although the session leader and co-facilitators were very mindful of including virtual attendees, the supportive technology did not function well within the space, and struggled to detect the direction of the sounds often highlighting people incorrectly on the video call. These learnings will be used in future hybrid iterations of our workshop.
At the end of the workshop, attendees and patient partners were asked to complete a survey to gather feedback on areas of success and improvement for future offerings of the workshop. Attendees expressed a high degree of satisfaction with the format and delivery of the workshop and felt strongly that the interactive patient partner panel was helpful to their learning (see Table 1). All patient partners involved in the peer view and workshop felt they were able to express their views freely, and that the development and delivery of the workshop was a collaborative experience. Overall, both attendees and patient partners expressed a high degree of enthusiasm, with most highlighting a desire for a longer workshop to facilitate a deeper discussion and exploration of patient engagement in research.
| Demographics/ Feedback | Attendees (n=29)a |
| Attendance | |
| In-Person | 18 (62%) |
| Virtual | 11 (38%) |
| Perspective brought to workshop* | |
| Researcher | 14 (48%) |
| Trainee | 14 (48%) |
| Other | 6 (21%) |
| “The interactive patient partner panel was helpful to my learning.” | |
| Strongly Agree | 18 (62%) |
| Agree | 9 (31%) |
| Neutral | 2 (7%) |
| Dissatisfied | - |
| Very Dissatisfied | - |
| Unsure | - |
| Overall satisfaction with the format and delivery of the Patient Engagement 101 workshop | |
| Very Satisfied | 12 (41%) |
| Satisfied | 13 (45%) |
| Neutral | 4 (14%) |
| Dissatisfied | - |
| Very Dissatisfied | - |
| “I learned something new today that will be useful for my future approach in patient partnership or patient engagement.” | |
| Strongly agree | 16 (55%) |
| Agree | 11 (38%) |
| Neutral | 2 (7%) |
| Disagree | -
|
| Strongly Disagree | - |
a. Responses reflect attendees that attended one of the workshops held over two days; *Can select more than one perspective.
Including patient partners in research not only enriches the research process, it ensures patient partners have a voice in shaping research that impacts their lives. Patient engagement provides opportunities for learning, skill development, and networking, and some patient partners also express personal fulfillment through meaningful contributions to the research process. This involvement ensures lived experiences are represented, fostering more patient-centered outcomes.
However, our discussions with researchers over the course of our workshops demonstrated that a gap remains in knowledge and understandings of patient partner engagement within the academic and research community, and especially in making a distinction between patient engagement versus patient participation in a study. This sentiment was shared by several of our patient partners and is highlighted by Trudy Flynn, a patient partner peer reviewer and workshop facilitator, who stated:
“Everyone seems to assume that patient partner engagement is common knowledge, but the reality is that many are still unaware of its importance and impact in healthcare. We need to break this assumption and foster a more inclusive understanding with workshops like this one. If we can educate just one researcher about the importance of patient partner engagement, it can create a snowball effect, leading to broader awareness and more meaningful collaboration in the research community."
For many trainees and researchers, this workshop was often their first exposure to any sort of formal education or knowledge dissemination about patient engagement. A continued emphasis on educational initiatives is needed if we are to ensure appropriate understanding and inclusion of patient partners within the research continuum.
This workshop highlighted the importance of patient engagement in shaping research to meet real-world needs. Effective communication between researchers and patient partners emerged as a crucial component for fostering collaboration. Additionally, the workshop underscored the value of diverse perspectives in enriching discussions and the importance of hybrid workshops when only hosted at one time. There may also be a benefit to hosting the content as virtual only or in-person only in certain settings, given that we can never truly mitigate the risk of issues with hybrid technology. Overall, the lessons learned will help improve future workshops and enhance the collaboration between researchers and patient partners.
Based on preliminary feedback, the most recent iterations of our workshop have shifted focus to offering a Q & A session in addition to, or instead of, case studies whereby attendees can use the latter portion of the workshop to ask questions about patient partner engagement within their respective areas of research. Indeed, in our most recent October 2024 workshop, we implemented these changes and received very positive feedback. It is also important to note that diversity in patient partner backgrounds and experiences can serve as a means to enrich the peer review and workshop process, highlighting gaps and areas of further exploration. The diversity of our patient partner backgrounds (including several youth members) underscored gaps in knowledge and brought forth additional considerations for equity and inclusion. Further, during enrollment there was a large number of researchers and trainees that expressed an interest in virtual attendance. Upcoming sessions of our workshop are now being offered virtually as a means to ensure accessible attendance for both patient partners and researchers across Canada.
Looking to the future, we aim to continue offering workshops on patient engagement and, ultimately, move the dial on patient engagement in research. However, achieving this goal necessitates ongoing efforts from the entire research community to establish a strong foundation of awareness and education regarding the significance of patient engagement in research.
For any questions related to our patient engagement workshop or for access to our peer review and workshop materials, please email: contact@pxphub.org or imha-iala@cihr-irsc.gc.ca
Canadian Institutes of Health Research. (2019). Patient engagement. https://cihr-irsc.gc.ca/e/45851.html
Crockett, L. K., Shimmin, C., Wittmeier, K. D. M., & Sibley, K. M. (2019). Engaging patients and the public in Health Research: experiences, perceptions and training needs among Manitoba health researchers. Research Involvement and Engagement, 5, 28. https://doi.org/10.1186/s40900-019-0162-2
Domecq, J. P., Prutsky, G., Elraiyah, T., Wang, Z., Nabhan, M., Shippee, N., Brito, J. P., Boehmer, K., Hasan, R., Firwana, B., Erwin, P., Eton, D., Sloan, J., Montori, V., Asi, N., Dabrh, A. M. A., & Murad, M. H. (2014). Patient engagement in research: A systematic review. BMC Health Services Research, 14(89), 1–9. https://doi.org/10.1186/1472-6963-14-89/FIGURES/3
Forsythe, L. P., Carman, K. L., Szydlowski, V., Fayish, L., Davidson, L., Hickam, D. H., Hall, C., Bhat, G., Neu, D., Stewart, L., Jalowsky, M., Aronson, N., & Anyanwu, C. U. (2019). Patient Engagement In Research: Early Findings From The Patient-Centered Outcomes Research Institute. Health Affairs (Project Hope), 38(3), 359–367. https://doi.org/10.1377/HLTHAFF.2018.05067
McVey, L., Frost, T., Issa, B., Davison, E., Abdulkader, J., Randell, R., Alvarado, N., Zaman, H., Hardiker, N., Cheong, V. L., & Woodcock, D. (2023). Working together: reflections on how to make public involvement in research work. Research Involvement and Engagement, 9(1). https://doi.org/10.1186/S40900-023-00427-4
Pratte, M. M., Audette-Chapdelaine, S., Auger, A. M., Wilhelmy, C., & Brodeur, M. (2023). Researchers’ experiences with patient engagement in health research: a scoping review and thematic synthesis. Research Involvement and Engagement, 9(1), 22. https://doi.org/10.1186/S40900-023-00431-8
Richards, D. P., Twomey, R., Flynn, T., Hunter, L., Lui, E., Stordy, A., Thomas, C., & Khan, K. (2024). Patient engagement in a Canadian health research funding institute: implementation and impact. BMJ Open, 14(7), e082502. https://doi.org/10.1136/BMJOPEN-2023-082502
In developing our rubric for the Patient Engagement 101 workshop, we integrated evidence from the literature and incorporated several design considerations that would support clear, reliable, and specific feedback. Our rubric was modeled around the Analytic Rubric, which has demonstrated increased reliability in comparison to alternative rubric formats (S. Brookhart, 2015; Jonsson & Svingby, 2007). Additionally, this type of rubric is designed to provide specific feedback as it provides scores on varying levels of performance. To assess levels of performance, we used an evaluative scale (e.g., Excellent, good, poor etc.) with six levels (i.e., 0-5) (S. Brookhart, 2015; S. M. Brookhart, 2018; Perlman, 2002). Each level of the rubric was associated with a whole number score (e.g., 5, 4, 3 etc.) instead of a range of scores (e.g., 8-10) or decimal point score (e.g., 3.2). The reasoning behind this was to ensure consistent scoring between reviewers for each level of the rubric and to increase meaningful differences between scores. When using decimal points, it can become challenging to identify what the meaningful difference is, say, between a score of 3.2 versus 3.3. Likewise, providing a range of scores for one level of a rubric can introduce subjectivity and decrease the reliability of scoring between reviewers because they now have multiple scores to choose from for one performance (e.g., Excellent = score of 8-10) (Perlman, 2002; Vercellotti & McCormick, 2021). With this in mind, each score on our rubric was accompanied by a very clear description of the type of performance that merited the score. While it can be time consuming to develop your own rubric, we felt this made the reviewing standards explicit and provided an accessible and easy tool for reviewers to assess workshop applicants.
Brookhart, S. (2015). How to Create and Use Rubrics for Formative Assessment and Grading (L. Bova, Ed.). ASCD.
Brookhart, S. M. (2018). Appropriate Criteria: Key to Effective Rubrics. Frontiers in Education, 3, 359715. https://doi.org/10.3389/FEDUC.2018.00022/BIBTEX
Jonsson, A., & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review, 2, 130–144. https://doi.org/10.1016/j.edurev.2007.05.002
Perlman, C. (2002). An introduction to performance assessment scoring rubrics. In C. Boston (Ed.), Understanding Scoring Rubrics (pp. 5–13). ERIC Clearinghouse on Assessment and Evaluation.
Vercellotti, M. Lou, & McCormick, D. E. (2021). Constructing Analytic Rubrics for Assessing Open-Ended Tasks in the Language Classroom. Teaching English as a Second or Foreign Language, 24(4), 1–19. https://www.tesl-ej.org/pdf/ej96/a2.pdf
English (UK) 
Français