Evaluating the benefit of early patient and public involvement for product development and testing with small companies

Abstract Introduction There is a growing understanding of the benefits of patient and public involvement (PPI), and its evaluation, in research. An online version of the CUBE PPI evaluation framework has been developed. We sought to use the CUBE to evaluate the value of early PPI with two small healthcare companies during product development. Methods Contributors were recruited online and had lived experience of either type 1 diabetes or obesity. Two 1‐h sessions were run with a company developing a smartphone application to manage diabetes (DEE‐EM): one with young people (YP; n = 5) and one with parents (n = 7). Two 1‐h sessions were run with a company developing a weight‐loss product, both with adults (n = 7 in each session). Sessions were facilitated by an independent University researcher and attended by company representatives, who presented their product. One facilitator led the evaluation of the session by giving a demonstration of the CUBE and asking simple questions in the YP session. Results A high proportion of contributors completed the CUBE (80.5% DEE‐EM; 93% Oxford Medical Products). Responses were positive to all four CUBE dimensions (in italics). Contributors felt there were diverse ways to contribute to the sessions, and that they had a strong voice to add to the discussion. Balance was achieved regarding whose concerns (public or company) led the agenda, and contributors felt that both companies would make changes based on the discussion. The supportive attitude of both companies resulted in most contributors feeling comfortable participating in PPI sessions with the industry, while recognising the profit‐making aspect of their work. Conclusions PPI with small healthcare companies is both feasible and worthwhile. The CUBE framework facilitated the evaluation of the interaction between experts in different knowledge spaces. We provide recommendations for future projects, including considerations of who should participate and the level of implicit endorsement of the product that participation implies. Patient or Public Contribution People with lived experience of type 1 diabetes or obesity were invited to contribute to one of four PPI sessions, which they then evaluated. One contributor agreed to contribute to the analysis of the evaluation data and interpretation and preparation of the manuscript.

important step is also emerging in Europe, through the support of the Innovative Medicines Initiative, 1 and also in the United States, as exemplified by the Patient-Centred Outcomes Research Institute (PCORI; https://www.pcori.org/). Funders encourage working closely with people with lived experience of health conditions as a mechanism to enhance the translation of health research findings into societal and economic benefits. Funding schemes supporting such collaboration, particularly from NIHR, require consideration as to how patients and the public will benefit from and be involved in research. 2 Concurrent with the rise of PPI in research is a call for greater evaluation of that PPI, to determine its value and quality. [3][4][5] The process of evaluation of PPI, allows PPI contributors to consider whether they have been given the opportunity to participate in a meaningful way, which can then have a positive impact on the research study in question. Through the evaluation, researchers gain insight into the contributors' experiences of the sessions, in turn giving the opportunity to improve aspects that may be less well received. Moreover, researchers and funders need to critically consider costs, benefits, and risks, as well as how best to conduct PPI for benefits to be realised. 3,6 A range of methodologies is available to evaluate PPI in research, often chosen based on the intended outcomes and the time frame available to conduct the research. 4 These approaches range in simplicity, from preparing an 'impact log' on the outcomes of the PPI, using the CUBE framework, 7 to the more comprehensive Public Involvement Impact Assessment Framework 8 or Realist Evaluation. 9 The CUBE framework was chosen for this project to reflect the consideration of the differing 'knowledge spaces' (the conversation space in which different types of expertise from the public, healthcare providers and other professionals are shared) that are important when evaluating interactions between the public and other organisations on healthcare issues. 4,10,11 In brief, the CUBE framework was developed through a combination of reviewing the theoretical literature on social inequality and practical workshops with members of the public. 7 The framework allows the researcher to evaluate PPI across four dimensions: voice (the extent to which contributors feel they have a weak or strong voice in decision-making); contribute (the number of ways to get involved to accommodate different contributors' needs); agenda (the balance between organisation and public contributor concerns); change (the willingness or resistance to change by the organisation). It can be used to compare the experience of PPI across different organisations. 7 Since the publication of this framework, Gibson  Companies have used market research, conducted in-house or by external market research agencies, to garner public opinion during the medical product lifecycle (MPLC). Market research is the passive extraction of information from the public, via focus groups, individual interviews, and surveys. By contrast, PPI encourages active involvement by patients and the public in a co-design process. It is recognised that opportunities exist for PPI throughout the MPLC and health technology assessment, 12,13 but there is a lack of consensus regarding how and when is best to engage with patient preferences. [14][15][16] Health preference research often employs discrete choice experiments to assess patient preferences, 14 however, it could be argued that alternative, discursive approaches, such as PPI, may be more appropriate at the early design phase of the MPLC. In the United States, for example, the Center for Drug Evaluation and Research has developed a programme to assist companies in 'Patient-Focused Drug Development', to 'ensure that patients' experiences, perspectives, needs and priorities are captured and meaningfully incorporated into drug development and evaluation'. 17 In the United Kingdom, the Aims 2 Trials have set up a steering committee of representatives with autism who actively advise on drug development and trials with industry partners and researchers. 18 The novel objectives of this project were twofold: (i) to use the CUBE framework to evaluate whether PPI sessions for small start-up companies were acceptable and valuable from the contributors' perspective and (ii) to assess whether PPI might be beneficial for companies who are developing products for healthcare and looking to partner with NIHR Biomedical Research Centres (BRCs) to move their plans forward. A further key question posed was whether the profit-making nature of a company changes the ethos of the PPI and therefore, whether the approach taken to PPI needs to be different with a company compared to with a research institution. Our final aim was to produce a set of recommendations on conducting PPI with small companies.

| METHODS
We set out to evaluate the introduction of PPI in the MPLC with two UK-based small technical companies. One company, DEE-EM, was in the initial product development phase of a novel smartphone app for type 1 diabetes management (https://www.dee-em.com/). The second, Oxford Medical Products (OMP), has developed a weightloss product, a hydrogel in capsule form which expands in the stomach with the aim of increasing fullness and reducing food intake and were at the beginning of the validation phase, planning their first in man safety and feasibility trial (https://www. oxfordmedicalproducts.com/). These two small companies were chosen as had approached the Bristol BRC for help in gaining public opinion from patients with relevant lived experience. We worked with these two companies in differing ways: for the DEE-EM project, only the company spokesperson and the PPI facilitator (M. B.) were actively involved in the session, whereas for the OMP project, the company spokesperson, PPI facilitator (E. C. H.) and a bariatric surgeon (chief investigator on the future trial) were involved.

| Contributors
Contributors were people with lived experience of either type 1 diabetes (DEE-EM sessions) or obesity (OMP sessions). Recruitment material was generated with input from each company. Advertisements were placed online with the support of People in Health West of England, Diabetes UK, Obesity UK and social media (Twitter and Facebook). For DEE-EM, two sessions were planned: the first with young people (YP) under 18 years with type 1 diabetes and the second with parents of YP with type 1 diabetes. For OMP, two sessions were also planned, both with adults with a lived experience of obesity. We aimed to recruit between six and eight contributors for each session. Following joint guidance from the National Research Ethics Service and NIHR INVOLVE initiative, 19 ethical approval was not sought for this project. Active involvement in PPI and its evaluation is conducted with the contributors as equal partners, rather than 'to, about, or for them', 12,p.1 as research participants. The PPI sessions were conducted with the utmost respect and care for contributors giving them the right to take part and share the details they choose to during the sessions. All contributors gave permission for the sessions to be recorded. The scores from the CUBE evaluation are collected anonymously, and all quotes from the recordings were also rendered anonymous before inclusion in this paper.

| Session design
The design of PPI sessions with both companies was of a similar afterwards with the details of how to evaluate the session if they wished. Contributors were provided with an honorarium to participate in sessions, based on the PPI rate advocated by the NIHR. 20  For all four sessions, contributors were asked to fill in the online version of the CUBE framework. 7 The four questions, one relating to each dimension, were adapted for this project (Table 1). Contributors Finally, contributors were asked an additional question: 'How did you feel during the session about contributing to the development of a product by a company rather than a university or healthcare provider?'

| Analysis plan
Scores from the questions given to the YP were summarised, and the scores on the CUBE for each session were collated and summarised by the median and interquartile range (IQR; due to the ordinal nature T A B L E 1 CUBE questions.

CUBE dimension Question
Contribute During this project, have there been different ways that have enabled you to contribute to the discussion?
Voice A strong voice participates in discussions and influences the discussion or decisions made. A weak voice participates in the discussion but has little chance of influencing the discussion. Do you feel you have had a weak or strong voice in this project?
Agenda Who sets the agenda? To what extent have you been able to express your concerns and thoughts during this project? Have the concerns of the organisers of the project dominated? You can move the marker between 'organisers concerns' and 'public concerns' and again add notes if you wish.

Change
To what extent do you believe the organisation will make changes after the session based on the comments made today? You can move the marker between 'organisers resistant to change' to 'organisers have been willing to change'.
F I G U R E 1 Screenshot of the CUBE. Questions for each dimension appear on the left of the screen with the slider to move the data point in blue. A comment box is provided under each question. The CUBE appears on the right, with each dimension on a different axis: Contribute on the x-axis, Voice on the y-axis, agenda on the z-axis and change as colour of the data point (yellow is neutral; red is resists change, green is willing to change). The cube can be rotated using the mouse so that contributors can explore the data points in relation to the different dimensions.
of the data). Comments made by contributors using the CUBE, and responses to the additional question, were collated into a table to summarise their experiences of each session and narratively synthesised across common themes between the sessions.

| RESULTS
The DEE-EM sessions took place early evenings of May 2021, and the OMP sessions took place in July 2021.

| DEE-EM YP session
Four YP with type 1 diabetes joined the call, with one parent in attendance in support of one of the YP.

| DEE-EM parent session
Seven parents joined the second call the following day.
OMP session 1: Seven contributors took part.
OMP session 2: Seven contributors took part (different people from OMP session 1).

| Evaluation of DEE-EM YP session
Contributors to the DEE-EM YP session (1)

| Summary of the CUBE results
A high proportion of the contributors did provide an evaluation of the session they attended using the CUBE: 75% from the DEE-EM YP session; 86% from the DEE-EM parent session; 93% of the adults who attended the OMP sessions. Figure 2 depicts the final CUBEs for each company. Figure 3 depicts the median values and IQR error bars for each of the four CUBE dimensions (see Table 1 for question details).

| CUBE dimension: Contribute
The mean scores in Figure 3 and data points in Figure 2 show that contributors reported that there were several different ways that F I G U R E 2 (A) Final CUBE with contributor's responses for DEE-EM; (B) final CUBE with contributor's responses for OMP. The CUBE appears on the right, with each dimension on a different axis: Contribute on the x-axis, Voice on the y-axis, agenda on the z-axis and change as colour of the data point (yellow is neutral; red is resists change and green is willing to change). The cube can be rotated using the mouse so that contributors can explore the data points in relation to the different dimensions.
enabled them to participate in the discussion ( Table 2,

| CUBE dimension: Voice
The scores in Figure 3 and quotes in Table 2 show that contributors to all the sessions felt listened to and had ample opportunity to get their 'voice' heard in different ways. The quotes in the table suggest the format of the online session may have influenced the extent to which some people felt able to contribute.

| CUBE dimension: Agenda
The comments provided by contributors on the extent to which company or public concerns dominated the session suggest that a good balance was achieved during sessions with both companies (  Figures 2 and 3), both companies achieved a good balance between their specific concerns and allowing the public contributors to bring their concerns into the discussion: 'I felt it was a good half and half, the organisation definitely has their concerns, but the publics were addressed as and when we got there…'

| CUBE dimension: Change
The final CUBE dimension captures the extent to which the contributors believed the organisation would make changes after the session based on the comments by the public made during the session. As Figure 3 shows, contributors gave moderately high scores towards the 'organisation have been willing to change' rather than resistant to change. The DEE-EM scores were particularly high on this dimension, again perhaps reflecting the emotional investment the company's representative has in developing the app for diabetes management. There was also a sense that for both companies, the sessions had been held early in the process, and that over time, it would become apparent if concerns had been addressed going forward, although one contributor expressed uncertainty regarding how they would find out (Table 2).

| Contributing to PPI for industry
The final question asked attendees to consider how they felt about contributing to the development of a product by a company rather than a university or healthcare provider. T A B L E 2 CUBE dimensions quote from the sessions with DEE-EM and OMP.

CUBE dimension DEE-EM sessions OMP sessions
Contribute 'We all made good points so I think they will take that into count'.
'By using chat was good as you can't always get your view over so using chat is easy' 'We were able to speak and to write in the chat box, which was very useful when being interrupted by an excited 3 year old!' 'Lots of questions were asked' 'You could click on the raise hand button or type in what you wanted to say'.
'I really enjoyed the mixture of engaging techniques used. Such as video, polls, chat box, contributing with the camera on' 'Chat feature on zoom, zoom discussion, initial thoughts over email'.
'Zoom hand and my hand' 'I thought the fact we could talk or write in the chat was great. If we weren't in a pandemic, I have no doubt a face to face option would be offered' 'So far, only discussion has been via the zoom meeting' Voice 'No one was ignored what shows that they listened'. 'In the middle I think as I'm always prepared to hear others views' 'I feel that my voice, thoughts and ideas were heard and valued'.

'I felt listened to'
'Felt that the recommendations were listened to and understood and considered for future developments to the app'.
'I felt very comfortable to contribute and the environment was very safe to contribute. It was great to see everyone who wanted to contribute to the discussion had the opportunity' 'Lots of valid points were made by all speakers. We all have a lot of shared opinions and ideas so it was important to listen as well as discuss, which the group seemed to do well, in my opinion'.

'Strong as I said what I needed to say'
'I am more vocal during face-to-face meetings'. 'I felt my voice was strong and I was heard, but I feel the same about the others also talking. Potentially, the people using the chat had a weaker voice' 'I made various points and felt free to do so' 'I felt able to participate wherever possible, notwithstanding time constraints' Agenda 'In the meeting we were encouraged to voice any concerns that we might have'.

'No concerns as it went really well'
'I feel that it was equal and everyone was given their chance to voice their concerns and opinions'.

'I think public concerns prevailed'
'Very open dialogue between all parties' 'Lots of opportunities to discuss. Ideas and suggestions seemed to be taken on board'.

'It was balanced'
'I think because of his circumstances, the organiser's concerns are also public concerns'.
'I felt it was a good half and half, the organisation definitely has their concerns but the publics were addressed as and when we got there. We were kept very well informed' 'I felt that the organisers were genuinely interested in our views, though not sure whether financial considerations will eventually outweigh "ideal" product' 'I felt the meeting was very much about the organisers making a great effort to discover public feelings and concerns' Change 'They wouldn't make the meeting for no reason'. 'Hopefully that our input has helped the research' 'I definitely feel that our thoughts were valued and that the ideas we had for the app would be considered as we were a good cross section of parents with different aged children using different kinds of technology so listening to our thoughts would allow the product to be useful to so many who have the same worries and issues that we have'.

| DISCUSSION
The primary aims of this project were to evaluate the use of the novel CUBE framework to assess whether PPI sessions for two UK-based small technological companies were acceptable and valuable from the contributor's perspective and to assess whether such PPI might be beneficial for those small companies who are developing products to improve healthcare. First, this was a feasible undertaking with sufficient people with relevant lived experience willing to contribute to the sessions and importantly a high proportion of those who attended the sessions completing the evaluation. Overall, The organisers were very interested in our opinions and ideas to improve the app. I am fully confident that if the ideas we had can be done then they will be added to the app'.
'I feel the organisation will have definitely taken the publics thoughts and opinions on and therefore will change what they can to do what they can to address our comments' 'Recognise its early days and adaptations and improvements can take time'.
'I felt organisers were genuinely interested in our views and I would like to think that theses will be taken fully into account in the design and launch of the product'  | 1167 process, we suggest providing the CUBE link immediately after the session while contributors are keen to give their evaluation of the session.
7. When interpreting the CUBE, consider that a higher score on a dimension may not be a better score; for example, a balance is optimum for the 'Agenda' dimension.
8. Ensure means to share the impact and outcomes of the PPI session, and the given contributions are planned and implemented when the novel healthcare products are tested and brought to market.
9. Use the evaluation by contributors to improve how future sessions are organised and run. patients and the public, by following published guidelines 13 and recommendations such as those presented here, should be a priority. 16 The positive impact of the contributions from the public to product development, reported by both companies in this evaluation, suggests that this is a fruitful application of PPI for the future. All authors commented and agreed on the final manuscript.