Original research••

Potential role of ChatGPT in simplifying and improving informed consent forms for vaccination: a pilot study conducted in Italy

•,,,,,,.

...

Abstract

Objectives Informed consent forms are important for assisting patients in making informed choices regarding medical procedures. Because of their lengthy nature, complexity and specialised terminology, consent forms usually prove challenging for the general public to comprehend. This pilot study aims to use Chat Generative Pretrained Transformer (ChatGPT), a large language model (LLM), to improve the readability and understandability of a consent form for vaccination.

Methods The study was conducted in Italy, within the Central Tuscany Local Health Unit. Three different consent forms were selected and approved: the standard consent form currently in use (A), a new form totally generated by ChatGPT (B) and a modified version of the standard form created by ChatGPT (C). Healthcare professionals in the vaccination unit were asked to evaluate the consent forms regarding adequacy, comprehensibility and completeness and to give an overall judgement. The Kruskal–Wallis test and Dunn’s test were used to evaluate the median scores of the consent forms across these variables.

Results Consent forms A and C achieved the top scores in every category; consent form B obtained the lowest score. The median scores were 4.0 for adequacy on consent forms A and C and 3.0 on consent form B. Consent forms A and C received high overall judgement ratings with median scores of 4.0, whereas consent form B received a median score of 3.0.

Conclusions The findings indicate that LLM tools such as ChatGPT could enhance healthcare communication by improving the clarity and accessibility of consent forms, but the best results are seen when these tools are combined with human knowledge and supervision.

What is already known on this topic

Informed consent forms are essential tools in public health, serving both medicolegal requirements and the empowerment of patients, yet they are often criticised for being overly complex and difficult to understand. Recent studies suggest that artificial intelligence tools such as Chat Generative Pretrained Transformer (ChatGPT) could improve the clarity and accessibility of informed consents; however, their effectiveness in this regard remains scarcely explored.

What this study adds

This pilot study demonstrates how informed consent forms created by combining ChatGPT with human experience are more readable and obtain higher ratings for adequacy, comprehensibility and completeness than informed consent forms generated solely by ChatGPT.

How this study might affect research, practice or policy

These findings highlight the potential of ChatGPT to improve healthcare communication by producing more understandable documents for patients when supplemented with human supervision. This approach could guide future policies aimed at enhancing health literacy and patient engagement, particularly in contexts requiring informed decision-making, such as vaccination programmes.

Introduction

Informed consent forms represent a fundamental tool for public health objectives, for healthcare professionals from a medical-legal perspective and, above all, for the autonomy of patients. Thanks to the consent form, in fact, the healthcare professional is legitimised to perform a healthcare act when the patient understands and consents to the benefits and risks.1–3 A consent form is not only a legal requirement, but a necessary aspect of ratifying the patient’s autonomy and trust in the healthcare system.4

The creation of patient-friendly consent forms is a long-standing challenge in the field of medicine and research.5 According to European Regulation, the aim of a consent form is to respect the autonomy of individuals in making decisions about their health by providing them with information that is ‘comprehensive, concise, clear, relevant and understandable’.6 It is important to note that not everyone has the skills to understand any kind of consent form and make decisions accordingly. This is often due to a low level of health literacy, which is the ability to access, understand, evaluate and apply health information to make the best decisions for their own health.7 Consent forms are often difficult to read due to various factors, such as their length, small font size or the presence of unexplained technical, medical and legal terms.8

Recent studies suggest that using chatbots based on large language models (LLM) could be a concrete way to improve the clarity and communicability of information in consent forms. These emerging tools can modify the health information contained in documents, making it more understandable, readable, accurate and complete for patients.9 10 Currently, one of the most sophisticated chatbots is Chat Generative Pretrained Transformer (ChatGPT), an LLM-based chatbot freely accessible via a web interface: https://chat.openai.com/.11

From the public health perspective, vaccines are highly effective in protecting populations from epidemics, reducing mortality and morbidity of many diseases, consequently increasing overall productivity.12 As with any other healthcare procedure, consent forms must be obtained before administering vaccines, so patients must be able to understand the object of treatment and make an informed decision about it.13 Regarding this, vaccine literacy is crucial for enabling people to make informed decisions about vaccines for themselves, their families and the community;14 thus, consent forms play a key role in this process. However, sometimes consent forms used for vaccinations do not convey enough information,15 and their readability does not match the reading ability of many people.16

For this reason, in this pilot study, we used OpenAI’s ChatGPT as a tool to modify the standard consent form currently in use in the Italian Prevention Department of the Central Tuscany Local Health Unit (LHU-CT) for vaccinations. Our aim was to create an improved, more user-friendly version that complied with all medico-legal requirements, assessing the completeness, adequacy and comprehensibility of each consent form examined.

Methods

A multiphase approach was adopted to evaluate different informed consent forms relating to vaccinations. In the first phase, starting with the consent form already in use (form A, online supplemental file 1), three other versions of the consent form for vaccination (B, C and D) were generated by ChatGPT (model 3.5, OpenAI 2024) by sending specific prompts. The four consent forms (A, B, C, D) were then evaluated by an expert panel in the field of vaccinations for inclusion in the study. As a result, one consent form (D) was excluded because it was deemed unsuitable, while the other three were considered worthy of being assessed for adequacy, comprehensibility, completeness and a final overall judgement by a convenience sample of healthcare professionals working in the Prevention Departments of the LHU-CT.

Identification of four informed consent forms

Consent form A was chosen for convenience, as it is the official consent form, fully human-generated, used in the LHU-CT area of our pilot study. By using consent form A as a template to begin with, two consent forms (C and D) were generated asking ChatGPT to rephrase and reformulate contents. Consent form B was entirely generated using ChatGPT, without considering consent form A. During the generation process, ChatGPT responded to specific prompts provided by a panel of experts (professors, researchers, medical residents and a student in Hygiene and Preventive Medicine) (table 1).

Table 1

•

Description of the informed consent forms used in the study

Selection of consent forms by a panel of experts

After obtaining the four consent forms, the panel of experts selected the ones to include in the study. Completeness, clarity and readability were considered inclusion criteria for comparison with the consent form A. Excessively colloquial tone was considered an exclusion criterion. To assess readability, the Gulpease Index17 was used as a non-binding support tool for the decision. The Gulpease Index is a readability index developed for the Italian language that considers the average number of reads per word and the average number of words per sentence, as well as the total number of sentences in the text. As an instrument developed to assess reading difficulty, the Gulpease Index determines how accessible a text is, generating a value that indicates the ease of reading for people with different degrees of education. A score above 60 indicates easy readability, while below 40 indicates high complexity.

Characteristics of questionnaires for the selected consent forms and their administration to healthcare professionals for evaluation

Each consent form was evaluated using a five-question questionnaire. The first question asked for the healthcare professional’s specific role (physician, healthcare assistant, nurse, other). The remaining four questions assessed adequacy, comprehensibility, completeness, and overall judgement:

Do you consider the consent to be adequate for the vaccination context?
Do you consider the consent to be understandable for users?
Do you consider the consent to be complete?
Overall, how do you rate the consent?

Adequacy concerns the ability to be sufficient for a particular purpose18 and the ability to comply with the requirements and timeframes set forth in the Code of the Good Clinical Practice19 so that the user can knowingly consent to the therapeutic choice; comprehensibility is the degree to which the text is usable and comprehensible;20 completeness refers to the presence of all the essential and specific information that must be present in a consent form.21

The response options for adequacy, comprehensibility and completeness (the definition of which was given in the questionnaire) were on a Likert scale with four options, each associated with a score: very much (4), quite a lot (3), a little (2) and not at all (1). The last question was answered by selecting an emoji from the four available. Each emoji was associated with a score ranging from 1 (sad emoji) to 4 (very happy emoji). Each questionnaire and its corresponding consent form were assigned a specific colour for clear and immediate distinction (online supplemental files 12).

The selected consent forms along with the questionnaires were distributed to 129 healthcare professionals working in the LHU-CT for their evaluation. The distribution was carried out via a Google Forms link, sent via WhatsApp or e-mail. Data collection began on 2 October 2023 and ended on 22 October 2023. Data were automatically saved and stored in an electronic database on Google Drive at the end of completion, preventing further access by participants. The estimated time for completion overall was about 5 min. No healthcare professional was aware of which consent form had been created or modified by ChatGPT.

Statistical analysis

For each psychometric variable and for the overall judgement, the median and IQR of obtained scores were calculated. The data were represented using bar plots. Medians of scores for each psychometric variable were calculated for different professional subgroups (healthcare assistants, nurses and physicians). The Kruskal–Wallis rank sum test was used to assess the equality of median scores with respect to the profession variable. In cases where the Kruskal–Wallis test yielded statistically significant results, the Dunn test with Bonferroni correction was employed to understand the direction of the difference. The alpha level considered for all analyses was 0.05. All analyses were conducted using RStudio V.2023.06.1 (Posit Software, PBC, Boston, MA. http://www.posit.co/.).

Results

Pre-selection of informed consent forms

Using the Gulpease Index, the scores obtained for each consent form were:

36 for consent form A, namely incomprehensible for elementary level, very difficult for middle and high school levels.
66 for consent form B, namely very difficult for elementary level, easy for middle and high school levels.
82 for consent form C, namely easy for the general population, very easy for higher education levels.
63 for consent form D, namely very difficult for elementary level, easy for middle and high school levels.

The panel of experts considered consent forms A, B and C to be positively selected based on readability. Although consent form D received a score of 63, which is equivalent to easy reading for the population with a middle school degree and above, it was discarded because it contained an excessively colloquial tone.

Analysis of psychometric variables and overall judgement

Forty (31%) healthcare professionals (healthcare assistants, nurses and physicians) working in the Prevention Departments of the LHU-CT responded to the questionnaires. Consent form A and consent form C obtained the highest scores in terms of adequacy, comprehensibility, completeness and overall judgement (table 2, figure 1).

Table 2

•

Adequacy, comprehensibility, completeness and overall judgement of the consents: descriptive analysis

Figure 1

Request permissions

Bar plots for adequacy, completeness, comprehensibility and overall judgement for the three different consent forms (A, B, C).

Consent form B, on the other hand, obtained the lowest scores. Regarding the adequacy, consent forms A and C obtained high ratings (median of 4.0 for both), while consent form B obtained a median of 3.0. For comprehensibility, consent form A had a median of 4.0 (IQR 3.0–4.0), consent form B a median of 3.0 (IQR 3.0–4.0) and consent form C a median of 3.5 (IQR 3.0–4.0). Completeness followed a similar trend, with consent form A receiving a median of 4.0 (IQR 3.0–4.0), consent form B 3.0 (IQR 3.0–4.0) and consent form C also scoring 3.0 (IQR 3.0–4.0). In terms of overall judgement, consent form A received a median of 3.0 (IQR 3.0–4.0), while consent form B scored 2.0 (IQR 2.0–3.0), and consent form C scored 3.0 (IQR 3.0–4.0). In all these cases, we always found a statistically significant difference between groups A and B and between B and C, but not between A and C.

The results of the analysis using the Kruskal–Wallis test to compare the median scores for consent forms A, B and C were significant for all variables considered. The test revealed a significant difference between the consent forms for the variables ‘adequacy’ (p<0.001), ‘completeness’ (p=0.01253), ‘comprehensibility’ (p=0.01374) and ‘overall judgement’ (p=0.002784). Specifically, with regard to adequacy, Dunn’s test with Bonferroni correction showed that consent form A differed significantly from consent form B (p<0.001) and consent form C (p=0.75). Consent forms B and C were also significantly different (p<0.01). With regard to the variable of completeness, the post Kruskal–Wallis analysis showed that consent form A differed from the B one (p=0.012), but not from the C one (p=1.0). In contrast, a significant difference emerged between consent forms B and C (p=0.017). Similarly, regarding the comprehensibility, Dunn’s test showed that consent form A differed significantly from the B one (p=0.005), but not from the C one (p=0.26), without finding significant differences between consent forms B and C (p=0.15). Regarding the overall judgement, the analyses revealed significant differences between consent forms A and B (p=0.009), and between consent forms B and C (p=0.002), while no significant differences were found between consent forms A and C (p=0.896).

Analysis by professional subgroups

The assessment for each psychometric variable and the overall judgement was analysed per professional subgroup (table 3).

Table 3

•

Adequacy, completeness, comprehensibility and overall judgement of the consents: descriptive analysis by professional subgroups

Regarding the evaluation of the ‘adequacy’ parameter, consent form A received the highest scores, with a median rating of 4.0 (IQR 4.0–4.0) from all three professional categories. Consent form C also received a median of 4.0 (IQR 3.5–4.0), specifically from healthcare assistants and physicians. In contrast, consent form B received the lowest score, with a median score of 3.0 (IQR 1.0–3.0) among healthcare assistants, 3.0 (IQR 2.5–3.0) among nurses and 3.0 (IQR 1.5–3.0) among physicians. Regarding the ‘completeness”, the pattern of scores mirrors that observed for ‘adequacy’. Consent forms A and C received the highest score, while form B was rated the lowest. The same trend was observed for the variable ‘comprehensibility’, where consent forms A and C had almost overlapping medians and interquartile ranges. In contrast, consent form B received consistently lower median scores from all three occupational groups. Finally, regarding the ‘overall judgement’ parameter, consent forms A and C received the highest scores. Specifically, consent form A achieved a median score of 4.0 (IQR 3.0–4.0) from nurses, while consent form C received a median score of 4.0 (IQR 3.0–4.0) from physicians. For both consent forms, the median scores were lower in the other professional categories.

Discussion

Existing literature supports the idea that artificial intelligence (AI) can improve the clarity and comprehensibility of complex documents. Studies have shown that AI-driven tools are able to modify information to better match the reading capabilities and health literacy levels of different populations, even improving health literacy levels themselves, which is critical in medical contexts where comprehension is essential.22 23

The aim of this pilot study was to explore the potential of LLM to improve informed consent forms for vaccinations currently used at the Prevention Department of the LHU-CT. We did this by assessing three psychometric properties (adequacy, completeness and comprehensibility) of three different vaccination consent forms: consent form A (the one already in use, constructed entirely by humans), consent form B (generated entirely by ChatGPT) and consent form C (consent form A further modified by ChatGPT).

The results highlight some important findings. First, the Gulpease index, used to support the selection of consent forms to be administered to healthcare professionals, revealed that consent form C scored the highest, being the most readable. This might suggest that the best results are achieved through the collaborative use of ChatGPT and human expertise. The subsequent score of the three consent forms by healthcare professionals showed that consent forms A and C achieved very high scores, almost the maximums for all the psychometric variables and in the overall judgement. Consent form B, generated entirely by ChatGPT, scored lower, suggesting that collaboration between AI and humans yields better results: ChatGPT alone might not be sufficient to produce the best and most accepted product, and that the best results are achieved through the collaborative use of LLMs and human expertise.

Our results are in line with findings indicating that AI-assisted modifications can produce consent forms easier to read and understand. Best practices recommend keeping consent forms at a reading level suitable for the average patient, generally at or below the eighth grade. Indeed, numerous studies have shown that consent forms are often drafted in overly technical and complex language, above the average level of comprehension.24 25 Other experiments in collaboration between humans and AI in creating more accessible informed consent forms have already been conducted. For instance, the effectiveness of using ChatGPT in simplifying surgical consent forms was investigated.22 Alongside the field of vaccines, surgical procedures represent a challenge for the average citizen, making it crucial to find the best solution that allows individuals to fully understand every aspect involved. Alternative methodologies to improve consent forms have also been tested, such as the involvement of expert groups or specific professional figures such as psychologists or ethicists, which could bring about further improvements.26 These approaches, combined with AI technology, could lead to the development of consent forms that are not only legally valid, but also more accessible and user-friendly.

This study has some limitations. First, consent form B is the result of impulses sent by humans. The process of proxying with humans in the realisation of this consent form may not have been completely effective, potentially affecting the quality of the consent form produced by ChatGPT. We used a free AI model (ChatGPT 3.5), and all responses were copied and pasted directly from the tool without further editing. This approach could limit the accuracy and contextualisation of the information provided. Second, we observe a ceiling effect in the answers. Many healthcare professionals provided very high ratings for all consent forms, making it difficult to discern the subtle differences between them. This effect indicates that although the AI-generated consent forms met a basic level of acceptability, more subtle distinctions in quality were obscured by uniformly high ratings. Thus, although our pilot study demonstrates the ability of LLMs to produce satisfactory consent forms, more precise evaluation methods may be needed to fully capture their effectiveness. Furthermore, ours was a convenience sample that may not be representative of the broader population of healthcare providers. The involvement of prompt engineering experts could refine the AI output, ensuring more accurate and contextually appropriate consent forms. This study also has a great strength in that it is a pioneering attempt to exploit LLM such as ChatGPT to improve the readability and comprehensibility of vaccination consent forms. Our aim was not to create a definitive tool, but rather to explore the potential of this technology to improve informed consent form processes. This preliminary attempt paves the way for further interactions and iterations with LLM, focusing on user involvement and refinement of the tool.

Conclusions

AI’s potential to improve health communication is crucial for addressing public health challenges such as vaccination coverage and immunisation. Our pilot study highlights the potential of AI in improving the implementation process of the informed consent form for vaccinations, making it more effective through human supervision. AI alone cannot replace human input27 and the forms of communication already produced and used by humans. The use of chatbots such as ChatGPT to improve the readability of texts could be a winning solution for public health, if integrated with the experience and judgement of healthcare professionals. In order to validate and extend our results, as well as to improve the process of generating the best possible consent forms, including through interactions between healthcare professionals and citizens themselves with LLM, further research is needed.

Supplementary PDF 1

Supplementary PDF 2

CL and MDR contributed equally.
Contributors: CC: conceptualisation, methodology, investigation, resources, data curation, writing-original draft. AR: investigation, resources, data curation, writing-review and editing. RC: investigation, resources, data curation, writing-review and editing. PZ: investigation, resources, data curation, writing-review and editing. GB: conceptualisation, investigation, resources, data curation, writing-review and editing, supervision, project administration. CL: conceptualisation, methodology, validation, investigation, resources, data curation, writing-review and editing, supervision, project administration. MDR: conceptualisation, methodology, validation, formal analysis, investigation, resources, data curation, writing-review and editing, supervision, project administration. CC is the guarantor. All authors have read and agreed to the published version of the manuscript.
Funding: The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests: None declared.
Provenance and peer review: Not commissioned; externally peer-reviewed.
Supplemental material: This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

Data availability statement

All data relevant to the study are included in the article or uploaded as supplementary information. Not applicable.

Ethics statements

Patient consent for publication:

Ethics approval:

Not applicable.

Beauchamp TL, Childress JF. Principles of Biomedical Ethics. Oxford, Oxford Univ. Press 2001;
Google Scholar
Schenker Y, Meisel A. Informed consent in clinical care: practical considerations in the effort to achieve ethical goals. JAMA 2011; 305:1130–1.
doi:10.1001/jama.2011.333•Google Scholar
Pietrzykowski T, Smilowska K. The reality of informed consent: empirical studies on patient comprehension-systematic review. Trials 2021; 22.
doi:10.1186/s13063-020-04969-w•Google Scholar
Reiss DR, Karako-Eyal N. Informed Consent to Vaccination: Theoretical, Legal, and Empirical Insights. Am J Law Med 2019; 45:357–419.
doi:10.1177/0098858819892745•Google Scholar
Fernandez A. Improving the quality of informed consent: it is not all about the risks. Ann Intern Med 2010; 153:342–3.
doi:10.7326/0003-4819-153-5-201009070-00011•Google Scholar
Regolamento (UE) n. 536/2014 del Parlamento Europeo e del Consiglio, del 16 aprile 2014, sulla sperimentazione clinica di medicinali per uso umano. e che abroga la direttiva 2001/20/CE. Gazzetta Ufficiale dell’Unione Europea 27 maggio 2014; L-158:1-76.
Available: here [Accessed 23 Dec 2024]
Google Scholar
Kickbusch I, Maag D. Health literacy, International Encyclopedia of Public Health. Academic Press 2008;
Google Scholar
Paasche-Orlow MK. The challenges of informed consent for low-literate populations, Understanding Health Literacy: Implications for Medicine and Public Health. Chicago, AMA Press 2005;
Google Scholar
Decker H, Trang K, Ramirez J, et al. Large Language Model-Based Chatbot vs Surgeon-Generated Informed Consent Documentation for Common Procedures. JAMA Netw Open 2023; 6.
doi:10.1001/jamanetworkopen.2023.36997•Google Scholar
Aggarwal A, Tam CC, Wu D, et al. Artificial Intelligence-Based Chatbots for Promoting Health Behavioral Changes: Systematic Review. J Med Internet Res 2023; 25.
doi:10.2196/40789•Google Scholar
Passanante A, Pertwee E, Lin L, et al. Conversational AI and Vaccine Communication: Systematic Review of the Evidence. J Med Internet Res 2023; 25.
doi:10.2196/42758•Google Scholar
Shukla VV, Shah RC. Vaccinations in Primary Care. Indian J Pediatr 2018; 85:1118–27.
doi:10.1007/s12098-017-2555-2•Google Scholar
Rajput M, Sharma L. Informed consent in vaccination in India: medicolegal aspects. Hum Vaccin 2011; 7:723–7.
doi:10.4161/hv.7.7.15411•Google Scholar
Lorini C, Del Riccio M, Zanobini P, et al. Vaccination as a social practice: towards a definition of personal, community, population, and organizational vaccine literacy. BMC Public Health 2023; 23.
doi:10.1186/s12889-023-16437-6•Google Scholar
Attena F, Valdes Abuadili A, Marino S, et al. The informed consent in Southern Italy does not adequately inform parents about infant vaccination. BMC Public Health 2014; 14.
doi:10.1186/1471-2458-14-211•Google Scholar
Institute of Medicine (US) Committee on Health Literacy. Health literacy: a prescription to end confusion. Washington (DC), National Academies Press (US) 2004;
Google Scholar
Lucisano P, Piemontese ME. GULPEASE: una formula per la predizione della difficoltà dei testi in lingua Italiana. Scuola e Città 1988;
Google Scholar
Oxford Learner’s Dictionaries. Dictionaries.
Available: here [Accessed 23 Jul 2024]
Google Scholar
Federazione Nazionale degli Ordini dei Medici Chirurghi e degli Odontoiatri, Codice di Buona Pratica Clinica (CDM). 2014;
Available: here [Accessed 23 Dec 2024]
Google Scholar
Oxford Learner’s Dictionaries. Dictionaries.
Available: here [Accessed 23 Dec 2024]
Google Scholar
Oxford Learner’s Dictionaries. Dictionaries.
Available: here [Accessed 23 Dec 2024]
Google Scholar
Ali R, Connolly ID, Tang OY, et al. Bridging the literacy gap for surgical consents: an AI-human expert collaborative approach. NPJ Digit Med 2024; 7.
doi:10.1038/s41746-024-01039-2•Google Scholar
Mendoza College of Business. Artificial intelligence tool could increase patient health literacy, study shows.
Available: here [Accessed 23 Dec 2024]
Google Scholar
Aldoory L. Informed consent and health literacy: workshop summary. Washington, D.C, National Academies Press 2015;
Available: here
Google Scholar
Bothun LS, Feeder SE, Poland GA, et al. Readability of Participant Informed Consent Forms and Informational Documents: From Phase 3 COVID-19 Vaccine Clinical Trials in the United States. Mayo Clin Proc 2021; 96:2095–2101.
Google Scholar
Fons-Martinez J, Ferrer-Albero C, Diez-Domingo J, et al. Assessment of the appropriateness of the i-CONSENT guidelines recommendations for improving understanding of the informed consent process in clinical studies. BMC Med Ethics 2021; 22.
doi:10.1186/s12910-021-00708-1•Google Scholar
Nutbeam D. Artificial intelligence and health literacy—proceed with caution. Health Literacy and Communication Open 2023; 1:2263355.
doi:10.1080/28355245.2023.2263355•Google Scholar