Accuracy and Reproducibility of Large Language Models in Answering Clinical Questions from the Japanese Clinical Practice Guidelines for Vascular Anomalies: A Preliminary Analysis

Key words: generative artificial intelligence, hemangioma, large language models, lymphatic abnormalities, practice guideline, propranolol, sirolimus, vascular malformations

Introduction

Large language models (LLMs) have rapidly emerged as powerful tools for natural language processing and information retrieval ⁽¹⁾. Recent evaluations have examined LLM performance in answering disease-specific clinical questions (CQs), showing that although LLMs can generate guideline-consistent recommendations, performance varies by clinical domain, question complexity, and guideline specificity ^{(2), (3)}. Vascular anomalies are a heterogeneous group of rare pediatric disorders with diverse presentations and management strategies ⁽⁴⁾. The Japanese clinical practice guidelines for vascular anomalies 2017 (the 2017 guideline) provide evidence-based recommendations for diagnosis, imaging, and treatment ⁽⁵⁾. LLM accuracy in answering guideline-derived CQs remains unclear. Therefore, we compared two LLMs regarding their accuracy in answering CQs on vascular anomalies.

Methods

Study design

A cross-sectional comparative evaluation of two LLMs, ChatGPT 5.1 Thinking (OpenAI) and Claude Opus 4.1 (Anthropic), was conducted using CQs derived from the 2017 guideline in both English and Japanese. Claude Opus 4.1 was selected as a comparator because it was a publicly accessible, high-performance LLM from a different developer and model family. Eight of the 33 CQs were selected for evaluation (Table 1) because they were retained in the subsequent Japanese clinical practice guidelines for vascular tumors, vascular malformations, lymphatic malformations, and lymphangiomatosis 2022 (the 2022 guideline) ⁽⁶⁾. The selected CQs covered four major categories: arteriovenous malformations (CQ1, CQ3), venous malformations (CQ9, CQ10), infantile hemangiomas (CQ14, CQ17), and lymphatic malformations/complex lymphatic anomalies (CQ27, CQ33). The 2017 guideline served as the reference standard for recommendations, recommendation strength, and evidence level ⁽⁵⁾.

For each CQ, five independent runs were performed per language and per model on different days in November 2025 ⁽⁷⁾. Each model was prompted separately with the English and Japanese versions of each CQ in a new session to prevent context carry-over. For ChatGPT, the “reference saved memories” option was disabled to prevent carry-over. The prompt required each model to provide a recommendation, recommendation strength, evidence level, and supporting references with links, using the uploaded GRADE guidelines ^{(8), (9)} and prespecified literature sources. The full prompt is provided in Supporting Information (Text S1).

Measures

The evaluation included four metrics: (1) correctness, defined as agreement with the guideline’s recommendation; (2) grade of recommendation concordance; (3) evidence-level concordance; and (4) reference accuracy, defined as the validity and relevance of cited literature.

For metric 1, correctness was scored on a five-point Likert scale (1–5) by a board-certified plastic surgeon (K.I.) ⁽¹⁰⁾. For metrics 2 and 3, the LLM output was classified as correct when it matched the guideline; when it differed, it was further categorized as overestimation (stronger recommendation grade or higher evidence level than the guideline) or underestimation (weaker recommendation grade or lower evidence level). For metric 4, reference accuracy, a response was judged correct when a proposed reference corresponded to a citation included in the guideline. In addition, we recorded whether the guideline article itself was cited by the LLMs. For link verification, all cited web links were checked and classified as valid when they resolved to the intended citation; otherwise, they were categorized as incorrect links.

To address repeated-output dependence, we evaluated consistency across the five runs for each CQ-model-language combination and performed a 32-unit aggregated sensitivity analysis using the median correctness score and majority classifications for recommendation-grade and evidence-level concordance.

Statistical analysis

Correctness (metric 1) and reference-related measures (metric 4) were summarized as median and interquartile ranges (IQRs) by model and language and compared using the Wilcoxon rank-sum test. Metrics 2 and 3 were summarized as counts and percentages in three categories (correct, overestimation, underestimation); for hypothesis testing, proportions of correct responses were compared using Fisher’s exact test, with overestimation and underestimation combined as incorrect. Analyses were performed using JMP version 18.2.1 (SAS Institute Inc., Cary, NC, USA).

Results

A total of 160 responses were evaluated (Table 2). Overall correctness was high (median 5 [IQR 4-5]); 159/160 (99.4%) responses scored ≥4, and 123/160 (76.9%) scored 5. Median correctness was 5 across groups, with no significant differences by model or language.

For recommendation grade (metric 2), 92/160 (57.5%) responses matched the guideline, 68/160 (42.5%) overestimated it, no underestimations occurred, and concordance did not differ by model or language.

For evidence level (metric 3), concordance was lower (40/160 [25.0%] correct; 110/160 [68.8%] overestimation; 10/160 [6.3%] underestimation) and did not differ by model or language.

For references (metric 4), the median number of citations per response was 6.5 (IQR 4-9); ChatGPT 5.1 Thinking cited more references than Claude Opus 4.1 in both languages (both p < 0.001). Median reference overlap with the guideline was 25.0% (IQR 0%-44.4%), with no significant differences by model or language. Link validity was higher for ChatGPT 5.1 Thinking than Claude Opus 4.1 in both languages (both p < 0.001). Neither model cited the guideline article itself. No Ichushi-Web–indexed references were cited by either model; however, ChatGPT 5.1 Thinking included Cochrane reviews a total of four times for Japanese prompts and twice for English prompts. When links were incorrect, they resolved to unrelated PubMed-indexed articles.

Reproducibility and aggregated sensitivity analyses are summarized in Supporting Information (Supplementary Tables 1 and 2). A representative example comparing an LLM-generated response with the guideline is provided in Supporting Information (Supplementary Table 3).

Discussion

In this guideline-based evaluation of CQs on vascular anomalies, both LLMs produced recommendations that were largely concordant with the guideline, with consistently high correctness scores across languages and no significant differences between models. These findings suggest that, for common management questions in this domain, contemporary LLMs can generate clinically plausible, guideline-aligned summaries that may be useful for rapid information support.

However, agreement with the guideline was substantially lower for the strength of recommendation and especially for the level of evidence, with frequent overestimation that could create unwarranted confidence without expert review. Reference-related performance also differed by model: ChatGPT 5.1 Thinking provided more citations and higher link validity; however, guideline reference overlap remained modest for both models, indicating that “plausible” citations may not reliably reflect the guideline evidence base ⁽¹¹⁾. Therefore, LLMs should be used only as adjuncts in vascular anomaly care, with verification of recommendations, grades, and references against primary guidelines and source articles.

The reproducibility and aggregated analyses suggested that the low evidence-level concordance was not solely attributable to treating repeated outputs as independent observations. The absence of Ichushi-Web–indexed references was also notable: 19 of 156 guideline references for the selected CQs were indexed in Ichushi-Web; however, neither model cited them. This may reflect limited retrieval or prioritization of Japanese-language domestic evidence by general-purpose LLMs.

This study has limitations. Only eight CQs from a single guideline were tested; however, the a priori selection was based on their inclusion in both the 2017 and 2022 guidelines, supporting their continued clinical relevance. Therefore, the results should not be interpreted as a comprehensive evaluation of all CQs in either guideline. Responses were generated under a fixed prompt and period, and performance may vary by model architecture, training data, prompt structure, input method, language, and future model versions. Correctness was assessed by a single expert without formal inter-rater reliability testing.

Overall, LLM outputs in vascular anomalies were generally consistent with guideline recommendations, but they should not substitute for guideline appraisal when determining evidence strength or selecting supporting references.

Article Information

Author Contributions

Kosuke Ishikawa: Conceptualization, Formal analysis, Investigation, Methodology, Writing―original draft; Yuki Sasaki: Data curation, Formal analysis, Investigation; Masahiro Hojo: Formal analysis, Investigation, Validation; Dongkyung Seo: Data curation, Formal analysis, Validation; Shunichi Oide: Formal analysis, Investigation, Visualization; and Taku Maeda: Project administration, Supervision, Writing―review and editing. All authors reviewed and approved the final manuscript.

Conflicts of Interest

None

Approval by Institutional Review Board

Ethics approval was not required because this study did not involve human participants, patient-identifiable data, or intervention.

Declaration of Generative AI and AI-assisted Technologies in the Writing Process

Language and readability of the manuscript were improved using ChatGPT 5.1 Thinking, and the authors have reviewed the content and assume full responsibility for the final manuscript.

References

1. 1.

   Douma H, McNamara C, Bakola M, et al. Leveraging ChatGPT to strengthen pediatric healthcare systems: a systematic review. Eur J Pediatr. 2025;184(8):478.

   • View Article
   • PubMed

2. 2.

   Uzun Bektas A, Bora B, Ünsal E. Comparative evaluation of ChatGPT and LLaMA for reliability, quality, and accuracy in familial Mediterranean fever. Eur J Pediatr. 2025;184(8):491.

3. 3.

   MacNevin W, Dawe N, Harkness L, et al. Evaluation of ChatGPT’s performance on answering pediatric urology questions based on association guidelines. Can Urol Assoc J. 2025;19(11):E362-7.

   • View Article
   • PubMed

4. 4.

   Adams DM, Brandão LR, Peterman CM, et al. Vascular anomaly cases for the pediatric hematologist oncologists-An interdisciplinary review. Pediatr Blood Cancer. 2018;65(1):e26716.

5. 5.

   Mimura H, Akita S, Fujino A, et al. Japanese clinical practice guidelines for vascular anomalies 2017. Pediatr Int. 2020;62(3):257-304.

   • View Article
   • PubMed

6. 6.

   Kinoshita Y, Ishikawa K, Akita S, et al. Japanese clinical practice guidelines for vascular tumors, vascular malformations, lymphatic malformations, and lymphangiomatosis 2022. Surg Today. 2026;56(5):567-672.

   • View Article
   • PubMed

7. 7.

   Shiraishi M, Tsuruda S, Tomioka Y, et al. Advancement of generative pre-trained transformer chatbots in answering clinical questions in the practical rhinoplasty guideline. Aesthet Plast Surg. 2025;49(7):1874-80.

8. 8.

   Balshem H, Helfand M, Schünemann HJ, et al. GRADE guidelines: 3. Rating the quality of evidence. J Clin Epidemiol. 2011;64(4):401-6.

   • View Article
   • PubMed

9. 9.

   Andrews J, Guyatt G, Oxman AD, et al. GRADE guidelines: 14. Going from evidence to recommendations: the significance and presentation of recommendations. J Clin Epidemiol. 2013;66(7):719-25.

10. 10.

    Clerici CA, Bernasconi A, Lasalvia P, et al. Being diagnosed with a rhabdomyosarcoma in the era of artificial intelligence: whom can we trust? Pediatr Blood Cancer. 2024;71(11):e31256.

    • View Article
    • PubMed

11. 11.

    Walters WH, Wilder EI. Fabrication and errors in the bibliographic citations generated by ChatGPT. Sci Rep. 2023;13(1):14045.

Supplement