BrainXAI's Substack

BrainX Waves: The Newsletter of BrainX Community

BrainXAI — Mon, 31 Aug 2026 11:41:07 GMT

Learn

Does Healthcare AI Always Need the Biggest Model?

The foundation model era has created a new challenge for healthcare: choosing the right language model. Accuracy remains important, but it is no longer the only criterion that matters. Cost, computational efficiency, ease of deployment, scalability and data privacy are becoming equally important considerations. Large closed-source models may deliver strong performance, yet dependence on external APIs can raise concerns around cost and sensitive clinical data. Open-source models offer greater control, but often require substantial infrastructure. Small Language Models (SLMs), such as Microsoft’s Phi family, promise a compelling middle ground—smaller, more efficient models that may be easier to deploy locally and operate with fewer resources.

A study by researchers at BrainXAI ReSearch, presented at IEEE ICHI 2026, puts that promise to the test. In healthcare-specific tasks including question answering and summarization, Phi-family SLMs generally underperformed state-of-the-art large language models. The most current Phi-4 model came closest to matching performance of LLMs. Healthcare pre-trained Phi-3.5 based models underperformed Phi-4 and LLMs. For high-stakes clinical reasoning, larger models may still hold an advantage. For privacy-sensitive, resource-constrained or narrowly defined workflows, smaller models may offer a practical alternative but need a lot more research and evaluation.

Reference

Awasthi, R., Shree, S., Bhattad, A.,et al. (2026). A Comprehensive Evaluation of the Phi Family of Small Language Models in Healthcare. Paper presented at the 2026 IEEE 14th International Conference on Healthcare Informatics (ICHI)

Learn More

Connect

Watch the recording of BrainX Community Live Webinar on Evidence synthesis using AI, as we explore how artificial intelligence is reshaping evidence synthesis, clinical question answering, and health professions education.

Presented by Dr. Ruthvik Edara, BrainXAI ReSearch, this interactive session examined four timely publications evaluating AI tools across real-world clinical queries, complex medical subspecialty scenarios, point-of-care decision support, and generative AI in health professions education.

This was followed by discussion with Dr. Robert Hoyt MD FACP on key related questions including how to critically evaluate what AI can do today, where it still falls short, and what comes next for trustworthy evidence synthesis in healthcare.

Related publications:

Connect More

Datasets (Open source)

General

PaJR Health Database

Includes more than 200 patient journeys with day-by-day diaries and case reports, 500+ community records, and 5000+ hospital records.

General/Generative AI/LLM

VITA HealthBench Benchmark and evaluation dataset

This dataset contains the full inputs, outputs, and grading results from a HealthBench-derived benchmark evaluation comparing five AI systems — GPT-5.5, VITA, Claude Opus-4.8, Gemini-3.5, and Grok-4.3 (all evaluated in non-thinking mode) — on 500 health/clinical questions. Each system’s response was graded against per-question rubric criteria by an LLM judge (DeepSeek-V4-Pro, via OpenRouter).

VITA HealthBench evaluation dataset: model responses and rubric-scoring outputs for 4,023 English-language questions

This dataset contains the complete set of responses generated by VITA, a retrieval-augmented generation system for context-specific clinical knowledge retrieval in India, across 4,023 English-language questions from HealthBench (OpenAI). It accompanies the manuscript “A corpus-specific clinical RAG system matches or outperforms frontier LLMs on HealthBench.”

Generative AI/LLM

MedVAL-Bench

MedVAL-Bench is a dataset for fine-tuning/evaluating the ability of language models to assess AI-generated medical text outputs (not their ability to generate input → output). A generator LM produces an output, and MedVAL assesses the output’s factual consistency with the input, while assigning a risk grade and determining its safety for deployment. 840 AI-generated medical outputs across 6 diverse medical tasks, annotated by 12 physicians using a clinically meaningful risk grading and error taxonomy.

Cardiology/General

Google Heart Rate Measurement Study Dataset (the “HR Dataset”)

The HR Dataset includes:Skin Tone Information, Facial Video Recording, Heart Rate.Facial video recordings captured during laboratory studies include video of study participants’ full faces. Facial video recordings captured under “free-living” conditions consist of de-identified patches from study participants’ forehead regions (downsampled to 10×5 pixel patches) to preserve participant privacy. The dataset additionally includes a heart rate estimation model (“PHRM-mini”) trained on the dataset.

More Data

Conferences

Additional BXC-featured publications

Generative AI/LLM

Development and Evaluation of Multi-Modal Universal CPR AI Assistance Response Engine (U-CARE)

Atreja, A., & Mathur, P.

Critical Care

Clinicians vs. artificial intelligence in patient outcome prediction in the intensive care unit

Kuang, C., Valderrama, C.E., Stelfox, H.T., et al.

Generative AI/LLM

A S.C.O.R.E. framework for evaluating open-ended responses from large language models in healthcare

Ting Fang Tan , Kabilan Elangovan , Jasmine Ong , et al.

Generative AI/Imaging/Radiology/LLM

Interpretable agentic AI system with localized reasoning for radiology

Chen, W., Dong, Y., Ding, Z. et al.

Join and follow the BrainX community!

Webpage: https://brainxai.org/

Newsletter: https://brainxai.substack.com/subscribe

LinkedIn: https://www.linkedin.com/groups/13599549/

Youtube: https://www.youtube.com/channel/UCua5EiLL6I29hpNrJsdv1rg

Share BrainXAI's Substack

BrainX Waves: The Newsletter of BrainX Community

BrainXAI — Thu, 30 Jul 2026 13:23:42 GMT

Subscribe now

Learn

Harnessing Generative AI for Responsible Evidence Synthesis in Health Professions Education

Generative artificial intelligence is creating a new opportunity to transform evidence synthesis from a slow, largely manual process into a more scalable and interactive component of healthcare education. A recent scoping review of 517 publications found rapidly expanding use of generative AI across medicine, nursing, dentistry, pharmacy and allied health education, particularly for generating educational materials and evaluating assessment items.¹ This growing body of work suggests that generative AI could help educators rapidly summarize literature, create case-based learning resources, tailor content to learner needs and support more continuous updating of curricula as evidence evolves. By reducing the time required to find, organize and translate information, these tools may enable educators and learners to devote greater attention to critical appraisal, clinical reasoning and the application of evidence rather than information retrieval alone.

The emerging evidence also shows that performance is highly dependent on the tool, task and evaluation setting. General-purpose frontier models outperformed specialized clinical tools on standardized benchmarks and selected physician questions, whereas a separate specialty-matched evaluation of real point-of-care queries found that a clinically customized platform was preferred by physicians for accuracy, usefulness, source quality, verifiability and completeness.^2,4A pilot study of complex subspecialty scenarios further reported modest accuracy and variable repeatability, despite improved performance with a deeper evidence-search mode.³ These apparently mixed findings are not a setback; rather, they demonstrate that evidence synthesis cannot be judged by a single benchmark. Larger independent studies, transparent reporting, representative clinical questions, expert evaluation and outcome-focused educational research are now needed. Together, these studies point toward a productive future in which generative AI complements,not replaces, human expertise, accelerating access to evidence while strengthening the role of educators and clinicians as critical interpreters and trusted validators.

References

Learn More

Connect

Join us for the next BrainXCommunity Live! Journal Club style Webinar , as we explore how artificial intelligence is reshaping evidence synthesis, clinical question answering, and health professions education.

Evidence Synthesis in the Age of AI: What Works, What Doesn’t, What’s Next

Presented by Dr. Ruthvik Edara, BrainXAI ReSearch, this interactive session will examine four timely publications evaluating AI tools across real-world clinical queries, complex medical subspecialty scenarios, point-of-care decision support, and generative AI in health professions education.

📅 Wednesday, August 26, 2026

🕚 11:00 AM–12:00 PM ET

Join the discussion and help us critically evaluate what AI can do today, where it still falls short, and what comes next for trustworthy evidence synthesis in healthcare.

Program Description:

1. Introductions

2. Journal Club: Dr. Sai Ruthvik Edara, Researcher, BrainXAI ReSearch, BrainX LLC., will present the following publications:

- The accuracy and repeatability of OpenEvidence on complex medical subspecialty scenarios: a pilot study

- General-purpose chatbots outperform clinical AI tools on physicians’ real-world questions

- Expert Evaluation of Clinical AI Tools on Real Point-of-Care Clinical Queries

- Generative AI Research in Health Professions Education: A Scoping Review

- Panel discussion with Q&A

Connect More

Datasets (Open source)

General

PMC-Patients

A novel dataset of patient summaries and relations called PMC-Patients to benchmark two ReCDS tasks: Patient-to-Article Retrieval (ReCDS-PAR) and Patient-to-Patient Retrieval (ReCDS-PPR). Specifically, we extract patient summaries from PubMed Central articles using simple heuristics and utilize the PubMed citation graph to define patient-article relevance and patient-patient similarity. PMC-Patients contains 167k patient summaries with 3.1M patient-article relevance annotations and 293k patient-patient similarity annotations, which is the largest-scale resource for ReCDS and also one of the largest patient collections.

Surgery

GR00T-H : Surgical robot data

GR00T-H post-trains GR00T N1.6 on surgical robot data from multiple institutions and robot platforms simultaneously. The core challenge is that each institution records data differently — different robots, coordinate conventions, frame rates, camera setups, and state/action representations. GR00T-H solves this by defining per-embodiment modality configs that convert each dataset into a common representation (REL_XYZ_ROT6D for EEF poses) while preserving robot-specific details like clutch handling and motion scaling. The dataset contains 778 hours of real and synthetic procedure episodes.

Emergency Medicine

OpenPOCUS: ED Lung US dataset

An open-source dataset of lung POCUS images derived from a multi-center study involving 226 adult patients presenting to emergency departments with respiratory symptoms. Images were acquired using a standardized scanning protocol (12-zone or modified 8-zone) with various POCUS devices. Videos were preprocessed to remove identifiers, and frames were extracted and standardized to 512×512 pixels using letterboxing to maintain aspect ratios. The dataset contains 1,871 video clips comprising 324,027 frames extracted and standardized to 512×512 pixels. Half of the participants (50%) had COVID-19 pneumonia.

More Data

Conferences

Additional BXC-featured publications

Course

HelloAI Modules

GE Healthcare

Generative AI/LLM

Addressing benchmarking gaps in large language models for health and medicine with dynamic red-teaming

Pan, J., Jian, B., Hager, P. et al.

Book

Artificial Intelligence in Medicine and Surgery – An Exploration of Current Trends, Potential Opportunities, and Evolving Threats, Volume 3

Stanislaw P. Stawicki, Thomas R. Wojda, Andries Engelbrecht

Imaging/Radiology

Health system learning enables generalist neuroimaging models

Kondepudi, A., Rao, A., Zhao, C. et al.

Generative AI/LLM

Evaluating the robustness and readiness of large frontier models in health AI applications

Gu, Y., Fu, J., Liu, X. et al.

Join and follow the BrainX community!

Webpage: https://brainxai.org/

Newsletter: https://brainxai.substack.com/subscribe

LinkedIn: https://www.linkedin.com/groups/13599549/

Youtube: https://www.youtube.com/channel/UCua5EiLL6I29hpNrJsdv1rg

Share BrainXAI's Substack

BrainX Waves: The Newsletter of BrainX Community

BrainXAI — Tue, 30 Jun 2026 16:31:00 GMT

Subscribe now

Learn

Responsible AI in the Era of Healthcare Agents

Source: How Responsible AI Changes in the Agent Era. Turing Post. ²

Artificial intelligence in healthcare is entering an agentic phase. Earlier clinical AI systems largely produced a prediction, classification, summary, or answer for a human to interpret. AI agents are different: they can pursue goals, call tools, retrieve evidence, draft documentation, interact with electronic health records, communicate with patients, and coordinate multistep workflows. This ability to act changes the safety problem. Responsible AI can no longer remain a set of principles applied before deployment or an output-review exercise performed after the fact. In healthcare, Responsible AI must become the operating infrastructure through which agents are allowed to act.^1,2

Responsible AI is commonly defined as the frameworks, principles, and practices that support ethical, safe, transparent, accountable, and fair AI development and deployment.¹ That definition remains necessary, but agentic healthcare systems require a more operational interpretation. When an AI system can trigger a workflow, influence a clinical decision, or alter the information presented to a patient or clinician, governance must move closer to the moment of action. The relevant question is not only, ‘Was the model accurate during validation?’ It is also, ‘What could the agent access, what was it permitted to do, how was risk recognized, who supervised it, and how would harm be detected and corrected?’

Scalable evaluation is another urgent need. LLM-as-a-judge methods are increasingly used to assess AI outputs because expert review is expensive, slow, and difficult to standardize. In healthcare, however, such methods must be used cautiously. The MedJUDGE scoping review found growing use of LLM-as-a-judge approaches but important gaps in validation rigor, bias assessment, temporal stability, and patient-context evaluation.³ Human-in-the-loop frameworks such as GamELY are important because they combine scalable LLM-based assessment with structured expert oversight.⁴ Emerging work on physician-like evaluation and theory-of-mind imitation asks whether LLMs can approximate clinician values and reasoning, but this should be viewed as an evaluation aid, not a replacement for clinical accountability.⁵

The promise of healthcare agents is substantial.Responsible AI in the agent era is not a policy document, a model card, or a retrospective benchmark. It is the clinical, technical, and ethical infrastructure that determines what agents can do, when they can do it, who remains accountable, and how health systems learn from every action. The future of healthcare AI will not be defined by whether agents can act, but by whether we can ensure that they act responsibly.

References

1. Stanford Institute for Human-Centered Artificial Intelligence. What is Responsible AI? Accessed June 29, 2026. https://hai.stanford.edu/ai-definitions/what-is-responsible-ai

2. Se K. How Responsible AI Changes in the Agent Era. Turing Post. Published June 20, 2026. Accessed June 29, 2026. https://www.turingpost.com/p/how-responsible-ai-changes-in-the-agent-era

3. Li C, Akhtar Z, Kwak M, et al. A Scoping Review of LLM-as-a-Judge in Healthcare and the MedJUDGE Framework. arXiv. Published April 30, 2026. Accessed June 29, 2026. https://arxiv.org/abs/2604.25933

4. Awasthi R, et al. GamELY: Human-in-the-Loop Framework for Scaling Human Evaluation of LLMs in Healthcare. ACM. Published December 10, 2025. Accessed June 29, 2026. https://dl.acm.org/doi/10.1145/3765612.3767224

5. Awasthi R, Mishra S, Raghu C, et al. Theory of Mind Imitation by LLMs for Physician-Like Human Evaluation. medRxiv. Published March 5, 2025. Accessed June 29, 2026. https://www.medrxiv.org/content/10.1101/2025.03.01.25323142.full

Learn More

Connect

Watch the recording of our recent webinar on AI Agent BrainX Community Live! June, 2026; featuring Chaitanya Mamillapalli, MD, MRCP, FAPCR, Chief of Endocrinology, Springfield Clinic, IL, and Ahmed Ibrahim El-Sonbaty, MD, Consultant, Cardiac Anaesthesia, King Fahd Armed Forces Hospital, Jeddah.

In this session, they showcase their work on developing healthcare education platforms such as MedEx, Anaesth-AI, and CCA-Pro. These platforms leverage Generative AI to provide near real time assessment and feedback. They provide clinician decision support tools such as medical calculators, guideline based response to crisis, journal club sessions, medical QA, and more.

Program Description:

– Introductions

– “Can AI Teach Clinical Judgment? Socratic Dialogue and Adaptive Learning in the Age of AI“: Chaitanya Mamillapalli, MD, MRCP, FAPCR

– “From OR to Code: Designing Point-of-Care AI Tools That Clinicians Actually Use“: Ahmed Ibrahim El-Sonbaty, MD

– Q & A

Connect More

Datasets (Open source)

Collection of datasets curated by EkaCare for development and evaluation of LLMs on healthcare domain. Listed below are a few:

– The Eka Medical ASR Evaluation Dataset enables comprehensive evaluation of automatic speech recognition systems designed to transcribe medical speech into accurate text—a fundamental component of any medical scribe system.

– The Eka Medical Records Parsing Dataset empowers evaluation of AI systems designed to extract structured information from unstructured medical documents, enabling true digitisation of healthcare data while maintaining clinical accuracy.

– The Eka Structured Clinical Note Generation Dataset facilitates evaluation of medical scribe systems capable of transforming transcribed medical conversations into structured, entity-level medical records. Comprising over 156 meticulously transcribed medical conversations between EkaCare’s internal doctors and team members serving as patients, this dataset captures diverse clinical interaction patterns.

More Data

Conferences

Additional BXC-featured publications

Generative AI/LLM/Neurology

Orchestrator multi-agent clinical decision support system for secondary headache diagnosis in primary care

Xizhi Wu, Hang Zhang, Yanshan Wang, et al.

General/Generative AI/LLM

Towards Conversational AI for Disease Management

Liévin, V., Palepu, A., Weng, WH. et al.

General/Generative AI/LLM

Towards autonomous medical artificial intelligence agents

Ferber, D., Hilgers, L., Höper, C. et al.

Generative AI/LLM

General-purpose large language models outperform specialized clinical AI tools on medical benchmarks

Vishwanath, K., Alyakin, A., Ghosh, M. et al.

Join and follow the BrainX community!

Webpage: https://brainxai.org/

Newsletter: https://brainxai.substack.com/subscribe

LinkedIn: https://www.linkedin.com/groups/13599549/

Youtube: https://www.youtube.com/channel/UCua5EiLL6I29hpNrJsdv1rg

Share BrainXAI's Substack

BrainX Waves: The Newsletter of BrainX Community

BrainXAI — Mon, 01 Jun 2026 00:40:31 GMT

Subscribe now

Learn

LLM-as-a-Judge in Healthcare: Can LLMs evaluate like a physician?

Large language models (LLMs) are increasingly being used not only as generators of clinical text but also as evaluators of AI-generated healthcare outputs. This emerging paradigm, often termed “LLM-as-a-judge”, offers a scalable alternative to costly, slow and sometimes inconsistent human evaluations. Early work in general natural language generation, including G-Eval, showed that GPT-4-based evaluators can achieve stronger alignment with human judgements than traditional reference-based metrics but also revealed risks of bias toward LLM-generated text. ¹ MedJUDGE synthesises the rapidly expanding literature and highlights that, despite broad adoption of LLM judges for benchmarking and evaluation, validation remains thin, fairness testing is uncommon, temporal robustness is largely unexamined and real-world deployment is rare. ² GamELY extends this discussion by proposing a human-in-the-loop framework in which LLM-based scoring is paired with targeted clinician auditing, preserving scalability while retaining human oversight for clinically consequential outputs. ³

Healthcare evaluation, however, cannot be reduced to linguistic fluency, factual overlap or generic preference alignment. Clinical judgement depends on context, uncertainty, patient safety, disease severity, medicolegal risk, feasibility, equity and the values embedded in shared decision-making. For this reason, specialized clinician expertise remains indispensable for defining evaluation rubrics, adjudicating ambiguous cases, detecting subtle harms and determining when apparent correctness may still be clinically unsafe. LLMs may approximate physician-like evaluation in constrained tasks, especially when criteria are explicit and risk is low, but they should not yet be assumed to evaluate like physicians or to fully match physician and patient values. Evidence from healthcare-specific judge frameworks suggests that LLMs can improve scale and consistency yet require risk-stratified governance, bias assessment, external validation and human-in-the-loop review. ²^,³ Moreover, recent work on value pluralism in clinical AI suggests that models may fall within the range of physician variation while still exhibiting stable, systematic value preferences, including potential underweighting of patient autonomy. ⁴ Thus, LLM judges should be viewed not as replacements for clinicians but as calibration tools: useful for triage, benchmarking and surveillance, while clinician experts remain the final arbiters of safety, appropriateness and human values in healthcare AI evaluation.

References

1. Liu Y, Iter D, Xu Y, Wang S, Xu R, Zhu C. G-Eval: NLG evaluation using GPT-4 with better human alignment. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics; 2023:2511-2522. doi:10.18653/v1/2023.emnlp-main.153

2. Li C, Akhtar Z, Kwak M, et al. A scoping review of LLM-as-a-judge in healthcare and the MedJUDGE framework. arXiv. Preprint posted April 3, 2026. doi:10.48550/arXiv.2604.25933

3. Awasthi R, Singh N, Mishra S, et al. GamELY: human-in-the-loop framework for scaling human evaluation of LLMs in healthcare. In: Proceedings of the 16th ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics. Association for Computing Machinery; 2025:Article 48, 1-6. doi:10.1145/3765612.3767224

4. Chandak P, Alkin V, Wu D, et al. What does the AI doctor value? Auditing pluralism in the clinical ethics of language models. arXiv. Preprint posted May 18, 2026. doi:10.48550/arXiv.2605.18738

Learn More

Connect

Watch the recording of our recent webinar on AI Agent BrainX Community Live! April 29, 2026; AI Agents for complex critically ill patients. In this webinar, the European Society of Intensive Care Medicine’s datathon award-winning team of Preethi Krishnan, Sai Deekshita Chaganty, Aryan Bhattacharya, Chao-Ping Wu, and Anirban Bhattacharyya present their work on: “Multi-agent Respiratory Orchestrator: A Safety-First AI Framework for ICU Mechanical Ventilation”. Learn about the hypotheses, research, and agent development work this team has done to orchestrate a safety-first framework for the management of ICU patients with respiratory illness.

Connect More

Datasets (Open source)

Generative AI/Surgery

MedVidBench

MedVidBench is a test benchmark for evaluating Video Large Language Models (VLMs) on medical and surgical video understanding. It covers 8 diverse tasks (with GPT-4 and Gemini variants for captioning tasks) across 8 surgical-video datasets. 6,245 test samples across 8 tasks (11 task variants). 8 surgical-video source datasets: AVOS, CholecT50, CholecTrack20, Cholec80_CVS, CoPESD, EgoSurgery, JIGSAWS, NurViD. 103,742 video frames (~18 GB) with per-sample FPS and temporal metadata. Bounding-box annotations for 306 region-caption samples.

Pathology

Haiku-demo-data

Demo data for 4 tissue regions (959 registered CODEX + H&E + text patches).

General

The All of Us Research Program’s wearables dataset

This dataset contains Fitbit data from more than 59,000 participants spanning 14 years with more than 39 million step observations and 31 million sleep observations. Nearly half (46%) of participants with Fitbit data also contributed electronic health records, physical measurements, genomics and survey data.

General/Generative AI

NHSEDataScience Synthetic Clinical Notes dataset

This dataset contains data generated from a pipeline using GPT-4o as the LLM. There are 70 patients, each with approximately 20 – 50 synthetic clinical notes. 50 patients are adults; 20 are paediatrics.

Generative AI/Surgery

LEMON: A Large Endoscopic MONocular Dataset and Foundation Model for Perception in Surgical Settings

LEMON is a dataset of over 4K surgical high-resolution videos totalling 938 hours from 35 diverse surgical procedure types. Each video is annotated for multi-label classification, indicating the surgical procedures carried out in the video, and for binary classification, indicating if it is robotic or non-robotic. The dataset’s annotations can be found in labels.json.

Critical Care/Pediatrics

SafeICU Clinical Database

SafeICU is a freely available database comprising de-identified health-related data from over 2,500 pediatric patients admitted to the pediatric intensive care unit at AIIMS (All India Institute of Medical Sciences), New Delhi, between 2015 and 2025. The AIIMS hosts one of the country’s leading pediatric intensive care units, an independent 8-bed unit that delivers advanced critical care to children with life-threatening illnesses. The SafeICU database includes information such as demographics, bedside vital signs (recorded every 15 seconds), laboratory test results, medications, caregiver notes, microbiology, and mortality. The unit cares for a broad clinical spectrum, with the most frequent reasons for admission being severe respiratory illnesses (such as pneumonia and respiratory failure), sepsis and septic shock, hypertensive crises, congenital heart disease, and acute kidney injury.

More Data

Conferences

Additional BXC-featured publications

Generative AI/LLM

Automation Bias in Large Language Model–Assisted Diagnostic Reasoning among Physicians Trained in AI Literacy — A Randomized Clinical Trial

Ihsan Ayyub Qazi, Ayesha Ali, Asad Ullah Khawaja, Muhammad Junaid Akhtar, Ali Zafar Sheikh, Muhammad Hamad Alizai

General

Clinical document metadata extraction: A scoping review

Kurt Miller, Qiuhao Lu, William Hersh , Kirk Roberts , Steven Bedrick , Andrew Wen , Hongfang Liu

Generative AI/LLM

Public use of a generalist LLM chatbot for health queries

Costa-Gomes, B., Tolmachev, P., Taysom, E. et al.

Course

AI Analytics (Data Literacy for Clinicians Course)

Robert Hoyt

Join and follow the BrainX community!

Webpage: https://brainxai.org/

Newsletter: https://brainxai.substack.com/subscribe

LinkedIn: https://www.linkedin.com/groups/13599549/

Youtube: https://www.youtube.com/channel/UCua5EiLL6I29hpNrJsdv1rg

Share BrainXAI's Substack

BrainX Waves: The Newsletter of BrainX Community

BrainXAI — Fri, 24 Apr 2026 07:50:35 GMT

Subscribe now

Learn

Patients, Prompts, and Pitfalls: Navigating the Promise and Perils of Generative AI in Healthcare

The rapid adoption of generative artificial intelligence (AI) is transforming how patients seek and interpret health information, with growing evidence that AI is becoming a first point of contact for medical queries. In an analysis of over 500,000 de-identified conversations with Microsoft Copilot, approximately 40% of queries focused on symptoms, medical conditions, and treatments, while 10.9% specifically sought interpretation of symptoms or diagnostic results (Microsoft AI, 2026). A large-scale study in Nature npj Health Systems found that nearly one in five conversations involved personal symptom assessment or condition discussion, with additional queries centered on specific treatments, suggesting that this may underestimate true personal health intent (Nature npj Health Systems, 2026). Notably, one in seven personal queries concerned another individual, indicating an emerging caregiving role.

However, the reliability of AI-generated health information remains a critical concern. A systematic audit published in BMJ Open evaluated five major chatbots, including ChatGPT, Gemini, Meta AI, Grok, and DeepSeek, and found that 49.6% of responses were problematic, including 30% somewhat problematic and 19.6% highly problematic (BMJ Open, 2026). Performance varied by topic, with stronger results in vaccines and cancer, but weaker performance in misinformation-prone areas such as nutrition and athletic performance. Chatbots consistently conveyed responses with high confidence despite inaccuracies, rarely refused to answer (0.8%), and demonstrated poor reference quality (median completeness 40%), with hallucinated citations limiting reliability. Additionally, readability levels were consistently above average patient comprehension, further constraining usability.

These findings highlight a critical gap between accessibility and trustworthiness in AI-enabled healthcare. Regulatory frameworks such as the European Union AI Act and GDPR provide patients with rights to explanations. (JMIR, 2026). Technical complexity, clinical nuance, and health literacy barriers further limit patient understanding, even when explanations are provided. Addressing this challenge requires a shift from compliance-driven transparency to outcome-oriented communication—ensuring that AI outputs are not only accurate but also interpretable and actionable. Without such efforts, the continued expansion of generative AI risks amplifying misinformation while simultaneously redefining patient engagement with healthcare systems.

Learn More

Connect

Join us for the upcoming webinar on AI Agent BrainX Community Live! April 29, 2026; 11 am – noon ET via Zoom: AI Agents for complex critically ill patients.

Registration link: https://us02web.zoom.us/meeting/register/wJpEaw-fSzmSJ0uhSoOWOQ#/registration

Program Description:

– Introduction

– “Multi-agent Respiratory Orchestrator: A Safety-First AI Framework for ICU Mechanical Ventilation”: Preethi Krishnan, Sai Deekshita Chaganty, Aryan Bhattacharya, Chao-Ping Wu, and Anirban Bhattacharyya

– Q and A

Connect More

Datasets (Open source)

EchoNext: A Dataset for Detecting Echocardiogram-Confirmed Structural Heart Disease from ECGs

This dataset contains a de-identified collection of 100,000 12-lead electrocardiograms (ECGs) with paired structural heart disease (SHD) labels derived from echocardiography, collected at Columbia University Irving Medical Center. Each ECG is provided with raw waveform data sampled at 250 Hz across all 12 leads, along with accompanying demographic and ECG-specific tabular metadata, including age, sex, heart rate, PR interval, QRS duration, and corrected QT interval. Each ECG is annotated with a binary label indicating the presence or absence of structural heart disease based on echocardiographic findings. This dataset was developed as part of the creation of the Columbia Mini-Model, a lightweight deep learning model for SHD detection from ECGs.

PGC Psychiatric GWAS Dataset

1 billion rows of genome-wide association study (GWAS) summary statistics from the Psychiatric Genomics Consortium (PGC). Data of 12 psychiatric disorders in this dataset: ADHD, Anxiety, OCD, Mood disorders, PTSD, Schizophrenia, and more…

SurgLaVi Dataset

SurgLaVi is the largest and most diverse surgical vision–language dataset to date, comprising nearly 240k clip–caption pairs from more than 200 procedures, and comprising hierarchical levels at coarse-, mid-, and fine-level. At the core of SurgLaVi lies a fully automated pipeline that systematically generates fine-grained transcriptions of surgical videos and segments them into coherent procedural units. Within this framework, the resulting captions are enriched with contextual detail, producing annotations that are both semantically rich and easy to interpret. To ensure accessibility, the researchers release SurgLaVi-β, an open-source derivative of ~113k clip–caption pairs constructed entirely from public data, which is over four times larger than existing surgical VLP datasets.

More Data

Conferences

Additional BXC-featured publications

General/Generative AI/LLM

A clinical environment simulator for dynamic AI evaluation

Luo, L., Kim, S.E., Zhang, X. et al.

Course

Multiagent Systems in Healthcare

Dr. Ruchi Saxena

Book

Multiagent Systems in Primary Health and Hospital Settings

Dr. Ruchi Saxena

General/Generative AI/LLM

Large Language Model Performance and Clinical Reasoning Tasks

Arya S. Rao; Kaiz P. Esmail; Richard S. Lee, et al.

General/Generative AI/LLM

Public use of a generalist LLM chatbot for health queries

Costa-Gomes, B., Tolmachev, P., Taysom, E. et al.

General/Generative AI

Generative artificial intelligence-driven chatbots and medical misinformation: an accuracy, referencing and readability audit

Nicholas B Tiller, Alessandro R Marcon, Marco Zenone, et al.

Generative AI

Benchmarking large language model-based agent systems for clinical decision tasks

Liu, Y., Carrero, Z.I., Jiang, X. et al.

General/Generative AI

Changes in Clinician Time Expenditure and Visit Quantity With Adoption of Artificial Intelligence–Powered Scribes

Lisa S. Rotenstein; A. Jay Holmgren; Robert Thombley; et al.

Join and follow the BrainX community!

Webpage: https://brainxai.org/

Newsletter: https://brainxai.substack.com/subscribe

LinkedIn: https://www.linkedin.com/groups/13599549/

Youtube: https://www.youtube.com/channel/UCua5EiLL6I29hpNrJsdv1rg

Share BrainXAI's Substack

BrainX Waves: The Newsletter of BrainX Community

BrainXAI — Sun, 29 Mar 2026 18:26:24 GMT

Subscribe now

Learn

Building communities of Clinicians and Data scientists for AI in healthcare

The American Medical Association’s annual Physician Survey on Augmented Intelligence assesses adoption, perceptions, and anticipated impacts of AI in medical practice. The 2026 results found more than four in five physicians (81 percent) use AI in their practices, more than double the 2023 rate (38 percent). Physicians are cautiously optimistic about the promise of AI—including its potential to combat burnout—but they also have concerns about patients using the technology to interpret complicated results without the assistance of a physician.

Education and training haven’t been extensive—respondents want more. Of those receiving any training, only 11% report receiving “a lot of” training from any source. This indicates that current education and training on AI has not been extensive despite some level of prior exposure. Survey respondents prefer training delivered within their existing workflows or alongside related training. Two-thirds selected continuing-education formats such as pre-recorded modules, online courses, or in-person sessions. Many also value hands-on sandbox environments and flexible self-paced modules. Training from AI companies themselves is the least preferred option.

Read the survey here: American Medical Association’s annual Physician Survey on Augmented Intelligence

Datathons are structured learning mediums where clinicians and data scientists from all backgrounds and levels of knowledge can learn about data science and AI. The Indian Society of Critical Care Medicine (ISCCM), in association with Kauvery Group of Hospitals, BrainX LLC., and the Indian Institute of Technology, Madras, conducted the first ever AI in Critical Care Datathon 2026 in India. The two-day datathon brought together critical care physicians and data scientists to work on real-world intensive care unit (ICU) challenges. Participants analysed clinical datasets, including continuous monitoring data, laboratory values, imaging and clinical notes, to develop predictive models for conditions such as sepsis, shock, acute kidney injury, and respiratory failure. Learn more here: ISCCM 2026 Datathon.

Want to join upcoming datathons and learn about AI applications in healthcare in a rich hands-on learning format? Join us for the upcoming Mayo Clinic and the FEPIMCTI Datathon 2026.

Learn more about upcoming AI in healthcare conferences here: https://brainxai.org/conference/

Join BrainX Community to stay up-to-date about such upcoming events: https://brainxai.org/

Learn More

Connect

Watch the recording of our recent BrainX Community Live! March 2026 event themed on State of Healthcare AI Research, featuring Piyush Mathur MD, Sai Ruthvik Edara MBBS, Avneesh Khare MBBS MD, and Anirban Bhattacharyya MD.

The speakers discussed the recently published scoping review on AI in healthcare research, an annual feature publication of BrainXAI Research. Through this presentation and publication one can learn about the key trends in research and publications addressing growing research related to AI in various healthcare specialities, key AI models being used, data trends and the use of foundation models for AI in healthcare research. These trends, which we have been measuring since 2019, show the key focus areas of this research work and foretell applications of AI in healthcare coming to the bedside.

Learn more about the event here: https://brainxai.org/connect/brainx-community-live-march-2026-state-of-healthcare-ai-research/

Read the related publication here: Artificial Intelligence in Healthcare: 2025 Year in Review. Edara, R., A. Khare, A. Atreja, R. Awasthi, B. Highum, N. Hakimzadeh, S. P. Ramachandran, S. Mishra, D. Mahapatra, S. Shree, A. Bhattacharyya, N. Singh, S. Reddy, J. B. Cywinski, A. K. Khanna, K. Maheshwari, F. A. Papay and P. Mathur (2026). medRxiv: 2026.2002.2023.26346888.

Related dataset: Artificial Intelligence in Healthcare: 2025 Year in Review Dataset. Edara, Ruthvik; Atreja, Aarit; Highum, Brianna; Hakimzadeh, Natalia; Prasad Ramachandran, Sai; Khare, Avneesh; et al. (2026). figshare. Dataset. https://doi.org/10.6084/m9.figshare.31395807.v1

Connect More

Datasets (Open source)

SynthVision dataset

Synthetic medical VQA pipeline: 119K images annotated by frontier VLMs, cross-validated at 93% agreement, fine-tuned on 3 model families (2-3B params)

RSNA Screening Mammography Breast Cancer Detection (RSNA-SMBC) Dataset

This dataset is highly enriched to contain 833 cancer examinations, 1188 biopsy-proven benign examinations, and 3897 examinations flagged as abnormal on screening but subsequently determined to be negative (BI-RADS 1) or benign (BI-RADS 2) at diagnostic imaging.

Cardiac Investigations Text dataset

Data consists of clinical test text data (ECGs, CMRs, rest and stress TTEs, ambulatory Holter monitors, CPXs). This study explored the potential of Articulate Medical Intelligence Explorer (AMIE), a large language model-based experimental medical artificial intelligence system, to augment clinical decision-making in this challenging context. We conducted a randomized controlled trial comparing large language model-assisted care with the usual care of complex patients suspected of having a genetic cardiomyopathy, and we curated a real-world dataset of complex cases from a subspecialist cardiology practice.

MIMIC-III-Ext-PPG: A PPG Benchmark Dataset for Cardiorespiratory Analysis

MIMIC-III-Ext-PPG is a large-scale, annotated, and quality-assessed dataset of photoplethysmogram (PPG) segments extracted from the MIMIC-III Waveform Database Matched Subset. It contains 4.9 million non-overlapping 30-second segments from 6,131 critically ill patients and serves as the largest public resource for heart rhythm classification. The dataset contains signals recorded during several heart rhythms: sinus rhythm, atrial fibrillation, atrial flutter, heart block, pacing, and several less common rhythms. In addition, simultaneous electrocardiogram (ECG, lead II), arterial blood pressure (ABP), and respiratory (RESP) signals are provided where available. Additional annotations were derived from these signals, including blood pressure, heart rate, and respiratory rate.

More Data

Conferences

Additional BXC-featured publications

General

Accelerating AI innovation in healthcare: real-world clinical research applications on the Mayo Clinic Platform

Yu, Y., Hu, X., Rajaganapathy, S. et al.

Critical Care/Generative AI

ADVANCING PREDICTIVE ANALYTICS IN CRITICAL CARE: THE ROLE OF TABULAR FOUNDATION MODELS

Naylor, Amanda; Awasthi, Raghav; Aggarwal, Avneep; Bhattacharyya, Anirban; Mishra, Shreya; Mathur, Piyush

General

Factors for Patient Trust and Acceptance of Medical Artificial Intelligence

Ana Bracic; Kayte Spector-Bagdady; Sophie Towle, et al.

General

The landscape of AI implementation in US hospitals

Hwang, YM., Ng, M.Y., Pillai, M. et al.

Journal

International Journal of Artificial Intelligence in Medicine

Editor in Chief: Dr Amit Kumar Dey

Join and follow the BrainX community!

Webpage: https://brainxai.org/

Newsletter: https://brainxai.substack.com/subscribe

LinkedIn: https://www.linkedin.com/groups/13599549/

Youtube: https://www.youtube.com/channel/UCua5EiLL6I29hpNrJsdv1rg

Share BrainXAI's Substack

BrainX Waves: The Newsletter of BrainX Community

BrainXAI — Thu, 26 Feb 2026 09:31:49 GMT

Subscribe now

Learn

Evaluation for implementation of AI in healthcare

Recently published studies demonstrate important issues with the feasibility and usability of some of the AI models. In a recently concluded trial by Kelsheiker, et al., for implementation of an AI stethoscope in routine primary care, the device-based AI did not significantly increase detection of heart failure or increase community-based diagnosis after 12 months of implementation. Although, AI stethoscope use was independently associated with significantly higher detection rates of heart failure, as well as atrial fibrillation and VHD. In another study by Bean, A.M.,et al., the researchers tested whether LLMs can assist members of the public in identifying underlying conditions and choosing a course of action (disposition) in ten medical scenarios. This controlled study with 1,298 participants identified user interactions as a challenge to the deployment of LLMs for medical advice. The researchers also found that standard benchmarks for medical knowledge and simulated patient interactions do not predict the failures we find with human participants.

Evaluation and validation of AI tools is essential to successful implementation in healthcare. Beyond leaderboard evaluation benchmarking and lab-orientated testing using traditional evaluation parameters, AI model testing is rapidly moving to workplace trials, feasibility and usability testing, as demonstrated in the above studies. These trials will provide key information to guide innovation and implementation of AI models in clinical workspace.

Learn More

Connect

Watch the recording of our recent BrainX Community Live! January 2026 event, featuring Fernando Eimbcke, MD, Alejandro Mauro M.D., Jaime de los Hoyos Moreno M.D., from Clínica Alemana de Santiago, Chile.

The speakers discussed promises and pitfalls of LLM-based agent deployment in healthcare and their innovative approaches for building, evaluating and deploying AI agents within their electronic health record. They also showcased successful use cases and clinician adoption of the deployed generative AI-based agents in EHR.

Connect More

Datasets (Open source)

REFLACX: Reports and eye-tracking data for localization of abnormalities in chest x-rays

This dataset provides eye-tracking data collected while radiologists dictated reports for frontal chest x-rays from the MIMIC-CXR dataset, paired with the timestamped transcription of the dictation. The dataset contains 3,032 cases of synchronised eye-tracking and transcription pairs labelled by five radiologists. For each case, labels for validating algorithms are derived from this dataset. These labels include image-level labels of anomalies found during the reading, ellipses localising these anomalies, and bounding boxes around lungs and heart. A subset of 109 chest x-rays has readings from all radiologists to estimate the variability of the provided labels.

VeriFact-BHC dataset

VeriFact-BHC contains long-form Brief Hospital Course (BHC) clinical narratives typically found in a discharge summary that have been decomposed into text proposition statements. From Remov100 patients in the MIMIC-III Clinical Database v1.4, we consider two types of BHC text: a human-written BHC and a LLM-generated BHC. A total of 13,070 propositions are annotated by multiple clinicians with a ground truth established via majority voting and manual adjudication.

ArchEHR-QA: A Dataset for Addressing Patient’s Information Needs related to Clinical Course of Hospitalization

ArchEHR-QA, an expert-annotated dataset of 134 cases from intensive care unit and emergency department settings to evaluate the grounding capabilities of models for responding to patient-initiated queries. The dataset consists of patient-initiated questions posted in the public domain, the corresponding clinician-interpreted questions, the excerpts of the EHRs annotated at the sentence level with relevance to the question, and clinician-generated free-text answers to the questions grounded with EHR sentences. We collect true patient health information needs expressed in real-world health forum messages, and we then align the messages to publicly accessible real EHRs. Derived from: MIMIC-III Clinical Database v1.4,MIMIC-IV-Note: Deidentified free-text clinical notes v2.2

PARROT dataset: Radiology reports

PARROT is a collaborative initiative to create a multilingual open dataset of radiological reports on which to test LLMs. The aim of PARROT is to represent the diversity of languages and reporting styles to promote the applicability of LLM-related research in non-English clinical settings. 2,658 reports, in 14 languages, from 76 authors, from 21 countries.

More Data

Conferences

Additional BXC-featured publications

Cardiology

Triple cardiovascular disease detection with an artificial intelligence-enabled stethoscope (TRICORDER) in the UK: a cluster-randomised controlled implementation trial

Kelshiker, Mihir A et al.

General/Generative AI/LLM

Reliability of LLMs as medical assistants for the general public: a randomized preregistered study

Bean, A.M., Payne, R.E., Parsons, G. et al.

Generative AI/LLM

Evaluation of electronic health record-integrated artificial intelligence chart review

Kahl, N.M., Frieden, M.J., Pope, Z.R. et al.-based

Join and follow the BrainX community!

Webpage: https://brainxai.org/

Newsletter: https://brainxai.substack.com/subscribe

LinkedIn: https://www.linkedin.com/groups/13599549/

Youtube: https://www.youtube.com/channel/UCua5EiLL6I29hpNrJsdv1rg

Share BrainXAI's Substack

BrainX Waves: The Newsletter of BrainX Community

BrainXAI — Sat, 31 Jan 2026 17:36:36 GMT

Subscribe now

Learn

2026: The year for AI in healthcare implementation

2022 witnessed the arrival of ChatGPT followed by massive research and development in understanding the use cases for Generative AI application in healthcare. In 2025, with the development of leaderboards such as MedHELM to evaluate LLM performance on healthcare data, much of the research focused on trials and evaluations. BrainXAI ReSearch also focused on development and deployment of AI evaluation tools such as HumanELY & GamELY.

With evaluation tools available and agentic AI frameworks available for implementation, in 2026 we are likely to see significant efforts directed towards implementation trials of various AI algorithms in healthcare. Building on the success of ambient scribe and the lessons learnt, this year has already started with many of the key companies such as OpenAI, Anthropic and Google releasing various solutions into the marketplace for use of healthcare data.

Our expectation is that strategic efforts and investments will be directed towards evaluation based implementation, clinician and patient education, and monitoring of these applications for success.

Learn More

Connect

Watch the recording of our recent BrainX Community Live! January 2026 event, featuring Fernando Eimbcke, MD, Alejandro Mauro M.D., Jaime de los Hoyos Moreno, MD. from Clínica Alemana de Santiago, Chile.

The speakers discussed promises and pitfalls of LLM based Agent deployment in healthcare, their innovative approaches for building, evaluation and deployment of AI agents within their electronic health record. They also showcased successful use cases and clinician adoption of the deployed generative AI based agents in EHR.

Connect More

Datasets (Open source)

VeriFact-BHC dataset

VeriFact-BHC contains long-form Brief Hospital Course (BHC) clinical narratives typically found in a discharge summary that have been decomposed into text proposition statements. From 100 patients in the MIMIC-III Clinical Database v1.4, we consider two types of BHC text: a human-written BHC and a LLM-generated BHC. A total of 13,070 propositions are annotated by multiple clinicians with a ground truth established via majority voting and manual adjudication.

ArchEHR-QA: A Dataset for Addressing Patient’s Information Needs related to Clinical Course of Hospitalization

ArchEHR-QA, an expert-annotated dataset of 134 cases from intensive care unit and emergency department settings to evaluate the grounding capabilities of models for responding to patient-initiated queries. The dataset consists of patient-initiated questions posted in public domain, the corresponding clinician-interpreted questions, the excerpts of the EHRs annotated at the sentence-level with relevance to the question, and clinician-generated free-text answers to the questions grounded with EHR sentences. We collect true patient health information needs expressed in real-world health forum messages, we then align the messages to publicly accessible real EHRs. Derived from:MIMIC-III Clinical Database v1.4,MIMIC-IV-Note: Deidentified free-text clinical notes v2.2

PARROT dataset: Radiology reports

PARROT is a collaborative initiative to create a multilingual open dataset of radiological reports on which to test LLMs. The aim of PARROT is to represent the diversity of languages and reporting styles to promote applicability of LLM-related research in non-English clinical settings. 2,658 reports, in 14 languages, from 76 authors, from 21 countries.

Medical-Reasoning-SFT

Medical SFT is a curated dataset designed to support the supervised fine-tuning of large language models (LLMs) for medical reasoning tasks. It comprises multi-turn dialogues, clinical case scenarios, and question-answer pairs that reflect the complex reasoning processes encountered in real-world clinical practice.The dataset is intended to help models develop key competencies such as differential diagnosis, evidence-based decision-making, patient communication, and guideline-informed treatment planning. II-Medical SFT is built using a combination of our custom synthetic data generation pipeline and publicly available medical reasoning datasets, ensuring both diversity and clinical relevance. The training dataset comprises 2,197,741 samples.

More Data

Conferences

Additional BXC-featured publications

General/Generative AI

Ambient Artificial Intelligence Scribes and Physician Financial Productivity

Holmgren AJ, Fenton CL, Thombley R, et al.

Book

Ethics in AI in Healthcare: Navigating the Future of Medicine

Ameed Khalid Abdul-Hamid MDS

Cancer/Generative AI/LLM

Artificial intelligence agents in cancer research and oncology

Truhn, D., Azizi, S., Zou, J. et al.

General/Generative AI/LLM

A foundation model for human-AI collaboration in medical literature mining

Wang, Z., Cao, L., Jin, Q. et al.

General/Generative AI/LLM

Responsible adoption of multimodal artificial intelligence in health care: promises and challenges

Ghazal Azarfar, Sara Naimimohasses, Sirisha Rambhatla, Matthieu Komorowski, Diana Ferro, Peter R Lewis, Darren Gates, Nawar Shara, Gregg M Gascon, Anthony Chang, Muhammad Mamdani, Mamatha Bhat

General/Generative AI/LLM

Holistic evaluation of large language models for medical tasks with MedHELM

Bedi, S., Cui, H., Fuentes, M. et al.

General/Generative AI/LLM

Implementation of large language models in electronic health records

Maxime Griot, Jean Vanderdonckt, Demet Yuksel

Course

Machine Learning in Healthcare: Foundations and Applications

Ghaith Habboub, MD

Join and follow the BrainX community!

Webpage: https://brainxai.org/

Newsletter: https://brainxai.substack.com/subscribe

LinkedIn: https://www.linkedin.com/groups/13599549/

Youtube: https://www.youtube.com/channel/UCua5EiLL6I29hpNrJsdv1rg

Share BrainXAI's Substack

BrainX Waves: The Newsletter of BrainX Community

BrainXAI — Thu, 18 Dec 2025 05:30:30 GMT

Subscribe now

Learn

AI in healthcare: An Agent of Change in 2026?

Growing research across many healthcare specialities and domains has established a clear and demonstrable interest for adoption of AI amongst the clinician community for adoption of AI. Patient use of AI tools such as ChatGPT has also grown exponentially validating the interest amongst them to get advice, although sometimes the potential risks are poorly understood by them. So, the key question is to understand the current state of adoption of AI in healthcare, especially amongst health systems to address this need.

In a study by Poon EG, et al., conducted as a survey of 67 health systems, 43 health systems responded with varying levels of adoption. Ambient AI scribe for clinical documentation has been an early use case for which many solutions are now present and in use. It was the only use case with 100% of respondents reporting adoption activities, and 53% reported a high degree of success with using AI for Clinical Documentation. With the highest number of FDA approved AI algorithms, Radiology emerged as the most widely deployed clinical AI use case, with 90% of organizations reporting at least partial deployment, although successes with diagnostic use cases were limited. The researchers also found that many organizations have deployed AI for clinical risk stratification such as early sepsis detection, but only 38% report high success in this area. Key barriers to adoption were Immature AI tools(77% of respondents), followed by financial concerns (47%) and regulatory uncertainty (40%).

So, what should we expect in 2026? Use of new models? New use cases? Agentic AI implementation? Poon EG ,et al., concluded that addressing these challenges through robust evaluations, shared strategies, and governance models will be essential to ensure effective integration and adoption of AI into healthcare practice. Watch our webinars on evaluation methodologies for Generative AI in healthcare and platform strategies for Generative AI in healthcare adoption to prepare yourself for implementation in 2026.

Related publications:

Learn More

Connect

Watch the recording of our recent BrainX Community Live! December 2025 event, featuring Dr. Suhana Bedi, Stanford University and Dr. Raghav Awasthi, BrainXAI ReSearch summarize evaluation methodologies for Generative AI in healthcare. Learn from the tools developed to perform LLM evaluations in healthcare including leaderboards, benchmark datasets, human evaluation, LLM-as-a-judge.

Connect More

Datasets (Open source)

Medical-Reasoning-SFT

Medical-Reasoning-SFT-GPT-OSS-120B

A high-quality synthetic dataset of synthetic medical reasoning conversations generated using OpenAI’s gpt-oss-120B model with reasoning effort set to high, designed for supervised fine-tuning of large language models in healthcare applications. Intelligent-Internet/II-Medical-Reasoning-SFT used as a seed dataset. Dataset Statistics: Total Samples: 200,927, Total Tokens: 539,165,577,User Messages: 200,847, Assistant Messages: 200,847, Average Tokens per Sample: 2,683.3, Average User Tokens per Sample: 114.1, Average Assistant Tokens per Sample: 2,569.2. Each conversation demonstrates structured medical thinking with step-by-step reasoning processes.

The Cancer Genome Atlas (TCGA) Multimodal Dataset

The Cancer Genome Atlas (TCGA) Multimodal Dataset is a comprehensive collection of clinical data, pathology reports, slide images, molecular data, and radiology images for cancer patients. This dataset aims to facilitate research in multimodal machine learning for oncology by providing embeddings generated using state-of-the-art models including GatorTron, MedGemma, Qwen, Llama, UNI, SeNMo, REMEDIS, and RadImageNet. 10,771 patient records per model for clinical data and 10,857 patient records per model for pathology reports.

Synthea Synthetic Patient Records (575K Patients)

A comprehensive synthetic healthcare dataset containing 575,415 patients with complete medical histories, generated using Synthea – an open-source synthetic patient generator. Generated by Richard Young, DeepNeuro.AI.

More Data

Conferences

Additional BXC-featured publications

General

Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance

Van Calster, Ben et al.

Journal

Computational Biomedicine

Editor in Chief: De-Shuang Huang

Generative AI/LLM

GamELY: Human-in-the loop Framework for Scaling Human Evaluation of LLMs in Healthcare

Raghav Awasthi, Nishant Singh, Shreya Mishra, Atharva Bhattad, Moises Auron, Charumathi Raghu Subramanian, Ashish Atreja, Kamal Maheshwari, Dwarikanath Mahapatra, Jacek B. Cywinski, Ashish Khanna, Francis Papay, Piyush Mathur

Generative AI/LLM

Automating expert-level medical reasoning evaluation of large language models

Zhou, S., Xie, W., Li, J. et al.

Course

Machine Learning in Healthcare: Foundations and Applications

Ghaith Habboub, MD

General/Generative AI/LLM

Promises and Pitfalls of Large Language Models use to interpret Healthcare Guidelines

Raghav Awasthi, Shreya Mishra, Ashish Atreja, Charumathi Raghu Subramanian, Moises Auron, Nishant Singh, Jacek B. Cywinski, Francis Papay, Kamal Maheshwari, Ashish K Khanna, Piyush Mathur

General/Generative AI/LLM

Ambient AI Scribes in Clinical Practice: A Randomized Trial

Paul J. Lukac, et al.

Anesthesiology

Artificial Intelligence in Anesthesiology: Regulatory Pathways and Analysis of FDA-Approved Devices

Almeida, Victor F.A.; Ozmen, Berk B. ; Tsai, Travis ; Ha, John Y. ; Dantas, Manoela ; Donato, Glaudir ; Mathur, Piyush

General

Assessment of patient and physician sentiment on artificial intelligence use in US healthcare

Wael Saasouh, et al.

Join and follow the BrainX community!

Webpage: https://brainxai.org/

Newsletter: https://brainxai.substack.com/subscribe

LinkedIn: https://www.linkedin.com/groups/13599549/

Youtube: https://www.youtube.com/channel/UCua5EiLL6I29hpNrJsdv1rg

Share BrainXAI's Substack

BrainX Waves: The Newsletter of BrainX Community

BrainXAI — Fri, 28 Nov 2025 08:53:52 GMT

Subscribe now

Learn

Will AI replace healthcare professionals?

There are ever increasing concerns about AI replacing the human workforce. In a recently published study from MIT,the researchers of Project Iceberg,defined the Iceberg Index as a measure of AI technical capabilities that overlap human occupational skills.¹ They concluded that in the US, AI technical capability extends to cognitive and administrative tasks spanning 11.7% of the labor market including in healthcare.¹ So, are healthcare professionals vulnerable? So, which tasks currently performed by healthcare professionals are likely to be replaced by AI? Dr. Maheshwari, recently wrote an interesting viewpoint article articulating the tasks currently performed by Anesthesiologists.² Similar to the Project Iceberg, he describes the overlap between AI technical capabilities and human skills across 12 domains.

Source: https://iceberg.mit.edu

Such projections and prophecies have been made in the past by many renowned scholars. But with the introduction of Generative AI tools, many tasks mimic human capabilities and such tools are seeing rapid adoption. Based on a recently conducted study, in which both physicians and patients were surveyed, the Dr. Saasouh noted varying degrees of comfort and trust from both physicians and patients with the use of AI in medicine.³These results are not different from a multinational survey of in-hospital patients,performed across 6 continents, the patients indicated a predominantly favorable general view of AI in health care.⁴ Thus, our patients are seeking and are comfortable with the use of AI in their healthcare.

So, how best can we prepare ourselves for the future? Or should we say near term present? With growing applications of AI in healthcare getting FDA approval, even in specialized fields such as Anesthesiology,⁵ there are some common recommendations being proposed by various think-tanks, as listed below.

Educate yourself about AI and potential uses in healthcare.⁶
Join a community for continuous learning in this fast evolving sector.⁷
Engage in development of new care delivery models leveraging AI.²
Partner with solution developers to guide, evaluate and validate the development of AI solutions for healthcare.⁸
Lead in development of implementation science of AI in healthcare.⁹
Educate the patients about appropriate use of AI technologies.⁷
Advocate for safe and cost-effective use of AI and human resources in healthcare.¹⁰

At BrainX (https://www.brainxai.com/), we are focused on following up with all of the above recommendations for implementation of AI in healthcare. Read more about our research here: https://www.brainxai.com/research

Reference publications:

Learn More

Connect

Watch the recording of our recent BrainX Community Live! November 2025 event, featuring Dr. Ashish Atreja, Founder, Genserve.AI. There is a growing need to adopt a platform based approach to adopt GenerativeAI at healthcare systems. 95% of GenAI pilots fail and many are due to the lack of a unified and systematic approach. Genserve.AI is a platform to deliver on launch with Enterprise-wide Secure Gen AI within 30 Days. It promised to bring trustworthy AI to all employees through our HIPAA-compliant GenAI delivery and vendor neutral monitoring platform. Learn more about their approach through this event recording.

Learn more about Genserve.ai

Connect More

Datasets (Open source)

STARC-9 (STAnford coloRectal Cancer WSI)

STARC-9 (STAnford coloRectal Cancer): A large-scale dataset with 630k samples across nine tissue types (~70k per class) collected from over 200 WSI for multi-class tissue classification task. Stanford and TCGA-CRC tile-level validation datasets. Pretrained model weights on STARC-9, including baseline, SOTA, and Pathology-specific foundation models.Open-source code repository for DeepCluster++ framework that can be used for image sample collection across domains.

MIMIC-IV-Ext-22MCTS: Time-Series Dataset

Clinical risk prediction based on machine learning algorithms plays a vital role in modern healthcare. A crucial component in developing a reliable prediction model is a high-quality dataset with time series clinical events. In this work, the authors release such a dataset that consists of 22,588,586 clinical time series events, which is term MIMIC-IV-Ext-22MCTS. The source data are discharge summaries selected from the well-known yet unstructured MIMIC-IV-Note. The authors then extract clinical events as short text spans from the discharge summaries, along with the timestamps of these events as temporal information by contextual retrieval and Llama-3.1-8B.

AI-READI dataset (Diabetes)

This dataset contains data from 2280 participants that was collected between July 19, 2023 and May 01, 2025. Data from multiple modalities are included. The AI-READI is a dataset consisting of data collected from individuals with and without Type 2 Diabetes Mellitus (T2DM) and harmonized across 3 data collection sites. The composition of the dataset was designed with future studies using AI/Machine Learning in mind. This included recruitment sampling procedures aimed at achieving approximately equal distribution of participants across diabetes severity, as well as the design of a data acquisition protocol across multiple domains (survey data, physical measurements, clinical data, imaging data, wearable device data, etc.) to enable downstream AI/ML analyses that may not be feasible with existing data sources such as claims or electronic health records data. The dataset contains 358,999 files and is around 3.87 TB in size.

HC4 (Healthcare Comprehensive Commons Corpus)

HC4 is a large-scale pretraining dataset containing over 65 billion tokens from diverse healthcare-related sources. The corpus was curated to enable systematic investigation of how data composition influences language model behavior, including potential demographic biases. 153GB (around 65 billion tokens). 9.7+ million documents from diverse sources including peer-reviewed scientific literature collected from PubMed Central, Semantic Scholar, OpenAlex repositories.

More Data

Conferences

Additional BXC-featured publications

Cardiology/Generative AI

Comprehensive echocardiogram evaluation with view primed vision language AI

Vukadinovic, M., Chiu, IM., Tang, X. et al.

Book

AI in Medicine: Clinical Updates & Physician Perspectives

Amit Kumar Dey, Ashlesha Tawde-Kelkar

Course

AI in Medicine Graduate Certificate

Book

Generative AI: Unlocking the Next Chapter in Healthcare

Rohit Mahajan , Ritu M. Uberoy

Generative AI/LLM

Evaluating clinical AI summaries with large language models as judges

Croxford, E., Gao, Y., First, E. et al.

Generative AI/LLM

Quantifying the reasoning abilities of LLMs on clinical cases

Qiu, P., Wu, C., Liu, S. et al.

Cardiology/Generative AI/LLM

Comprehensive echocardiogram evaluation with view primed vision language AI

Vukadinovic, M., Chiu, IM., Tang, X. et al.

Join and follow the BrainX community!

Webpage: https://brainxai.org/

Newsletter: https://brainxai.substack.com/subscribe

LinkedIn: https://www.linkedin.com/groups/13599549/

Youtube: https://www.youtube.com/channel/UCua5EiLL6I29hpNrJsdv1rg

Share BrainXAI's Substack

BrainX Waves: The Newsletter of BrainX Community

BrainXAI — Sat, 25 Oct 2025 07:11:34 GMT

Subscribe now

Learn

The Illusion of Readiness: Stress Testing Large Frontier Models on Multimodal Medical Benchmarks

For successful implementation of Generative AI solutions in healthcare, benchmarking of models and evaluation of their outputs is extremely important. Growing evidence suggests that foundation models like GPT-5 now achieve top scores on medical benchmarks. But in this study from Microsoft Research, their stress tests tell a different story. They found that the leading models often guess correctly even when key inputs like images are removed, flip answers under trivial prompt changes, and fabricate convincing yet flawed reasoning.Through clinician-guided rubric evaluation, the authors demonstrate that benchmarks vary widely in what they truly measure yet are treated interchangeably, masking failure modes. The authors caution that medical benchmark scores do not directly reflect real-world readiness.

Therefore, at BrainXAI ReSearch (https://www.brainxai.com/research), we have built freely available tools such as HumanELY(https://www.brainxai.com/humanely) to provide for comprehensive, comparable, and quantifiable assessment of LLMs in healthcare. We will be releasing additional tools shortly which allow scaling of evaluations to overcome challenges of human evaluation.

Reference publication:The Illusion of Readiness: Stress Testing Large Frontier Models on Multimodal Medical Benchmarks.https://www.arxiv.org/pdf/2509.18234

Learn More

Connect

Watch the recording of our recent BrainX Community Live! October 2025 event, featuring Dr. Pawan Jindal,CEO, Prompt Opinion, and Dr. Kamal Maheshwari. In this presentation, Dr. Jindal addressed key terminologies related to Generative AI, HL7 FHIR, SMART on FHIR, Model Context Protocol(MCP), EHR integration of AI models, Agent to Agent(A2A) protocols, and so much more.

Register to join our upcoming BrainX Community Live, November 2025 event themed, “Enterprise AI platform for Healthcare”, on Nov 17, 2025 04:00 PM Eastern Time (US and Canada)/ 01:00 PM Pacific Time (US and Canada) via Zoom for free. This event will feature Dr. Ashish Atreja, “The App Doctor,” is a nationally recognized physician executive and founder of GenServe.AI., who will present, “AI Centers of Excellence: The Path from AI Innovation to Impact”.

Registration link: https://us02web.zoom.us/meeting/register/NYgSHiUyR9iJx5SCCpgqlw

Program Agenda:

Introduction
“AI Centers of Excellence: The Path from AI Innovation to Impact”: Dr. Ashish Atreja, “The App Doctor,” is a nationally recognized physician executive and founder of GenServe.AI.
Q&A

Learn more about Genserve.ai

Connect More

Datasets (Open source)

WoundcareVQA: A multilingual visual question answering benchmark dataset for wound care

A multilingual dataset of 477 wound care cases, 768 responses, 748 images, 3k structured data labels, 1362 translation instances, and 10k judgments.

GPTNERMED: NER dataset for German Medical text

GPTNERMED is a novel open synthesized dataset and neural named-entity-recognition (NER) model for German texts in medical natural language processing (NLP).This dataset contains the synthetic German sentences with annotated entities (Medikation, Dosis, Diagnose) from the GPTNERMED project. The sentences as well as the annotations are not manually validated by medical professionals and therefore this dataset is no gold standard dataset.The dataset consists of 9,845 sentences (121,027 tokens by SpaCy Tokenizer, 245,107 tokens by the GPT tokenizer).

MedPix 2.0 dataset

MedPix 2.0 is derived from a freely open-access source MedPix. It offers a balanced variety of CT and MRI scans of different body parts.For each image, a complete structured clinical case is provided. MedPix® is a free open-access multimodal online database of medical images, teaching cases, and clinical topics, managed by the National Library of Medicine (NLM) of the National Institutes of Health (NIH). It mainly serves as a support system for Continuing Medical Education (CME) of physicians, nurses, and healthcare students. The database collects clinical cases related to more than 12,000 patients. Each case contains at least one medical image, and the corresponding findings, discussion notes, diagnosis, differential diagnosis, treatment, and follow up. Textual information is reported in a semi-structured format. Attached to the clinical case, there is the topic section, where the disease under investigation is discussed in detail from an academic and general perspective.

FFA-IR : Fundus Fluorescein Angiography Images and Reports dataset

Large-scale medical dataset. FFA-IR collects 766 reports along with 47,247 FFA images from clinical practice. Explainable annotation. FFA-IR annotates 46 categories of lesions with a total of 12,166 regions. Bilingual reports. FFA-IR provides both English and Chinese reports for each case.

More Data

Conferences

Additional BXC-featured publications

General

The STARD-AI reporting guideline for diagnostic accuracy studies using artificial intelligence

Sounderajah, V., Guni, A., Liu, X. et al.

Anesthesiology/Surgery

Multicenter validation of a scalable, interpretable, multitask prediction model for multiple clinical outcomes

Yoon, HK., Kim, B.R., Kim, H.Y. et al.

General/Generative AI/LLM

Zero shot health trajectory prediction using transformer (ETHOS:Enhanced Transformer for Health Outcome Simulation)

Renc, P., Jia, Y., Samir, A.E. et al.

General/Generative AI/LLM

Foundation model of electronic medical records for adaptive risk estimation (ARES: Adaptive Risk Estimation System)

Pawel Renc, Michal K Grzeszczyk, Nassim Oufattole,et al.

General/Generative AI/LLM

A foundation model for human-AI collaboration in medical literature mining

Wang, Z., Cao, L., Jin, Q. et al.

General

Benefit-Risk Reporting for FDA-Cleared Artificial Intelligence−Enabled Medical Devices

John C. Lin; Bhav Jain; Jay M. Iyer, et al

Course

Machine Learning in Healthcare 10-742 CMU Course

Adam Berger

Join and follow the BrainX community!

Webpage: https://brainxai.org/

Newsletter: https://brainxai.substack.com/subscribe

LinkedIn: https://www.linkedin.com/groups/13599549/

Youtube: https://www.youtube.com/channel/UCua5EiLL6I29hpNrJsdv1rg

Share BrainXAI's Substack

BrainX Waves: The Newsletter of BrainX Community

BrainXAI — Mon, 01 Sep 2025 14:26:54 GMT

Subscribe now

Learn

Can AI lead to loss of skills amongst the healthcare workforce? Should we be concerned?

In a first-of-its-kind study, Marcin Romańczyk, et al., found a negative impact of regular AI use on healthcare professionals' ability to complete a patient-relevant task in medicine. The researchers conducted a retrospective, observational study at four endoscopy centres in Poland taking part in the ACCEPT (Artificial Intelligence in Colonoscopy for Cancer Prevention) trial. These centres introduced AI tools for polyp detection at the end of 2021, after which colonoscopies had been randomly assigned to be conducted with or without AI assistance according to the date of examination.

Image Source: Biffi, C., Salvagnini, P., Dinh, N.N. et al. A novel AI device for real-time optical characterization of colorectal polyps. npj Digit. Med. 5, 84 (2022).

Amongst the colonoscopies performed between Sept 8, 2021, and March 9, 2022, 1443 patients underwent non-AI-assisted colonoscopy before (n=795) and after (n=648) the introduction of AI (median age 61 years [IQR 45–70], 847 [58·7%] female, 596 [41·3%] male). The adenoma detection rate (ADR) of standard colonoscopy decreased significantly from 28·4% (226 of 795) before to 22·4% (145 of 648) after exposure to AI, corresponding with an absolute difference of –6·0% (95% CI –10·5 to –1·6; p=0·0089). The authors concluded that continuous exposure to AI might reduce the ADR of standard non-AI-assisted colonoscopy, suggesting a negative effect on endoscopist behaviour.

It is clear that given the research, development and adoption of AI in medicine are rapidly spreading, this could be a cause of concern. Evolution in skills is a major part of any technological advancement. In prior studies, AI has also shown skill enhancement with the use of AI, such as performance of ultrasound for fetal age estimation or echocardiography. We need more research in this area to understand the impact of AI adoption in healthcare on skills and patient care delivery outcomes.

Reference publication: Endoscopist deskilling risk after exposure to artificial intelligence in colonoscopy: a multicentre, observational study. Budzyń, Krzysztof et al. The Lancet Gastroenterology & Hepatology

Learn More

Connect

Register to join our upcoming BrainX Community Live event themed, ‘Reporting Guidelines for Generative AI’, on September 16, 2025; 11am - noon ET via Zoom for free. This event will feature Bright Huo MD & Giovanni Cacciamani MD, who will present the recent CHART reporting guidelines, and we will discuss other reporting guidelines for Generative AI research and publication.

Registration link: https://us02web.zoom.us/meeting/register/99GG_Vb6ReS2c_hAN-64GA#/registration

Program Agenda:

– Introduction

– “Reporting guidelines for chatbot health advice studies: the Chatbot Assessment Reporting Tool (CHART) statement”: Bright Huo MD; Giovanni Cacciamani MD

– Q&A

Related publications: Reporting guideline for chatbot health advice studies: the Chatbot Assessment Reporting Tool (CHART) statement. Chart Collaborative. BMJ Medicine. 2025;4:e001632.

Watch the video of our BrainX Community Live event themed: ‘Radiology platforms for AI implementation’, featuring Drs. Pranav Rajpurkar, Co-founder of a2z Radiology AI, and Vidur Mahajan, CEO for CARPL.ai.

Connect More

Datasets (Open source)

Consumer-health-research:Personal Health Large Language Model (PH-LLM) dataset

All input data and associated expert responses for all 857 case studies in sleep and fitness used for training and evaluation, along with all model responses to the 100 holdout case studies, are stored in JSON format. All input and target prediction data, which include bucketized age, daily sensor readings for 20 sensors over 15 days and 16 binary survey responses for all 4,163 samples in the PRO task, are stored in JSON format.

FactEHR: A Benchmark for Fact Decomposition of Clinical Notes

FactEHR is a benchmark dataset designed to evaluate the ability of large language models (LLMs) to perform factual reasoning over clinical notes. It includes:

2,168 deidentified notes from multiple publicly available datasets
8,665 LLM-generated fact decompositions
987,266 entailment pairs evaluating precision and recall of facts
1,036 expert-annotated examples for evaluation

FactEHR supports LLM evaluation across tasks like information extraction, entailment classification, and model-as-a-judge reasoning.

More Data

Conferences

Additional BXC-featured publications

Generative AI/LLM

Large Language Model Analysis of Reporting Quality of Randomized Clinical Trial Articles: A Systematic Review

Apoorva Srinivasan; Jacob Berkowitz; Nadine A. Friedrich,et al

Generative AI/LLM

Generative Medical Event Models Improve with Scale

Shane Waxler, Paul Blazek, Rahul Shah, et al.

Endocrinology

Type 2 Diabetes Pharmacotherapy De-Escalation Through AI-Enabled Lifestyle Modifications: A Randomized Clinical Trial

Kevin M. Pantalone, Huijun Xiao, and Jeffrey I. Mechanick, et al.

Generative AI/LLM

Reporting guidelines for chatbot health advice studies: explanation and elaboration for the Chatbot Assessment Reporting Tool (CHART)

Bright Huo and The CHART Collaborative

Generative AI/LLM

Fidelity of Medical Reasoning in Large Language Models

Suhana Bedi; Yixing Jiang; Philip Chung, et al

General/Generative AI/LLM

A personal health large language model for sleep and fitness coaching

Khasentino, J., Belyaeva, A., Liu, X. et al.

General/Generative AI/LLM

Evaluating Hospital Course Summarization by an Electronic Health Record–Based Large Language Model

William R. Small; Jonathan Austrian; Luke O’Donnell, et al.

Join and follow the BrainX community!

Webpage: https://brainxai.org/

Newsletter: https://brainxai.substack.com/subscribe

LinkedIn: https://www.linkedin.com/groups/13599549/

Youtube: https://www.youtube.com/channel/UCua5EiLL6I29hpNrJsdv1rg

Share BrainXAI's Substack

BrainX Waves: The Newsletter of BrainX Community

BrainXAI — Wed, 30 Jul 2025 10:56:12 GMT

Subscribe now

Learn

AI and Robotics ready to transform perioperative patient care

During a trip to China, Co-founder and CEO of Nvidia Jensen Huang was asked by a journalist: “If you are a 22-year-old version of Jensen [who] just graduated today in 2025 but with the same ambition, what would you focus on?”

To that, the Nvidia CEO said: “For the young, 20-year-old Jensen, that’s graduated now, he probably would have chosen ... more of the physical sciences than the software sciences,” adding that he actually graduated two years early from college, at age 20.

Applications of physical reasoning include predicting outcomes, such as where a ball will roll, understanding how much force is needed to grip an object without damaging it and inferring the presence of a pedestrian behind a car.

“And when you take that physical AI and then you put it into a physical object called a robot, you get robotics,” he added. “This is really, really important for us now, because we’re building plants and factories all over the United States.”

Ref: https://www.cnbc.com/2025/07/18/nvidia-ceo-jensen-huang-study-field-computer-science-software-gpu-alexnet-generative-physical-ai-university.html

Extending the same argument, we are now seeing combined applications of deep learning and robotics come to the operating rooms.

In a recently published study, a Robotic Flexible Intubation System (RFIS) which uses Convolutional Neural Network(CNN) was trialed for airway intubation on a simulation manikin.

Connect

Register today to join us on August 19, 2025 at 11 am – Noon ET/8:30-9:30 PM India time via Zoom. This session is themed: “Radiology platforms for AI implementation”, features Drs. Pranav Rajpurkar, Co-founder of a2z Radiology AI, and Vidur Mahajan, CEO for CARPL.ai

https://us02web.zoom.us/meeting/register/Un6VwGWsQXyvo048unjyVA#/registration

Connect More

Datasets (Open source)

Mediflow: Synthetic clinical dataset

A large-scale synthetic instruction dataset of 2.5M rows (~700k unique instructions) for clinical natural language processing covering 14 task types and 98 fine-grained input clinical documents

PadChest-GR (Grounded-Reporting)

A public bilingual dataset of 4,555 CXR studies with substantiated reports (3,099 abnormal and 1,456 normal), each containing complete lists of sentences describing individual positive and negative findings in English and Spanish. In total, PadChest-GR contains 7,037 sentences of positive findings and 3,422 sentences of negative findings. This dataset is derived from the PadChest dataset.

More Data

Conferences

Additional BXC-featured publications

Surgery

Explainable Artificial Intelligence: Enhancing Decision-Making in Plastic Surgery

Berk B. Ozmen, Victor Almeida, Piyush Mathur, Graham S. Schwarz

Dermatology

A multimodal vision foundation model for clinical dermatology (PanDerm)

Yan, S., Yu, Z., Primiero, C. et al.

Book

Machine Learning in Medical Diagnosis: A Framework for a Normative Evaluation of Benefits and Risks

Leslye Denisse Dias Duran

Join and follow the BrainX community!

Webpage: https://brainxai.org/

Newsletter: https://brainxai.substack.com/subscribe

LinkedIn: https://www.linkedin.com/groups/13599549/

Youtube: https://www.youtube.com/channel/UCua5EiLL6I29hpNrJsdv1rg

Share BrainXAI's Substack

BrainX Waves: The Newsletter of BrainX Community

BrainXAI — Mon, 23 Jun 2025 10:18:25 GMT

Subscribe now

Learn

Multinational Attitudes Toward AI in Health Care and Diagnostics Among Hospital Patients

This is an important study that survey’s hospital patients’ attitudes toward the use of artificial intelligence (AI) in health care across diverse socio demographic contexts. This cross-sectional study developed and implemented an anonymous quantitative survey between February 1 and November 1, 2023, using a nonprobability sample at 74 hospitals in 43 countries. Participants included hospital patients 18 years of age or older who agreed with voluntary participation in the survey presented in 1 of 26 languages. The primary outcome was participant responses to a 26-item instrument containing a general data section (8 items) and 3 dimensions (trust in AI, AI and diagnosis, preferences and concerns toward AI) with 6 items each.

In total, 13 806 patients participated. Their median (IQR) age was 48 (34-62) years, and 6973 (50.5%) were male. The survey results indicated a predominantly favorable general view of AI in healthcare, with 57.6% of respondents (7775 of 13 502) expressing a positive attitude. However, attitudes exhibited notable variation based on demographic characteristics, health status, and technological literacy. Female respondents (3511 of 6318 [55.6%]) exhibited fewer positive attitudes toward AI use in medicine than male respondents (4057 of 6864 [59.1%]), and participants with poorer health status exhibited fewer positive attitudes toward AI use in medicine (eg, 58 of 199 [29.2%] with rather negative views) than patients with very good health (eg, 134 of 2538 [5.3%] with rather negative views). Conversely, higher levels of AI knowledge and frequent use of technology devices were associated with more positive attitudes. Notably, fewer than half of the participants expressed positive attitudes regarding all items pertaining to trust in AI. The lowest level of trust was observed for the accuracy of AI in providing information regarding treatment responses (5637 of 13 480 respondents [41.8%] trusted AI). Patients preferred explainable AI (8816 of 12 563 [70.2%]) and physician-led decision-making (9222 of 12 652 [72.9%]), even if it meant slightly compromised accuracy.

Findings indicate that tailored AI implementation strategies should take patient demographics, health status, and preferences for explainable AI and physician oversight into account.

Learn More

Connect

Register today to join us on June 25, 2025; 11 am – 12 noon ET ( 8:30 – 9:30 PM India time) for the BrainX Community Live event featuring Dr. Avneesh Khare, Dr. Amit Kumar Dey , Dr. S.V. Kulkarni, Maj (Dr) Ashlesha Tawde-Kelkar. The sessions features DoctorsAI and is themed on continuous AI learning opportunities for healthcare professionals.

Register here for the free Zoom event: https://us02web.zoom.us/meeting/register/MkOxLL0lTNa-g5d11nTOQA#/registration

Connect More

Datasets

Artificial Intelligence in Healthcare: 2024 Year in Review Dataset

A dataset based on PubMed search of 28,180 total articles; 1,693 were initially labeled mature, after which 1,551 articles were analyzed after exclusions.The most prevalent specialties within our PubMed search originated from imaging (407), head and neck (127), and General (122). Analysis of AI model types showed that the Large Language Model (LLM) was the most popular utilized in 479 publications, followed by AI General (448), and DL (372). Qualitative data was obtained on the data types, and it was revealed that the image data was predominant and used in 57.0% of the mature sources, followed by text (33.1%), followed by tabular (7.59%). The utilization of Large Language Models (LLMs) is the highest in publications associated with education at 18.6%, followed by General at 13.6%.

Dataset lung cancer 2024

This study examined the relationships between coping styles, stress levels, and pain levels in lung cancer patients, and whether these relationships were moderated by place of residence (small town vs large city). A sample of 97 lung cancer patients completed questionnaires measuring coping styles (Stress Coping Inventory), perceived stress levels (Perceived Stress Scale), and pain levels (Short Form McGill Pain Questionnaire).

COMFORT study dataset: Multinational attitudes towards AI in healthcare and diagnostics among hospital patients

The survey was administered to a nonprobability convenience sample at 74 COMFORT network hospitals across 43 countries.In total, 13 806 patients participated, including 8951 (64.8%) in the Global North and 4855 (35.2%) in the Global South. Their median (IQR) age was 48 (34-62) years, and 6973 (50.5%) were male. The survey instrument consisted of 3 dimensions: trust in AI, AI and diagnosis, and preferences and concerns toward AI, each with 6 items, complemented by a general data section with 8 items (self-reported gender, age, highest educational level, weekly use of technological devices, health status, AI knowledge, and general attitudes toward AI use in medicine and health care).

Hillel Yaffe Glaucoma Dataset (HYGD)

The Hillel Yaffe Glaucoma Dataset (HYGD) addresses a critical limitation in existing GON datasets: the lack of gold-standard annotations. The dataset was curated by the Hillel Yaffe Ophthalmology Department Glaucoma Unit, Hadera, Israel, between 2022-2024. The dataset consists of 747 DFIs, including:548 glaucomatous DFIs (73%) and 199 non-glaucomatous DFIs (27%). HYGD’s labels are based on comprehensive ophthalmic examinations, including visual acuity assessment, intraocular pressure measurement, optical coherence tomography (OCT), visual field tests, and at least one year of follow-up monitoring.The dataset is structured to include both the DFIs and a labels file containing patient IDs, GON classifications and image quality scores. All DFIs were taken using a TOPCON DRI OCT Triton retinal camera with a 45° FOV and underwent deidentification and standardization processing.

More Data

Conferences

Additional BXC-featured publications

Surgery

Development of a novel artificial intelligence clinical decision support tool for hand surgery: HandRAG

Berk B. Ozmen, Nishant Singh, Kavach Shah, Ibrahim Berber, Damanjit Singh, Eugene Pinsky, Antonio Rampazzo, Graham S. Schwarz

Course

AI Passport

University of Florida

Anesthesiology/Surgery/Critical Care

Artificial intelligence for the prediction of postoperative complications in the critically ill

Vanessa Moll, Ashish K Khanna , Piyush Mathur

Surgery

Evidence-based artificial intelligence: Implementing retrieval-augmented generation models to enhance clinical decision support in plastic surgery

Berk B. Ozmen, Piyush Mathur

Critical Care/General/Generative AI/LLM

Development and prospective implementation of a large language model based system for early sepsis prediction

Supreeth P. Shashikumar, Sina Mohammadi, Rishivardhan Krishnamoorthy, Avi Patel, Gabriel Wardi, Joseph C. Ahn, Karandeep Singh, Eliah Aronoff-Spencer & Shamim Nemati

Join and follow the BrainX community!

Webpage: https://brainxai.org/

Newsletter: https://brainxai.substack.com/subscribe

LinkedIn: https://www.linkedin.com/groups/13599549/

Youtube: https://www.youtube.com/channel/UCua5EiLL6I29hpNrJsdv1rg

Share BrainXAI's Substack

BrainX Waves: The Newsletter of BrainX Community

BrainXAI — Sat, 31 May 2025 09:43:47 GMT

Subscribe now

Learn

Voices of Innovation – AI: Fulfilling the Promise of AI in Healthcare

Edited By Edward W. Marx, Sakshika Dhingra, Frank Papay, Piyush Mathur

As an industry, we have clearly made great technological advancements, and yet we find ourselves behind peer industries when it comes to innovation with patient care, customer experience, augmented intelligence, virtual care, and cybersecurity. One reason for the lack of innovation velocity is the need for a universally adopted model or best practice framework. The time has come for Voices of Innovation – AI. Artificial intelligence is what some call the fourth transformative revolution in human history. Healthcare is among the many industries with significant opportunities for the use of AI and machine learning, as the convergence of technology and healthcare will result in significant innovation. This book is both practical and inspirational. Using the HIMSS model for innovation as the structural framework, Voices of Innovation – AI will showcase the great AI innovations being implemented across healthcare globally.

With contributions from leading authorities in this field, this book will become the de facto resource for any organization seeking to leverage AI effectively. Loaded with numerous case studies and stories of successful innovation projects, this book helps the reader understand how to leverage AI to help fulfill the promise of technology in enabling superior business and clinical outcomes.

Learn More

Connect

At the BrainX Community Live! May 2025 event on the real-world application of Generative AI in clinical practice, Prof. Ngiam Kee Yuan, National University Hospital, Singapore, shared his experiences with the implementation of AI in clinical practice in a talk titled, “Implementing AI in Clinical Practice”. Watch his presentation, which detailed building an infrastructure to support the implementation of AI in a health system, real-world use cases in current deployment and how LLMs are being integrated to support clinicians.

Connect More

Datasets

Bridge2AI-Voice: Voice dataset linked to health information

Bridge2AI-Voice v2.0 contains data for 19,271 recordings collected from 442 participants across five sites in North America. Participants were selected based on known conditions which manifest within the voice waveform including voice disorders, neurological disorders, mood disorders, and respiratory disorders.

BRIDGE-Open Dataset

The BRIDGE-Open dataset is an open-access subset of the datasets used by the BRIDGE benchmark. This dataset includes the 55 tasks of the open-access datasets that can be released to the public. Due to privacy and security considerations of clinical data, regulated-access datasets can not be directly published. However, all detailed task descriptions and their corresponding data sources are available in the BRIDGE paper, Supplementary Section 5. Importantly, all 87 datasets have been verified to be either fully open-access or publicly accessible via reasonable request in the systematic review.

OpenPOCUS (Lung ultrasound images)

The largest annotated lung ultrasound (LUS) repositories to date—over 300,000 de-identified frames, drawn from 226 patients with diverse respiratory diagnoses. Includes both COVID and non-COVID pneumonia, pulmonary edema, COPD, healthy controls, and more.

Adult NHANES 2011-2012

This dataset consists of 4571 patients (rows) with 73 features (columns). It includes extensive demographics, past history, extensive lab work, activity levels, light levels, cog scores, and many more variables. The final column is diabetes (0, 1) that could be used in a classification model.

Symptom2Disease dataset

This file contains the dataset. It has 24 diseases each with 50 symptoms giving a total of 1200 data points.

More Data

Conferences

Additional BXC-featured publications

Critical Care/ General/ Generative AI/ LLM

Development and prospective implementation of a large language model based system for early sepsis prediction

Supreeth P. Shashikumar, Sina Mohammadi, Rishivardhan Krishnamoorthy, Avi Patel, Gabriel Wardi, Joseph C. Ahn, Karandeep Singh, Eliah Aronoff-Spencer & Shamim Nemati

General/ Generative AI/ LLM

BRIDGE: Benchmarking Large Language Models for Understanding Real-world Clinical Practice Text

Jiageng Wu, Bowen Gu, et al.

General/ Generative AI/ LLM

A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation (CREOLA)

Elham Asgari, Nina Montaña-Brown, Magda Dubois, Saleh Khalil, Jasmine Balloch, Joshua Au Yeung & Dominic Pimenta

General/ Generative AI/ LLM

Physician- and Large Language Model–Generated Hospital Discharge Summaries

Christopher Y. K. Williams, Charumathi Raghu Subramanian, Syed Salman Ali, et al.

Join and follow the BrainX community!

Webpage: https://brainxai.org/

Newsletter: https://brainxai.substack.com/subscribe

LinkedIn: https://www.linkedin.com/groups/13599549/

Youtube: https://www.youtube.com/channel/UCua5EiLL6I29hpNrJsdv1rg

Share BrainXAI's Substack

BrainX Waves: The Newsletter of BrainX Community

BrainXAI — Wed, 30 Apr 2025 07:25:06 GMT

Subscribe now

Learn

Understanding Large Language Models in Healthcare: A Guide to Clinical Implementation and Interpreting Publications

This review hopes to equip the healthcare professionals with the knowledge they need to understand LLM in healthcare research. In this review, the authors discuss the development and evolution of LLMs, especially in healthcare. It provides a description of the key terminologies associated with LLMs to improve the understanding of these terms and their application context in healthcare. It also provides a description of evaluation methodologies used in LLM research. Through illustrative examples of research in the application of LLMs in healthcare, the authors showcase opportunities to leverage this state-of-the-art artificial intelligence (AI) technique with considerations for clinical and administrative adoption by both the patients and healthcare professionals.

Perspective
Artificial Intelligence at the Bedside
By David L Brown, MD; Former Institute Chair Anesthesiology, Cleveland Clinic.
dbrown@clearconsults.com
Clinician-scientists I respect are devoting serious effort to integrating artificial intelligence (AI) into clinical decision-making at the bedside. At first glance, this mirrors the broader trend of AI adoption across our culture.
After nearly 40 years in academic medicine, I’ve had the privilege of mentoring many young physicians. Understandably, those early in training often sought black-and-white answers to clinical questions, preferring simplified steps over nuance. Inexperience naturally nudges many toward streamlined decision-making.
AI, in ordinary terms, builds algorithmic approaches to care by analyzing data from many patients—a population—to guide treatment for an individual. My concern is that the tendency of physicians to seek clarity and speed may align too well with AI’s standardized pathways. Over time, this may erode the nuance essential to individualized care.
This isn’t just theoretical. Having survived a rare and critical illness myself, I know firsthand that subtlety in care—nuance—can make the difference between life and death. When clinical decisions are filtered through population-level algorithms, there is a risk that care becomes normed rather than personalized. Patient autonomy may be diminished by the persuasive authority of the AI-powered bell curve.
The physician-patient relationship has always been sacred because it embraces complexity, not simplicity. Yet, with growing pressure to “produce”—to see more patients in less time—AI may become a tempting shortcut. And the time required for true clinical reflection may continue to shrink.
Most physicians I know genuinely want to do the right thing. But if “the right thing” becomes “what the AI suggests,” especially for early-career clinicians under pressure, will we lose something essential? Will we lose the capacity to see each patient as unique?
My goal—as a long-time physician and now as a patient—is to receive individualized care, not care stripped of nuance. As AI becomes more central to bedside decision-making, we must ask: Will following AI prompts become synonymous with good care? Or will we guard the space for nuance, reflection, and autonomy?
Time will tell.

Learn More

Connect

Register to join our next BrainX Community Live! May 2025 event on Real world application of Generative AI in clinical practice. May 12, 6:00 - 7:00 PM ET via Zoom.

Prof. Ngiam Kee Yuan, National University Hospital, Singapore, will share his experiences with the implementation of LLM in clinical practice in a talk titled “Implementing AI in Clinical Practice”.

Watch the recording of our recent BrainX Community Live! April 2025 event on Vision Model and Agentic AI application in Healthcare.

Speakers: Dileep Raman, MD & Dileep Unnikrishnan, MD, Cloudphysician

In this engaging presentation, the speakers from Cloudphysician discuss applications of Vision AI models and AI agents in healthcare practice. From discussing the opportunities to the technical aspects of the implementation of these state-of-the-art models, this presentation provides key insights into the promises and potential of these models to improve healthcare delivery.

Connect More

Datasets

Tahoe-100M: a giga-scale single-cell atlas

Tahoe-100M is a giga-scale single-cell perturbation atlas consisting of over 100 million transcriptomic profiles from 50 cancer cell lines exposed to 1,100 small-molecule perturbations. Generated using Vevo Therapeutics’ Mosaic high-throughput platform, Tahoe-100M enables deep, context-aware exploration of gene function, cellular states, and drug responses at unprecedented scale and resolution. This dataset is designed to power the development of next-generation AI models of cell biology, offering broad applications across systems biology, drug discovery, and precision medicine.

INSPIRE, an INformative Surgical Patient dataset for Innovative Research Environment

A publicly available research dataset in perioperative medicine, which includes approximately 130,000 cases (50% of all surgical cases) who underwent anesthesia for surgery at an academic institution in South Korea between 2011 and 2020. This comprehensive dataset includes patient characteristics such as age, sex, American Society of Anesthesiologists physical status classification, diagnosis, surgical procedure code, department, and type of anesthesia. It also includes vital signs in the operating theatre, general wards, and intensive care units (ICUs), laboratory results from six months before admission to six months after discharge, and medication during hospitalization. Complications include total hospital and ICU length of stay and in-hospital death.

MOVER: Medical Informatics Operating Room Vitals and Events Repository

MOVER is a repository containing data associated with hospital visits for patients undergoing surgery at the University of California, Irvine (UCI) Medical Center in Orange County, California. The data included in MOVER was collected over 4 years and contains comprehensive EMR records and waveforms for each patient encounter. These records include general information about each patient and their medical history, and specific information regarding the surgical procedure being performed, including: medicines used, lines or drains used, and post-operative complications. The repository includes 58,799 unique patients with data from 83,468 surgeries. MOVER is freely available for download for all researchers who sign a data usage agreement and is intended to advance a wide variety of healthcare research.

PubMed dataset for summarization

Dataset for summarization of long documents, using Pubmed articles summarized with abstracts. The dataset has 3 splits: train, validation, and test. Train data with 119,924 instances, validation data with 6,633 instances and test data with 6,658 instances.

More Data

Conferences

Additional BXC-featured publications

Book

Voices of Innovation – AI Fulfilling the Promise of AI in Healthcare

Edward W. Marx, Sakshika Dhingra, Frank Papay, Piyush Mathur

Pulmonary

Status and Opportunities of Machine Learning Applications in Obstructive Sleep Apnea: A Narrative Review

Matheus Lima Diniz Araujo, Trevor Winger, Samer Ghosn, Carl Saab, Jaideep Srivastava, Louis Kazaglis, Piyush Mathur, Reena Mehra

General/ Generative AI/ LLM

Towards accurate differential diagnosis with large language models

Daniel McDuff, Mike Schaekermann, Vivek Natarajan, et al.

General/ Generative AI/ LLM

Towards conversational diagnostic artificial intelligence

Tao Tu, Mike Schaekermann,Vivek Natarajan, et al.

Join and follow the BrainX community!

Webpage: https://brainxai.org/

Newsletter: https://brainxai.substack.com/subscribe

LinkedIn: https://www.linkedin.com/groups/13599549/

Youtube: https://www.youtube.com/channel/UCua5EiLL6I29hpNrJsdv1rg

Share BrainXAI's Substack

BrainX Waves: The Newsletter of BrainX Community

BrainXAI — Mon, 31 Mar 2025 11:13:22 GMT

Subscribe now

Learn

The TRIPOD-LLM reporting guideline for studies using large language models

Large language models (LLMs) are rapidly being researched and adopted in healthcare, necessitating standardized reporting guidelines. The authors present transparent reporting of a multivariable model for individual prognosis or diagnosis (TRIPOD)-LLM, an extension of the TRIPOD + artificial intelligence statement, addressing the unique challenges of LLMs in biomedical applications. TRIPOD-LLM provides a comprehensive checklist of 19 main items and 50 subitems, covering key aspects from title to discussion. The guidelines introduce a modular format accommodating various LLM research designs and tasks, with 14 main items and 32 subitems applicable across all categories.

Developed through an expedited Delphi process and expert consensus, TRIPOD-LLM emphasizes transparency, human oversight, and task-specific performance reporting. There is also an interactive website (https://tripod-llm.vercel.app/) facilitating easy guideline completion and PDF generation for submission.

Learn More

Connect

BrainX Community, March 2025, Live event on State of AI in Healthcare Research

The BrainX Community Live, March 2025 event, featured a review of the 2024 AI in Healthcare research landscape, as detailed in the related publication "Artificial Intelligence in Healthcare: 2024 Year in Review" by Raghav Awasthi, Sai Prasad Ramachandran, Shreya Mishra, et al. (Medrxiv, doi:10.1101/2025.02.26.25322978).

Key findings from the 2024 review include the dominance of imaging in AI healthcare publications, the rise of large language models, the increased use of text data, and the emergence of multimodal models. The review concludes that AI research publications in healthcare are maturing and growing, with foundation models generating excitement.

Connect More

Datasets

Pathology Images of Scanners and Mobilephones (PLISM) – Original Whole Slide Images Dataset

The Pathology Images of Scanners and Mobilephones (PLISM) dataset was created for the evaluation of AI models’ robustness to domain shifts. PLISM is the first group-wised pathological image dataset that encompasses diverse tissue types stained under 13 H&E conditions with multiple imaging media, including smartphones (7 scanners and 6 smartphones).

In PLISM-sm, smartphone images were used as queries to create image groups for each staining condition corresponding to each tile image. The PLISM-sm subset contains a total of 57,902 images.
In PLISM-wsi, smartphone images were used as queries to create image groups for each staining condition corresponding to each tile image. The PLISM-wsi subset contains a total of 310,947 images
Color and texture in digital pathology images are affected by H&E stain conditions (e.g., Harris or Carrazi) and digitalization devices (e.g., slide scanners or smartphones), which cause inter-institutional domain shifts.

MC-MED: Emergency department physiological waveforms dataset

118,385 emergency department visits with continuous physiological waveforms.

Multimodal Clinical Monitoring in the Emergency Department (MC-MED), a comprehensive, multimodal, and de-identified clinical and physiological dataset. MC-MED includes 118,385 adult ED visits to an academic medical centre from 2020 to 2022. Data include continuously monitored vital signs, physiologic waveforms (electrocardiogram, photoplethysmogram, respiration), patient demographics, medical histories, orders, medication administrations, laboratory and imaging results, and visit outcomes. MC-MED is the first dataset to combine detailed physiologic monitoring with clinical events and outcomes for a large, diverse ED population.

More Data

Conferences

Additional BXC-featured publications

Cardiology/ Gastroenterology/ Generative AI

Opportunistic Screening of Chronic Liver Disease with Deep-Learning–Enhanced Echocardiography

Yuki Sahashi, Milos Vukadinovic, Fatemeh Amrollahi, David Ouyang, Alan C. Kwan, et al.

February 27, 2025

Book

ROI for AI in Medicine: Unlocking the Value: Quantifying the Return on Investment of Healthcare Artificial Intelligence

Rubin Pillay MD PhD

General/ Generative AI/ LLM

Theory of Mind Imitation by LLMs for Physician-Like Human Evaluation

Raghav Awasthi, Shreya Mishra, Charumathi Raghu, Auron Moises, Ashish Atreja, Dwarikanath Mahapatra, Nishant Singh, Ashish K. Khanna, Jacek B. Cywinski, Kamal Maheshwari, Francis A. Papay, Piyush Mathur

Critical Care

Elucidating the causal relationship of mechanical power and lung injury: a dynamic approach to ventilator management

Wu, C., Canakoglu, A., Vine, J. et al.

Exciting Announcement

Introducing the Indian College of AI in Medicine (ICAIM)

We’re excited to share that one of our close collaborators, Doctors AI Community, has officially announced their academic wing—the Indian College of Artificial Intelligence in Medicine (ICAIM).

ICAIM will focus on structured learning, research, and innovation to empower medical professionals in the evolving field of AI in healthcare.

Stay tuned for upcoming learning opportunities, certifications, and collaborative programs from ICAIM!

Join and follow the BrainX community!

Webpage: https://brainxai.org/

Newsletter: https://brainxai.substack.com/subscribe

LinkedIn: https://www.linkedin.com/groups/13599549/

Youtube: https://www.youtube.com/channel/UCua5EiLL6I29hpNrJsdv1rg

Share BrainXAI's Substack

BrainX Waves: The Newsletter of BrainX Community

BrainXAI — Fri, 28 Feb 2025 10:48:03 GMT

Subscribe now

Learn

MedArena: LLM Arena for Clinicians

MedArena is a free platform for clinicians to compare different large language models (LLMs) in an unbiased, head-to-head competition. Similar to other leaderboards, such as those from Huggingface, MedArena is a platform to compare LLMs for different tasks using the Elo rating system to rank models based on human preferences.

For human evaluation of LLM outputs, HumanELY was created based on comprehensive, comparable, and quantifiable metrics using Likert scale.

Learn More

Connect

BrainX Community, January 2025, Live event on Evaluating Large Language Models with frameworks and tools for healthcare

The BrainX Community Live, January 2025 event, featured Suhana Bedi, Stanford University; Dr. Ashish Atreja, UC Davis; VALIDAI & Dr. Yanshan Wang, University of Pittsburgh. The session was moderated by Dr. Piyush Mathur, Cleveland Clinic; BrainX. At the panel discussion, key aspects of human evaluation of LLMs in healthcare, such as metrics for evaluation, evaluation methods, including who and how these need to be performed, and frameworks, were reviewed. Future directions, such as new frameworks, tools, and the possibility of LLMs-as-judge, were also explored.

Connect More

Datasets

DiagSet: a dataset for prostate cancer histopathological image classification

The dataset consists of three different partitions: DiagSet-A, containing over 2.6 million tissue patches extracted from 430 fully annotated scans; DiagSet-B, containing 4675 scans with assigned binary diagnosis; and DiagSet-C, containing 46 scans with diagnosis given independently by a group of histopathologists.

More Data

Conferences

Additional BXC-featured publications

General/ Generative AI/ LLM

Multimodal Medical Code Tokenizer

General/ Generative AI

The TRIPOD-LLM reporting guideline for studies using large language models

General

FUTURE-AI: international consensus guideline for trustworthy and deployable artificial intelligence in healthcare

Join and follow the BrainX community!

Webpage: https://brainxai.org/

Newsletter: https://brainxai.substack.com/subscribe

LinkedIn: https://www.linkedin.com/groups/13599549/

Youtube: https://www.youtube.com/channel/UCua5EiLL6I29hpNrJsdv1rg

Share BrainXAI's Substack

BrainX Waves: The Newsletter of BrainX Community

BrainXAI — Fri, 31 Jan 2025 18:10:44 GMT

Subscribe now

Learn

With a growing number of research, publications, and applications related to Large Language Models in healthcare, there is an ever-growing need for evaluation frameworks and tools. The publications featured below provide key recommendations, frameworks, and tools for evaluation of LLMs in healthcare. These key publications were discussed at the BrainX Community, January 2025 Live event.

Awasthi, R., S. Mishra, D. Mahapatra, A. Khanna, K. Maheshwari, J. Cywinski, F. Papay and P. Mathur (2023). “HumanELY: Human evaluation of LLM yield, using a novel web-based evaluation tool.” medRxiv: 2023.2012.2022.23300458.

Bedi S, Liu Y, Orr-Ewing L, et al. Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review. JAMA. Published online October 15, 2024. doi:10.1001/jama.2024.21700

Tam, T.Y.C., Sivarajkumar, S., Kapoor, S. et al. A framework for human evaluation of large language models in healthcare derived from literature review. npj Digit. Med. 7, 258 (2024). https://doi.org/10.1038/s41746-024-01258-7

Learn More

Connect

BrainX Community,January 2025, Live event on Evaluating Large Language Models with frameworks and tools for healthcare

The BrainX Community Live, January 2025 event, featured Suhana Bedi, Stanford University; Dr. Ashish Atreja, UC Davis; VALIDAI & Dr. Yanshan Wang, University of Pittsburgh. The session was moderated by Dr. Piyush Mathur, Cleveland Clinic;BrainX. At the panel discussion, key aspects of human evaluation of LLMs in healthcare, such as metrics for evaluation, evaluation methods, including who and how these need to be performed, and frameworks, were reviewed. Future directions, such as new frameworks, tools, and the possibility of LLMs-as-judge, were also explored.

Connect More

Datasets

MEDEC Dataset (MEDICAL ERROR DETECTION AND CORRECTION IN CLINICAL NOTES)

It includes 3,848 clinical texts from the MS and University of Washington hospital collections covering five types of errors (Diagnosis, Management, Treatment, Pharmacotherapy, and Causal Organism).

BiomedParseData: A foundation model dataset for joint segmentation, detection and recognition of biomedical objects

BiomedParseData by combining 45 biomedical image segmentation datasets and using GPT-4 to generate the canonical semantic label for each segmented object. GPT-4 was used to create a unifying biomedical object ontology for image analysis and harmonise natural language descriptions with this ontology. This ontology encompasses three main categories (histology, organ, and abnormality), 15 meta-object types, and 82 specific object types.

The resulting BiomedParseData contains 3.4 million distinct image-mask-label triples, spanning nine imaging modalities and 25 anatomic sites, representing a large-scale and diverse dataset for semantic-based biomedical image analysis. The images include CT scan, MRI, chest X-ray, ultrasound, skin lesion photos, endoscopy images, pathology whole slide images, and eye OCTs.

More Data

Podcast

In this episode, we feature Dr. Amol M. Joshi. Dr. Joshi is the Thomas H. Davis Professor in Business at Wake Forest University (WFU). He holds a joint faculty appointment as an Associate Professor of Strategic Management in the WFU School of Business and as an Associate Professor of Innovation & Commercialisation in the WFU School of Medicine.

Dr. Joshi is an inventor who helps others reinvent themselves. His passion is guiding aspiring entrepreneurs, executives, and students of all levels to pursue their business dreams. He has trained startup founders, corporate managers, and industry leaders in the US, Austria, Denmark, and Vietnam.

A recognised expert on innovation, his interdisciplinary research examines how inventors create and commercialise new technologies and launch and fund new ventures. With a 13-year prior career as an engineer and entrepreneur in venture capital-funded startups in Silicon Valley, Dr. Joshi co-invented two patents for AI-based voice assistant products. He co-founded BeVocal, a speech recognition software startup acquired by Nuance Communications, and served as VP of Sales & Marketing from 1999 to 2003.

Dr. Joshi has advised Federal agencies, including NASA, NSF, NIH, and the US Department of Energy, on national innovation policies and R&D grant programs for small businesses. Dr. Joshi earned a PhD in Business Administration from UNC Chapel Hill, an MBA and MS in Engineering Sciences from Dartmouth College, and a BS in Electrical Engineering from Georgia Tech with highest honours.

Listen More

Conferences

Additional BXC-featured publication

General

Determinants and performance outcomes of artificial intelligence adoption: Evidence from U.S. Hospitals

Join and follow the BrainX community!

Webpage: https://brainxai.org/

Newsletter: https://brainxai.substack.com/subscribe

LinkedIn: https://www.linkedin.com/groups/13599549/

Youtube: https://www.youtube.com/channel/UCua5EiLL6I29hpNrJsdv1rg

Share BrainXAI's Substack

BrainX Waves: The Newsletter of BrainX Community

BrainXAI — Mon, 30 Dec 2024 09:59:26 GMT

Subscribe now

Happy Holidays and Happy New Year!

Learn

2024 was the year of strategies. After the release of multiple LLMs and frameworks last year, the key focus for 2024 was on developing strategies for their adoption. Many of the strategic guidance were focused on educating ourselves about the technical needs and the value proposition, and many others on evaluation and implementation. Below are some of the key BrainXAI ReSearch publications and more focused on these strategies from this year. 2025 is likely to focus on evaluation of the concepts, the experiments and the solutions we developed for adoption of AI in healthcare.

Navigating AI: A Quick Start Guide for Healthcare Professionals

Strategies for Artificial Intelligence in Healthcare: Real-World Solutions for Reshaping Healthcare Through Artificial Intelligence and Beyond

Strategic and Value-based Implementation of Generative AI in Healthcare

A framework for human evaluation of large language models in healthcare derived from literature review

Learn More

Connect

BrainX Community Live! December 2024: Foundation models in Cardiology – Automating Echocardiography with Echoprime.

The BrainX Community Live event featured a presentation on AI for comprehensive transthoracic echocardiogram exam interpretation, showcasing the EchoPrime model's capabilities in automating echocardiography analysis and report generation.

EchoPrime, a multi-view, view-informed, video-based vision-language foundation model trained on over 12 million video-report pairs. The presenters discussed the model's technical aspects, training process, and performance, highlighting its potential to improve healthcare outcomes and democratize access to medical expertise.

The model was trained on 2 Nvidia Rtx A 6,000 Gpus with a batch size of 32. The training took approximately one month. The model was trained to analyze echocardiograms and use synthesized information across the whole study. The results showed that as more data was included, the accuracy progressively improved. EchoPrime outperformed previous models on all tasks, particularly those requiring temporal information.

The presenters also compared EchoPrime with task-specific models and found that EchoPrime either outperformed or matched their performance. The event concluded with discussions on the future directions of AI in healthcare, including the integration of more medical modalities and the importance of collaboration between clinicians and engineers in model development and validation.

Presenters: Milos Vukadnovic, PhD Candidate at UCLA; I-Min Chiu, MD, PhD

Related publication: EchoPrime: A Multi-Video View-Informed Vision-Language Model for Comprehensive Echocardiography Interpretation. Milos Vukadinovic, Xiu Tang, Neal Yuan, Paul Cheng, Debiao Li, Susan Cheng, Bryan He, David Ouyang.

Join our January 2025, BrainX Community Live event on Evaluating Large Language Models with frameworks and tools for healthcare

Registration Link: https://us02web.zoom.us/meeting/register/n79nZz6xTZOIqaISs9qDSA

Program agenda: Panel discussion on tools and frameworks for LLM evaluations in healthcare.

Featured panelists:

- Dr. Ashish Atreja, UC Davis; VALIDAI

- Dr. Yanshan Wang, University of Pittsburgh

Moderator: Dr. Piyush Mathur, Cleveland Clinic; BrainX

Connect More

Datasets

UniTox: drug-induced toxicity dataset

A unified dataset of 2,418 FDA-approved drugs with drug-induced toxicity summaries and ratings created by using GPT-4o to process FDA drug labels. UniTox spans eight types of toxicity: cardiotoxicity, liver toxicity, renal toxicity, pulmonary toxicity, hematological toxicity, dermatological toxicity, ototoxicity, and infertility.

MedConceptsQA

An open source benchmark for medical concepts question answering. The benchmark comprises questions of various medical concepts across different vocabularies: diagnoses, procedures, and drugs. The questions are categorized into three levels of difficulty: easy, medium, and hard.

More Data

Podcast

A recognized expert on innovation, his interdisciplinary research examines how inventors create and commercialize new technologies and launch and fund new ventures. With a 13-year prior career as an engineer and entrepreneur in venture capital-funded startups in Silicon Valley, Dr. Joshi co-invented two patents for AI-based voice assistant products. He co-founded BeVocal, a speech recognition software startup acquired by Nuance Communications, and served as VP of Sales & Marketing from 1999 to 2003.

Dr. Joshi has advised Federal agencies including NASA, NSF, NIH, and the US Department of Energy on national innovation policies and R&D grant programs for small businesses. Dr. Joshi earned a PhD in Business Administration from UNC Chapel Hill, an MBA and MS in Engineering Sciences from Dartmouth College, and a BS in Electrical Engineering from Georgia Tech with highest honors.

Listen More

Conferences

The Doctors AI Global Summit on AI in Healthcare, held virtually on December 14 and 15, 2024, was a landmark event that brought together over 1,000 attendees from around the globe. The summit featured in-depth discussions on the transformative role of artificial intelligence in healthcare through 8 keynote addresses, 14 plenary sessions, and 8 panel discussions. More than 100 esteemed faculty members and 30 international experts across various specialties contributed to rich dialogues on a range of topics, from AI-driven patient care to responsible AI innovations.

Keynote presentations explored critical issues such as the current state and future directions of AI in healthcare, ethical considerations in AI-driven healthcare, and the potential and challenges of validating and deploying Large Language Models (LLMs) effectively within medical contexts. The summit also highlighted the integration of AI in various medical subspecialties, public health transformation, AI education for medical professionals, and revolutionary changes AI is bringing to drug discovery and development.

A notable panel on AI ethics emphasized the importance of maintaining ethical standards in the deployment of AI technologies in healthcare settings. The event also showcased an oral presentation competition that spotlighted cutting-edge research, including predictive AI models for surgical outcomes. Overall, the summit emphasized AI’s role in enhancing healthcare delivery, fostering ethical innovation, and improving patient outcomes, marking a significant advancement in the global healthcare landscape.

Additional BXC-featured publications

Cardiology/General

Cost-Effectiveness of Artificial Intelligence-Enabled Electrocardiograms for Early Detection of Low Ejection Fraction

General

Tackling algorithmic bias and promoting transparency in health datasets: the STANDING Together consensus recommendations

Generative AI/LLM

Matching patients to clinical trials with large language models (TrialGPT)

Book/General

Artificial Intelligence and Machine Learning in Health Care and Medical Science: Best Practices and Pitfalls

Generative AI/LLM

MedMobile: A mobile-sized language model with expert-level clinical capabilities

Generative AI/LLM

Large Language Model Influence on Diagnostic Reasoning: A Randomized Clinical Trial

BrainXAI's Substack

BrainX Waves: The Newsletter of BrainX Community

Does Healthcare AI Always Need the Biggest Model?

Reference

Datasets (Open source)

General

General/Generative AI/LLM

Generative AI/LLM

Cardiology/General

Additional BXC-featured publications

Generative AI/LLM

Atreja, A., & Mathur, P.

Critical Care

Kuang, C., Valderrama, C.E., Stelfox, H.T., et al.

Generative AI/LLM

Ting Fang Tan , Kabilan Elangovan , Jasmine Ong , et al.

Generative AI/Imaging/Radiology/LLM

Chen, W., Dong, Y., Ding, Z. et al.

Join and follow the BrainX community!

BrainX Waves: The Newsletter of BrainX Community

References

Datasets (Open source)

General

Surgery

Emergency Medicine

Additional BXC-featured publications

GE Healthcare

Pan, J., Jian, B., Hager, P. et al.

Stanislaw P. Stawicki, Thomas R. Wojda, Andries Engelbrecht

Kondepudi, A., Rao, A., Zhao, C. et al.

Gu, Y., Fu, J., Liu, X. et al.

Join and follow the BrainX community!

BrainX Waves: The Newsletter of BrainX Community

Datasets (Open source)

General

Surgery

Emergency Medicine

General/LLM

Additional BXC-featured publications

Xizhi Wu, Hang Zhang, Yanshan Wang, et al.

Liévin, V., Palepu, A., Weng, WH. et al.

Ferber, D., Hilgers, L., Höper, C. et al.

Vishwanath, K., Alyakin, A., Ghosh, M. et al.

Join and follow the BrainX community!

BrainX Waves: The Newsletter of BrainX Community

Datasets (Open source)

Generative AI/Surgery

Pathology

General

General/Generative AI

Generative AI/Surgery

Critical Care/Pediatrics

Additional BXC-featured publications

Ihsan Ayyub Qazi, Ayesha Ali, Asad Ullah Khawaja, Muhammad Junaid Akhtar, Ali Zafar Sheikh, Muhammad Hamad Alizai

Kurt Miller, Qiuhao Lu, William Hersh , Kirk Roberts , Steven Bedrick , Andrew Wen , Hongfang Liu

Costa-Gomes, B., Tolmachev, P., Taysom, E. et al.

Robert Hoyt

Join and follow the BrainX community!

BrainX Waves: The Newsletter of BrainX Community

Datasets (Open source)

Additional BXC-featured publications

Luo, L., Kim, S.E., Zhang, X. et al.

Dr. Ruchi Saxena

Dr. Ruchi Saxena

Arya S. Rao; Kaiz P. Esmail; Richard S. Lee, et al.

Costa-Gomes, B., Tolmachev, P., Taysom, E. et al.

Nicholas B Tiller, Alessandro R Marcon, Marco Zenone, et al.

Liu, Y., Carrero, Z.I., Jiang, X. et al.

Lisa S. Rotenstein; A. Jay Holmgren; Robert Thombley; et al.

Join and follow the BrainX community!

BrainX Waves: The Newsletter of BrainX Community

Datasets (Open source)

Additional BXC-featured publications

Yu, Y., Hu, X., Rajaganapathy, S. et al.

Naylor, Amanda; Awasthi, Raghav; Aggarwal, Avneep; Bhattacharyya, Anirban; Mishra, Shreya; Mathur, Piyush

Ana Bracic; Kayte Spector-Bagdady; Sophie Towle, et al.

Hwang, YM., Ng, M.Y., Pillai, M. et al.

Editor in Chief: Dr Amit Kumar Dey

Join and follow the BrainX community!

BrainX Waves: The Newsletter of BrainX Community