PubMedQA

A Dataset for Biomedical Research Question Answering

About

The task of PubMedQA is to answer research questions with yes/no/maybe (e.g.: Do preoperative statins reduce atrial fibrillation after coronary artery bypass grafting?) using the corresponding abstracts.

For more details about PubMedQA, please refer to this paper:

Dataset

PubMedQA has 1k expert labeled, 61.2k unlabeled and 211.3k artificially generated QA instances.

Please visit our GitHub repository to download the dataset:

Submission

To submit your model, please follow the instructions in the GitHub repository.

Citation

If you use PubMedQA in your research, please cite our paper by:

@inproceedings{jin2019pubmedqa,
  title={PubMedQA: A Dataset for Biomedical Research Question Answering},
  author={Jin, Qiao and Dhingra, Bhuwan and Liu, Zhengping and Cohen, William and Lu, Xinghua},
  booktitle={Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  pages={2567--2577},
  year={2019}
}
Leaderboard (reasoning-required setting)
Model Code Size Accuracy (%) Macro-F1 (%)
1
Nov 28, 2023
GPT-4 (Medprompt)
Microsoft
(Nori et al. 2023)
NA 82.0 NA
2
May 16, 2023
Med-PaLM 2
Google Research & DeepMind
(Singhal et al. 2023)
NA 81.8 NA
3
Nov 27, 2023
MEDITRON
EPFL
(Chen et al. 2023)
70B 81.6 NA
4
Jul 6, 2023
Palmyra-Med
Writer Inc.
(Kamble et al. 2023)
40B 81.1 NA
5
Dec 1, 2023
AntGLM-Med
Ant Group
(Li et al. 2023)
10B 80.6 NA
6
Apr 12, 2023
GPT-4-base
Microsoft & OpenAI
(Nori et al. 2023)
NA 80.4 NA
7
Mar 4, 2024
Claude 3
Anthropic
(Anthropic, 2024)
NA 79.7 NA
8
Jan 11, 2023
GPT-3.5 + Z-Code++
Microsoft Azure AI
(He et al. 2022)
175B 79.6 55.8
9
Dec 26, 2022
Flan-PaLM (3-shot)
Google Research & DeepMind
(Singhal et al. 2022)
540B 79.0 NA
10
Mar 14, 2024
HEAL
DeepScribe Inc
(Yuan et al. 2024)
13B 78.4 NA
11
Dec 20, 2022
Codex (5-shot)
Technical University of Denmark & Copenhagen University Hospital
(LiƩvin et al. 2022)
175B 78.2 NA
12
Sep 13, 2019
Human Performance
University of Pittsburgh & Carnegie Mellon University
(Jin et al. 2019)
NA 78.0 72.2
13
Nov 16, 2022
Galactica
Meta AI
(Taylor et al. 2022)
120B 77.6 NA
14
May 22, 2023
GatorTronGPT
University of Florida & NVIDIA
(Peng et al. 2023)
20B 77.6 NA
15
Mar 4, 2024
MediSwift-XL
Cerebras Systems
(Thangarasa et al. 2024)
1.2B 76.8 NA
16
Mar 20, 2023
GPT-4
Microsoft & OpenAI
(Nori et al. 2023)
NA 75.2 NA
17
Apr 18, 2024
Reka Core
Reka
(Ormazabal et al. 2024)
NA 74.6 NA
18
Dec 15, 2022
PubMedGPT
Stanford University
(Bolton et al. 2022)
2.7B 74.4 NA
19
Oct 17, 2022
DRAGON
Stanford University & EPFL
(Yasunaga et al. 2022)
360M 73.4 NA
20
Apr 27, 2023
PMC-LLaMA
Shanghai Jiao Tong University
(Wu et al. 2023)
7B 73.4 NA
21
Mar 29, 2022
BioLinkBERT (large)
Stanford University
(Yasunaga et al. 2022)
340M 72.2 NA
22
Mar 29, 2022
BioLinkBERT (base)
Stanford University
(Yasunaga et al. 2022)
110M 70.2 NA
23
Sep 13, 2019
BioBERT (multi-phase tuning)
University of Pittsburgh & Carnegie Mellon University
(Jin et al. 2019)
110M 68.1 52.7
24
June, 2021
BioELECTRA
SAAMA AI Research Lab
(Kanakarajan et al. 2019)
110M 64.0 NA
25
July 31, 2020
PubMedBERT
Microsoft Research
(Gu et al. 2020)
110M 55.8 NA