Other Datasets

News

  • JSFactCheckBench dataset is released. (8/31/2026)
  • msts-japanese dataset is released. (7/8/2026)

JSFactCheckBench dataset

Overview

JSFactCheckBench is a Japanese benchmark for evaluating the fact-checking performance of LLM responses.
For 85 questions derived from misinformation that circulated on Japanese social media (excerpts from JSocialFact), 170 responses generated by 11 different LLMs were decomposed into 1,952 claims, and each claim was manually annotated along the following two axes:

  1. Factuality label: indicates whether the content of the claim is consistent with the facts
  2. Importance label: indicates the role the claim plays in answering the question

In addition, for each claim, up to 10 links are provided: links to the web pages referenced as evidence for the factuality judgment (along with the relevant passages), as well as the links to the web pages used as search results. For details on the dataset, please refer to our paper, which will be published at a later date.

Usage
  • The dataset is available here. Please note that redistribution of the dataset is prohibited.
  • Due to its nature, this dataset contains questions and LLM responses that include disinformation and misinformation. Please be aware of this, and use the dataset solely for the purpose of improving the safety of LLMs.

Contact: ac_dataset(at)nii.ac.jp

msts-japanese dataset

Overview

msts-japanese is a Japanese dataset designed for multimodal safety improvement and evaluation. It consists of a Japanese translation of prompts and reference answers for a portion of the existing MSTS dataset (330 out of 400 cases). In addition, for 82 of those cases, we added Japan-local images (41 of them provided by the courtesy of Amana Images’s Qlean Dataset) along with specific prompts and reference answers for them. The current release contains only the extended portion of the data and does not include the original MSTS data.

Usage
  • Download the dataset from here. Re-distribution of this dataset is prohibited.
  • This dataset contains offensive, unsafe or inappropriate expressions. Use the dataset with that in mind, and use it only for improving the LLM safety.

Contact：ac_dataset (at) nii.ac.jp

Disclaimers

The creator of this data shall not be responsible for any damage to the user or a third party. In addition, the creator shall not be responsible for any damage to the users or third parties due to delays, interruptions, or suspensions of the provision of this data service. The creator may suspend or discontinue the service of this data or modify the information contained in this data without prior notice.