Fairness Evaluator

fair-eval is a pipeline for extracting and evaluating FAIR evidence in scientific articles.

At this stage, the project is mainly focused on creating and preparing a reference dataset for FAIRness annotation. The current pipeline is a base for future work toward more automatic FAIRness assessment methods.

Fair refers to the FAIR data principles for data management and stewardship: Findable, Acessible, Interoperable and Reusable.

The pipeline, as of right now, can do the following:

• Convert PDFs to the TEI XML format using GROBID
• generate FAIR annotations using the configured Ollama endpoint and models
• combine model outputs and define a final FAIR score before export
• export the resolved dataset to CSV
• create Label Studio tasks for manual review

Project context

This project was developed during an internship at FCUP - Faculty of Sciences of the University of Porto, as part of the Computational Biology (CompBio) group at CIBIO - Research Centre in Biodiversity and Genetic Resources, supervised by Nuno Fonseca.

Table of Contents

• Prerequisites
• Installation
• Configuration
• Usage
• Label Studio Setup
• Project Structure

Prerequisites

• Git
• Conda-compatible environment manager
• GROBID for PDF to XML conversion
• an Ollama API endpoint configured in config/default.yaml for creating FAIR scores with LLMs
• Label Studio for optional manual review

Useful links:

• GROBID: https://grobid.readthedocs.io/en/latest/Install-Grobid/
• Ollama: https://ollama.com/download (don't forget to add the necessary drivers to run on the gpu)
• Label Studio: https://labelstud.io/guide/install.html

Installation

Clone the repo:

git clone https://github.com/joaoplazevedo/fair-eval
cd fair-eval

Create the conda environment:

conda env create -f environment.yaml
conda activate fair-eval

Check the CLI:

python pipeline.py --help

Configuration

The pipeline reads its settings from:

config/default.yaml

Before running it, check the input and output paths:

paths:
  pdf_dir: "data/pdfs_v1"
  xml_dir: "data/xmls_v1"
  ollama_output_dir: "outputs/fair_principles_v1"
  labelstudio_output: "outputs/tasks_v1.json"
  csv_output: "outputs/fair_dataset_v1.csv"

Put the PDFs to process inside pdf_dir. If pdf_dir contains subfolders, the same folder structure is preserved in xml_dir.

Ollama is currently the only supported LLM provider. The API endpoint and models can be changed in the ollama section of config/default.yaml.

grobid:
  url: "http://localhost:8070/api/processFulltextDocument"

ollama:
  url: "http://localhost:11434/api/generate"
  models:
    - name: "gpt-oss:20b"
      dir_name: "gpt-oss_20b"

A local config can also be created:

config/local.yaml

Warning: config/local.yaml is not a partial override. If you create it, copy the full structure from config/default.yaml and edit the values you need.

Usage

All commands read from the config files.

Command	Description	Flags
python pipeline.py convert-pdf-to-xml	convert PDFs to TEI XML	--overwrite, -o
python pipeline.py generate-llm-annotations	generate FAIR annotations	--overwrite, -o
python pipeline.py export-to-csv	export the resolved dataset to CSV	none
python pipeline.py create-labelstudio-tasks	create Label Studio tasks	none
python pipeline.py run	run the full pipeline	--overwrite-xml, --overwrite-llm

Global flag:

python pipeline.py --verbose <command>

Label Studio Setup

The Label Studio exporter currently supports local PDF files only. Cloud storage requires changing fair_eval/create_labelstudio_tasks.py script directly.

The generated tasks follow the label studio requirement for local storage:

/data/local-files/?d=path/file

Take a look at the docs: https://labelstud.io/guide/storage_local (In the future, I will add more details about this)

The task file and interface template are defined in:

outputs/tasks_v1.json
config/label-studio-template.xml

Tasks are imported manually through the Label Studio gui.

Project structure

fair-eval/
├── archive/                           # old scripts for OpenAlex API access, rule-based pipeline, etc...
├── config/
│   ├── default.yaml                   # main config
│   ├── local.yaml                     # optional local config, gitignored
│   └── label-studio-template.xml      # Label Studio interface
├── fair_eval/
│   ├── config.py                      # loads config
│   ├── convert_grobid.py              # PDF -> TEI XML
│   ├── fair_llm.py                    # LLM FAIR annotations
│   ├── resolve_outputs.py             # merge/resolve model outputs
│   ├── export_csv.py                  # export resolved dataset to CSV
│   └── create_labelstudio_tasks.py    # export Label Studio tasks
├── prompts/
│   ├── fair_principles_prompt.txt
│   └── fair_principles_system.txt
├── pipeline.py                        # CLI/pipeline entry point
├── environment.yaml
└── README.md