Evaluation Results Recorder#

Record your evaluation results to W&B (wandb) with wandb_writer.py.

With wandb_writer.py, you can:

  • visualize the changes of evaluation metrics of your model during the training process Metrics

  • make a leaderboard to compare the metrics of different models Leaderboard

Usage#

python wandb_writer.py --config <config_file> [--print-only]

  • config_file: path to the configuration file (see Configuration for details)

  • --print-only: only print the result to command line, do not write to wandb

Configuration#

We provided three example files in config folder for three different cases.

The general format is as follows:

project: <str>   # your wandb project name
base_url: <str>  # your wandb instance url
# other specific configuration items

Parse from HELM output#

The following configuration is used to parse evaluation results from HELM output folder and record them to wandb.

# general configurations
# ...

evals:  # evaluations to record
  - eval_type: helm    # only support helm for now
    model_name: <str>  # your model name
    source: helm  # use helm to parse from helm output directory
    helm_output_dir: <your helm output dir path>
    helm_suite_name: <your helm suite name>
    token_per_iteration: <tokens per iteration in billions>
    benchmarks:  # benchmark metrics to be recorded, and below are some examples
      - name: mmlu
        metrics:
          - EM
      - name: boolq
        metrics:
          - EM
      - name: narrative_qa
        metrics:
          - F1
      - name: hellaswag
        metrics:
          - EM
      - ...

We use 16 core metrics of HELM as the default benchmarks if the benchmarks field is not provided, the 16 metrics are as follows:

mmlu.EM, raft.EM, imdb.EM, truthful_qa.EM, summarization_cnndm.ROUGE-2, summarization_xsum.ROUGE-2, boolq.EM, msmarco_trec.NDCG@10, msmarco_regular.RR@10, narrative_qa.F1, natural_qa_closedbook.F1, natural_qa_openbook_longans.F1, civil_comments.EM, hellaswag.EM, openbookqa.EM

Parse from configuration file#

The scores of metrics can be given directly in the configuration file, the following is an example.

# general configurations
# ...

evals:  # evaluations to record
  - eval_type: helm
    model_name: llama-7B  # your model name
    source: file  # use file to parse from configuration
    token_num: 1000
    eval_result:  # evaluation results to be recorded
      mmlu:
        EM: 0.345
      boolq:
        EM: 0.751
      narrative_qa:
        F1: 0.524
      hellaswag:
        EM: 0.747
      ...

Make leaderboard#

The following configuration is used to make a leaderboard.

# general configurations
# ...
leaderboard: True
leaderboard_metrics:  # metrics required for the leaderboard
  - mmlu.EM
  - boolq.EM
  - quac.F1
  - hellaswag.EM
  - ...
excluded_models:   # models that do not participate in the leaderboard
  - <model to exclude>
  - ...

We use 16 core metrics of HELM as the default leaderboard metrics if the leaderboard_metrics field is not provided, the 16 metrics are as same as the default benchmark metrics.