Auto Evaluation Toolkit#

Automatically evaluate your model and monitor changes of metrics during the training process.

Preparation#

  1. Multiple GPU machines (at least 2, one for evaluation, the other for training).

  2. Mount a shared file system (e.g., NAS) to the same path (e.g., /mnt/shared) on the above machines.

  3. Install Data-Juicer in the shared file system (e.g., /mnt/shared/code/data-juicer).

  4. Install thirdparty dependencies (Megatron-LM and HELM) accoroding to thirdparty/README.md on each machine.

  5. Prepare your dataset and tokenizer, preprocess your dataset with Megatron-LM into mmap format (see README of Megatron-LM for more details) in the shared file system (e.g., /mnt/shared/dataset).

  6. Run Megatron-LM on training machines and save the checkpoint in the shared file system (e.g., /mnt/shared/checkpoints).

Usage#

Use evaluator.py to automatically evaluate your models with HELM and OpenAI API.

python tools/evaluator.py  \
    --config <config>      \
    --begin-iteration     <begin_iteration>     \
    [--end-iteration      <end_iteration>]      \
    [--iteration-interval <iteration_interval>] \
    [--check-interval <check_interval>]         \
    [--model-type     <model_type>]             \
    [--eval-type      <eval_type>]

  • config: a yaml file containing various settings required to run the evaluation (see Configuration for details)

  • begin_iteration: iteration of the first checkpoint to be evaluated

  • end_iteration: iteration of the last checkpoint to be evaluated. If not set, continuously monitor the training process and evaluate the generated checkpoints.

  • iteration_interval: iteration interval between two checkpoints, default is 1000 iterations

  • check_interval: time interval between checks, default is 30 minutes

  • model_type: type of your model, support megatron and huggingface for now

    • megatron: evaluate Megatron-LM checkpoints (default)

    • huggingface: evaluate HuggingFace model, only support gpt eval type

  • eval-type: type of the evaluation to run, support helm and gpt for now

    • helm: evaluate your model with HELM (default), you can change the benchmarks to run by modifying the helm specific template file

    • gpt: evaluate your model with OpenAI API, more details can be found in gpt_eval/README.md.

e.g.,

python evaluator.py --config <config_file> --begin-iteration 2000 --iteration-interval 1000 --check-interval 10

will use HELM to evaluate a Megatron-LM checkpoint every 1000 iterations starting from iteration 2000, and check whether there is a new checkpoint meets the condition every 10 minutes.

After running the evaluator.py, you can use recorder/wandb_writer.py to visualize the evaluation results, more details can be found in recorder/README.md.

Configuration#

The format of config_file is as follows:

auto_eval:
  project_name: <str> # your project name
  model_name: <str>   # your model name
  cache_dir: <str>    # path of cache dir
  megatron:
    process_num: <int>     # number of process to run megatron
    megatron_home: <str>   # root dir of Megatron-LM
    checkpoint_path: <str> # path of checkpoint dir
    tokenizer_type: <str>  # support gpt2 or sentencepiece for now
    vocab_path: <str>      # configuration for gpt2 tokenizer type, path to vocab file
    merge_path: <str>      # configuration for gpt2 tokenizer type, path to merge file
    tokenizer_path: <str>  # configuration for sentencepiece tokenizer type, path to model file
    max_tokens: <int>      # max tokens to generate in inference
    token_per_iteration: <float> # billions tokens per iteration
  helm:
    helm_spec_template_path: <str> # path of helm spec template file, default is tools/evaluator/config/helm_spec_template.conf
    helm_output_path: <str>  # path of helm output dir
    helm_env_name: <str>     # helm conda env name
  gpt_evaluation:
    # openai config
    openai_api_key: <str>       # your api key
    openai_organization: <str>  # your organization
    # files config
    question_file: <str>  # default is tools/evaluator/gpt_eval/config/question.jsonl
    baseline_file: <str>  # default is tools/evaluator/gpt_eval/answer/openai/gpt-3.5-turbo.jsonl
    prompt_file: <str >   # default is tools/evaluator/gpt_eval/config/prompt.jsonl
    reviewer_file: <str>  # default is tools/evaluator/gpt_eval/config/reviewer.jsonl
    answer_file: <str>    # path to generated answer file
    result_file: <str>    # path to generated review file
On this page

This Page