GSgnnMrrLPEvaluator

class graphstorm.eval.GSgnnMrrLPEvaluator(eval_frequency, eval_metric_list=None, use_early_stop=False, early_stop_burnin_rounds=0, early_stop_rounds=3, early_stop_strategy='average_increase')

Bases: GSgnnBaseEvaluator, GSgnnLPRankingEvalInterface

Evaluator for Link Prediction tasks using mrr as metric.

A built-in evaluator for Link Prediction tasks. It uses mrr as the default eval metric, which implements the GSgnnLPRankingEvalInterface.

To create a customized Link Prediction evaluator that use an evaluation metric other than mrr, users might need to 1) define a new evaluation interface if the evaluation method requires different input arguments; 2) inherite the new evaluation interface in a customized Link Prediction evaluator; 3) define a customized Link Prediction Trainer/Inferrer to call the customized Link Prediction evaluator.

Parameters

eval_frequency: int

The frequency (number of iterations) of doing evaluation.

eval_metric_list: list of string

Evaluation metrics used during evaluation. Default: [“mrr”].

use_early_stop: bool

Set true to use early stop. Default: False.

early_stop_burnin_rounds: int

Burn-in rounds (number of evaluations) before starting to check for the early stop condition. Default: 0.

early_stop_rounds: int

The number of rounds (number of evaluations) for validation scores used to decide early stop. Default: 3.

early_stop_strategy: str

The early stop strategy. GraphStorm supports two strategies: 1) consecutive_increase, and 2) average_increase. Default: average_increase.

evaluate(val_rankings, test_rankings, total_iters)

GSgnnLinkPredictionTrainer and GSgnnLinkPredictionInferrer will call this function to compute validation and test mrr scores.

Parameters

val_rankings: dict of tensors

Rankings of positive scores of validation edges for each edge type in the format of {etype: ranking}.

test_rankings: dict of tensors

Rankings of positive scores of test edges for each edge type in the format of {etype: ranking}.

total_iters: int

The current iteration number.

Returns

val_score: dict

Validation mrr score in the format of {“mrr”: val_score}. If the val_ranking is None, return {“mrr”: “N/A”}.

test_score: dict

Test mrr score in the format of {“mrr”: test_score}. If the test_ranking is None, return {“mrr”: “N/A”}.

compute_score(rankings, train=True)

Compute mrr evaluation score.

Parameters

rankings: dict of tensors

Rankings of positive scores in the format of {etype: ranking}

train: boolean

If in model training.

Returns

return_metrics: dict

Evaluation mrr score of in the format of {“mrr”: score}.