GSgnnPerEtypeMrrLPEvaluator
- class graphstorm.eval.GSgnnPerEtypeMrrLPEvaluator(eval_frequency, eval_metric_list=None, major_etype='ALL', use_early_stop=False, early_stop_burnin_rounds=0, early_stop_rounds=3, early_stop_strategy='average_increase')
Bases:
GSgnnBaseEvaluator,GSgnnLPRankingEvalInterfaceEvaluator for Link Prediction tasks using
mrras metric, and return per edge typemrrscores.Parameters
- eval_frequency: int
The frequency (number of iterations) of doing evaluation.
- eval_metric_list: list of string
Evaluation metrics used during evaluation. Default: [“mrr”].
- major_etype: tuple
A canonical edge type used for selecting the best model. Default: will use the summation of
mrrvalues of all edge types.- use_early_stop: bool
Set true to use early stop. Default: False.
- early_stop_burnin_rounds: int
Burn-in rounds (number of evaluations) before starting to check for the early stop condition. Default: 0.
- early_stop_rounds: int
The number of rounds (number of evaluations) for validation scores used to decide early stop. Default: 3.
- early_stop_strategy: str
The early stop strategy. GraphStorm supports two strategies: 1)
consecutive_increase, and 2)average_increase. Default:average_increase.
- evaluate(val_rankings, test_rankings, total_iters)
GSgnnLinkPredictionTrainerandGSgnnLinkPredictionInferrerwill call this function to compute validation and testmrrscores.Parameters
- val_rankings: dict of tensors
Rankings of positive scores of validation edges for each edge type in the format of {etype: ranking}.
- test_rankings: dict of tensors
Rankings of positive scores of test edges for each edge type in the format of {etype: ranking}.
- total_iters: int
The current iteration number.
Returns
- val_score: dict of dict
Validation
mrrscore in the format of {“mrr”: {etype: val_score}}. If theval_rankingis None, return {“mrr”: “N/A”}.- test_score: dict of dict
Test
mrrscore in the format of {“mrr”: {etype: test_score}}. If thetest_rankingis None, return {“mrr”: “N/A”}.
- compute_score(rankings, train=True)
Compute per edge type
mrrevaluation score.Parameters
- rankings: dict of tensors
Rankings of positive scores in the format of {etype: ranking}.
- train: boolean
If in model training.
Returns
- return_metrics: dict of dict
Per edge type evaluation
mrrscore in the format of {“mrr”: {etype: score}}.
- get_val_score_rank(val_score)
Get the rank of the validation score of the
major_etypeinitialized in class initialization by comparing its value to the existing historical values. If use the defaultmajor_etype, will use the summation of validation values of all edge types to get the rank.Parameters
- val_score: dict of dict
A dict in the format of {“mrr”: {etype: score}}.
Returns
- rank: int
The rank of the validation score of the given
major_etypeinitialized in class initialization. If using the defaultmajor_etype, the rank will be computed based on the summation of validation scores for all edge types.