convlab2.evaluator package¶
Submodules¶
convlab2.evaluator.evaluator module¶
-
class
convlab2.evaluator.evaluator.Evaluator¶ Bases:
object-
add_goal(goal)¶ init goal and array.
- args:
- goal:
dict[domain] dict[‘info’/’book’/’reqt’] dict/dict/list[slot]
-
add_sys_da(da_turn)¶ add sys_da into array.
- args:
- da_turn:
dict[domain-intent] list[slot, value]
-
add_usr_da(da_turn)¶ add usr_da into array
- args:
- da_turn:
dict[domain-intent] list[slot, value]
-
book_rate(ref2goal=True, aggregate=True)¶ judge if the selected entity meets the constraint
-
domain_success(domain, ref2goal=True)¶ judge if the domain (subtask) is successfully completed
-
final_goal_analyze()¶ judge whether the final goal satisfies the database constraints
-
inform_F1(ref2goal=True, aggregate=True)¶ judge if all the requested information is answered
-
task_success(ref2goal=True)¶ judge if all the domains are successfully completed
-
convlab2.evaluator.multiwoz_eval module¶
-
class
convlab2.evaluator.multiwoz_eval.MultiWozEvaluator¶ Bases:
convlab2.evaluator.evaluator.Evaluator-
add_goal(goal)¶ init goal and array
- args:
- goal:
dict[domain] dict[‘info’/’book’/’reqt’] dict/dict/list[slot]
-
add_sys_da(da_turn)¶ add sys_da into array
- args:
- da_turn:
list[intent, domain, slot, value]
-
add_usr_da(da_turn)¶ add usr_da into array
- args:
- da_turn:
list[intent, domain, slot, value]
-
book_rate(ref2goal=True, aggregate=True)¶ judge if the selected entity meets the constraint
-
domain_reqt_inform_analyze(domain, ref2goal=True)¶
-
domain_success(domain, ref2goal=True)¶ judge if the domain (subtask) is successfully completed
-
final_goal_analyze()¶ percentage of domains, in which the final goal satisfies the database constraints. If there is no dialog action, returns 1.
-
inform_F1(ref2goal=True, aggregate=True)¶ judge if all the requested information is answered
-
task_success(ref2goal=True)¶ judge if all the domains are successfully completed
-