tatk.evaluator package¶
Submodules¶
tatk.evaluator.evaluator module¶
-
class
tatk.evaluator.evaluator.Evaluator¶ Bases:
object-
__init__()¶ Initialize self. See help(type(self)) for accurate signature.
-
add_goal(goal)¶ init goal and array.
- args:
- goal:
dict[domain] dict[‘info’/’book’/’reqt’] dict/dict/list[slot]
-
add_sys_da(da_turn)¶ add sys_da into array.
- args:
- da_turn:
dict[domain-intent] list[slot, value]
-
add_usr_da(da_turn)¶ add usr_da into array
- args:
- da_turn:
dict[domain-intent] list[slot, value]
-
book_rate(ref2goal=True, aggregate=True)¶ judge if the selected entity meets the constraint
-
domain_success(domain, ref2goal=True)¶ judge if the domain (subtask) is successfully completed
-
inform_F1(ref2goal=True, aggregate=True)¶ judge if all the requested information is answered
-
task_success(ref2goal=True)¶ judge if all the domains are successfully completed
-
tatk.evaluator.multiwoz_eval module¶
-
class
tatk.evaluator.multiwoz_eval.MultiWozEvaluator¶ Bases:
tatk.evaluator.evaluator.Evaluator-
__init__()¶ Initialize self. See help(type(self)) for accurate signature.
-
add_goal(goal)¶ init goal and array
- args:
- goal:
dict[domain] dict[‘info’/’book’/’reqt’] dict/dict/list[slot]
-
add_sys_da(da_turn)¶ add sys_da into array
- args:
- da_turn:
dict[domain-intent] list[slot, value]
-
add_usr_da(da_turn)¶ add usr_da into array
- args:
- da_turn:
dict[domain-intent] list[slot, value]
-
book_rate(ref2goal=True, aggregate=True)¶ judge if the selected entity meets the constraint
-
domain_success(domain, ref2goal=True)¶ judge if the domain (subtask) is successfully completed
-
inform_F1(ref2goal=True, aggregate=True)¶ judge if all the requested information is answered
-
task_success(ref2goal=True)¶ judge if all the domains are successfully completed
-