convlab2.util package¶
Subpackages¶
Submodules¶
convlab2.util.allennlp_file_utils module¶
Copy from allennlp https://github.com/allenai/allennlp/blob/master/allennlp/common/file_utils.py Utilities for working with the local dataset cache.
-
class
convlab2.util.allennlp_file_utils.
Tqdm
¶ Bases:
object
-
static
set_slower_interval
(use_slower_interval: bool) → None¶ If
use_slower_interval
isTrue
, we will dramatically slow downtqdm's
default output rate.tqdm's
default output rate is great for interactively watching progress, but it is not great for log files. You might want to set this if you are primarily going to be looking at output through log files, not the terminal.
-
static
tqdm
(*args, **kwargs)¶
-
static
-
convlab2.util.allennlp_file_utils.
cached_path
(url_or_filename: Union[str, pathlib.Path], cache_dir: Optional[str] = None) → str¶ Given something that might be a URL (or might be a local path), determine which. If it’s a URL, download the file and cache it, and return the path to the cached file. If it’s already a local path, make sure the file exists and then return the path.
-
convlab2.util.allennlp_file_utils.
filename_to_url
(filename: str, cache_dir: Optional[str] = None) → Tuple[str, str]¶ Return the url and etag (which may be
None
) stored for filename. RaiseFileNotFoundError
if filename or its stored metadata do not exist.
-
convlab2.util.allennlp_file_utils.
get_from_cache
(url: str, cache_dir: Optional[str] = None) → str¶ Given a URL, look for the corresponding dataset in the local cache. If it’s not there, download it. Then return the path to the cached file.
-
convlab2.util.allennlp_file_utils.
get_s3_resource
()¶
-
convlab2.util.allennlp_file_utils.
is_url_or_existing_file
(url_or_filename: Optional[Union[str, pathlib.Path]]) → bool¶ Given something that might be a URL (or might be a local path), determine check if it’s url or an existing file path.
-
convlab2.util.allennlp_file_utils.
read_set_from_file
(filename: str) → Set[str]¶ Extract a de-duped collection (set) of text from a file. Expected file format is one item per line.
-
convlab2.util.allennlp_file_utils.
s3_get
(url: str, temp_file: IO) → None¶ Pull a file directly from S3.
-
convlab2.util.allennlp_file_utils.
s3_request
(func: Callable)¶ Wrapper function for s3 requests in order to create more helpful error messages.
-
convlab2.util.allennlp_file_utils.
session_with_backoff
() → requests.sessions.Session¶ We ran into an issue where http requests to s3 were timing out, possibly because we were making too many requests too quickly. This helper function returns a requests session that has retry-with-backoff built in.
convlab2.util.file_util module¶
-
convlab2.util.file_util.
cached_path
(file_path, cached_dir=None)¶
-
convlab2.util.file_util.
dump_json
(content, filepath)¶
-
convlab2.util.file_util.
get_root_path
()¶
-
convlab2.util.file_util.
read_zipped_json
(zip_path, filepath)¶
-
convlab2.util.file_util.
write_zipped_json
(zip_path, filepath)¶
convlab2.util.module module¶
module interface.
-
class
convlab2.util.module.
Module
¶ Bases:
abc.ABC
-
from_cache
(*args, **kwargs)¶ restore internal state for multi-turn dialog
-
init_session
()¶ Init the class variables for a new session.
-
test
(*args, **kwargs)¶ Model testing entry point
-
to_cache
(*args, **kwargs)¶ save internal state for multi-turn dialog
-
train
(*args, **kwargs)¶ Model training entry point
-
convlab2.util.train_util module¶
-
convlab2.util.train_util.
init_logging_handler
(log_dir, extra='')¶
-
convlab2.util.train_util.
to_device
(data)¶