multiml.saver module

Saver module.

class multiml.saver.Saver(save_dir=None, serial_id=None, mode='zarr', recreate=False)

Bases: object

Miscellaneous object management class.

Dictionary to save miscellaneous objects, and provides utility methods to manage ML metadata. There are two type of backends, zarr and dict, to store objects. zarr mode stores persistent objects, and dict mode stores temporary objects.

Examples

>>> from multiml import Saver
>>> saver = Saver()
>>> print(saver.save_dir)
>>> saver['key0'] = obj0
>>> saver['key1'] = obj1
__init__(save_dir=None, serial_id=None, mode='zarr', recreate=False)

Initialize Saver and create the base directory.

Parameters:

  • save_dir (str) – directory path to save objects. If None is given, a temporary directory is created automatically by tempfile.mkdtemp() method.

  • serial_id (int) – suffix of save_dir, i.e. save_dir.*serial_id*. If None is given, serial_id is incremented automatically based on existence of the directory.

  • mode (str) – zarr or dict for default metadata management.

  • recreate (bool) – recreate zarr database if True.

__len__()

Returns the number of stored objects in zarr and dict.

Returns:

the total number of stored objects in zarr and dict.

Return type:

int

__setitem__(key, obj)

Set key and store object to the default backend.

key and obj are stored to the default backend, zarr or dict.

Parameters:

  • key (str) – unique identifier of given object.

  • obj (obj) – arbitrary object to be stored.

__getitem__(key)

Returns object for given key.

key is searched from the both zarr and dict backends regardless of the default backend mode.

Parameters:

key (str) – unique identifier to retrieve object.

Returns:

arbitrary object..

Return type:

obj

__delitem__(key)

Delete key and object from backends.

Parameters:

key (str) – unique identifier to be deleted.

init_zarr(recreate=False)

Initialize zarr database and confirm connection.

Parameters:

recreate (bool) – If recreate is True, existing database is overwritten by an empty database.

set_mode(mode)

Set default database (backend) mode.

Parameters:

mode (str) – zarr or dict.

keys(mode=None)

Return registered keys in backends.

Parameters:

mode (str) – If zarr is given, keys in zarr database are returned. If dict is given, keys in dict database are returned. If None (default) all keys stored in the both backends are returned.

Returns:

list of registered keys.

Return type:

list

add(key, obj, mode=None, check=False)

Add object to given backend by key.

If given key already exists in the given backend, object is overwritten. If the key already exists in the other backend, raises error, which can be avoided by setting check = False. If mode is None, the default backend is used to store object.

Parameters:

  • key (str) – unique identifier of given object.

  • obj (obj) – arbitrary object to be stored.

  • mode (str) – zarr or dict to specify the backend database.

  • check (bool) – If True, consistency between the backends is checked.

delete(key, mode=None)

Delete key and object from the backends.

Parameters:

key (str) – unique identifier to be deleted.

save()

Save the objects registered in dict to zarr.

to_memory(key)

Move object from zarr to dict.

Parameters:

key (str) – the unique identifier to be moved.

to_storage(key)

Move object from dict to storage.

Parameters:

key (str) – the unique identifier to be moved.

property save_dir

Returns the name of base directory of Saver.

Returns:

the name of base directory.

Return type:

str

dump_ml(key, suffix=None, ml_type=None, **kwargs)

Dump machine learning models and parameters.

Parameters:

  • key (str) – the unique identifier to store metadata.

  • suffix (str) – arbitrary suffix to key (e.g. job_id, epoch) to avoid conflicts.

  • ml_type (str) – keras or pytorch or None. If it is None, just kwargs are dumped, which means ML model is not dumped.

  • kwargs – arbitrary arguments. Only standard types (int, float, str, list, dict) are dumped due to a limitation of pickle.

load_ml(key, suffix=None)

Load machine learning models and parameters.

Parameters:

  • key (str) – the unique identifier to load metadata.

  • suffix (str) – arbitrary suffix to key (e.g. job_id, epoch).

Returns:

arbitrary object.

Return type:

obj