tidypath package

Submodules

tidypath.config module

Config specs for the decorators ‘savedata’, ‘savefig’.

tidypath.config.reload_config()

Reload the configuration from environment variables, if necessary.

tidypath.decorators module

Main uses: save and cache data, save figures, change figure color, timing.

tidypath.decorators.savedata(keys_or_function=None, include_classes='file', ext='lzma', keys='kwargs_full', funcname_in_filename=False, overwrite=False, save=True, load_opts_default_save=True, load_opts={}, **save_opts)

Decorator for automatically saving output and then loading cached data. Default behavior:

1st function call: stores the data with extension ‘ext’. rest: loads stored data if the key args passed to the function coincide.

NOTE (!) decorated funcs will have extra arguments:

• save (bool). whether to save the output of the function.

• overwrite (bool). whether to overwrite the saved version.

• funcname_in_filename(bool) whether to include funcname in the filename.

• keys (dict/str/iterable) · dict: specifies filename of the form k1-v1_k2-v2_…kn_vn.

  k_i do not have to be arguments of the function.

  · str: key of a keyword argument. Example: keys = ‘x’.

  “kwargs_defaults” => default kwargs that were not modified. “kwargs” => kwargs passed during function call. “kwargs_full” => kws_defaults + kws. “pos_only” => length of *args “args” => attrs != **kwargs, *args. “all” if config.KEYS_ADD_POSONLY_TO_ALL => all attrs

  else => all attrs except pos_only: kwargs_full + args

  can combine options using “+” or “-”. Example: “args+z”, “x+y+kwargs”, “all-y”.

  · iterable: containing a combination of the available string keys (above). [“k1”, “k2”] == “k1+k2”.

Attrs:

• function: function to which the decorator is applied

• ext: storing extension. Selects ‘storage’ functions save_ext, load_ext.

  Supported: ‘lzma’ (default), ‘bz2’, ‘json’, ‘csv’, ‘npz’.

• include_classes: include class tree in saving_path.

• load_opts: kws for storage.load_ext. default kws are those of ‘saving_options’, those specified update saving_options dict.

• save_opts: kws for storage.save_ext

• load_opts_default_save: use save_opts as default for load_opts.

• rest: default behavior for decorated funcs extra arguments (above).

Returns: Function decorator

tidypath.decorators.savefig(keys_or_function=None, include_classes='file', ext='png', keys='kwargs_full', funcname_in_filename=True, return_fig=False, overwrite=True, save=True, **save_opts)

Generates figsaver decorator. Figure returned by function is automatically saved. Compatible with matplotlib and plotly.

NOTE (!) decorated funcs will have extra arguments:

• return_fig (bool) whether to return the figure (output of function).

• save (bool). whether to save the figure.

• overwrite (bool). whether to overwrite the saved version.

• funcname_in_filename(bool) whether to include funcname in the filename.

• keys (dict/str/iterable) · dict: specifies filename of the form k1-v1_k2-v2_…kn_vn.

  k_i do not have to be arguments of the function.

  · str: key of a keyword argument. Example: keys = ‘x’.

  “kwargs_defaults” => default kwargs that were not modified. “kwargs” => kwargs passed during function call. “kwargs_full” => kws_defaults + kws. “pos_only” => length of *args. Also self and cls are counted as pos_only arguments. “args” => attrs != **kwargs, *args. “all” if config.KEYS_ADD_POSONLY_TO_ALL => all attrs

  else => all attrs except pos_only: kwargs_full + args

  can combine options using “+” or “-”. Example: “args+z”, “x+y+kwargs”, “all-y”.

  · iterable: containing a combination of the available string keys (above). [“k1”, “k2”] == “k1+k2”.

Attrs:

• function: function to which the decorator is applied

• ext: storing extension. ‘eps’ recommended for articles.

  Supported: any extension supported by matplotlib/plotly. Example: ‘png’, ‘eps’, ‘html’ (plotly), etc.

• include_classes: include class tree in saving_path.

• save_opts: kws for saving function.

• rest: default behavior for decorated funcs extra arguments (above).

Returns: Function decorator

tidypath.fmt module

Encode dictionaries in strings and decode them. Useful for automatically asigning filenames.

tidypath.fmt.decoder(x, iterables=(<class 'list'>, <class 'tuple'>, <class 'numpy.ndarray'>))

string version of x -> x

tidypath.fmt.dict_to_id(*args, ndigits=2, join_char='_', **kwargs)

Generate ID of the form k1-v1_k2-v2… for k_i, v_i keys and values of the dictionary d or the kwargs.

tidypath.fmt.encoder(x, ndigits=2, iterables=(<class 'list'>, <class 'tuple'>, <class 'numpy.ndarray'>), iterable_maxsize=3)

x -> string version of x

tidypath.fmt.getopt_printer(opts)

Prints getopt input in a readable way.

tidypath.fmt.id_renamer(update_dict, parentDir, key=None, mode='add')

Modifies id of files in parentDir by updating the underlying dict. Attrs:

• update_dict: dict to use for updating the id

• parentDir: folder where files are located.

• key: string contained in the file for it to be modified.

• mode: - “add”: add update_dict to the id.

  • “delete”: delete update_dict from the id.

Returns #modified files. NOTE: If update_dict={} => filenames will be rearranged according to other_utils.dict_to_id.

tidypath.fmt.id_to_dict(identifier)

Inverse of dict_to_id.

tidypath.fmt.id_updater(filename, update_dict, mode='add')

Modifies filename by updating the underlying dict. Attrs:

• filename: id to be modified

• update_dict: dict to use for updating the id. if update_dict={} => filename rearranged according to other_utils.dict_to_id.

• mode: - “add”: add update_dict to the id.

  • “delete”: delete update_dict from the id.

Returns modified filename.

tidypath.inspection module

Analyze variables passed to functions, parent class, merge wrapper and wrapped signatures.

tidypath.inspection.classify_call_attrs(func, args, kwargs, add_pos_only_to_all=False)

Classify function args and kwargs passed during function call

NOTE: Probably could be done more efficiently relying more on inspect.getargspec(func) or inspect.signature(func), but realized later. NOTE2: For position-only arguments f(*args),

usually referred in the docs as f(pos1, pos2, /, pos_or_kwd,…) (‘/’ indicates end of position-only-args), only provides the number of *args.

Example: dict()

Returns: dict containing

“kwargs_defaults” => default kwargs that were not modified. “kwargs” => kwargs passed during function call. “kwargs_full” => kws_defaults + kws. “pos_only” => length of *args (position-only arguments) “args” => position-or-keyword arguments (are called static in the code). “all” => add_pos_only_to_all == False => all attrs except pos_only: kwargs_full + args

else => all attrs: kwargs_full + args + pos_only

tidypath.inspection.delete_cls_self_arg(args)

tidypath.inspection.get_class_that_defined_method(meth, default=None)

tidypath.inspection.merge_wrapper_signatures(wrapper, wrapper_params)

Add parameters from the wrapper function (called by @functools.wraps) to its signature. Wrapper params are assumed to be keyword-only.

Attrs:

• wrapper: wrapper function

• wrapper_params: params of wrapper to be added to the signature.

tidypath.paths module

Creates paths for automatically organizing files. Allows filename modification: If arg added (deleted) to func => add (delete) arg to stored function output files.

Two schemes:

1. By module => Module -> submodules -> class tree (until classes in module by default) -> func

2. By class => Class organization is a class to be inherited. Sets path: parentDir -> subfolder (optionally specified) -> inheritance_tree -> func_name.

1. is the most appropiate.

class tidypath.paths.ClassPath

Bases: Organizer

Updates fig and data dir of the class as follows: parentDir -> subfolder -> inheritance_tree -> func_name

classmethod create_dataDir()

classmethod create_dir(dirKey, depth=2)

Creates tree: parentDir -> subfolder -> inheritance_tree -> func_name

classmethod create_figDir()

classmethod inheritance_path(skip=2)

class tidypath.paths.Organizer

Bases: object

dataDir = 'data'

figDir = 'figs'

subfolder = ''

tidypath.paths.add_arg(func=None, directory=None, check_first=True, overwrite=False, **args)

Encodes new function args into filename. Useful when:

1. Adding a new arg to a function wrapped by savedata. Allows loading the available data without recomputing.

2. Modifying figure names produced by a function.

Attrs:

• args (kwargs): new arguments to encode into filename.

• func (callable): function (wrapped by savedata or savefig) for retrieving the corresponding directory where the outputs are stored.

• directory (str): directory where files are to be modified. It is recommended to pass the function instead of the directory.

• check_first (bool): whether to check the new filenames before applying any changes.

• overwrite (bool): whether to overwrite files when in conflict.

  Example: add an arg -> two filenames are the same -> one remains.

tidypath.paths.datapath(ext='lzma', **kwargs)

tidypath.paths.delete_arg(func=None, arg=None, directory=None, check_first=True, overwrite=False)

Delete encoded function arg from filename. Useful when:

1. Deleting an arg from a function wrapped by savedata. Allows loading the available data without recomputing.

2. Modifying figure names produced by a function.

Attrs:

• arg (str or iterable): key of an arg (k in k=v) to be deleted, or an iterable containing keys.

• func (callable): function (wrapped by savedata or savefig) for retrieving the corresponding directory where the outputs are stored.

• directory (str): directory where files are to be modified. It is recommended to pass the function instead of the directory.

• check_first (bool): whether to check the new filenames before applying any changes.

• overwrite (bool): whether to overwrite files when in conflict.

  Examples: Delete an arg -> two filenames are the same -> one remains.

tidypath.paths.figpath(ext='png', **kwargs)

tidypath.paths.filename_modifier(process_filename, func=None, directory=None, check_first=True, overwrite=False, **args)

Base function for adding/deleting/modifying function args encoded in the output filenames.

Attrs:

• process_filename: function f(file, k_name, k, v) -> new_filename.

  • k, v are the key and value of an arg.

  • k_name is the string encoding of k. (replace ‘_’ -> ‘-‘)

• func (callable): function (wrapped by savedata or savefig) for retrieving the corresponding directory where the outputs are stored.

• directory (str): directory where files are to be modified. It is recommended to pass the function instead of the directory.

• check_first (bool): whether to check the new filenames before applying any changes.

• overwrite (bool): whether to overwrite files when in conflict.

  Examples: add/modify/delete an arg -> two filenames are the same -> one remains.

• args (kwargs): Arguments to add/delete/modify in the filename.

tidypath.paths.modify_arg(func=None, directory=None, check_first=True, overwrite=False, **args)

Modified encoded function arg from filename. Useful when:

1. Renaming an arg value with a more suitable one (more comprehensible, better defined…), leaving the func output unchanged. Allows loading the available data without recomputing.

2. Modifying figure names produced by a function.

Attrs:

• args (kwargs): arguments to modify in the filename.

• func (callable): function (wrapped by savedata or savefig) for retrieving the corresponding directory where the outputs are stored.

• directory (str): directory where files are to be modified. It is recommended to pass the function instead of the directory.

• check_first (bool): whether to check the new filenames before applying any changes.

• overwrite (bool): whether to overwrite files when in conflict.

  Examples: Modify an arg -> two filenames are the same -> one remains.

tidypath.paths.module_path(func=None, depth=3, include_classes='file', skip=1)

returns module path and func_name.

include_classes: ‘file’: path: module -> class tree contained in func file -> func_name.

‘all’: path: module -> full class tree -> func_name.

tidypath.paths.saving_path(Dir, ext, keys={}, subfolder='', return_dir=False, funcname_in_filename=False, **kwargs)

Tree path: subfolder -> module -> (classes) -> func_name.

tidypath.storage module

Handle data (store/move/delete/load…)

class tidypath.storage.NpEncoder(*, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None)

Bases: JSONEncoder

default(obj)

Implement this method in a subclass such that it returns a serializable object for o, or calls the base implementation (to raise a TypeError).

For example, to support arbitrary iterators, you could implement default like this:

def default(self, o):
    try:
        iterable = iter(o)
    except TypeError:
        pass
    else:
        return list(iterable)
    # Let the base class default method raise the TypeError
    return JSONEncoder.default(self, o)

tidypath.storage.compress_files(rootDir='data', extension='.json', compress_to='lzma', min_size=0, loading_key=None)

Searches in a directory and all its subdirectories files with a certain extension and compresses them.

Attributes: - rootDir: The root directory. - extension: Extension setting which files should be compressed. Available options: ‘.json’, ‘.npz’. - compress_to: File type after compression. Available options: ‘lzma’, ‘bz2’ (or pbz2). - min_size: Minimum size for applying compression (in MB). - loading_key: Key for retrieving the data in a npz file. If None, retrieves the data corresponding to the first key.

Returns: Dict containing the name of the files that could not be processed (‘bad compression’) or those that were corrupted and had to be deleted(‘deleted files’).

tidypath.storage.create_loading_func(file_opener, file_loader, extra_processing=None, apply_defaults=None)

• extra_processing = List of process that could be applied to the file.

• apply_defaults = dict:{str: bool}. The key is set as a function variable, the value indicates whether to apply the process named by the key.

tidypath.storage.create_saving_func(file_opener, file_writer, saving_type, **kwargs)

tidypath.storage.int_keys(dictionary)

tidypath.storage.load_bz2(path)

tidypath.storage.load_csv(path, mode='pandas', **kwargs)

tidypath.storage.load_json(path, int_keys=True)

tidypath.storage.load_lzma(path)

tidypath.storage.load_npz(path, key=None)

tidypath.storage.parent_dir = 'data'

1. Save

tidypath.storage.save_bz2(data, filename, parent_dir='data')

Can specify directory in filename or in parent_dir.

tidypath.storage.save_csv(data, filename, parent_dir='data', **df_kwargs)

If data is a numpy array => if 2D => data

1D => [data]

(required by np.savetxt)

tidypath.storage.save_json(data, filename, parent_dir='data')

Can specify directory in filename or in parent_dir.

tidypath.storage.save_lzma(data, filename, parent_dir='data')

Can specify directory in filename or in parent_dir.

tidypath.storage.save_npz(arr, filename, parent_dir='data', key='arr')

tidypath.storage.save_preprocess(filename, saving_type, parent_dir)

Creates parent dir if does not exist and ensures filename has the saving_type extension.

Module contents

tidypath.delete_arg(func=None, arg=None, directory=None, check_first=True, overwrite=False)

Delete encoded function arg from filename. Useful when:

1. Deleting an arg from a function wrapped by savedata. Allows loading the available data without recomputing.

2. Modifying figure names produced by a function.

Attrs:

• arg (str or iterable): key of an arg (k in k=v) to be deleted, or an iterable containing keys.

• func (callable): function (wrapped by savedata or savefig) for retrieving the corresponding directory where the outputs are stored.

• directory (str): directory where files are to be modified. It is recommended to pass the function instead of the directory.

• check_first (bool): whether to check the new filenames before applying any changes.

• overwrite (bool): whether to overwrite files when in conflict.

  Examples: Delete an arg -> two filenames are the same -> one remains.

tidypath.modify_arg(func=None, directory=None, check_first=True, overwrite=False, **args)

Modified encoded function arg from filename. Useful when:

1. Renaming an arg value with a more suitable one (more comprehensible, better defined…), leaving the func output unchanged. Allows loading the available data without recomputing.

2. Modifying figure names produced by a function.

Attrs:

• args (kwargs): arguments to modify in the filename.

• func (callable): function (wrapped by savedata or savefig) for retrieving the corresponding directory where the outputs are stored.

• directory (str): directory where files are to be modified. It is recommended to pass the function instead of the directory.

• check_first (bool): whether to check the new filenames before applying any changes.

• overwrite (bool): whether to overwrite files when in conflict.

  Examples: Modify an arg -> two filenames are the same -> one remains.

tidypath.savedata(keys_or_function=None, include_classes='file', ext='lzma', keys='kwargs_full', funcname_in_filename=False, overwrite=False, save=True, load_opts_default_save=True, load_opts={}, **save_opts)

Decorator for automatically saving output and then loading cached data. Default behavior:

1st function call: stores the data with extension ‘ext’. rest: loads stored data if the key args passed to the function coincide.

NOTE (!) decorated funcs will have extra arguments:

• save (bool). whether to save the output of the function.

• overwrite (bool). whether to overwrite the saved version.

• funcname_in_filename(bool) whether to include funcname in the filename.

• keys (dict/str/iterable) · dict: specifies filename of the form k1-v1_k2-v2_…kn_vn.

  k_i do not have to be arguments of the function.

  · str: key of a keyword argument. Example: keys = ‘x’.

  “kwargs_defaults” => default kwargs that were not modified. “kwargs” => kwargs passed during function call. “kwargs_full” => kws_defaults + kws. “pos_only” => length of *args “args” => attrs != **kwargs, *args. “all” if config.KEYS_ADD_POSONLY_TO_ALL => all attrs

  else => all attrs except pos_only: kwargs_full + args

  can combine options using “+” or “-”. Example: “args+z”, “x+y+kwargs”, “all-y”.

  · iterable: containing a combination of the available string keys (above). [“k1”, “k2”] == “k1+k2”.

Attrs:

• function: function to which the decorator is applied

• ext: storing extension. Selects ‘storage’ functions save_ext, load_ext.

  Supported: ‘lzma’ (default), ‘bz2’, ‘json’, ‘csv’, ‘npz’.

• include_classes: include class tree in saving_path.

• load_opts: kws for storage.load_ext. default kws are those of ‘saving_options’, those specified update saving_options dict.

• save_opts: kws for storage.save_ext

• load_opts_default_save: use save_opts as default for load_opts.

• rest: default behavior for decorated funcs extra arguments (above).

Returns: Function decorator

tidypath.savefig(keys_or_function=None, include_classes='file', ext='png', keys='kwargs_full', funcname_in_filename=True, return_fig=False, overwrite=True, save=True, **save_opts)

Generates figsaver decorator. Figure returned by function is automatically saved. Compatible with matplotlib and plotly.

NOTE (!) decorated funcs will have extra arguments:

• return_fig (bool) whether to return the figure (output of function).

• save (bool). whether to save the figure.

• overwrite (bool). whether to overwrite the saved version.

• funcname_in_filename(bool) whether to include funcname in the filename.

• keys (dict/str/iterable) · dict: specifies filename of the form k1-v1_k2-v2_…kn_vn.

  k_i do not have to be arguments of the function.

  · str: key of a keyword argument. Example: keys = ‘x’.

  “kwargs_defaults” => default kwargs that were not modified. “kwargs” => kwargs passed during function call. “kwargs_full” => kws_defaults + kws. “pos_only” => length of *args. Also self and cls are counted as pos_only arguments. “args” => attrs != **kwargs, *args. “all” if config.KEYS_ADD_POSONLY_TO_ALL => all attrs

  else => all attrs except pos_only: kwargs_full + args

  can combine options using “+” or “-”. Example: “args+z”, “x+y+kwargs”, “all-y”.

  · iterable: containing a combination of the available string keys (above). [“k1”, “k2”] == “k1+k2”.

Attrs:

• function: function to which the decorator is applied

• ext: storing extension. ‘eps’ recommended for articles.

  Supported: any extension supported by matplotlib/plotly. Example: ‘png’, ‘eps’, ‘html’ (plotly), etc.

• include_classes: include class tree in saving_path.

• save_opts: kws for saving function.

• rest: default behavior for decorated funcs extra arguments (above).

Returns: Function decorator