FileFinderSource

class FileFinderSource(params=None, **kwargs)

Bases: MultiFileSource, CachedModule

Multifiles manager using Filefinder.

Written for datasets comprising of many datafiles, either because of they have long time series, or many parameters. The user has to define two methods. One returning the root directory containing all the datafiles (get_root_directory()). And another one returning the filename pattern (get_filename_pattern()). Using methods allows to return a different directory or pattern depending on the parameters.

Note

It is important to note that only parameters in the filename pattern can take multiple values in a single get_source() call. To get files from different root directories and merge the results, the current solution is to use get_data_sets().

The filename pattern specify the parts of the datafiles that vary from file to file using a powerful syntax. See the filefinder package documentation for the details.

The parameters that are specified in the filename pattern, and thus correspond to variations from file to file (the date in daily datafiles for instance) are called ‘fixables’. If they are not set, the filemanager will select all files, which is okay for finding files and opening the corresponding data. If the user ‘fix’ them to parameters to be set, for instance to generate a specific filename. a value, only part of the files will be selected. Some operation require all

Parameters:

params (t.Any | None)

_lines()

Human readable description.

Return type:

list[str]

property datafiles: list[str]

Datafiles available.

Use the filefinder object to scan for files corresponding to the filename pattern.

property filefinder: Finder

Filefinder instance to scan for datafiles.

Is also used to create filenames for a specific set of parameters.

property fixable: list[str]

List of parameters that can vary in the filename.

Found automatically from a filefinder.Finder instance. This correspond to the list of the group names in the Finder (without duplicates).

get_filename(relative=False, **fixes)

Create a filename corresponding to a set of parameters values.

All parameters must be defined, either by the parent DatasetAbstract.params, or by the fixes arguments.

Parameters:

• relative (bool) – If True, make the file relative to the root directory. Default is False.

• fixes – Parameters to fix to specific values. Only parameters defined in the filename pattern can be fixed. Will take precedence over the parent params attribute.

Return type:

str

get_filename_pattern()

Return the filename pattern.

The filename pattern specify the parts of the datafiles that vary from file to file. See the filefinder package documentation for the details.

Not implemented:

implement in your DataManager class.

Return type:

str

property unfixed: list[str]

List of varying parameters whose value is not fixed.

Considering the current set of parameters of the dataset. Parameters set to None or set to a sequence of values are considered unfixed.