1 clim_recal.convert

clim_recal.convert

Convert UKHADS data and UKCP18 data.

  • UKCP18 is converted temporally from a 360 day calendar to a standard (365/366 day) calendar and projected to British National Grid (BNG) EPSG:27700 from its original rotated polar grid.
  • UKHADS is converted spatially from 1km to 2.2km in BNG aligned with the projected UKCP18

1.1 Classes

Name Description
CPMConvert CPM specific changes to HADsConvert.
CPMConvertManager Class to manage processing CPM resampling.
HADsConvert Manage resampling HADs datafiles for modelling.
HADsConvertManager Class to manage processing HADs resampling.
IterCalcBase Base class to inherit for HADs and CPM.
IterCalcManagerBase Base class to inherit for HADs and CPM converter managers.

1.1.1 CPMConvert

clim_recal.convert.CPMConvert(self, *, input_path=RAW_CPM_TASMAX_PATH, output_path=CONVERT_OUTPUT_PATH / CPM_OUTPUT_PATH, variable_name=VariableOptions.default(), input_files=None, cpus=None, final_crs=BRITISH_NATIONAL_GRID_EPSG, input_file_extension=NETCDF_EXTENSION_STR, export_file_extension=NETCDF_EXTENSION_STR, start_index=0, stop_index=None, _result_paths=dict(), _iter_calc_method_name='range_to_reprojection', prior_time_series=None)

CPM specific changes to HADsConvert.

1.1.1.1 Attributes

Name Type Description
input_path PathLike | None Path to CPM files to process.
output Path to save processed CPM files.
input_files Path or Paths of NCF files to reproject.
cpus Number of cpu cores to use during multiprocessing.
resampling_func Function to call on self.input_files.
crop Path or file to spatially crop input_files with.
final_crs Coordinate Reference System (CRS) to return final format in.
input_file_extension File extensions to glob input_files with.

1.1.1.2 Examples

>>> if not is_data_mounted:
...     pytest.skip(mount_doctest_skip_message)
>>> resample_test_cpm_output_path: Path = getfixture(
...         'resample_test_cpm_output_path')
>>> cpm_converter: CPMConvert = CPMConvert(
...     input_path=RAW_CPM_TASMAX_PATH,
...     output_path=resample_test_cpm_output_path,
... )
>>> cpm_converter
<CPMConvert(count=..., max_count=...,...
    ...input_path='.../tasmax/01/latest',...
    ...output_path='.../test-run-results_..._.../cpm')>
>>> pprint(cpm_converter.input_files)
(...Path('.../tasmax/01/latest/tasmax_...-cpm_uk_2.2km_01_day_19801201-19811130.nc'),
 ...Path('.../tasmax/01/latest/tasmax_...-cpm_uk_2.2km_01_day_19811201-19821130.nc'),
 ...
 ...Path('.../tasmax/01/latest/tasmax_...-cpm_uk_2.2km_01_day_20791201-20801130.nc'))

1.1.2 CPMConvertManager

clim_recal.convert.CPMConvertManager(self, *, input_paths=RAW_CPM_PATH, output_paths=CONVERT_OUTPUT_PATH / CPM_OUTPUT_PATH, variables=(VariableOptions.default()), sub_path=CPM_SUB_PATH, start_index=0, stop_index=None, start_date=CPM_START_DATE, end_date=CPM_END_DATE, configs=list(), config_default_kwargs=dict(), calc_class=CPMConvert, cpus=None, _input_path_dict=dict(), _output_path_dict=dict(), _strict_fail_if_var_in_input_path=True, _allow_check_fail=False, runs=RunOptions.preferred())

Class to manage processing CPM resampling.

1.1.2.1 Attributes

Name Type Description
input_paths PathLike | Sequence[PathLike] Path or Paths to CPM files to process. If Path, will be propegated with files matching
output_paths PathLike | Sequence[PathLike] Path or Paths to to save processed CPM files to. If Path will be propagated to match input_paths.
variables Which VariableOptions to include.
runs Sequence[RunOptions | str] Which RunOptions to include.
crop_regions RegionOptions (like Manchester, Scotland etc.) to crop results to.
crop_paths Where to save region crop files.
sub_path Path Path to include at the stem of input_paths.
start_index Index to begin iterating input files for resampling or cropping.
stop_index Index to to run from start_index to when resampling or cropping. If None, iterate full list of paths.
start_date date | None Not yet implemented, but in future from what date to generate start index from.
end_date date | None Not yet implemented, but in future from what date to generate stop index from.
configs list[CPMConvert] List of HADsConvert instances to iterate resampling or cropping.
config_default_kwargs Parameters passed to all running self.configs.
calc_class type[CPMConvert] class to construct all self.configs instances with.
cpus Number of cpu cores to use during multiprocessing.

1.1.2.2 Examples

>>> if not is_data_mounted:
...     pytest.skip(mount_doctest_skip_message)
>>> resample_test_cpm_output_path: Path = getfixture(
...         'resample_test_cpm_output_path')
>>> cpm_converter_manager: CPMConvertManager = CPMConvertManager(
...     stop_index=9,
...     output_paths=resample_test_cpm_output_path,
...     )
>>> cpm_converter_manager
<CPMConvertManager(variables_count=1, runs_count=4,
                     input_paths_count=4)>
>>> configs: tuple[CPMConvert, ...] = tuple(
...     cpm_converter_manager.yield_configs())
>>> pprint(configs)
(<CPMConvert(count=9, max_count=100,
               input_path='.../tasmax/05/latest',
               output_path='.../cpm/tasmax/05')>,
 <CPMConvert(count=9, max_count=100,
               input_path='.../tasmax/06/latest',
               output_path='.../cpm/tasmax/06')>,
 <CPMConvert(count=9, max_count=100,
               input_path='.../tasmax/07/latest',
               output_path='.../cpm/tasmax/07')>,
 <CPMConvert(count=9, max_count=100,
               input_path='.../tasmax/08/latest',
               output_path='.../cpm/tasmax/08')>)

1.1.3 HADsConvert

clim_recal.convert.HADsConvert(self, *, input_path=RAW_HADS_TASMAX_PATH, output_path=CONVERT_OUTPUT_PATH / HADS_OUTPUT_PATH, variable_name=VariableOptions.default(), input_files=None, cpus=None, final_crs=BRITISH_NATIONAL_GRID_EPSG, input_file_extension=NETCDF_EXTENSION_STR, export_file_extension=NETCDF_EXTENSION_STR, start_index=0, stop_index=None, _result_paths=dict(), _iter_calc_method_name='range_to_reprojection', input_file_x_column_name=HADS_XDIM, input_file_y_column_name=HADS_YDIM, cpm_for_coord_alignment=RAW_CPM_TASMAX_PATH, cpm_for_coord_alignment_path_converted=False)

Manage resampling HADs datafiles for modelling.

1.1.3.1 Attributes

Name Type Description
input_path PathLike | None Path to HADs files to process.
output Path to save processed HADS files.
input_files Iterable[PathLike] | None Path or Paths of NCF files to convert.
resampling_func Function to call on self.input_files.
crop Path or file to spatially crop input_files with.
final_crs str Coordinate Reference System (CRS) to return final format in.
input_file_x_column_name str Column name in input_files or input for x coordinates.
input_file_y_column_name str Column name in input_files or input for y coordinates.
input_file_extension NETCDF_OR_TIF File extensions to glob input_files with.
start_index First index of file to iterate processing from.
stop_index Last index of files to iterate processing from as a count from start_index. If None, this will simply iterate over all available files.
cpm_for_coord_alignment T_Dataset | PathLike | None CPM Path or Dataset to match alignment with.
cpm_for_coord_alignment_path_converted bool Whether a Path passed to cpm_for_coord_alignment should be processed.

1.1.3.2 Examples

>>> if not is_data_mounted:
...     pytest.skip(mount_doctest_skip_message)
>>> resample_test_hads_output_path: Path = getfixture(
...         'resample_test_hads_output_path')
>>> hads_converter: HADsConvert = HADsConvert(
...     output_path=resample_test_hads_output_path,
... )
>>> hads_converter
<HADsConvert(...count=504,...
    ...input_path='.../tasmax/day',...
    ...output_path='...run-results_..._.../hads')>
>>> pprint(hads_converter.input_files)
(...Path('.../tasmax/day/tasmax_hadukgrid_uk_1km_day_19800101-19800131.nc'),
 ...Path('.../tasmax/day/tasmax_hadukgrid_uk_1km_day_19800201-19800229.nc'),
 ...,
 ...Path('.../tasmax/day/tasmax_hadukgrid_uk_1km_day_20211201-20211231.nc'))

1.1.3.3 Methods

Name Description
set_cpm_for_coord_alignment Check if cpm_for_coord_alignment is a Dataset, process if a Path.
1.1.3.3.1 set_cpm_for_coord_alignment

clim_recal.convert.HADsConvert.set_cpm_for_coord_alignment()

Check if cpm_for_coord_alignment is a Dataset, process if a Path.

1.1.4 HADsConvertManager

clim_recal.convert.HADsConvertManager(self, *, input_paths=RAW_HADS_PATH, output_paths=CONVERT_OUTPUT_PATH / HADS_OUTPUT_PATH, variables=(VariableOptions.default()), sub_path=HADS_SUB_PATH, start_index=0, stop_index=None, start_date=HADS_START_DATE, end_date=HADS_END_DATE, configs=list(), config_default_kwargs=dict(), calc_class=HADsConvert, cpus=None, _input_path_dict=dict(), _output_path_dict=dict(), _strict_fail_if_var_in_input_path=True, _allow_check_fail=False, cpm_for_coord_alignment=RAW_CPM_TASMAX_PATH, cpm_for_coord_alignment_path_converted=False)

Class to manage processing HADs resampling.

1.1.4.1 Attributes

Name Type Description
input_paths PathLike | Sequence[PathLike] Path or Paths to CPM files to process. If Path, will be propegated with files matching
output_paths PathLike | Sequence[PathLike] Path or Paths to to save processed CPM files to. If Path will be propagated to match input_paths.
variables Which VariableOptions to include.
crop_regions RegionOptions (like Manchester, Scotland etc.) to crop results to.
crop_paths Where to save region crop files.
sub_path Path Path to include at the stem of input_paths.
start_index Index to begin iterating input files for resampling or cropping.
stop_index Index to to run from start_index to when resampling or cropping. If None, iterate full list of paths.
start_date date | None Not yet implemented, but in future from what date to generate start index from.
end_date date | None Not yet implemented, but in future from what date to generate stop index from.
configs list[HADsConvert] List of HADsConvert instances to iterate resampling or cropping.
config_default_kwargs dict[str, Any] Parameters passed to all running self.configs.
calc_class type[HADsConvert] class to construct all self.configs instances with.
cpus Number of cpu cores to use during multiprocessing.
cpm_for_coord_alignment T_Dataset | PathLike CPM Path or Dataset to match alignment with.
cpm_for_coord_alignment_path_converted bool Whether a Path passed to cpm_for_coord_alignment should be processed.

1.1.4.2 Examples

>>> if not is_data_mounted:
...     pytest.skip(mount_doctest_skip_message)
>>> resample_test_hads_output_path: Path = getfixture(
...         'resample_test_hads_output_path')
>>> hads_converter_manager: HADsConvertManager = HADsConvertManager(
...     variables=VariableOptions.all(),
...     output_paths=resample_test_hads_output_path,
...     )
>>> hads_converter_manager
<HADsConvertManager(variables_count=3, input_paths_count=3)>

1.1.4.3 Methods

Name Description
set_cpm_for_coord_alignment Check if cpm_for_coord_alignment is a Dataset, process if a Path.
yield_configs Generate a CPMConvert or HADsConvert for self.input_paths.
1.1.4.3.1 set_cpm_for_coord_alignment

clim_recal.convert.HADsConvertManager.set_cpm_for_coord_alignment()

Check if cpm_for_coord_alignment is a Dataset, process if a Path.

1.1.4.3.2 yield_configs

clim_recal.convert.HADsConvertManager.yield_configs()

Generate a CPMConvert or HADsConvert for self.input_paths.

1.1.5 IterCalcBase

clim_recal.convert.IterCalcBase(self, *, input_path=Path(), output_path=CONVERT_OUTPUT_PATH, variable_name=VariableOptions.default(), input_files=None, cpus=None, final_crs=BRITISH_NATIONAL_GRID_EPSG, input_file_extension=NETCDF_EXTENSION_STR, export_file_extension=NETCDF_EXTENSION_STR, start_index=0, stop_index=None, _result_paths=dict(), _iter_calc_method_name='range_to_reprojection')

Base class to inherit for HADs and CPM.

1.1.5.1 Attributes

Name Description
max_count Maximum length of self.input_files ignoring start_index and start_index.

1.1.5.2 Methods

Name Description
execute Run all steps for processing
set_input_files Replace self.input and process self.input_files.
1.1.5.2.1 execute

clim_recal.convert.IterCalcBase.execute(**kwargs)

Run all steps for processing

1.1.5.2.2 set_input_files

clim_recal.convert.IterCalcBase.set_input_files(new_input_path=None)

Replace self.input and process self.input_files.

1.1.6 IterCalcManagerBase

clim_recal.convert.IterCalcManagerBase(self, *, input_paths=Path(), output_paths=Path(), variables=(VariableOptions.default()), sub_path=Path(), start_index=0, stop_index=None, start_date=None, end_date=None, configs=list(), config_default_kwargs=dict(), calc_class=None, cpus=None, _input_path_dict=dict(), _output_path_dict=dict(), _strict_fail_if_var_in_input_path=True, _allow_check_fail=False)

Base class to inherit for HADs and CPM converter managers.

1.1.6.1 Attributes

Name Description
input_folder Return self._input_path set by set_input_paths().
max_count Maximum length of self.input_files ignoring start_index and start_index.
output_folder Return self._output_path set by set_output_paths().

1.1.6.2 Classes

Name Description
VarirableInBaseImportPathError Checking import path validity for self.variables.
1.1.6.2.1 VarirableInBaseImportPathError

clim_recal.convert.IterCalcManagerBase.VarirableInBaseImportPathError()

Checking import path validity for self.variables.

1.1.6.3 Methods

Name Description
check_paths Check and set input, convert and crop paths.
execute_configs Run all converter configurations
set_input_paths Propagate self.input_paths if needed.
set_output_paths Propagate self.output_paths if needed.
yield_configs Generate a CPMConvert or HADsConvert for self.input_paths.
1.1.6.3.1 check_paths

clim_recal.convert.IterCalcManagerBase.check_paths(run_set_data_paths=True)

Check and set input, convert and crop paths.

1.1.6.3.2 execute_configs

clim_recal.convert.IterCalcManagerBase.execute_configs(multiprocess=False, cpus=None, return_converters=False, return_path=True, description_func=path_print_progress, **kwargs)

Run all converter configurations

1.1.6.3.2.1 Parameters
Name Type Description Default
multiprocess bool If True run parameters in convert_configs with multiprocess_execute. False
cpus int | None Number of cpus to pass to multiprocess_execute. None
return_converters bool Return instances of generated HADsConvert or CPMConvert, or return the results of each execute call. False
return_path bool Return Path to results object if True, else converted Dataset. True
kwargs Parameters to path to sampler execute calls. {}
1.1.6.3.3 set_input_paths

clim_recal.convert.IterCalcManagerBase.set_input_paths()

Propagate self.input_paths if needed.

1.1.6.3.4 set_output_paths

clim_recal.convert.IterCalcManagerBase.set_output_paths()

Propagate self.output_paths if needed.

1.1.6.3.5 yield_configs

clim_recal.convert.IterCalcManagerBase.yield_configs()

Generate a CPMConvert or HADsConvert for self.input_paths.

1.2 Functions

Name Description
reproject_2_2km_file_name Return tweaked path to indicate standard day projection.
reproject_standard_calendar_file_name Return tweaked path to indicate standard day projection.

1.2.1 reproject_2_2km_file_name

clim_recal.convert.reproject_2_2km_file_name(path)

Return tweaked path to indicate standard day projection.

1.2.2 reproject_standard_calendar_file_name

clim_recal.convert.reproject_standard_calendar_file_name(path)

Return tweaked path to indicate standard day projection.

Back to top