1 clim_recal.convert
clim_recal.convert
Convert UKHADS data and UKCP18 data.
- UKCP18 is converted temporally from a 360 day calendar to a standard (365/366 day) calendar and projected to British National Grid (BNG) EPSG:27700 from its original rotated polar grid.
- UKHADS is converted spatially from 1km to 2.2km in BNG aligned with the projected UKCP18
1.1 Classes
Name | Description |
---|---|
CPMConvert | CPM specific changes to HADsConvert. |
CPMConvertManager | Class to manage processing CPM resampling. |
HADsConvert | Manage resampling HADs datafiles for modelling. |
HADsConvertManager | Class to manage processing HADs resampling. |
IterCalcBase | Base class to inherit for HADs and CPM . |
IterCalcManagerBase | Base class to inherit for HADs and CPM converter managers. |
1.1.1 CPMConvert
clim_recal.convert.CPMConvert(self, *, input_path=RAW_CPM_TASMAX_PATH, output_path=CONVERT_OUTPUT_PATH / CPM_OUTPUT_PATH, variable_name=VariableOptions.default(), input_files=None, cpus=None, final_crs=BRITISH_NATIONAL_GRID_EPSG, input_file_extension=NETCDF_EXTENSION_STR, export_file_extension=NETCDF_EXTENSION_STR, start_index=0, stop_index=None, _result_paths=dict(), _iter_calc_method_name='range_to_reprojection', prior_time_series=None)
CPM specific changes to HADsConvert.
1.1.1.1 Attributes
Name | Type | Description |
---|---|---|
input_path | PathLike | None | Path to CPM files to process. |
output | Path to save processed CPM files. |
|
input_files | Path or Paths of NCF files to reproject. |
|
cpus | Number of cpu cores to use during multiprocessing. |
|
resampling_func | Function to call on self.input_files . |
|
crop | Path or file to spatially crop input_files with. |
|
final_crs | Coordinate Reference System (CRS) to return final format in. | |
input_file_extension | File extensions to glob input_files with. |
1.1.1.2 Examples
>>> if not is_data_mounted:
... pytest.skip(mount_doctest_skip_message)>>> resample_test_cpm_output_path: Path = getfixture(
'resample_test_cpm_output_path')
... >>> cpm_converter: CPMConvert = CPMConvert(
=RAW_CPM_TASMAX_PATH,
... input_path=resample_test_cpm_output_path,
... output_path
... )>>> cpm_converter
<CPMConvert(count=..., max_count=...,...
='.../tasmax/01/latest',...
...input_path='.../test-run-results_..._.../cpm')>
...output_path>>> pprint(cpm_converter.input_files)
'.../tasmax/01/latest/tasmax_...-cpm_uk_2.2km_01_day_19801201-19811130.nc'),
(...Path('.../tasmax/01/latest/tasmax_...-cpm_uk_2.2km_01_day_19811201-19821130.nc'),
...Path(
...'.../tasmax/01/latest/tasmax_...-cpm_uk_2.2km_01_day_20791201-20801130.nc')) ...Path(
1.1.2 CPMConvertManager
clim_recal.convert.CPMConvertManager(self, *, input_paths=RAW_CPM_PATH, output_paths=CONVERT_OUTPUT_PATH / CPM_OUTPUT_PATH, variables=(VariableOptions.default()), sub_path=CPM_SUB_PATH, start_index=0, stop_index=None, start_date=CPM_START_DATE, end_date=CPM_END_DATE, configs=list(), config_default_kwargs=dict(), calc_class=CPMConvert, cpus=None, _input_path_dict=dict(), _output_path_dict=dict(), _strict_fail_if_var_in_input_path=True, _allow_check_fail=False, runs=RunOptions.preferred())
Class to manage processing CPM resampling.
1.1.2.1 Attributes
Name | Type | Description |
---|---|---|
input_paths | PathLike | Sequence[PathLike] | Path or Paths to CPM files to process. If Path , will be propegated with files matching |
output_paths | PathLike | Sequence[PathLike] | Path or Paths to to save processed CPM files to. If Path will be propagated to match input_paths . |
variables | Which VariableOptions to include. |
|
runs | Sequence[RunOptions | str] | Which RunOptions to include. |
crop_regions | RegionOptions (like Manchester, Scotland etc.) to crop results to. |
|
crop_paths | Where to save region crop files. | |
sub_path | Path | Path to include at the stem of input_paths . |
start_index | Index to begin iterating input files for resampling or cropping . |
|
stop_index | Index to to run from start_index to when resampling or cropping . If None , iterate full list of paths. |
|
start_date | date | None | Not yet implemented, but in future from what date to generate start index from. |
end_date | date | None | Not yet implemented, but in future from what date to generate stop index from. |
configs | list[CPMConvert] | List of HADsConvert instances to iterate resampling or cropping . |
config_default_kwargs | Parameters passed to all running self.configs . |
|
calc_class | type[CPMConvert] | class to construct all self.configs instances with. |
cpus | Number of cpu cores to use during multiprocessing. |
1.1.2.2 Examples
>>> if not is_data_mounted:
... pytest.skip(mount_doctest_skip_message)>>> resample_test_cpm_output_path: Path = getfixture(
'resample_test_cpm_output_path')
... >>> cpm_converter_manager: CPMConvertManager = CPMConvertManager(
=9,
... stop_index=resample_test_cpm_output_path,
... output_paths
... )>>> cpm_converter_manager
<CPMConvertManager(variables_count=1, runs_count=4,
=4)>
input_paths_count>>> configs: tuple[CPMConvert, ...] = tuple(
... cpm_converter_manager.yield_configs())>>> pprint(configs)
<CPMConvert(count=9, max_count=100,
(='.../tasmax/05/latest',
input_path='.../cpm/tasmax/05')>,
output_path<CPMConvert(count=9, max_count=100,
='.../tasmax/06/latest',
input_path='.../cpm/tasmax/06')>,
output_path<CPMConvert(count=9, max_count=100,
='.../tasmax/07/latest',
input_path='.../cpm/tasmax/07')>,
output_path<CPMConvert(count=9, max_count=100,
='.../tasmax/08/latest',
input_path='.../cpm/tasmax/08')>) output_path
1.1.3 HADsConvert
clim_recal.convert.HADsConvert(self, *, input_path=RAW_HADS_TASMAX_PATH, output_path=CONVERT_OUTPUT_PATH / HADS_OUTPUT_PATH, variable_name=VariableOptions.default(), input_files=None, cpus=None, final_crs=BRITISH_NATIONAL_GRID_EPSG, input_file_extension=NETCDF_EXTENSION_STR, export_file_extension=NETCDF_EXTENSION_STR, start_index=0, stop_index=None, _result_paths=dict(), _iter_calc_method_name='range_to_reprojection', input_file_x_column_name=HADS_XDIM, input_file_y_column_name=HADS_YDIM, cpm_for_coord_alignment=RAW_CPM_TASMAX_PATH, cpm_for_coord_alignment_path_converted=False)
Manage resampling HADs datafiles for modelling.
1.1.3.1 Attributes
Name | Type | Description |
---|---|---|
input_path | PathLike | None | Path to HADs files to process. |
output | Path to save processed HADS files. |
|
input_files | Iterable[PathLike] | None | Path or Paths of NCF files to convert. |
resampling_func | Function to call on self.input_files . |
|
crop | Path or file to spatially crop input_files with. |
|
final_crs | str | Coordinate Reference System (CRS) to return final format in. |
input_file_x_column_name | str | Column name in input_files or input for x coordinates. |
input_file_y_column_name | str | Column name in input_files or input for y coordinates. |
input_file_extension | NETCDF_OR_TIF | File extensions to glob input_files with. |
start_index | First index of file to iterate processing from. | |
stop_index | Last index of files to iterate processing from as a count from start_index . If None , this will simply iterate over all available files. |
|
cpm_for_coord_alignment | T_Dataset | PathLike | None | CPM Path or Dataset to match alignment with. |
cpm_for_coord_alignment_path_converted | bool | Whether a Path passed to cpm_for_coord_alignment should be processed. |
1.1.3.2 Examples
>>> if not is_data_mounted:
... pytest.skip(mount_doctest_skip_message)>>> resample_test_hads_output_path: Path = getfixture(
'resample_test_hads_output_path')
... >>> hads_converter: HADsConvert = HADsConvert(
=resample_test_hads_output_path,
... output_path
... )>>> hads_converter
<HADsConvert(...count=504,...
='.../tasmax/day',...
...input_path='...run-results_..._.../hads')>
...output_path>>> pprint(hads_converter.input_files)
'.../tasmax/day/tasmax_hadukgrid_uk_1km_day_19800101-19800131.nc'),
(...Path('.../tasmax/day/tasmax_hadukgrid_uk_1km_day_19800201-19800229.nc'),
...Path(
...,'.../tasmax/day/tasmax_hadukgrid_uk_1km_day_20211201-20211231.nc')) ...Path(
1.1.3.3 Methods
Name | Description |
---|---|
set_cpm_for_coord_alignment | Check if cpm_for_coord_alignment is a Dataset , process if a Path . |
1.1.3.3.1 set_cpm_for_coord_alignment
clim_recal.convert.HADsConvert.set_cpm_for_coord_alignment()
Check if cpm_for_coord_alignment
is a Dataset
, process if a Path
.
1.1.4 HADsConvertManager
clim_recal.convert.HADsConvertManager(self, *, input_paths=RAW_HADS_PATH, output_paths=CONVERT_OUTPUT_PATH / HADS_OUTPUT_PATH, variables=(VariableOptions.default()), sub_path=HADS_SUB_PATH, start_index=0, stop_index=None, start_date=HADS_START_DATE, end_date=HADS_END_DATE, configs=list(), config_default_kwargs=dict(), calc_class=HADsConvert, cpus=None, _input_path_dict=dict(), _output_path_dict=dict(), _strict_fail_if_var_in_input_path=True, _allow_check_fail=False, cpm_for_coord_alignment=RAW_CPM_TASMAX_PATH, cpm_for_coord_alignment_path_converted=False)
Class to manage processing HADs resampling.
1.1.4.1 Attributes
Name | Type | Description |
---|---|---|
input_paths | PathLike | Sequence[PathLike] | Path or Paths to CPM files to process. If Path , will be propegated with files matching |
output_paths | PathLike | Sequence[PathLike] | Path or Paths to to save processed CPM files to. If Path will be propagated to match input_paths . |
variables | Which VariableOptions to include. |
|
crop_regions | RegionOptions (like Manchester, Scotland etc.) to crop results to. |
|
crop_paths | Where to save region crop files. | |
sub_path | Path | Path to include at the stem of input_paths . |
start_index | Index to begin iterating input files for resampling or cropping . |
|
stop_index | Index to to run from start_index to when resampling or cropping . If None , iterate full list of paths. |
|
start_date | date | None | Not yet implemented, but in future from what date to generate start index from. |
end_date | date | None | Not yet implemented, but in future from what date to generate stop index from. |
configs | list[HADsConvert] | List of HADsConvert instances to iterate resampling or cropping . |
config_default_kwargs | dict[str, Any] | Parameters passed to all running self.configs . |
calc_class | type[HADsConvert] | class to construct all self.configs instances with. |
cpus | Number of cpu cores to use during multiprocessing. |
|
cpm_for_coord_alignment | T_Dataset | PathLike | CPM Path or Dataset to match alignment with. |
cpm_for_coord_alignment_path_converted | bool | Whether a Path passed to cpm_for_coord_alignment should be processed. |
1.1.4.2 Examples
>>> if not is_data_mounted:
... pytest.skip(mount_doctest_skip_message)>>> resample_test_hads_output_path: Path = getfixture(
'resample_test_hads_output_path')
... >>> hads_converter_manager: HADsConvertManager = HADsConvertManager(
=VariableOptions.all(),
... variables=resample_test_hads_output_path,
... output_paths
... )>>> hads_converter_manager
<HADsConvertManager(variables_count=3, input_paths_count=3)>
1.1.4.3 Methods
Name | Description |
---|---|
set_cpm_for_coord_alignment | Check if cpm_for_coord_alignment is a Dataset , process if a Path . |
yield_configs | Generate a CPMConvert or HADsConvert for self.input_paths . |
1.1.4.3.1 set_cpm_for_coord_alignment
clim_recal.convert.HADsConvertManager.set_cpm_for_coord_alignment()
Check if cpm_for_coord_alignment
is a Dataset
, process if a Path
.
1.1.4.3.2 yield_configs
clim_recal.convert.HADsConvertManager.yield_configs()
Generate a CPMConvert
or HADsConvert
for self.input_paths
.
1.1.5 IterCalcBase
clim_recal.convert.IterCalcBase(self, *, input_path=Path(), output_path=CONVERT_OUTPUT_PATH, variable_name=VariableOptions.default(), input_files=None, cpus=None, final_crs=BRITISH_NATIONAL_GRID_EPSG, input_file_extension=NETCDF_EXTENSION_STR, export_file_extension=NETCDF_EXTENSION_STR, start_index=0, stop_index=None, _result_paths=dict(), _iter_calc_method_name='range_to_reprojection')
Base class to inherit for HADs
and CPM
.
1.1.5.1 Attributes
Name | Description |
---|---|
max_count | Maximum length of self.input_files ignoring start_index and start_index . |
1.1.5.2 Methods
Name | Description |
---|---|
execute | Run all steps for processing |
set_input_files | Replace self.input and process self.input_files . |
1.1.5.2.1 execute
clim_recal.convert.IterCalcBase.execute(**kwargs)
Run all steps for processing
1.1.5.2.2 set_input_files
clim_recal.convert.IterCalcBase.set_input_files(new_input_path=None)
Replace self.input
and process self.input_files
.
1.1.6 IterCalcManagerBase
clim_recal.convert.IterCalcManagerBase(self, *, input_paths=Path(), output_paths=Path(), variables=(VariableOptions.default()), sub_path=Path(), start_index=0, stop_index=None, start_date=None, end_date=None, configs=list(), config_default_kwargs=dict(), calc_class=None, cpus=None, _input_path_dict=dict(), _output_path_dict=dict(), _strict_fail_if_var_in_input_path=True, _allow_check_fail=False)
Base class to inherit for HADs
and CPM
converter managers.
1.1.6.1 Attributes
Name | Description |
---|---|
input_folder | Return self._input_path set by set_input_paths() . |
max_count | Maximum length of self.input_files ignoring start_index and start_index . |
output_folder | Return self._output_path set by set_output_paths() . |
1.1.6.2 Classes
Name | Description |
---|---|
VarirableInBaseImportPathError | Checking import path validity for self.variables . |
1.1.6.2.1 VarirableInBaseImportPathError
clim_recal.convert.IterCalcManagerBase.VarirableInBaseImportPathError()
Checking import path validity for self.variables
.
1.1.6.3 Methods
Name | Description |
---|---|
check_paths | Check and set input , convert and crop paths. |
execute_configs | Run all converter configurations |
set_input_paths | Propagate self.input_paths if needed. |
set_output_paths | Propagate self.output_paths if needed. |
yield_configs | Generate a CPMConvert or HADsConvert for self.input_paths . |
1.1.6.3.1 check_paths
clim_recal.convert.IterCalcManagerBase.check_paths(run_set_data_paths=True)
Check and set input
, convert
and crop
paths.
1.1.6.3.2 execute_configs
clim_recal.convert.IterCalcManagerBase.execute_configs(multiprocess=False, cpus=None, return_converters=False, return_path=True, description_func=path_print_progress, **kwargs)
Run all converter configurations
1.1.6.3.2.1 Parameters
Name | Type | Description | Default |
---|---|---|---|
multiprocess |
bool | If True run parameters in convert_configs with multiprocess_execute . |
False |
cpus |
int | None | Number of cpus to pass to multiprocess_execute . |
None |
return_converters |
bool | Return instances of generated HADsConvert or CPMConvert , or return the results of each execute call. |
False |
return_path |
bool | Return Path to results object if True, else converted Dataset . |
True |
kwargs |
Parameters to path to sampler execute calls. |
{} |
1.1.6.3.3 set_input_paths
clim_recal.convert.IterCalcManagerBase.set_input_paths()
Propagate self.input_paths
if needed.
1.1.6.3.4 set_output_paths
clim_recal.convert.IterCalcManagerBase.set_output_paths()
Propagate self.output_paths
if needed.
1.1.6.3.5 yield_configs
clim_recal.convert.IterCalcManagerBase.yield_configs()
Generate a CPMConvert
or HADsConvert
for self.input_paths
.
1.2 Functions
Name | Description |
---|---|
reproject_2_2km_file_name | Return tweaked path to indicate standard day projection. |
reproject_standard_calendar_file_name | Return tweaked path to indicate standard day projection. |
1.2.1 reproject_2_2km_file_name
clim_recal.convert.reproject_2_2km_file_name(path)
Return tweaked path
to indicate standard day projection.
1.2.2 reproject_standard_calendar_file_name
clim_recal.convert.reproject_standard_calendar_file_name(path)
Return tweaked path
to indicate standard day projection.