Utilities

src.pipelines.tensorflow_v2.helpers.utilities.save_prcurve_csv(run_name, mask, pred, type)

Saves the result of a precision recall curve as a json. The elements are precision, recall, and the threshold. If there are more than 1000000 thresholds, the elements are downsampled to 1000 items.

Parameters

• run_name (str) – the run name to use when saving the file

• mask (List[array]) – the masks

• pred (List[array]) – the predictions

• type (str) – the type of prcurve to be used in the save name; e.g. test or val

Return type

None

Returns

None

src.pipelines.tensorflow_v2.helpers.utilities.save_predictions(prediction, folder_path, filename)

Saves a tf model prediction

Parameters

• prediction (array) – the tf model prediction

• folder_path (str) – the folder path to save the image

• filename (str) – the filename to save the image

Return type

None

Returns

None

src.pipelines.tensorflow_v2.helpers.utilities.save_compilation_dict(answers, lr, save_path)

Saves the a json of the compilation used for a training run.

Parameters

• answers (Dict) – a dictionary of answers which contains the user choices for the model, loss, optimiser, filter multiple, loss weight, initialiser, and activation

• lr (float) – the learning rate

• save_path (str) – where to save the json

Return type

None

Returns

None

src.pipelines.tensorflow_v2.helpers.utilities.get_sorted_list(search_string)

Returns a sorted list of file paths given a search string.

Parameters

search_string (str) – the search string to use; this should be a complete path with regex expressions for filenames

Return type

List[str]

Returns

list of file paths

src.pipelines.tensorflow_v2.helpers.utilities.configure_for_performance(ds, batch_size=2, buffer_size=100)

Shuffles a tf dataset and batches it. It also configures the amount of batches to prefetch which improves performance.

Parameters

• ds (DatasetV2) – the tf dataset

• batch_size (int) – the batch size

• buffer_size (int) – the buffer size; this controls the extent that the dataset will be shuffled

Return type

None

Returns

tf dataset configured for performance

src.pipelines.tensorflow_v2.helpers.utilities.tune_model(hyperModel, train_dataset, val_dataset, results_dir, run_name, input_shape, output_channels=1)

Performs Bayesian hyperparameter optimization using the input HyperModel.

Parameters

• hyperModel (HyperModel) – a kerastuner HyperModel

• train_dataset (DatasetV2) – the training dataset to use in the optimization

• val_dataset (DatasetV2) – the validation set to use in the optimization

• results_dir (str) – where the results of the optimisation should be stored

• run_name (int) – the run name, which is used when saving the results

• input_shape (Tuple[int]) – the input shape of HyperModel input

• output_channels (int) – the number of output channels of the HyperModel output

Return type

None

Returns

None

src.pipelines.tensorflow_v2.helpers.utilities.save_lrt_results(lr_range_test, save_path)

Saves the result of a learning rate range test as a json.

Parameters

• lr_range_test – a LRRangeTest object

• save_path (str) – where to save the json

Return type

None

Returns

None

src.pipelines.tensorflow_v2.helpers.utilities.get_class_weight(training_path, incl_aug)

Gets the % of embolism pixels to use as a class weight in a weighted loss function

Parameters

• training_path (str) – the path to the training images

• incl_aug (bool) – whether augmented images should be included

Return type

float

Returns

the % of embolism pixels

src.pipelines.tensorflow_v2.helpers.utilities.global_contrast_normalization(img, s=1, lamb=10, eps=1e-07)

Applies global contrast normalisation, which aims to standardise the contrast across pixels in the image.

Parameters

• img (Tensor) – input image

• s (int) – scale parameter; the standard deviation across all pixels is equal to s

• lamb (int) – lambda, which is regularisation parameter

• eps (float) – episilon, the minimum size of the denomintor; this can be used instead of lambda

Return type

Tensor

Returns

a normalized image

src.pipelines.tensorflow_v2.helpers.utilities.read_file(img_path, shift_256, transform_uint8)

Reads in an image from the file path and converts it to a tf tensor.

Parameters

• img_path (str) – file path of the image to read in

• shift_256 (bool) – whether to shift the image by 256

• transform_uint8 (bool) – whether to transform the image to a uint8 format

Return type

Tensor

Returns

an image as tf.float32 tensor

src.pipelines.tensorflow_v2.helpers.utilities.parse_numpy_image(img, batch_shape)

Converts an np.array to tf.float32 tensor, normalises the image by dividing by 255, and reshapes the image according to the batch shape.

Parameters

• img (array) – the image to be parsed

• batch_shape (int) – the batch shape

Return type

Tensor

Returns

a parsed image as a tf.float32 tensor

src.pipelines.tensorflow_v2.helpers.utilities.parse_image_fc(leaf_shape, mask_shape, test=False, shift_256=False, transform_uint8=False)

A function closure which creates leaf, mask samples. This is used as a preparation step for a tf.Dataset.

Parameters

• leaf_shape (Tuple[int, int, int]) – the shape of the leaf

• mask_shape (Tuple[int, int, int]) – the shape of the leaf

• test (bool) – whether test set samples are being parsed (currently this is not used)

• shift_256 (bool) – whether to shift the image by 256

• transform_uint8 (bool) – whether to transform the image to a uint8 format

Returns

the inner function which expects an leaf path and mask path

src.pipelines.tensorflow_v2.helpers.utilities.im2_lt_im1(pred, input_image)

Converts all pixels in the prediction to 0 in locations where the input image pixels are positive

Parameters

• pred (array) – the prediction to process

• input_image (array) – the input image corresponding the prediction

Return type

array

Returns

a processed prediction

src.pipelines.tensorflow_v2.helpers.utilities.gaussian_blur(img, kernel=(5, 5), base=1)

Applies a Gaussain blur to an image. Pixels which are greater than zero are set the base pixel provided. This is intended to be used to blur masks and predictions.

Parameters

• img (array) – the image to blur

• kernel (Tuple[int]) – Gaussian kernel, the first element is kernel height and the second element is kernel width; both elements must be positive and odd

• base (int) – the value to set pixels greater than zero to, after blurring

Return type

array

Returns

a blurred image

src.pipelines.tensorflow_v2.helpers.utilities.threshold(img, thresh)

Thresholds a prediction, by converting all pixels greater than the threshold to 1 and the remainder to 0.

Parameters

• img (array) – the image prediction to threshold

• thresh (float) – the threshold

Return type

array

Returns

a prediction with pixel intensities of 0 or 1

src.pipelines.tensorflow_v2.helpers.utilities.check_model_save(model, new_model, new_loss, new_opt, answers, metrics, model_save_path, check_opt=False)

Checks if a model have been saved correctly.

Parameters

• model (Model) – the old model to compare to

• new_model (Model) – the saved model

• new_loss (Loss) – an instance of tf loss to attach to the new model

• new_opt (OptimizerV2) – an instance of tf optimiser to attach to the new model; this must be instantiated with the correct learning rate

• answers (Dict) – the answers from the input prompt used in the training run

• metrics (List[Metric]) – the metrics to attach to the model

• model_save_path (str) – the save path of the model (where the weights are located)

• check_opt – whether the optimiser state should be checked

Return type

None

Returns

None

src.pipelines.tensorflow_v2.helpers.utilities.get_model_pred_batch_loss(model, leaf_shape, mask_shape)

Gets the prediction and batch loss for an input and mask of zeros.

Parameters

• model (Model) – the model to make a prediction with

• leaf_shape (Tuple[int, int, int]) – the leaf shape

• mask_shape (Tuple[int, int, int]) – the leaf shape

Return type

Tuple[array, float]

Returns

the prediction and the batch loss

src.pipelines.tensorflow_v2.helpers.utilities.parse_arguments()

Argument parser

Return type

Namespace

Returns

An argparse namespace

src.pipelines.tensorflow_v2.helpers.utilities.print_options_dict(output_dict)

Print a formatted version of the results of the output_dict generated by the input prompt

Parameters

output_dict – the output dict to print

Returns

None

src.pipelines.tensorflow_v2.helpers.utilities.interactive_prompt()

Interactive prompt which allows users to interact with the code base. The results are returned as a dict. There is an option, through the prompt, to save the input as a json to be used again.

Return type

None

Returns

None