This is a suggestion for further improvement of the file dosed/functions/compute_metrics_dataset.py. When working with the framework on detecting two different types of events, it was necessary to adapt the function in order to achieve a good detection.
The current implementation averages the metrics for each type of event. If there are five records with ten events of each event type and the network detects all ten events in one of the records and none of the events in the other nine records, the metrics are still good. This is because a record with no predicted events results in an empty list, which is not added to metrics_test[event_num][metric] and therefore not included when averaging the result over all ten records.
for event_num in range(network.number_of_classes - 1):
for metric in metrics.keys():
metrics_test[event_num][metric] = np.nanmean(np.array(metrics_test[event_num][metric]))
return metrics_testI suggest to adapt the function as follows. This will weight the metrics with the proportion of records where events of this type are found. With this adaption, the network learns to find the most possible number of events in each record and not only to maximize the results in one record. Further this suggestion weights the prediction of no events of one event type with -1 to achieve a more balanced prediction of the two event types.
for event_num in range(network.number_of_classes - 1):
for metric in metrics.keys():
num_records_with_found_events = len(metrics_test[event_num][metric])
num_records = len(test_dataset.records)
if (num_records_with_found_events > 0):
metrics_test[event_num][metric] = np.nanmean(np.array(metrics_test[event_num][metric])) * (num_records_with_found_events / num_records)
else:
metrics_test[event_num][metric] = -1Maybe this suggestion could improve the prediction process.
This is a suggestion for further improvement of the file dosed/functions/compute_metrics_dataset.py. When working with the framework on detecting two different types of events, it was necessary to adapt the function in order to achieve a good detection.
The current implementation averages the metrics for each type of event. If there are five records with ten events of each event type and the network detects all ten events in one of the records and none of the events in the other nine records, the metrics are still good. This is because a record with no predicted events results in an empty list, which is not added to metrics_test[event_num][metric] and therefore not included when averaging the result over all ten records.
I suggest to adapt the function as follows. This will weight the metrics with the proportion of records where events of this type are found. With this adaption, the network learns to find the most possible number of events in each record and not only to maximize the results in one record. Further this suggestion weights the prediction of no events of one event type with -1 to achieve a more balanced prediction of the two event types.
Maybe this suggestion could improve the prediction process.