-
Notifications
You must be signed in to change notification settings - Fork 3
Application user guide
File:Page1.png ''Figure 1 - the main page''<br />
The Main Page of the Statistics Tool serves as the central hub for report management, analysis, and comparison.
On the main page of the statistics tool, users have the option to perform the following actions:
- Create new report: By clicking on the "New Report" button, users can initiate the process of creating a new report. This action allows them to input the necessary information and parameters to generate a new report within the tool. It typically involves providing data sources, selecting variables, and configuring analysis settings.
- Load existing report(s): By clicking on the "Load Report" button, users can access and analyze previously created reports. This action allows them to retrieve and view reports that have been saved or stored within the tool. They can select from a list of existing reports and load them for further analysis or exploration.
These options provide users with the flexibility to either create new reports or access and analyze existing reports, empowering them to effectively utilize the statistics tool based on their specific needs and requirements.
Users can create new reports by utilizing the "'''New Report'''" feature on the Main Page. This allows them to define analysis parameters and generate insights for specific use cases.
By clicking the "'''New Report'''" button [Figure 1 - '''A'''] on the Main Page, users are redirected to the New Report page. This page provides a user-friendly interface for creating new reports and defining the necessary
analysis parameters.
Users can load existing reports for analysis and comparison. Multiple reports can be loaded simultaneously, enabling users to assess model performance across datasets, compare different models, and evaluate the impact of model versions or parameter settings.
Loaded reports can be analyzed using the tool's comprehensive features, including visualizations, statistical metrics, and performance indicators. Users can explore metrics such as accuracy, precision, recall, and utilize intuitive visualizations to gain insights and make data-driven decisions.
By clicking the "'''Load Report'''" button [Figure 1 - '''D'''] on the Main Page, users are redirected to the View Reports page. This page serves as a central repository for all loaded reports, providing a comprehensive overview and access to various analysis and comparison functionalities. <br />
File:St new report.png ''Figure 2- new report'' <br />
Below is detailed information for all the data required to create a new report for the statistical tool.
Task configuration in the Statistics Tool [Figure 2 - '''C'''] allows users to customize and define various options for the current tool run. The configuration is stored in a JSON file located in the "'''StatisticsTool/configs'''" folder.
In the user interface (UI), users can select the desired configuration file from the available options in the folder. Each configuration file represents a unique set of user-defined options for the tool run.
The configuration file is a JSON object containing the following properties:
https://github.com/microsoft/StatisticsTool/wiki/User-Defined-Functions
Users can modify these properties in the configuration file to tailor the analysis and calculations based on their specific requirements. <br />
<div> The Prediction Directory [Figure 2 - '''D'''] is the location where the user specifies the directory containing log files from the previous run. These files are processed based on the configuration settings, filtering out irrelevant ones. The directory can be local or within the configured blob storage (directory under Predictions blobs prefix in the blob). This ensures accurate statistics and comparisons.<div>
<div>
- The Ground Truth Directory [Figure 2 - '''E'''] allows the user to specify a local directory for the ground truth file. It is an optional field and can be left blank if not applicable.
- '''Using Annotation Store''' - When the "Using Annotation Store" field is left blank, the ground truth (GT) will be retrieved from the annotation store as configured in the configuration file. The name of the GT file is derived from the name of the video file mentioned in the prediction log header. However, if the prediction log files were generated from local videos, the tool may not be able to accurately determine the corresponding GT file in the annotation store. In such cases, the user should provide a local directory for the ground truth (GT) files.
- '''Using Local GT Folder''' - If the user provides a local directory for the ground truth (GT) files, the tool will search for the GT files within that folder. The local GT folder should adhere to one of the following format: '''Flat Structure''': All the GT files are placed directly in the local GT folder without any subfolders. The name of each GT file should match the name of the corresponding video file, with a .json extension. Example: File:LocalGT.png'''Nested Structure''': Each video has its own subfolder named after the video file (without extension). Within each subfolder, the GT file shares the same name as the prediction log file. Example: File:Local2.pngBy organizing the GT files in one of these formats within the local GT folder, the tool will be able to properly associate the GT files with the corresponding video files and perform the necessary comparisons and evaluations.
- '''Example for local GT folders''' - If the local GT folder is located at "c:\folder\gt" and the video file is "Experiments\Recording\vid1.mp4" with the corresponding prediction log file at 'predictions/vid1/algo_log.json', the tool will search for the GT file in one of the following formats:
- GT File Format 1:
- Path: c:\folder\gt\vid1.json
- Here, the GT file is directly placed in the local GT folder with the same name as the video file (vid1) and a .json extension.
- GT File Format 2:
- Path: c:\folder\gt\vid1\algo_log.json
- In this format, a subfolder named "vid1" is created within the local GT folder. Inside the "vid1" subfolder, the GT file has the same name as the prediction log file (algo_log.json). By following either of these two formats within the local GT folder, the tool will be able to locate and use the appropriate GT file for evaluation and comparison with the prediction log file.
- GT File Format 1:
The Report Output Directory [Figure 2 - '''F'''] is the directory where the report outputs will be saved. Each run of the tool will create a folder with the current date and time (DATETIME) as the folder name. Within this folder, the main report file in .pkl format, which should be used to load the report, will be saved under the path: ouput_directory/DATETIME/saved by user/<configuration_name>.pkl. The <configuration_name> represents the name of the configuration used for the current run of the tool. This allows for easy organization and retrieval of the generated report outputs, ensuring that each run's results are stored separately under the corresponding timestamped folder. <br />
- The "Show" button [Figure 2 - '''G'''] allows users to view the content of the selected configuration. When the button is clicked, the content of the configuration is displayed in the gray cube located on the right side of the page [Figure 2 - '''I''']. This feature provides users with a convenient way to access and review the details of a specific configuration without the need to navigate to external files or sources.
- Add button functionality: By clicking on the "Add" button [Figure 2 - '''G'''], users can create a new configuration. The gray cube on the right side of the page allows users to input the required information for the new configuration. Once the necessary information is provided and the configuration is added, it will be saved as a JSON file under the "StatisticsTool/configs" directory. Additionally, the new configuration will be added to the Task Configuration list for easy access and selection. Here is the information needed for creating a new configuration:
- Configuration Name: [Specify the name for the configuration]
- Log Names to Evaluate: [Specify the names of the logs to evaluate]
- File Reading Function: [Specify the function for reading the files]
- Overlap Function: [Specify the overlap function to use]
- Transform Function (Empty for None): [Specify the transformation function, if applicable]
- Threshold: [Specify the threshold value]
- Evaluation Function: [Specify the evaluation function to use]
- Statistics Functions: [Specify the statistics functions to calculate]
- Partitioning Functions: [Specify the partitioning functions to use] By providing this information and clicking the "Add" button, users can easily create and manage multiple configurations for their tasks. Here is a screenshot when the user click on Add button - for adding new configuration:File:AddConfig.png<br />
Running the new report requires the user to provide all the necessary data that detailed above, including the configuration, predictions directory, ground truth directory, and reporter output directory. Once all the required information is filled, the user can initiate the report generation process by clicking the "Run" button. After the tool completes the calculations, a new report is generated. The user can access and view the generated report by clicking on the "Go To Reporter" link [Figure 3 - '''A'''], which will redirect them to the Report Viewer page.File:Gotoreporter new.png''Figure 3 - Go to Reporter''
The "Back" button on the New Report page that allows users to return to the main page [Figure 2 - '''A'''].
The main feature of the Statistic Tool is the ability to view reports. This feature allows users to analyze and compare experiment files stored in different directories.
- '''Main Directory''': This text field [Figure 1 - '''B'''] requires the user to enter the directory path where the main experiment files (e.g., .pkl files) are stored. This field is mandatory and must be filled in.
- '''Reference Directory''': This text field [Figure 1 - '''C'''] is optional and contains the directory path where the reference experiment files are stored. If the reference directory is provided, the user will have the option to compare the files between the main and reference directories.
The Report Viewer page provides a platform to load and analyze the files from both directories, enabling users to perform comparisons and gain insights from the experiment data.
To initiate the report viewing process, the user simply clicks on the "Load Report" button. This action redirects them to the Report Viewer page, where they can explore and analyze the loaded files.
</div>File:Report viewer1 new.png''Figure 4 - Report Viewer''<br />
To access the Report Viewer page in the Statistics Tool, there are two scenarios:
- When the user clicks on the "Go To Reporter" link from the Create New Report page.
- When the user clicks on the "Load Report" button on the main page.
In both cases, the user will be redirected to the Report Viewer page, where they can view and analyze the generated reports.
The Report Viewer page in the Statistics Tool provides various functionalities for analyzing and exploring evaluation reports. The following are the key components and features of the Report Viewer page:
The Toolbar contains various interactive elements and controls to enhance the user experience and provide quick access to essential functions.
Clicking on the Back button [figure 4 - '''A'''] redirects the user back to the main page, allowing for easy navigation between different sections of the tool.
The Reports Dropdown List [figure 4 - '''B'''] displays all the report files (e.g., .PKL files) found in the main directory. Users can select a specific report from the list to load it in the viewer for analysis and visualization.
A Template is a JSON file that defines the structure and segmentation of the report.
It consists of multiple '''View Segmentation-''' a View is a specific segment within a template that defines a subset of rows and columns for displaying the report data. Views provide a way to organize and present different sections of the report in a structured manner.
Templates allow for customizable views and layouts of the report data.
template example:File:Template1.pngThe "Template 1" consists of three segmentations: Total, View X position, and View Y position. Each segmentation has rows and columns segmentations. This hierarchical structure helps organize and display the report data effectively. Users can analyze different aspects of the data by selecting specific segmentations.
The Templates Dropdown List [figure 4 - '''C'''] contains all the template files found in the main directory. Templates are JSON files that provide information on how to segment the report, including rows and columns segmentation. If no templates are found in the main directory, the tool will search for templates in the reference directory. Additionally, a default template is always available for use.
Users can select a report from a list, and the tool will load and render the views of that report based on the chosen template.
Users can choose a template from a list. Upon selecting a template, the tool will render the currently selected report views according to the chosen template.
Users have the ability to update a specific view by selecting or deselecting its segmentations (horizontal and vertical). When segmentations are selected, the view will be rendered accordingly, reflecting the chosen segmentations.
The Add View button [figure 4 - '''D'''] in the toolbar that allows users to add a new view. When the user clicks on this button, a new view is added to the page. The user can provide a name for the view and select the desired segmentations for it, defining its layout and structure.File:ADD VIEW 1.png''Figure 5 - Add View'' In this example, when the user clicks on the "Add View" button in the toolbar, a new view is added at the bottom of the page. The user can expand or collapse the views to manage their visibility and organization. Once the view is added, the user has the option to name it, such as "My New View" [Figure 5 - '''A''']. Additionally, the user can select the desired segmentations, defining the horizontal and vertical divisions
[Figure 5 - '''B,C'''] or arrangements within the view.
Upon selecting the segmentations, the view will be rendered accordingly, displaying the content and layout based on the chosen segmentations. This allows the user to customize and visualize data within the newly added view, tailoring it to their specific requirements.
<br />
User can remove a View by clicking on the X button [Figure 5 - '''D''']
By clicking the "Save Template" button [figure 4 - '''E'''], users can save the templates along with all the changes made to the views.
When the "Save Template" button is clicked, a dialog box will open. Within this dialog, users have the option to update the template name. If the user chooses to change the template name while saving, a new template will be created with the new name.
It's important to note that if the user intends to save the default template, they must provide a new name for the template, as the default template cannot be overwritten. This ensures that the original default template remains intact while allowing users to create customized templates with unique names.File:Save template.png
If a user selects a report that exists in both the main and reference directories, they can compare the two versions and calculate the unique information.
To initiate the calculation of unique information, the user needs to activate it by using the slider in the toolbar [figure 4 - '''F''']. It's important to note that the slider is only active when the user selects the reference directory.
Once the unique calculation is activated, the tool will render the views with the unique information. This means that the views will display data or insights that are specific to either the main version or the reference version of the selected report.
By comparing the main and reference versions and calculating the unique information, users can gain valuable insights into the differences and distinct aspects of the selected report.File:Unique1.png''Figure 6 - Unique''
In the example above, the user can calculate the unique information by activating the Unique feature. By activating the Unique feature, the tool will perform the necessary calculations and render the views with the unique information.
User can access a list of TP (True Positive), FP (False Positive), and FN (False Negative) examples for each report (main and reference), as well as for each partition and category.
For each report and category, there is a link that displays the number of examples in that category. When the user clicks on the link, the examples list corresponding to the clicked category will open.
For instance, based on the example above, there are 2,479 TP [Figure 6 - '''A'''] examples in the Main report and 2,846 TP [Figure 6 - '''B'''] examples in the Reference report.
Furthermore, the example highlights that in the main report, there are 2 unique examples categorized as FP, while in the reference report, there are 7 unique [Figure 6 - '''C'''] examples for the TP category.
This functionality allows users to explore and analyze specific examples within different categories, providing a more detailed understanding of the reports and their corresponding classifications.
The Drawer functionality allows users to interact with the Examples List and perform additional actions. Here's an overview of its features:
- Examples List: When the user clicks on one of the links within the views, the Examples List will be opened in the drawer. The Examples List is presented on the left side of the drawer. It displays a list of examples categorized by video and grouped based on their comparison result. This organization provides an intuitive way to browse and navigate through the examples.
- Image Display: When the user selects an example from the Examples List, the corresponding image will be displayed on the right side of the drawer. The image will include its prediction and the ground truth (GT) annotation. This visualization helps users analyze and compare the specific example within the context of its prediction and ground truth.
- Export to Sheldon: The Drawer also includes an "Export to Sheldon" button. When clicked, a jump file for Sheldon is created. The jump file contains relevant information about the selected example. The path for the created file will be printed below the button, providing users with the location and accessibility of the exported file for further analysis or processing with Sheldon.
The Drawer functionality enhances the user experience by providing an organized view of examples, allowing image display, and facilitating the export of relevant information to Sheldon for further analysis.File:CaptureDrawer11.png
The Local Store functionality allows users to specify a local storage location from which the tool will load images. In the toolbar, there is an option to set the local storage configuration.
If the user sets the local storage information [figure 4 - '''H'''], the tool will search for and preview the images from that specified local storage. By default, if no local storage is specified, the tool will retrieve the images from the blob repository.
This feature provides users with flexibility in choosing the source of the images for preview and analysis within the tool. They can either use the default blob repository or configure a local storage location to access the images.
<br />
The statistics tool is built using Flask, Dash, and Angular. Flask is used for the backend server, Dash handles the interactive visualizations, and Angular provides a dynamic user interface. Together, they create a powerful and user-friendly tool for analyzing statistical data.
'''Flask''' serves as the backend for the statistics tool, handling server-side logic and data processing. Additionally, Flask also provides the client-side functionality by serving the HTML files for the main page, create new report page, example list, and image preview. This integration allows Flask to handle both the server-side operations and the delivery of client-side content, resulting in a cohesive and unified web application.
'''Angular''' - In the statistics tool, the '''Report Viewer''' page is built using the Angular framework. This includes the implementation of the Drawer component. However, the content within the views of the Report Viewer page is provided by Flask as Dash elements. These Dash elements are embedded within an iframe in the Angular page, allowing for the integration of Flask's server-side functionality with Angular's front-end framework.
Additionally, the Example List and the images within the Drawer are both HTML files that are returned from Flask. These HTML files are also embedded within iframes in the Angular page, enabling the seamless display and interaction of the Example List and images within the Drawer.
By leveraging Angular's framework for the Report Viewer page and incorporating Flask's Dash elements and HTML files within iframes, the statistics tool combines the strengths of both frameworks to deliver a comprehensive and interactive user experience.
To compile the Angular part of the statistics tool and generate the necessary production files, follow these steps:
- Open the terminal and navigate to the directory where the Angular project is located. In this case, it would be the "StatisticsTool/ng-client/statToolAngularApp" directory.
- Once inside the project directory, run the following command: '''npm run build''' This command will initiate the build process for the Angular application.
- After the build process completes, it will generate the necessary production files. One of these files is typically called "main.js," which contains the compiled code for the Angular application to run in a production environment. The "main.js" file can then be used to deploy and run the Angular part of the statistics tool in a production setting. <br /> </div></div> Category:Surface IL Data Science - Tools