report hung radostrace ops on interrupt

[zhhuabj]

Handle SIGINT without exiting immediately so radostrace can dump operations still pending in the BPF ops map before cleanup.

• add SIGINT state tracking in radostrace
• iterate the ops BPF map and print unfinished operations
• include stuck duration, client, tid, target osd, object, and rw mode
• return cleanly after interrupt/timeout-triggered shutdown

[zhhuabj]

test result - https://paste.ubuntu.com/p/FvpMC8XsCX/

[pponnuvel]

exit(3) isn't a async-signal-safe function: https://www.man7.org/linux/man-pages/man7/signal-safety.7.html

better to use _exit() instead.

The same concern is applicable for clog too. We need to review "signal safety" across the codease. But that can be done separately later.

[pponnuvel]

To get the first key that's not in the map, bpf_map_get_next_key needs to be called with NULL as key. But a valid "key" is passed now (key is zero-initalized but it's not NULL), so this is likely to miss the correct first key.

struct client_op_k *cur = NULL;
while (bpf_map_get_next_key(map_fd, cur, &next_key) == 0) {
    if (bpf_map_lookup_elem(map_fd, &next_key, &val) == 0) {
        ...
    }
    key = next_key;
    cur = &key;
}

would fix it.

[pponnuvel]

I also think "hung ops" better describes than "hang ops", so suggest name changes as such.

[taodd]

We also need to add a test to deliberately cause a hang Op

[taodd]

Let's keep the original code here

[zhhuabj]

done

[zhhuabj]

test result in microceph - https://paste.ubuntu.com/p/9NqMgrcz4K/
test result in openstack - https://paste.ubuntu.com/p/Dtg36rVNx8/

[taodd]

Thanks @zhhuabj

1. The incomplete operations are missing the pid value, we can get the pid value from the send_op function instead of the finish_op function.
2. To simplify the output, instead of naming the new column as "status" and filled with words "complete" and "incomplete", we can name the column to "Complete" with value of "1" or "0" to indicate whether the operation is complete or not.
3. Another suggestion is that we don't need to separate the output between complete and incomplete ops. Put all of them together would be fine and easy to process by the analyzer

[taodd]

Also, the patch needs to be rebased

[zhhuabj]

Thanks @zhhuabj

1. The incomplete operations are missing the pid value, we can get the pid value from the send_op function instead of the finish_op function.
2. To simplify the output, instead of naming the new column as "status" and filled with words "complete" and "incomplete", we can name the column to "Complete" with value of "1" or "0" to indicate whether the operation is complete or not.
3. Another suggestion is that we don't need to separate the output between complete and incomplete ops. Put all of them together would be fine and easy to process by the analyzer

Hi @dongdong, Thank you for the feedback! I've addressed all three suggestions:

1, pid from send_op: Moved val->pid = get_pid() to uprobe_send_op so incomplete ops also have a valid pid.
2, Complete column with 0/1: Renamed the status column to Complete and replaced "complete"/"incomplete" strings with "1"/"0" for easier filtering.
3, Unified output: Merged complete and incomplete ops into a single output stream — completed ops print with Complete=1 as they arrive, incomplete ops print with Complete=0 at flush time.

and here is the latest test result - https://paste.ubuntu.com/p/3sSGqb3WGv/