HRI Task - SM1-3 fully tested and working on the robot#443
Merged
Conversation
Final checks to see if it works consistently needs to be carried out
Fadi-Mostefai
added a commit
that referenced
this pull request
Jun 15, 2026
* Testing HRI SM2 * LLM package with venv * Fixed SM3 * RVIZ debug for HRI testing * Fixed lasr_llm by removing broken llama_cpp_python * Successfull SM2 of HRI * WIP - SM2 to SM3 * WIP: CHECKS * WIP - SM1 -> SM3 Final checks to see if it works consistently needs to be carried out * Ported recieve Object * Temporary detect3d fix * Working SM1-3 of HRI --------- Co-authored-by: Aldrich-Fernandes <aldrich.fernandes05@gmail.com>
Fadi-Mostefai
added a commit
that referenced
this pull request
Jun 18, 2026
* Ros2 Port Language Package * Introduction testing * Modifications. fix init before final push * Message filter * Launch file * State machine and setup.py modifications * Fix register face and recognise * Port HRI introduce states to YASMIN * Fix YOLO detections in introduce. * Initial code for pick and place * Bug fixes * Fixes after testing on the robot * Black formatting * HRI Task - SM1-3 fully tested and working on the robot (#443) * Testing HRI SM2 * LLM package with venv * Fixed SM3 * RVIZ debug for HRI testing * Fixed lasr_llm by removing broken llama_cpp_python * Successfull SM2 of HRI * WIP - SM2 to SM3 * WIP: CHECKS * WIP - SM1 -> SM3 Final checks to see if it works consistently needs to be carried out * Ported recieve Object * Temporary detect3d fix * Working SM1-3 of HRI --------- Co-authored-by: Aldrich-Fernandes <aldrich.fernandes05@gmail.com> * New VLM service to be used in HRI (#444) * vlm service * service update * Added back license and readme * Formatted files with black --------- Co-authored-by: rajhirym <rym.rajhi@icloud.com> Co-authored-by: Yara Alkhelaiwi <yaralkhelaiwi@gmail.com> * VLM service for HRI task (#445) * Working VLM Replaced old vqa for describing people with new vlm service * Formatted files with black * Fixed accidental merge errors * WIP - Setting up and Testing HRI task * Fix introduce and remove testing code * WIP - HRI task semi-working SM1-4 works correctly, however during SM4 it doesn't look at the seated guests * Change LLM to Ollama got HRI get anem and drink * Updated Seat Guest (SM3) * WIP - HRI SM4 being refactored * Verbal introduction is correct * Debugging SM4 face person errors * Formatted files with black --------- Co-authored-by: Yara Alkhelaiwi <yara.alkhelaiwi@kcl.ac.uk> Co-authored-by: Yara Alkhelaiwi <yaralkhelaiwi@gmail.com> Co-authored-by: rajhirym <rym.rajhi@icloud.com> Co-authored-by: Yara Alkhelaiwi <Yarakmk> Co-authored-by: Aldrich-Fernandes <aldrich.fernandes05@gmail.com> Co-authored-by: Ma'ayan Armony <maayan.armony@gmail.com>
Aldrich-Fernandes
added a commit
that referenced
this pull request
Jun 19, 2026
* Ros2 Port Language Package * Introduction testing * Modifications. fix init before final push * Message filter * Launch file * State machine and setup.py modifications * Fix register face and recognise * Port HRI introduce states to YASMIN * Fix YOLO detections in introduce. * Initial code for pick and place * Bug fixes * Fixes after testing on the robot * Black formatting * HRI Task - SM1-3 fully tested and working on the robot (#443) * Testing HRI SM2 * LLM package with venv * Fixed SM3 * RVIZ debug for HRI testing * Fixed lasr_llm by removing broken llama_cpp_python * Successfull SM2 of HRI * WIP - SM2 to SM3 * WIP: CHECKS * WIP - SM1 -> SM3 Final checks to see if it works consistently needs to be carried out * Ported recieve Object * Temporary detect3d fix * Working SM1-3 of HRI --------- Co-authored-by: Aldrich-Fernandes <aldrich.fernandes05@gmail.com> * New VLM service to be used in HRI (#444) * vlm service * service update * Added back license and readme * Formatted files with black --------- Co-authored-by: rajhirym <rym.rajhi@icloud.com> Co-authored-by: Yara Alkhelaiwi <yaralkhelaiwi@gmail.com> * VLM service for HRI task (#445) * Working VLM Replaced old vqa for describing people with new vlm service * Formatted files with black * Fixed accidental merge errors * WIP - Setting up and Testing HRI task * Fix introduce and remove testing code * WIP - HRI task semi-working SM1-4 works correctly, however during SM4 it doesn't look at the seated guests * Change LLM to Ollama got HRI get anem and drink * Updated Seat Guest (SM3) * WIP - HRI SM4 being refactored * Verbal introduction is correct * Debugging SM4 face person errors * Formatted files with black --------- Co-authored-by: Yara Alkhelaiwi <yara.alkhelaiwi@kcl.ac.uk> Co-authored-by: Yara Alkhelaiwi <yaralkhelaiwi@gmail.com> Co-authored-by: rajhirym <rym.rajhi@icloud.com> Co-authored-by: Yara Alkhelaiwi <Yarakmk> Co-authored-by: Aldrich-Fernandes <aldrich.fernandes05@gmail.com> Co-authored-by: Ma'ayan Armony <maayan.armony@gmail.com>
Fadi-Mostefai
added a commit
that referenced
this pull request
Jun 23, 2026
* HRI task (now SM1-5) (#447) * Ros2 Port Language Package * Introduction testing * Modifications. fix init before final push * Message filter * Launch file * State machine and setup.py modifications * Fix register face and recognise * Port HRI introduce states to YASMIN * Fix YOLO detections in introduce. * Initial code for pick and place * Bug fixes * Fixes after testing on the robot * Black formatting * HRI Task - SM1-3 fully tested and working on the robot (#443) * Testing HRI SM2 * LLM package with venv * Fixed SM3 * RVIZ debug for HRI testing * Fixed lasr_llm by removing broken llama_cpp_python * Successfull SM2 of HRI * WIP - SM2 to SM3 * WIP: CHECKS * WIP - SM1 -> SM3 Final checks to see if it works consistently needs to be carried out * Ported recieve Object * Temporary detect3d fix * Working SM1-3 of HRI --------- Co-authored-by: Aldrich-Fernandes <aldrich.fernandes05@gmail.com> * New VLM service to be used in HRI (#444) * vlm service * service update * Added back license and readme * Formatted files with black --------- Co-authored-by: rajhirym <rym.rajhi@icloud.com> Co-authored-by: Yara Alkhelaiwi <yaralkhelaiwi@gmail.com> * VLM service for HRI task (#445) * Working VLM Replaced old vqa for describing people with new vlm service * Formatted files with black * Fixed accidental merge errors * WIP - Setting up and Testing HRI task * Fix introduce and remove testing code * WIP - HRI task semi-working SM1-4 works correctly, however during SM4 it doesn't look at the seated guests * Change LLM to Ollama got HRI get anem and drink * Updated Seat Guest (SM3) * WIP - HRI SM4 being refactored * Verbal introduction is correct * Debugging SM4 face person errors * Formatted files with black --------- Co-authored-by: Yara Alkhelaiwi <yara.alkhelaiwi@kcl.ac.uk> Co-authored-by: Yara Alkhelaiwi <yaralkhelaiwi@gmail.com> Co-authored-by: rajhirym <rym.rajhi@icloud.com> Co-authored-by: Yara Alkhelaiwi <Yarakmk> Co-authored-by: Aldrich-Fernandes <aldrich.fernandes05@gmail.com> Co-authored-by: Ma'ayan Armony <maayan.armony@gmail.com> * Added REID fallback Also made minor changes to accomodate gpu * Test HRI on GPU * Updated HRI to say attributes Also added in a SafeGoToLocation and moved StartDoorSM to skills * Debugging HRI task * Added gpu requirements for lasr_vision_reid * Testing of HRI with GPU * Fixing bugs * Fixing of EyeTracker * Working eye tracker, yolo is bottleneck * Fully fixed eye tracker * Adding in host * Fixed introduce for host * Formatted files * Merge branch 'ros2-hri-sm6' into ros2-gpu-and-reid
Fadi-Mostefai
added a commit
that referenced
this pull request
Jun 26, 2026
* Testing HRI SM2 * LLM package with venv * Fixed SM3 * RVIZ debug for HRI testing * Fixed lasr_llm by removing broken llama_cpp_python * Successfull SM2 of HRI * WIP - SM2 to SM3 * WIP: CHECKS * WIP - SM1 -> SM3 Final checks to see if it works consistently needs to be carried out * Ported recieve Object * Temporary detect3d fix * WIP: Follow Person Experimental * Merge branch 'ros2-hri-testing' into ros2-hri-sm6 * WIP: Planning SM6 * update waitforperson to use blackboard data as well * WIP: debugging stuck in loop * ran black formatting * Merge branch 'ros2' into ros2-hri-sm6 * Fixed broken merge * WIP: Debugging tracking * testing tf_listener instead of amcl pose for robot pose * Fixed tracking * Fixed following navigation * Fixed inconsistant following logic * Fixed bugs related to handle_sigint and service * Seperated updateable navigation to prevent conflicts * Fixed merge error and removed duplicate file * Fixed blocking logic * Added debug config for rviz * Ported additional States * HRI task (now SM1-5) (#447) * Ros2 Port Language Package * Introduction testing * Modifications. fix init before final push * Message filter * Launch file * State machine and setup.py modifications * Fix register face and recognise * Port HRI introduce states to YASMIN * Fix YOLO detections in introduce. * Initial code for pick and place * Bug fixes * Fixes after testing on the robot * Black formatting * HRI Task - SM1-3 fully tested and working on the robot (#443) * Testing HRI SM2 * LLM package with venv * Fixed SM3 * RVIZ debug for HRI testing * Fixed lasr_llm by removing broken llama_cpp_python * Successfull SM2 of HRI * WIP - SM2 to SM3 * WIP: CHECKS * WIP - SM1 -> SM3 Final checks to see if it works consistently needs to be carried out * Ported recieve Object * Temporary detect3d fix * Working SM1-3 of HRI --------- Co-authored-by: Aldrich-Fernandes <aldrich.fernandes05@gmail.com> * New VLM service to be used in HRI (#444) * vlm service * service update * Added back license and readme * Formatted files with black --------- Co-authored-by: rajhirym <rym.rajhi@icloud.com> Co-authored-by: Yara Alkhelaiwi <yaralkhelaiwi@gmail.com> * VLM service for HRI task (#445) * Working VLM Replaced old vqa for describing people with new vlm service * Formatted files with black * Fixed accidental merge errors * WIP - Setting up and Testing HRI task * Fix introduce and remove testing code * WIP - HRI task semi-working SM1-4 works correctly, however during SM4 it doesn't look at the seated guests * Change LLM to Ollama got HRI get anem and drink * Updated Seat Guest (SM3) * WIP - HRI SM4 being refactored * Verbal introduction is correct * Debugging SM4 face person errors * Formatted files with black --------- Co-authored-by: Yara Alkhelaiwi <yara.alkhelaiwi@kcl.ac.uk> Co-authored-by: Yara Alkhelaiwi <yaralkhelaiwi@gmail.com> Co-authored-by: rajhirym <rym.rajhi@icloud.com> Co-authored-by: Yara Alkhelaiwi <Yarakmk> Co-authored-by: Aldrich-Fernandes <aldrich.fernandes05@gmail.com> Co-authored-by: Ma'ayan Armony <maayan.armony@gmail.com> * Added Follow Host to HRI * Fixed recovery states not being reached * Fixed looking at person while tracking * Fixed Detect_keypoints and started implementing place_bag * Added drop point detection * WIP: Place bag MVP * Removed potentially breaking handle_sigints * Updated GetPersonPoint logic and added goal forgiving * Updated wait times and removed redundent check * Merging fixed gpu version of HRI (#454) * HRI task (now SM1-5) (#447) * Ros2 Port Language Package * Introduction testing * Modifications. fix init before final push * Message filter * Launch file * State machine and setup.py modifications * Fix register face and recognise * Port HRI introduce states to YASMIN * Fix YOLO detections in introduce. * Initial code for pick and place * Bug fixes * Fixes after testing on the robot * Black formatting * HRI Task - SM1-3 fully tested and working on the robot (#443) * Testing HRI SM2 * LLM package with venv * Fixed SM3 * RVIZ debug for HRI testing * Fixed lasr_llm by removing broken llama_cpp_python * Successfull SM2 of HRI * WIP - SM2 to SM3 * WIP: CHECKS * WIP - SM1 -> SM3 Final checks to see if it works consistently needs to be carried out * Ported recieve Object * Temporary detect3d fix * Working SM1-3 of HRI --------- Co-authored-by: Aldrich-Fernandes <aldrich.fernandes05@gmail.com> * New VLM service to be used in HRI (#444) * vlm service * service update * Added back license and readme * Formatted files with black --------- Co-authored-by: rajhirym <rym.rajhi@icloud.com> Co-authored-by: Yara Alkhelaiwi <yaralkhelaiwi@gmail.com> * VLM service for HRI task (#445) * Working VLM Replaced old vqa for describing people with new vlm service * Formatted files with black * Fixed accidental merge errors * WIP - Setting up and Testing HRI task * Fix introduce and remove testing code * WIP - HRI task semi-working SM1-4 works correctly, however during SM4 it doesn't look at the seated guests * Change LLM to Ollama got HRI get anem and drink * Updated Seat Guest (SM3) * WIP - HRI SM4 being refactored * Verbal introduction is correct * Debugging SM4 face person errors * Formatted files with black --------- Co-authored-by: Yara Alkhelaiwi <yara.alkhelaiwi@kcl.ac.uk> Co-authored-by: Yara Alkhelaiwi <yaralkhelaiwi@gmail.com> Co-authored-by: rajhirym <rym.rajhi@icloud.com> Co-authored-by: Yara Alkhelaiwi <Yarakmk> Co-authored-by: Aldrich-Fernandes <aldrich.fernandes05@gmail.com> Co-authored-by: Ma'ayan Armony <maayan.armony@gmail.com> * Added REID fallback Also made minor changes to accomodate gpu * Test HRI on GPU * Updated HRI to say attributes Also added in a SafeGoToLocation and moved StartDoorSM to skills * Debugging HRI task * Added gpu requirements for lasr_vision_reid * Testing of HRI with GPU * Fixing bugs * Fixing of EyeTracker * Working eye tracker, yolo is bottleneck * Fully fixed eye tracker * Adding in host * Fixed introduce for host * Formatted files * Merge branch 'ros2-hri-sm6' into ros2-gpu-and-reid * Added in host logic to SM * Testing HRI * Working HRI * Bug fixes * HRI MVP * Formatted files * Merge branch 'ros2' into ros2-hri-sm6
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SM 1-3 for the HRI task has been completed, marking the end to #417 and the first part of #418. Next steps are to add in SM4 (introduction and reid i.e. solving issues #418, #419), as well as SM5 (grabbing the bag), and SM6 (follow host). #420 has been temporarily solved by using the outdated clip vqa model, this is to be replaced as soon as possible by the new VLM service once that is merged onto the ros2 branch.
SM5 has already been completed by @Aldrich-Fernandes on a seperate branch, and will be fully integrated into the HRI state machine soon. Same goes for SM4 by @yarakmk.