Yara/pick and place yolo#460
Merged
yarakmk merged 25 commits intoJul 3, 2026
Merged
Conversation
) * Testing HRI SM2 * LLM package with venv * Fixed SM3 * RVIZ debug for HRI testing * Fixed lasr_llm by removing broken llama_cpp_python * Successfull SM2 of HRI * WIP - SM2 to SM3 * WIP: CHECKS * WIP - SM1 -> SM3 Final checks to see if it works consistently needs to be carried out * Ported recieve Object * Temporary detect3d fix * Working SM1-3 of HRI --------- Co-authored-by: Aldrich-Fernandes <aldrich.fernandes05@gmail.com>
* vlm service * service update * Added back license and readme * Formatted files with black --------- Co-authored-by: rajhirym <rym.rajhi@icloud.com> Co-authored-by: Yara Alkhelaiwi <yaralkhelaiwi@gmail.com>
* Working VLM Replaced old vqa for describing people with new vlm service * Formatted files with black
* Ros2 Port Language Package * Introduction testing * Modifications. fix init before final push * Message filter * Launch file * State machine and setup.py modifications * Fix register face and recognise * Port HRI introduce states to YASMIN * Fix YOLO detections in introduce. * Initial code for pick and place * Bug fixes * Fixes after testing on the robot * Black formatting * HRI Task - SM1-3 fully tested and working on the robot (LASR-at-Home#443) * Testing HRI SM2 * LLM package with venv * Fixed SM3 * RVIZ debug for HRI testing * Fixed lasr_llm by removing broken llama_cpp_python * Successfull SM2 of HRI * WIP - SM2 to SM3 * WIP: CHECKS * WIP - SM1 -> SM3 Final checks to see if it works consistently needs to be carried out * Ported recieve Object * Temporary detect3d fix * Working SM1-3 of HRI --------- Co-authored-by: Aldrich-Fernandes <aldrich.fernandes05@gmail.com> * New VLM service to be used in HRI (LASR-at-Home#444) * vlm service * service update * Added back license and readme * Formatted files with black --------- Co-authored-by: rajhirym <rym.rajhi@icloud.com> Co-authored-by: Yara Alkhelaiwi <yaralkhelaiwi@gmail.com> * VLM service for HRI task (LASR-at-Home#445) * Working VLM Replaced old vqa for describing people with new vlm service * Formatted files with black * Fixed accidental merge errors * WIP - Setting up and Testing HRI task * Fix introduce and remove testing code * WIP - HRI task semi-working SM1-4 works correctly, however during SM4 it doesn't look at the seated guests * Change LLM to Ollama got HRI get anem and drink * Updated Seat Guest (SM3) * WIP - HRI SM4 being refactored * Verbal introduction is correct * Debugging SM4 face person errors * Formatted files with black --------- Co-authored-by: Yara Alkhelaiwi <yara.alkhelaiwi@kcl.ac.uk> Co-authored-by: Yara Alkhelaiwi <yaralkhelaiwi@gmail.com> Co-authored-by: rajhirym <rym.rajhi@icloud.com> Co-authored-by: Yara Alkhelaiwi <Yarakmk> Co-authored-by: Aldrich-Fernandes <aldrich.fernandes05@gmail.com> Co-authored-by: Ma'ayan Armony <maayan.armony@gmail.com>
…a/pick_and_place # Conflicts: # common/language/lasr_llm/lasr_llm/llm_inference.py # common/language/lasr_llm/lasr_llm/nodes/hri_task_service.py # common/language/lasr_llm/requirements.in # common/language/lasr_llm/requirements.txt # common/language/lasr_llm/setup.py # common/vision/lasr_vision_eye_tracker/lasr_vision_eye_tracker/eye_tracker_action_server.py # skills/setup.py # skills/src/lasr_skills/ask_and_listen.py # skills/src/lasr_skills/describe_people.py # skills/src/lasr_skills/detect_3d.py # skills/src/lasr_skills/detect_all_in_polygon.py # skills/src/lasr_skills/eye_tracker.py # skills/src/lasr_skills/go_to_location.py # skills/src/lasr_skills/vision/crop_image_3d.py # skills/src/lasr_skills/vision/get_image.py # skills/src/lasr_skills/wait_for_person_in_area.py # tasks/HRI/HRI/states/get_name_and_drink.py # tasks/HRI/HRI/states/greet.py # tasks/HRI/HRI/states/hri_learn_faces.py # tasks/HRI/HRI/states/seat_guest.py # tasks/HRI/config/debug.rviz # tasks/HRI/config/lab.yaml # tasks/HRI/launch/HRI.launch.py # tasks/pick_and_place/config/config.yaml # tasks/pick_and_place/pick_and_place/state_machine.py # tasks/pick_and_place/pick_and_place/states/choose_shelf.py # tasks/pick_and_place/pick_and_place/states/classify_category.py # tasks/pick_and_place/pick_and_place/states/compute_approach.py # tasks/pick_and_place/pick_and_place/states/detect_objects.py # tasks/pick_and_place/pick_and_place/states/find_and_go_to_table.py # tasks/pick_and_place/pick_and_place/states/instruct_pick.py # tasks/pick_and_place/pick_and_place/states/instruct_place.py # tasks/pick_and_place/pick_and_place/states/scan_shelves.py # tasks/pick_and_place/pick_and_place/states/select_and_visualize_object.py # tasks/pick_and_place/pick_and_place/states/start.py # tasks/pick_and_place/pick_and_place/test_detect.py # tasks/pick_and_place/setup.py # tasks/pick_and_place/test/test_copyright.py # tasks/pick_and_place/test/test_flake8.py # tasks/pick_and_place/test/test_pep257.py
* start of GPSR package * Refactor GPSR state machine to use YASMIN, add AnnouncePlan state, and update TTS handling * Refactor: Black reformat * Add configuration files and update parameters for GPSR task modified prompt * - Introduced world.py to manage skills, locations, objects, people, and general knowledge for the GPSR task. - Added general knowledge configuration in general_knowledge.yaml. * Add prompts for skill selection, refinement, and planning; update locations, objects, and people configurations * Refactor GPSR task to enhance LLM integration and streamline planning process - Refactored agent.py to support cloud and local LLM dispatching, including new planning and querying functionalities. - Introduced planner.py for skill selection and planning pipeline. - Enhanced prompts for skill selection and planning. - Updated state machine to remove unnecessary states and streamline command processing. - Improved world management with new functions for loading and formatting skills, locations, and objects. - Removed deprecated states and configurations to simplify the codebase. * implemented go to skill and readme * Refactor: Black format --------- Co-authored-by: Ma'ayan Armony <maayan.armony@gmail.com>
* Testing HRI SM2 * LLM package with venv * Fixed SM3 * RVIZ debug for HRI testing * Fixed lasr_llm by removing broken llama_cpp_python * Successfull SM2 of HRI * WIP - SM2 to SM3 * WIP: CHECKS * WIP - SM1 -> SM3 Final checks to see if it works consistently needs to be carried out * Ported recieve Object * Temporary detect3d fix * WIP: Follow Person Experimental * Merge branch 'ros2-hri-testing' into ros2-hri-sm6 * WIP: Planning SM6 * update waitforperson to use blackboard data as well * WIP: debugging stuck in loop * ran black formatting * Merge branch 'ros2' into ros2-hri-sm6 * Fixed broken merge * WIP: Debugging tracking * testing tf_listener instead of amcl pose for robot pose * Fixed tracking * Fixed following navigation * Fixed inconsistant following logic * Fixed bugs related to handle_sigint and service * Seperated updateable navigation to prevent conflicts * Fixed merge error and removed duplicate file * Fixed blocking logic * Added debug config for rviz * Ported additional States * HRI task (now SM1-5) (LASR-at-Home#447) * Ros2 Port Language Package * Introduction testing * Modifications. fix init before final push * Message filter * Launch file * State machine and setup.py modifications * Fix register face and recognise * Port HRI introduce states to YASMIN * Fix YOLO detections in introduce. * Initial code for pick and place * Bug fixes * Fixes after testing on the robot * Black formatting * HRI Task - SM1-3 fully tested and working on the robot (LASR-at-Home#443) * Testing HRI SM2 * LLM package with venv * Fixed SM3 * RVIZ debug for HRI testing * Fixed lasr_llm by removing broken llama_cpp_python * Successfull SM2 of HRI * WIP - SM2 to SM3 * WIP: CHECKS * WIP - SM1 -> SM3 Final checks to see if it works consistently needs to be carried out * Ported recieve Object * Temporary detect3d fix * Working SM1-3 of HRI --------- Co-authored-by: Aldrich-Fernandes <aldrich.fernandes05@gmail.com> * New VLM service to be used in HRI (LASR-at-Home#444) * vlm service * service update * Added back license and readme * Formatted files with black --------- Co-authored-by: rajhirym <rym.rajhi@icloud.com> Co-authored-by: Yara Alkhelaiwi <yaralkhelaiwi@gmail.com> * VLM service for HRI task (LASR-at-Home#445) * Working VLM Replaced old vqa for describing people with new vlm service * Formatted files with black * Fixed accidental merge errors * WIP - Setting up and Testing HRI task * Fix introduce and remove testing code * WIP - HRI task semi-working SM1-4 works correctly, however during SM4 it doesn't look at the seated guests * Change LLM to Ollama got HRI get anem and drink * Updated Seat Guest (SM3) * WIP - HRI SM4 being refactored * Verbal introduction is correct * Debugging SM4 face person errors * Formatted files with black --------- Co-authored-by: Yara Alkhelaiwi <yara.alkhelaiwi@kcl.ac.uk> Co-authored-by: Yara Alkhelaiwi <yaralkhelaiwi@gmail.com> Co-authored-by: rajhirym <rym.rajhi@icloud.com> Co-authored-by: Yara Alkhelaiwi <Yarakmk> Co-authored-by: Aldrich-Fernandes <aldrich.fernandes05@gmail.com> Co-authored-by: Ma'ayan Armony <maayan.armony@gmail.com> * Added Follow Host to HRI * Fixed recovery states not being reached * Fixed looking at person while tracking * Fixed Detect_keypoints and started implementing place_bag * Added drop point detection * WIP: Place bag MVP * Removed potentially breaking handle_sigints * Updated GetPersonPoint logic and added goal forgiving * Updated wait times and removed redundent check * Merging fixed gpu version of HRI (LASR-at-Home#454) * HRI task (now SM1-5) (LASR-at-Home#447) * Ros2 Port Language Package * Introduction testing * Modifications. fix init before final push * Message filter * Launch file * State machine and setup.py modifications * Fix register face and recognise * Port HRI introduce states to YASMIN * Fix YOLO detections in introduce. * Initial code for pick and place * Bug fixes * Fixes after testing on the robot * Black formatting * HRI Task - SM1-3 fully tested and working on the robot (LASR-at-Home#443) * Testing HRI SM2 * LLM package with venv * Fixed SM3 * RVIZ debug for HRI testing * Fixed lasr_llm by removing broken llama_cpp_python * Successfull SM2 of HRI * WIP - SM2 to SM3 * WIP: CHECKS * WIP - SM1 -> SM3 Final checks to see if it works consistently needs to be carried out * Ported recieve Object * Temporary detect3d fix * Working SM1-3 of HRI --------- Co-authored-by: Aldrich-Fernandes <aldrich.fernandes05@gmail.com> * New VLM service to be used in HRI (LASR-at-Home#444) * vlm service * service update * Added back license and readme * Formatted files with black --------- Co-authored-by: rajhirym <rym.rajhi@icloud.com> Co-authored-by: Yara Alkhelaiwi <yaralkhelaiwi@gmail.com> * VLM service for HRI task (LASR-at-Home#445) * Working VLM Replaced old vqa for describing people with new vlm service * Formatted files with black * Fixed accidental merge errors * WIP - Setting up and Testing HRI task * Fix introduce and remove testing code * WIP - HRI task semi-working SM1-4 works correctly, however during SM4 it doesn't look at the seated guests * Change LLM to Ollama got HRI get anem and drink * Updated Seat Guest (SM3) * WIP - HRI SM4 being refactored * Verbal introduction is correct * Debugging SM4 face person errors * Formatted files with black --------- Co-authored-by: Yara Alkhelaiwi <yara.alkhelaiwi@kcl.ac.uk> Co-authored-by: Yara Alkhelaiwi <yaralkhelaiwi@gmail.com> Co-authored-by: rajhirym <rym.rajhi@icloud.com> Co-authored-by: Yara Alkhelaiwi <Yarakmk> Co-authored-by: Aldrich-Fernandes <aldrich.fernandes05@gmail.com> Co-authored-by: Ma'ayan Armony <maayan.armony@gmail.com> * Added REID fallback Also made minor changes to accomodate gpu * Test HRI on GPU * Updated HRI to say attributes Also added in a SafeGoToLocation and moved StartDoorSM to skills * Debugging HRI task * Added gpu requirements for lasr_vision_reid * Testing of HRI with GPU * Fixing bugs * Fixing of EyeTracker * Working eye tracker, yolo is bottleneck * Fully fixed eye tracker * Adding in host * Fixed introduce for host * Formatted files * Merge branch 'ros2-hri-sm6' into ros2-gpu-and-reid * Added in host logic to SM * Testing HRI * Working HRI * Bug fixes * HRI MVP * Formatted files * Merge branch 'ros2' into ros2-hri-sm6
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.