Problems with Cloudsim

[osrf-migration]

Original report (archived issue) by Sophisticated Engineering (Bitbucket: sopheng).

---

We have problems with cloudsim on the portal.

Tests from last night are running very long and have the Status “Deleting Pods”. And some have additionally “Error: Admin Review”.

I’ve send an Email to subt-help.

Is this problem also seen by other teams?

[osrf-migration]

Original comment by Chris Fotache (Bitbucket: chrisfotache).

---

Yes. Reminds me of last time.

[osrf-migration]

Original comment by Arthur Schang (Bitbucket: Arthur Schang).

---

It seems CloudSim was under a very heavy load last night. Please try resubmitting a couple of your runs.

[osrf-migration]

Original comment by Hector Escobar (Bitbucket: hector_escobar).

---

I just uploaded my images like 30 min ago and got the same Error:Admin Review. Now it changed to “Error: InitializationFailed”. My images run ok with docker-compose, is this due to the loading in the cloud?

[osrf-migration]

Original comment by Malcolm Stagg (Bitbucket: malcolmst7).

---

I see the same behavior here

[osrf-migration]

Original comment by Martin Dlouhy (Bitbucket: robotikacz).

---

The same problem (Terminated, Error: InitializationFailed) for all robotika latest simulations (ver55). I would almost change the priority from “major” to “blocker”. We are trying to create workaround for missing messages, and there is no way how to test it now …

[osrf-migration]

Original comment by Malcolm Stagg (Bitbucket: malcolmst7).

---

Does anyone see any update on this? My recent runs seem to have been restarted. Most are pending, one is LaunchingPods. That one only has real-time logs (nice feature btw) for one robot, which shows:

ROS_MASTER_URI=http://10.46.56.2:11311
]2;/home/developer/subt_ws/install/share/subt_ros/launch/x2_description.launch http://10.46.56.2:11311�
No processes to monitor
shutting down processing monitor...
... shutting down processing monitor complete

Not sure that’s a good sign…

[Update] it failed, admin review

[osrf-migration]

Original comment by Arthur Schang (Bitbucket: Arthur Schang).

---

I believe part of the Initialization failure/error messages is based upon submitting runs in rapid succession. Please try resubmitting in a slower manner and space your simulations submissions apart. I don't have a good answer for how long to wait between submissions but waiting to submit subsequent submissions until the first submission is Running should be sufficient. If you're using the CLI and submitting a handful of submissions at once, please add some arbitrarily lengthy sleep (5 minutes is a good spacing) between each submission for a temporary hands-off fix or monitor your submissions on the web portal and submit subsequent simulations after the prior has the status Running.

This is related to an issue bumping into an AWS limit on CloudSim with subsequent loading of a large number of simulation requests at a single time. We're working on a more permanent fix that does not require competitors to alter their submission workflow.

[osrf-migration]

Original comment by Sophisticated Engineering (Bitbucket: sopheng).

---

Two of our tests are meanwhile “running “ . The others are “Terminated GazeboError”

[osrf-migration]

Original comment by Arthur Schang (Bitbucket: Arthur Schang).

---

GazeboError is a different issue, please re-run the experiments.

[osrf-migration]

Original comment by Sarah Kitchen (Bitbucket: snkitche).

---

Arthur Schang (Arthur Schang) It appears that there is an auto-restart process that happens in some cases - specifically, I observe in some cases when I receive an error, but the state is not given as Terminated, the run will relaunch after a few hours. This is really nice to have, but could it also be affecting how often we are seeing the Initialization Failures in the last 24 hours?

‌

[osrf-migration]

Original comment by Hector Escobar (Bitbucket: hector_escobar).

---

Mine restarted but again went into Admin Review

[osrf-migration]

Original comment by Martin Dlouhy (Bitbucket: robotikacz).

---

robotika

1/30/20, 5:01 PM

ver55p3

f0da93a2-1f41-4398-be24-28d6cdbaa975-r-1

DeletingPods

Error: AdminReview

[osrf-migration]

Original comment by Arthur Schang (Bitbucket: Arthur Schang).

---

Sarah, I am not aware of the logic behind the restart process. I will defer to someone else to answer that question. If a batch of runs are all restarted at the same time it will almost certainly result in another initialization error/failure at the moment.

[osrf-migration]

Original comment by Malcolm Stagg (Bitbucket: malcolmst7).

---

I can confirm I had 6 runs restart (all at the same time, though 5 were initially “pending”) and all 6 failed with initialization error/admin review. Now that those are all done I’m going to carefully try just one, hoping for the best…

[osrf-migration]

Original comment by Malcolm Stagg (Bitbucket: malcolmst7).

---

The portal is now displaying “Unknown Error” for me and is not displaying any simulation results. Is anyone else seeing that? I tried logging out and in again.

I was just going to start a new simulation but maybe I’d better wait a bit first.

[Edit] Looks ok now, maybe just a temporary server issue