patch: fix /tmp permissions for aws eks deployments

[patriciareinoso]

🏷️ Type of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation update
• Tooling and CI
• Dependencies upgrade or change
• Chores / refactoring

📝 Description

The /tmp directory has the wrong permissions when deployed in EKS or Canonical k8s.

This PR sets the proper permissions for that directory for the k8s branch.

We need to protect the storage_attached event from WorkloadExecError exception, otherwise we see:

unit-rs-fix-0: 12:23:25 WARNING unit.rs-fix/0.logs-storage-attached   File "/var/lib/juju/agents/unit-rs-fix-0/charm/venv/lib/python3.12/site-packages/single_kernel_mongo/events/lifecycle.py", line 307, in on_storage_attached
unit-rs-fix-0: 12:23:25 WARNING unit.rs-fix/0.logs-storage-attached     self.dependent.prepare_storage()
unit-rs-fix-0: 12:23:25 WARNING unit.rs-fix/0.logs-storage-attached   File "/var/lib/juju/agents/unit-rs-fix-0/charm/venv/lib/python3.12/site-packages/single_kernel_mongo/managers/mongodb_operator.py", line 1082, in prepare_storage
unit-rs-fix-0: 12:23:25 WARNING unit.rs-fix/0.logs-storage-attached     self.workload.exec(["chmod", "1777", f"{self.workload.paths.tmp_path}"])
unit-rs-fix-0: 12:23:25 WARNING unit.rs-fix/0.logs-storage-attached   File "/var/lib/juju/agents/unit-rs-fix-0/charm/venv/lib/python3.12/site-packages/single_kernel_mongo/core/k8s_workload.py", line 176, in exec
unit-rs-fix-0: 12:23:25 WARNING unit.rs-fix/0.logs-storage-attached     raise WorkloadExecError(
unit-rs-fix-0: 12:23:25 WARNING unit.rs-fix/0.logs-storage-attached single_kernel_mongo.exceptions.WorkloadExecError: cmd failed (-1) - cmd=chmod 1777 /tmp, stdout=Pebble client can't connect to the socket., stderr=
unit-rs-fix-0: 12:23:25 ERROR juju.worker.uniter.operation hook "logs-storage-attached" (via hook dispatching script: dispatch) failed: exit status 1
unit-rs-fix-0: 12:23:25 INFO juju.worker.uniter awaiting error resolution for "storage-attached" hook
unit-rs-fix-0: 12:23:35 INFO juju.worker.uniter awaiting error resolution for "storage-attached" hook
unit-rs-fix-0: 12:23:36 INFO unit.rs-fix/0.juju-log charmlibs.rollingops._rollingops_manager:Etcd relation configured but no cluster_id yet. Using peer backend until cluster_id provided.
unit-rs-fix-0: 12:23:36 INFO unit.rs-fix/0.juju-log charm_refresh:This unit's workload container digest is not available from the Kubernetes API. Unable to save versions to app databag (for next refresh). Will retry next Juju event
unit-rs-fix-0: 12:23:37 INFO unit.rs-fix/0.juju-log root:Component MongoDBRoles.REPLICATION is not a shard, cannot be integrated to a config-server.
unit-rs-fix-0: 12:23:37 INFO unit.rs-fix/0.juju-log single_kernel_mongo.events.lifecycle:Not ready to start. MongoDB workload is not present yet.
unit-rs-fix-0: 12:23:37 ERROR unit.rs-fix/0.juju-log root:Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-rs-fix-0/charm/venv/lib/python3.12/site-packages/single_kernel_mongo/core/k8s_workload.py", line 148, in exec
    process = self.container.exec(
              ^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-rs-fix-0/charm/venv/lib/python3.12/site-packages/ops/model.py", line 3249, in exec
    return self._pebble.exec(
           ^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-rs-fix-0/charm/venv/lib/python3.12/site-packages/ops/pebble.py", line 3132, in exec
    resp = self._request('POST', '/v1/exec', body=body)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-rs-fix-0/charm/venv/lib/python3.12/site-packages/ops/pebble.py", line 2182, in _request
    response = self._request_raw(method, path, query, headers, data)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-rs-fix-0/charm/venv/lib/python3.12/site-packages/ops/pebble.py", line 2235, in _request_raw
    raise ConnectionError(
ops.pebble.ConnectionError: Could not connect to Pebble: socket not found at '/charm/containers/mongod/pebble.socket' (container restarted?)

In my tests, the charms it is able to recover but ideally we should not see the storage-attached hook fail.

Note

Small fix in execute_on_server , this function executed the command only in the leader unit.
This was removed, and now it executes the command on the unit passed as parameter.
This function was only used in test_check_max_tasks which iterated on every unit

🧪 Manual testing steps

Build and deploy on Canonical k8s

Manually deployed several times

🔬 Automated testing steps

• Added a UT
• Added a check in the integration tests on the permission for /tmp in microk8s we may not see the difference but it more relevant when we migrate to Canonical K8s

✅ Checklist

• My code follows the code style of this project.
• I have added or updated any relevant documentation.
• I have read the CONTRIBUTING document.
• I have added tests to cover my changes.
• All new and existing tests passed.