ES PVC Not deploying cleanly in quicklab

[csibbitt]

This is probably more of an issue with the default storageclass config in quicklab, because this works fine on minishift; but I wanted to capture it here so we can track it.

SAF will not deploy cleanly in quicklab because the ES PVC fails to bind.

[csibbitt@laptop deploy (master) (⎈ |sa-telemetry/openshift-internal-safperf002-lab-rdu2-cee-redhat-com:443/system:admin)]$ oc get pvc
NAME                                         STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS        AGE
elasticsearch-elasticsearch-cdm-bmr16sp3-1   Pending                                                                                            2m
prometheus-white-db-prometheus-white-0       Bound     pvc-c5f2b95b-ef59-11e9-945a-fa163e0af8aa   2Gi        RWO            glusterfs-storage   2m

[csibbitt@laptop deploy (master) (⎈ |sa-telemetry/openshift-internal-safperf002-lab-rdu2-cee-redhat-com:443/system:admin)]$ oc describe pvc elasticsearch-elasticsearch-cdm-bmr16sp3-1
Name:          elasticsearch-elasticsearch-cdm-bmr16sp3-1
Namespace:     sa-telemetry
StorageClass:
Status:        Pending
Volume:
Labels:        <none>
Annotations:   <none>
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
Events:
  Type    Reason         Age               From                         Message
  ----    ------         ----              ----                         -------
  Normal  FailedBinding  1m (x19 over 6m)  persistentvolume-controller  no persistent volumes available for this claim and no storage class is set
  Normal  FailedBinding  12s (x6 over 1m)  persistentvolume-controller  no persistent volumes available for this claim and no storage class is set

We should be able to work around this by setting an appropriate storage class for elasticsearch_storage_storage_class_name on the eleasticsearch resource. There is no global recommendation for this setting, and your cloud SHOULD take care of setting a default; but we should at least document the quicklab workaround somewhere.

[leifmadsen]

@csibbitt try just passing this instead of a full storageClassName. This will result in ephemeral storage (probably good enough for testing purposes).

apiVersion: logging.openshift.io/v1
kind: Elasticsearch
metadata:
  name: elasticsearch
  namespace: sa-telemetry
spec:
  managementState: Managed
  nodeSpec:
    image: [...]
    resources:
      limits:
        memory: 1Gi
      requests:
        memory: 512Mi
  nodes:
  - nodeCount: 1
    roles:
    - client
    - data
    - master
    storage: {}
  redundancyPolicy: ZeroRedundancy

[leifmadsen]

(once you've generated the manifests with ansible, just drop everything under the existing storage parameter and add {})

[csibbitt]

Finally got around to testing this

Broken

NAME                                            READY     STATUS              RESTARTS   AGE
cloud1-notify-smartgateway-1-bnsrt              0/1       CrashLoopBackOff   12         43m
elasticsearch-cdm-bed5aewa-1-754988cf8f-8z2tn   0/2       Pending            0          2m

NAME                                         STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS        AGE
elasticsearch-elasticsearch-cdm-bed5aewa-1   Pending                                                                                            12h

Fix!

$ oc delete elasticsearch --all
$ vim deploy/service-assurance/elasticsearch/elasticsearch.yaml  # storageClassName: "glusterfs-storage"
$ oc create -f deploy/service-assurance/elasticsearch/elasticsearch.yaml

NAME                                         STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS        AGE
elasticsearch-elasticsearch-cdm-updjx5gn-1   Bound     pvc-063245fc-07b3-11ea-b442-fa163e45c75d   2Gi        RWO            glusterfs-storage   26s

NAME                                            READY     STATUS              RESTARTS   AGE
cloud1-notify-smartgateway-1-bnsrt              1/1       Running   148        12h
elasticsearch-cdm-updjx5gn-1-7cd948f55f-bxn48   2/2       Running   6          39m

No doubt disabling the storage entirely also works, but this quick workaround proves that we should also be able to fix this at deploy time with -e elasticsearch_storage_storage_class_name=glusterfs-storage (or as appropriate for a customer deployment)

I think I'm just happy to just close this issue for now; but perhaps this belongs in the documentation. @leifmadsen, I didn't see anything about ES storage sizing in the docs (expected it near here: https://github.com/redhat-service-assurance/saf-documentation/blob/fea71509f613065204c0627d067f448baabbbcab/doc-Service-Assurance-Framework/modules/proc_generating-the-manifests-for-saf.adoc). Do you have any additional events related content in the wings, or should I go ahead and add this detail to the docs there?

[leifmadsen]

@csibbitt please add to the docs there! I agree we should have some sizing information. We have a task for this. https://trello.com/c/vS1lkKvb/779-add-scaling-performance-and-sizing-section-to-saf-documentation

I've added a small checklist with some possible areas to address in a section like that.

I think the only documentation change in that section, is that it's obviously not clear enough from my Tip that there are other options you can pass.

If you think it's warranted to specifically call out the ElasticSearch storage (or even a generic "storage" section right around that area) that makes sense to me. I'd keep it a 1 point task though and then I'm happy to have this issue resolved!

[leifmadsen]

@csibbitt I feel like we can close this issue now. I have provided a config that can be passed to quickstart.sh to start a full SAF in quicklab.

[csibbitt]

I'll test it out and will close if it WFM

[csibbitt]

I finally got a quicklab to provision and I confirmed this works. If you don't mind I'll keep it open until I can retest after infrawatch/service-telemetry-operator#27

[leifmadsen]

That'd be great. Thanks!

[leifmadsen]

Just realized this will require a change probably. I'll submit.