More fine-grained statuses for MicroCluster

[troyanov]

MicroCluster requires all nodes in a cluster to have the same schema/API version and if update will be triggered when one of the nodes is offline, the update will never finish.

I was thinking if we can prevent such update from happening by adding some guard rails (deb preinst or snap pre-refresh) before doing an update? But it seems to be tricky.

1. member1 checks cluster status before upgrade, if all online, then it proceeds

+---------+-------------+
|  NAME   |   STATUS    |
+---------+-------------+
| member1 | ONLINE      |
+---------+-------------+
| member2 | ONLINE      |
+---------+-------------+
| member3 | ONLINE      |
+---------+-------------+

2. member1 is updated

cluster status from member1

+---------+---------------+
|  NAME   |   STATUS      |
+---------+---------------+
| member1 | UPGRADING     |
+---------+---------------+
| member2 | NEEDS UPGRADE |
+---------+---------------+
| member3 | NEEDS UPGRADE |
+---------+---------------+

cluster status from member2

+---------+-------------+
|  NAME   |   STATUS    |
+---------+-------------+
| member1 | UNREACHABLE |
+---------+-------------+
| member2 | ONLINE      |
+---------+-------------+
| member3 | ONLINE      |
+---------+-------------+

Any reason the heartbeat endpoint couldn't be made to return a status indicating the member1 is online but waiting for upgrade instead of unreachable?

P.S. I am using this as a current workaround:

var healthCmd = rest.Endpoint{
    Path:              "health",
    AllowedBeforeInit: true,
    Post:              rest.EndpointAction{Handler: cmdPost, AllowUntrusted: true},
}

func cmdPost(state state.State, r *http.Request) response.Response {
  return response.SyncResponse(true, state.Database().Status())
}