Add metrics endpoint for prometheus monitoring

[cpeel]

Add a very lightweight endpoint for monitoring with prometheus. This uses the default /metrics/ endpoint used by prometheus exporters.

This PR renames the mis-named ConnectionPool.pool variable to ConnectionPool.engine since that's actually what it contains. This variables does not appear to be used anywhere outside of the class that I can tell, except for @zachjesus's incoming PR #232

            self._fts = FullTextSearch(cherrypy.engine.pool.pool)

Example output:

autocat3_total_threads 24
autocat3_memory_kb_self 75436
autocat3_memory_kb_children 0
autocat3_httpserver_socket_queue_size 10
autocat3_httpserver_accepted_queue_size -1
autocat3_httpserver_pool_minsize 20
autocat3_httpserver_pool_maxsize 20
autocat3_httpserver_pool_available 19
autocat3_httpserver_pool_size 20
autocat3_httpserver_pool_used 1
autocat3_httpserver_pool_queued 0
autocat3_sqlalchemy_pool_maxsize 20
autocat3_sqlalchemy_pool_available 1
autocat3_sqlalchemy_pool_used 0
autocat3_sqlalchemy_pool_overflow -19

[eshellman]

the /diagnostics/ endpoint returns some of the same information:

  "sessions": 2,
  "sessions_storage": 1643,
  "allocated_blocks": 491158,
  "rusage_self": [1.981567, 1.009967, 79282176, 0, 0, 0, 62621, 9367, 0, 0, 0, 80, 68, 0, 11939, 9679],
  "rusage_children": [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
  "thread_info": [
    "MainThread",
    "HTTPServer Thread-1",
    "CP Server Thread-2",
    "CP Server Thread-3",
    "CP Server Thread-4",
    "CP Server Thread-5",
    "CP Server Thread-6",
    "CP Server Thread-7",
    "CP Server Thread-8",
    "CP Server Thread-9",
    "CP Server Thread-10",
    "CP Server Thread-11",
    "CP Server Thread-12",
    "CP Server Thread-13",
    "CP Server Thread-14",
    "CP Server Thread-15",
    "CP Server Thread-16",
    "CP Server Thread-17",
    "CP Server Thread-18",
    "CP Server Thread-19",
    "CP Server Thread-20",
    "CP Server Thread-21",
    "timer",
    "Session cleanup"
  ]
}```

[eshellman]

do we need both?

[eshellman]

we were looking at the growth of sessions back then - every hit from a bot would result in a session, and there was a bug in CherryPy in the session management code that we helped to fix.

[cpeel]

We either need both or we just need /metrics/ because the /diagnostics/:

1. isn't in a form that prometheus can scrape
2. is "heavy" in that computing the size of RamSession.cache is, relatively speaking, computationally intensive and the endpoint takes around a second to return

[eshellman]

let's remove diagnostics, and replace its documentation with that for /metrics/

[cpeel]

let's remove diagnostics, and replace its documentation with that for /metrics/

Sounds good. I can get to that on Monday.

[cpeel]

New /metrics/ endpoint updated to include the number of sessions. And a second commit to remove the /diagnostics/ endpoint and update the docs to use /metrics/ instead.

[eshellman]

could you avoid force pushing changes to a branch ready for review? It's easier to review if I can see the incremental changes. and not have to force push myself.

[cpeel]

could you avoid force pushing changes to a branch ready for review? It's easier to review if I can see the incremental changes. and not have to force push myself.

I can as long as you let me rebase and selectively squash commits before you merge a PR together so that the commits are coherant.