Application for migrating REST endpoints into ElasticSearch.
Logstash HTTP input plugin can only be used to fetch a static URL. What about cases where the URL needs to be updated between requests?
This application provides the mechanisms necessary to crawl a web service. The context from the previous request is passed to the logic for creating the next URL.
Add new configuration object to configs array
{
rest: {
/**
* String identify file for storing REST context.
* File is used to provide initial context upon process start.
* File is updated after each successful http-request/es-insert cycle.
*/
contextFilename: '',
/**
* Sleep period when REST call returns no data or an error occurs
*/
sleep: 300000,
/**
* Create the URL for next GET request.
* Use the context from the previous response to craft the URL address
*
* @method next
* @param prevContext {Object} Previous response context containing the min and max values for each property defined in watchlist.
* @return {string} next URL to fetch
next: function(prevContext) {},
/**
* Array of strings containing property names that will be placed in the prevContext object utilized by the next function.
*/
watchlist: []
},
es: {
host: 'ES ip address',
index: 'ES index',
type: 'ES index type'
},
/**
* Some REST endpoints do not return an Array of rows.
* This function provides the ability to traverse the response and return an Array of rows.
*
* @method extractRows
* @param respBody {Object} Response body
* @return {Array}
*/
extractRows: function(respBody) {},
/**
* Transform REST document into ElasticSearch document
*
* @method transfrom
* @param doc {Object} row returned from REST web service array.
* @param callback {function} Function to call when transformation is complete.
*/
transform: function(doc, callback) {}
}