Add new frame: JSON

[jordansissel]

Frame header 1J

Proposed frame body:

• 32bit unsigned sequence number
• 32bit unsigned length in bytes (representing length of the JSON content)
• the JSON content exactly length in bytes.

[jordansissel]

If we allow the full logstash json schema here, a client should ship fields within @metadata and Logstash could make use of that for routing/filtering decisions. This would be useful for Packetbeat, for example, to say what kind of document is being sent through (packet dissection or topology update).

[urso]

Adapting the logstash input plugin I wonder if this change is really required. Don't want to drop the current frame encoding format for backwards compatibility yet.

Instead of having a json decoder, senders might send only one field with key = "line" with the full json encoded content in the value. Thusly users MUST configure the json codec for the logstash plugin.

[urso]

For example possible implementation for integrating json:
https://github.com/urso/logstash-input-lumberjack/tree/feature/json-input

[urso]

So the only advantage of inserting a json data frames is: we could keep similar semantics for line field in input plugin.

That is if received json has a 'line' field, this field will be analyzed by the input plugin its configured codec => users updating from logstash-forwarder to filebeat won't have to change their configuration. Otherwise users have to configure 'codec = json' and whatever codec different from 'plain' has been used must be applied in the filtering phase.

conclusion:

• adding json data frame users switching from LSF to filebeat will have update the input plugin only
• not adding json data frame users switching from LSF to filebeat have to update their logstash configurations.