Publishing a JSON Schema for Developers

[Ruhrpottpatriot]

As developers we face the problem, that we don't know when the contents of an api request have changed. The v2/items node for example are very detailed and if this would change for whatever reason we wouldn't immediately know what has changed. As an example: a few months ago an additional type of consumables was added and surely broke code in strongly typed languages.

Of course ANet does not do this out of spite for use community developers, but things can happen that were not intended. With the release of HoT imminent we api developers face this problem again. What if the changelog missed menntioning a small enum change? The worst case would be to go though many items and look for ourselves what was changed. This introduces unecessary work, which can easily be avoided.

JSON-Schema was published for this use case. It offers us developers an easy and consistent way to check the results of an api query. If ANet publishes a JSON-Schema for each endpoint validating the result would be a breeze.

Example (in C# and with the JSON.NET Schema library):

public class JsonValidator
{
    /// The json retrieved from the api is passed to this method,
    /// as well as the schema for this particular node.     
    public ValidateResponse Valiate(ValidateRequest request)
    {
        // load schema
        JSchema schema = JSchema.Parse(request.Schema);
        JToken json = JToken.Parse(request.Json);

        // validate json
        IList<ValidationError> errors;
        bool valid = json.IsValid(schema, out errors);

        // return error messages and line info to the browser
        return new ValidateResponse
        {
            Valid = valid,
            Errors = errors
        };
    }
}

public class ValidateRequest
{
    public string Json { get; set; }
    public string Schema { get; set; }
}

public class ValidateResponse
{
    public bool Valid { get; set; }
    public IList<ValidationError> Errors { get; set; }
}

While developers could validate the json returned by checking the wiki, which already offers a rudimentary schema, after each patch this is tedious and tiresome.
Of course not every language might have a validation library as powerful as JSON.NET Schema, however the possibility of validating the response against a formalized schema alone makes this even more useful.

So what am I proposing in this pull request:

1. Formalize the response data by introducing a JSON-Schema for each node.
2. Add a new node called v2/schemas which provides the users with an array of all schemas (i.e. ["schemas_root", "build", "items", "items_detail", ...]). Furthermore a child node called v2/schemas"{schema name} will give the user the schema for the node with that particular name.

This PR now only contains the schema for the v27schema root node and the v2/build node. Further will be added later on, if the discussion proves fruitful.

[Ruhrpottpatriot]

Here are two examples created by using the JSON-Schema validator:
The first is a passing validation, against the build schema:

The second is a failing example because an additional property is present in the data, that is not present in the schema

[lye]

Hmm, that's actually not a bad idea. TBH I don't like using JSON schema for runtime data validation (I am not a good programmer), but it could be integrated into my current integration test suite and shared. That'd be lovely.

[Ruhrpottpatriot]

The most widespread languages have a schema validation library, so in many cases the user needn't to write a validation for himself. In my eyes it really takes off much work.

I'm currently working on a more complex example which represents the colors endpoint. I hope to share it tomorrow.

[Ruhrpottpatriot]

Commit 931b82a introduces a more complex example to show how json schema would deal with repetitive data.

I might add, that I'm totally ok with hard coding the schema and not generating them on the fly. This could actually be faster, but more error prone.

[lye]

hard coding the schema and not generating them on the fly

That's almost certainly how we'd do it. There's no way we'd be able to generate them on the fly -- the backend is a massive mismash of C++ and Javascript across multiple completely decoupled backend components.

[Ruhrpottpatriot]

the backend is a massive mismash of C++ and Javascript across multiple completely decoupled backend components

Ah ok. That explains a lot. Although JS apparently has a good library for JSON schema.

I'll upload a couple more schemas for existing nodes in the next few days.

[Ruhrpottpatriot]

I did some consolidation work, merging separate schemas into one and added a new schema for v2/files.

[SamHurne]

👍 for this. Representing the schema with actual data structures so that we can have our own tests to validate endpoints (to know when something has changed and what changed) would be great.

[Ruhrpottpatriot]

With this commits the misc and trading post APIs are covered as far as Wiki information is concerned.

[Ruhrpottpatriot]

I just added a whole bunch of new schemas to the branch. The commits cover the whole authenticated endpoint. The Items endpoint is up next.

[lye]

Sweet, thanks for working on this. I haven't had a chance to integrate these into my unit tests yet, but rest assured it's going to happen at some point.

[Ruhrpottpatriot]

If you integrate them and somehow make the public (via a schemas endpoint), could you tell me the url in advance, then I'll update the schemas with a proper Id.

I'll probably update the current ones before uploading new ones, since working with ASP.NET 5 I have found out some great things about these schemas.

[lye]

I'm probably not going to put 'em on an endpoint*, but rather put them into the github repo (and keep it sync'd with the copies used by my unit tests). I've got half of the ones you've posted integrated already (basically all of the unauthenticated ones); just need to finish the other half then I'll merge.

* Let me know if there's a compelling reason to put them on an endpoint. It seems kinda strange to me.

[sliekens]

There are code generation tools out there that can generate an object model from schema files. It would be convenient for that purpose if the schemas had a permalink.

Example: http://www.jsonschema2pojo.org/

[Ruhrpottpatriot]

@lye There is no compelling reason to put them on an endoint. As long as they have a web acessible URL it is very much fine. The only reason to put them on a (static) endpoint (maybe generated from the git repository) would be to have unified access to them via the api, as in predictability for developers.

@StevenLiekens Any file in the Repository has a permalink via the RAW view of the file (at least on GitHub).

I'll try to get some more into this PR the next few days when I finish my Go programming and logic tasky for university. So stay tuned.

[Artanis]

@Ruhrpottpatriot, It would be better to avoid hotlinking the raw view directly. That view is not a direct link to a static asset, and needs to be rendered by the Rails app that powers Github. Further, these views are served as text/plain with type sniffing disabled (see the blog post Heads up: nosniff header support coming to Chrome and Firefox for more information).

Instead, the schema could be published with Github Pages. Truly static files would then be available at a https://arenanet.github.io/api-cdi.

[Ruhrpottpatriot]

Ah, didn't know about the nosniff thing.
But then again, a github page would probably be more work than a static API Endpoint