RFC: /v2/commerce/history

[alliepiper]

Summary

This RFC adds historical data as well as buy / sell quantity information to the Guild Wars 2 commerce API. While serving a complete transaction history for all items from the trading post is understandably impractical, a statistical summary of the transactions may be feasible.

Motivation

Historical data regarding trading post activity is is currently unavailable through the current API, and applications that wish to display this information in plots, etc. must currently scrape the existing commerce endpoints to build their own databases, which can only provide data from when the scrapes begin and can result in incomplete data due to network outages, server uptime, etc. This approach is also sensitive to sampling errors due to high frequency fluctuations in strongly demanded items. Adding summaries of this data to the API would allow these applications to make fewer calls against the existing APIs to update their local caches, and allow them to import data from before the services became active.

Additionally, the number of items actually bought and sold are invaluable pieces of information for various types of analysis, and this information cannot currently be obtained in any accurate fashion. The best way to approximate this information using the current API is to take the differences in supply and demand between scrapes. This approach is subject to inaccuracies from sampling artifacts and misinterprets buy/sell orders that have been removed as completed, rather than aborted, transactions.

Providing this information would add much greater depth to the market data available through the Guild Wars 2 API without incurring the storage and computational expenses associated with serving individual transactions.

Implementation

This approach would require hourly, daily, and monthly jobs to produce / condense the summaries.

The hourly job would iterate through the transaction logs to produce a summary of the previous hour's market activity, while the daily and monthly jobs would condense the summaries older than one month or one year, respectively.

The summaries will include the following statistics, calculated as described:

Buy and sell price minima, maxima, and means for each item. In pseudocode,

// Calculate price statistics for the last hour.
// If no transactions, just use current listing/order price.
minPrice = lastHourTransactionLog.first().price
maxPrice = lastHourTransactionLog.first().price
meanPrice = 0
for transaction in lastHourTransactionLog:
  minPrice = min(transaction.price, minPrice)
  maxPrice = max(transaction.price, maxPrice)
  meanPrice += transaction.price * transaction.millisecondsToNextTransaction()
meanPrice /= 1000 * 60 * 60 // ms per hour

The same pattern would be used for supply and demand quantites. Buy and sell quantities are simply summations of all transaction quantities.

The hourly and daily summaries can be merged into daily and weekly summaries by simply taking the min/max of the individual mins/maxes, averaging the temporal means, and summing the buy/sell quantities.

[lye]

I think the response format looks fine. I'd also add a variance field to the buy_price/sell_price/demand/supply objects. I think that'll give additional insight into the aggregated distributions.

The big blocker on this endpoint would be setting up a proper time-series database and piping real-time transaction records through it. I'll ask around on Monday to see what approaches we might have and how feasible they are to implement, but it's probably going to be a long while before everything's in place to even begin implementing this.

[alliepiper]

I wanted to add deviation info originally, but I'm rusty enough at stats that I couldn't remember how easy it would be to condense those ;)

It would be a large project, for sure, but a very valuable one. Thanks for looking into it! 👍