constraints of input datasets for each data-mining algorithm

[HimmelStein]

when a user select a dataset, and moves on to the data-mining service. Indigo shall only display data-mining algorithms which can be applied for the selected dataset.

so, please describe constraints of input datasets for your developed data-mining algorithm (send me through email before this Thursday).

[wk0206]

@larjohn as described in mail, I will add the constraints of each data-mining algorithm in dam.json.
So that when you visit dam in the first time, you can get the list.
Now I am focus on Timeseries and using it as sample:

Facts: at least one time dimension with years as values, and three or more years available
Aggregates: at least one time dimension as drilldown

I changed the dam.json to following format,

• add "conditions" as the same level as "aggregate"

• add "dimension","numberRestriction","formatRestriction","description" as value

• all value use String format

• "three or more years available" becomes "3+" in String.

> "time_series": {
>     "configurations": {
>       "aggregate": {
>         "inputs": {
>         },
>         "outputs": {
>         },
>         "prompt": XX,
>         "method": XX,
>         "endpoint": XX,
>         "name": "aggregate",
>         "title": "Timeseries of aggregated fiscal data"
>       },
>       "conditions": {
>         "Facts": {
>           "dimension": "year",
>           "numberRestriction": "3+",
>           "formatRestriction": "",
>           "description":"at least one time dimension with years as values, and three or more years available"
>         },
>         "Aggregates": {
>           "dimension": "time",
>           "numberRestriction": "",
>           "formatRestriction": "drilldown",
>           "description":"at least one time dimension as drilldown"
>         }
>       }
>     },
>     "name": "time_series",
>     "title": "Time Series",
>     "description": XX
>   }

The difficult part is how to understand the "Semantics meaning" of " time dimension with years as values", should us only check the "dimension title" to make sure it has "year" inside, or we have to focus on the value, guarantee the regex filter like "^(19|20)\d{2}$".

Actually , how to write/read this condition, we have to listen more to your opinion.
If you prefer some other format, such as put the condition as the same level as "input" or "endpoint", it is OK for me too.

[larjohn]

@wk0206 time dimensions should be found in the package query, with datetime dimension type:

{
  "model": {
    "dimensions": {
      "global__functionalClassification__78f10": {
        "dimensionType": "classification"
      },
      "global__economicClassification__569a2": {
        "dimensionType": "classification"
      },
      "global__budgetPhase__afd93": {
        "dimensionType": "classification"
      },
      "global__administrativeClassification__854d0": {
        "dimensionType": "classification"
      },
      "global__classification__9ddd4": {
        "dimensionType": "classification"
      },
      "global__currency__1a842": {
        "dimensionType": "classification"
      },
      "global__fiscalPeriod__28951": {
        "dimensionType": "datetime"
      },
      "global__operationCharacter__0c040": {
        "dimensionType": "classification"
      },
      "global__organization__0eba1": {
        "dimensionType": "location"
      },
      "global__administrativeClassification__70a05": {
        "dimensionType": "classification"
      },
      "global__administrativeClassification__f9d35": {
        "dimensionType": "classification"
      },
      "global__administrativeClassification__38bee": {
        "dimensionType": "classification"
      },
      "global__administrativeClassification__13968": {
        "dimensionType": "classification"
      },
      "global__date__99de8": {
        "dimensionType": "classification"
      }
    },
    "measures": {
      "global__amount__0397f": {
        "currency": "EUR",
        "title": "Global amount"
      }
    }
  },
  "countryCode": null,
  "cityCode": null,
  "name": "global",
  "title": "Global dataset"
}

Regarding the constraints, please put them inside the configuration, as each configuration (usually facts vs aggregates) could have different starting points, requiring different thins, so they can't have the same constraints every time.

Also note, that most of the constraints might be better to be applied at DAM level, so that instead of requiring the algorithms with a generic request, indigo should request per dataset and only get the datasets that apply. A good strategy would be to cache the constraints analysis to avoid overhead.

[HimmelStein]

@larjohn why there is a random tail at each global key? e.g. what is the function of "__0eba1" for "global__organization__0eba1"?

[HimmelStein]

@larjohn let us take the data-mining function 'time series' as the example. The applicable datasets must have a dimension 'fiscalPeriod' and there shall be 3 different values in the dimension 'fiscalPeriod'.

"time_series": {
"configurations": {
"aggregate": {
"inputs": {
},
"outputs": {
},
"endpoint": <..>,
"name": "aggregate",
"title": "Timeseries of aggregated fiscal data"
},
"conditions": {
"Facts": {
"dimension": "datetime",
"numberRestriction": "3+",
"formatRestriction": "",
"description":"at least one dimension of type "datetime" with three or more different values available"
},
"Aggregates": {
"dimension": "datetime",
"numberRestriction": "",
"formatRestriction": "drilldown",
"description":"at least one time dimension as drilldown"
}
}
},
"name": "time_series",
"title": "Time Series"
}