Test case creation via wk CLI

[mikko-n-esystems]

Current way to create test cases manually and exclusively via Workato UI, now combined with possibilities to automate recipe creation/editing with AI creates a big mismatch for capabilities. Harnessing AI to create comprehensive test sets for recipes would be awesome and add some quality / robustness stops to CI/CD.

Could it be possible to add a way to create/edit test cases for recipes via API / CLI tooling, too? I understand this is not a feature offered by current developer API and wk cli is first intended to have parity with current api, but it's still something I think would benefit the platform as a part of enterprise infrastructure done professionally.

[zaynelt]

Thanks for raising this @mikko-n-esystems! I was just looking at this area yesterday & wondering about it. What format(s) would you want to be able to use when you are working with test cases? How large are your test case data sets, typically?

[mikko-n-esystems]

Hm. Initially I was thinking there really should be a way to have CRUD operations for test cases in API / other tooling in addition to just running tests for a recipe. Defining tests as json, as they currently are when exporting a package is a good starting point. I would kind of expect there's similar support in AI skills to create and modify test cases as there are with actual recipes. Currently adding tests to greenfield development is really cumbersome, requires lots of manual steps, exporting and importing and as suggested, has a productivity mismatch when compared to creating recipes.

When exporting recipe from UI, I can include all my test cases to the export, but (re)creating or adding to these test case json definitions is something current tooling doesn't really support. If I clone and modify a previously created test case manually and add those cloned tests to package, I can add arbitrary number of test cases to recipe via importing modified asset zip. This can ofc be at least partially automated, but tooling does not support actually creating those tests. Json schema for defining tests isn't really well documented thing either.

Our test case data set sizes depend quite a lot on the recipe itself, eg. what needs to be mocked and how many possible branches the recipe being tested has etc. Ideally we'd like test all the execution branches, but in reality, having a comprehensive enough regression test set for key parts / recipes is good enough.