#766: Added SHACL-AF rules to SHACL-SPARQL

[HolgerKnublauch]

Closes #766

• See this document rendered online here

[HolgerKnublauch]

This is ready for review now. One open issue that the new section mentions is the handling of iteration. Currently we only specify a single iteration. In practice, it is trivial to do a fix point iteration that keeps running. The main concern there is when new bnodes or infinite amounts of triples are created continuously. This could be defined as an undefined scenario, while in practice implementations may just set a max iteration count and stop with a warning when that happens.

Practically speaking, it would be great to have SOME version of this on the main branch and then make smaller increments from there, with dedicated follow-up PRs. The text that is here right now is essentially what was on SHACL-AF for years.

[liviorobaldo]

Hi @HolgerKnublauch, yes, that's exactly what I was pointing out in my email. I was suggesting that we briefly mention here that "SPARQL-based rules may lead to infinite loops when repeatedly executed until saturation, and may also exhibit non-deterministic behaviour due to operators such as NOT EXISTS".

To avoid infinite loops, implementations can impose a maximum iteration count. Avoiding non-determinism, however, is the user's responsibility, who must use sh:order to that end.

Nevertheless, based on my experience with use cases I worked on, I would say that relying on sh:order is not only verbose and tedious, but also risky and error-prone. Users may run a set of rules on one implementation, observe that everything works as expected, and conclude that there is no need to specify sh:order. They then run the same rules on another implementation and obtain different results... because the need for an explicit order was always there: it was simply "masked" by the fact that the first implementation happened to execute the rules in the order the user expected. In other words, users can easily overlook the need to establish an ordering that is portable across implementations.

Therefore, the fact that rule execution yields the same results across implementations must be actually formally demonstrated. This is precisely what I had to do in my paper on the Time Ontology, in order to get it published.

SHACL 1.2 rules avoid these issues altogether by defining stratification and run-once rules.

I think we should explain this briefly in this section. What do you both (and anyone else following the discussion) think?

[afs]

The name "SHACL Transformation Rules" can reflect that the control is with the user - and include the following changes:

• include INSERT-DELETE-WHERE
• pre-binding is based on a mechanism that is engine neutral, then all standards-based triple stores can be used.

[HolgerKnublauch]

@afs SPIN did support INSERT/DELETE. INSERT doesn't add value over CONSTRUCT, so this is all about DELETE. I am not aware that support for deletes was used or requested much. In practice such use cases are better handled using scripting such as our JavaScript integration where updates are determined dynamically.

FWIW since your current draft is essentially a subset of SPARQL, there could be a profile of SHACL-SPARQL Rules that is limited to that same subset.

[HolgerKnublauch]

I think we should explain this briefly in this section. What do you both (and anyone else following the discussion) think?

Yes we should explain and highlight these problems in the document. This language has a design philosophy similar to imperative languages. For example, anyone can easily do endless loops using while(true); but this doesn't render these programming languages useless. The responsibility is shifted to the developer and make sure these scenarios are avoided. The rule language itself doesn't prevent them, and people could easily do infinite loops using

CONSTRUCT {
    ?b rdf:type rdfs:Resource .
}
WHERE {
    BIND (BNODE() AS ?b) .
}

If you have experience and best practices from the other rules document then we should incorporate that.

Practically speaking, I suggest we try to get this first version into the main branch soon and create individual follow-up tickets to address the things that we want to improve, esp the handling of iteration. It will become easier to make incremental changes when the diff becomes smaller and focused.

[afs]

Suggested change

	<h3>SPARQL Rules</h3>
	<h3>SPARQL CONSTRUCT Rules</h3>

to focus on what it does
or

Suggested change

	<h3>SPARQL Rules</h3>
	<h3>SHACL SPARQL Rules</h3>

to be clear it is part of the framework.

[HolgerKnublauch]

There is much mix-up potential here. SHACL-AF rules have used the term SPARQL Rules for many years, even declares sh:SPARQLRule and actually uses SPARQL and not RULE...WHERE like the other draft. So at least within the SHACL document SHACL-SPARQL Rules is equivalent to SPARQL Rules. We also don't say "SHACL Shapes" everywhere. SHACL 1.2 Rules should become RDF 1.2 Rules (or some other term, but neither SHACL nor SPARQL).

[liviorobaldo]

I am not aware that support for deletes was used or requested much. In practice such use cases are better handled using scripting such as our JavaScript integration where updates are determined dynamically.

@HolgerKnublauch, I think DELETE would be extremely useful for removing redundant triples, as discussed in the past. Of course, the same can already be achieved with a standard DELETE WHERE SPARQL query, JavaScript, or other mechanisms, so I don't see this as a blocker. However, extending SHACL-SPARQL and/or SHACL 1.2 to support it would be a worthwhile enhancement.

Redundant triples are often generated when intermediate results are needed to compute the final results. Once the final results have been obtained, however, those intermediate triples are no longer needed and can safely be removed, even though they remain logically consistent with the final graph. This is the motivation behind supporting DELETE WHERE rules. I have already encountered a couple of real-world use cases where this functionality would have been beneficial.

This is something that could still be added in the future, together with aggregates and the other "future work" ideas mentioned by @afs in one of our recent meetings. The main challenge with DELETE is that it naturally exacerbates the risk of infinite loops. However, making all DELETE rules run once by definition, and placing them in the last stratum by default, could be a reasonable compromise.

[HolgerKnublauch]

@liviorobaldo Ok I see what you mean, and also had use cases for temporary triples in the past, for example to remember which source resource maps to which target resource, and whether during OWL2SHACL conversion an OWL axiom was already covered or not.

With RDF 1.2 reification we now have additional ways of marking such triples for later clean up. For example, we could introduce a dedicated property such as <<( A B C )>> sh:tempTriple true, just like we may want to add a generic provenance mechanism that can be used to annotate which rules have produced which triples.

This does sound like a separate issue though, and should become its own thread IMHO.

[liviorobaldo]

For example, anyone can easily do endless loops using while(true);

Indeed, I also often use while (true) as an example to illustrate that avoiding infinite loops is ultimately the responsibility of the programmer (or user).

If you have experience and best practices from the other rules document then we should incorporate that.

Well, I think run-once rules are the best option. If one needs a fixed number of iterations, e.g., two or three, one can simply define two or three run-once rules explicitly, with each rule triggering on the results produced by the previous one. In real-world use cases, however, one never needs an infinite number of iterations, even though an infinite sequence of entailments may exist in theoretical or abstract terms.

As for other "best practices", as I said, one should formally demonstrate that infinite loops cannot occur. The way I do this (and, in fact, the only way that has come to mind) is to use NOT EXISTS clauses that prevent rules from firing once they have already produced their outputs. In effect, however, this is just a SHACL-SPARQL workaround for emulating run-once rules, because you are essentially saying: "run this rule only if its outputs do not already exist in the graph".

Practically speaking, I suggest we try to get this first version into the main branch soon and create individual follow-up tickets to address the things that we want to improve, esp the handling of iteration. It will become easier to make incremental changes when the diff becomes smaller and focused.

Ok, I was about to create a PR to incorporate the points discussed above, but from what I understand, you'd prefer that we finish this PR before opening any new ones. So I'll wait.

[liviorobaldo]

This does sound like a separate issue though, and should become its own thread IMHO.

@HolgerKnublauch, I completely agree, including with the point about reification. Perhaps for the time being @afs could just add it to the list of future work (assuming there is one, that's what I understood).

[liviorobaldo]

Does this imply that other types of rules may be defined elsewhere? Is this intended to accommodate future rule types (e.g., JavaScript-based rules) that are not currently defined? If so, I think we should add a sentence clarifying this here. When I first read this, my immediate reaction was: "Where can I find the other rule types?"

[mgberg]

And if the intent is to support other types of rules, that would presumably have an impact on the above discussion regarding the naming of this feature.

[HolgerKnublauch]

My assumption is that the current SHACL 1.2 Rules draft will be renamed. It has nothing to do with SHACL or shapes, and would be better named RDF Rules. With that, the name SHACL Rules is clearly and uniquely defined here.

[HolgerKnublauch]

And yes, the intent here is that there can and will be other rule types. The bulk of the syntax and the main algorithm is independent from SPARQL. Therefore the general language here can be named "SHACL Rules", while the specific SPARQL-based rule type can be either SHACL-SPARQL Rules or (shorter) SPARQL Rules (because we are in the context of the SHACL document this should be clear. We also don't spell out "SHACL Shape" everywhere even though ShEx shapes exist too).

[liviorobaldo]

And yes, the intent here is that there can and will be other rule types.

@HolgerKnublauch, ok, so why couldn't these other types of rules also include the SHACL 1.2 rules (or whatever we want to call them: "RDF rules", "stratified run-once rules", etc.)?

I think this could accommodate everyone's wishes and needs.

SPARQL-based rules are needed for backward compatibility, as well as for the other reasons discussed in issue #766. The SHACL 1.2 rules, on the other hand, are safer because stratification and run-once semantics relieve users of the burden of formally demonstrating that rule execution is both deterministic and terminating.

sh:condition allows users to specify shapes that are validated before rules are executed, while the shapes embedding the rules are validated after rule execution. This addresses the question of execution order (i.e., whether shapes should be validated before or after rules).

The only feature that I think is still missing is a mechanism for associating a set of rules with a set of shapes. In the current design, a set of rules can only be associated with a single shape. As a result, if the same rules need to be executed for multiple shapes, they must be duplicated. An implementation can, of course, internally group the rules and execute them only once before validating all associated shapes. However, I think this grouping should also exist at the logical level, meaning that SHACL should provide a mechanism for expressing such associations between sets of rules and sets of shapes explicitly.

[liviorobaldo]

Yes, I am also a bit confused about the distinction between owl:imports and sh:imports. I have just opened a new issue on this topic. I think we should clarify our position on this matter.

[HolgerKnublauch]

On rule sets, I had suggested elsewhere that the named graphs should count as rule sets by default. This is how SHACL-AF rules worked. So any rule in the shapes graph are in the default rule set. Whether another type of rule set is needed is another question but there is no need for an explicit triple (and users will likely forget it anyway).

[liviorobaldo]

Hi @HolgerKnublauch,

I'm not sure I fully understand how this would work.

At the moment, sh:rule links a single shape to a single rule. A shape can be linked to multiple rules, which together constitute the shape's "set of rules".

Are you suggesting that, instead of linking individual rules, sh:rule should link to a named graph, for example:

ex:Graph1 {
    ...
}

If so, I'm not sure what the "..." inside the named graph would contain, i.e., which triples would define the rule set.

By the way, I was also thinking that sh:rule should have maxCount = 1 when it links a shape to a set of rules (regardless of whether the set is represented by a sh:RuleSet or by a named graph). In other words, I do not think a shape should be associated with multiple rule sets (possibly even of different types), since this would allow the rule sets to interact with one another, making the inference process much harder (if not impossible) to understand and manage.

Finally, what are your thoughts on introducing sets of shapes?

[HolgerKnublauch]

There is no need to do anything here. SHACL is based on the data graph and the shapes graph. It doesn't care how these are constructed, but the shapes graph is the named graph containing the shapes and their rules. I see no need to complicate this further. The rule set is simply defined by the triples that exist in the shapes graph. An application could pre-process the shapes graph to hide certain triples, or add sh:deactivated to certain rules to define subsets.

sh:rule links a shape to one or more rules, so there cannot be a sh:maxCount 1. For example, a Rectangle may have one rule to compute the area, and another to compute the circumference.

Sets of shapes are equally unnecessary to me. They are just the shapes graph.

[liviorobaldo]

The sh:maxCount 1 constraint on sh:rule was only proposed under the assumption that sh:rule would link a shape to a set of rules. If the objects of sh:rule are restricted to individual rules, then I agree that we should not impose sh:maxCount 1.

In any case, if we assume that the (single) set of shapes and the (single) set of rules are simply those present in the graph, I think we should at least require that all rules are of the same type. This was my main concern in the previous comment. We should not allow SHACL-SPARQL rules and SHACL 1.2 rules to be mixed in the same graph. A graph should contain either only SHACL-SPARQL rules or only SHACL 1.2 rules; otherwise, the inference process would become too difficult to define and implement, if not impossible.

[afs]

Perhaps for the time being @afs could just add it to the list of future work (assuming there is one, that's what I understood).

AFAIK the process is that the WG collects future work as issues on this repo. It will at least be recorded if nothing else.

[mgberg]

And if the intent is to support other types of rules, that would presumably have an impact on the above discussion regarding the naming of this feature.

[mgberg]

It may be clearer for implementers if it was additionally explicitly stated if/when inferred triples can be seen.

[HolgerKnublauch]

This does sound like a separate issue though, and should become its own thread IMHO.

@HolgerKnublauch, I completely agree, including with the point about reification. Perhaps for the time being @afs could just add it to the list of future work (assuming there is one, that's what I understood).

Opened #995 to outsource this topic for now.

[afs]

It would seem quite natural as a new document and call it "SHACL 1.2 Rules Framework". The framework part is important.
It will likely grow as work happens on the framework execution algorithm.

That said:

"6. SHACL Rules Framework"
This isn't just about one rule type.

6.1 to 6.4 unchanged.

"6.5 SHACL SPARQL Rules"
(and informally after that within the document "SPARQL Rules")

Other options:

"6.5 SHACL SPARQL CONSTRUCT Rules"
It isn't using SPARQL Update. Say which bit of SPARQL it is based one.

or in keeping with the other parts of the SHACL SPARQL Extensions document:
"6.5 SPARQL-based Inference Rules"

[HolgerKnublauch]

The most natural thing, and original plan, is to have a single unified document called SHACL 1.2 Rules that is actually about SHACL Rules.