Fix fastrun bugs, resolve TODOs and update the documentation

README.md

@@ -75,6 +75,11 @@ wikibaseintegrator~=0.11.3
     - [Modify an existing item](#modify-an-existing-item)
     - [A bot for Mass Import](#a-bot-for-mass-import)
 - [Examples (in "fast run" mode)](#examples-in-fast-run-mode)
+    - [The base filter](#the-base-filter)
+    - [Checking if a write is required](#checking-if-a-write-is-required)
+    - [Options](#options)
+    - [Checking labels, descriptions and aliases](#checking-labels-descriptions-and-aliases)
+    - [Limitations](#limitations)
 - [Debugging](#debugging)
 
 # WikibaseIntegrator / WikidataIntegrator #
@@ -91,7 +96,8 @@ The main differences between these two libraries are :
 * Add OAuth 2.0 login method
 * Add logging module support
 
-But WikibaseIntegrator lack the "fastrun" functionality implemented in WikidataIntegrator.
+WikibaseIntegrator also provides a rewritten version of the "fast run" mode of WikidataIntegrator, which avoids
+unnecessary API writes (see [Examples (in "fast run" mode)](#examples-in-fast-run-mode)).
 
 # Documentation #
 
@@ -799,42 +805,61 @@ for entrez_id, ensembl in raw_data.items():
 
 # Examples (in "fast run" mode) #
 
+The fast run mode is designed for bots synchronising an external resource with a Wikibase instance, where most of the
+entities are usually already up to date. Instead of loading every entity through the MediaWiki API to compare it with
+the local data, the fast run mode preloads the current state of all the entities of the data corpus with a few paginated
+SPARQL queries, then decides locally, entity per entity, if a write is required. When nothing changed (the most common
+case), no MediaWiki API call is made at all for that entity.
+
 In order to use the fast run mode, you need to know the property/value combination which determines the data corpus you
 would like to operate on. E.g. for operating on human genes, you need to know
 that [P351](https://www.wikidata.org/entity/P351) is the NCBI Entrez Gene ID and you also need to know that you are
 dealing with humans, best represented by the [found in taxon property (P703)](https://www.wikidata.org/entity/P703) with
 the value [Q15978631](https://www.wikidata.org/entity/Q15978631) for Homo sapiens.
 
-IMPORTANT: In order for the fast run mode to work, the data you provide in the constructor must contain at least one
-unique value/id only present on one Wikidata element, e.g. an NCBI entrez gene ID, Uniprot ID, etc. Usually, these would
-be the same unique core properties used for defining domains in wbi_core, e.g. for genes, proteins, drugs or your custom
-domains.
+IMPORTANT: In order for the fast run mode to work on entities without a known entity ID, the data you provide must
+contain at least one unique value/id only present on one Wikibase entity, e.g. an NCBI entrez gene ID, Uniprot ID, etc.
+This unique value is used to identify the target entity. If the entity ID is already known (e.g. the entity was loaded
+with `wbi.item.get()`), this requirement does not apply.
+
+## The base filter ##
+
+The base filter defines the data corpus and is a list of datatype instances (the same classes used to create claims).
+Three forms are supported:
 
-Below, the normal mode run example from above, slightly modified, to meet the requirements for the fast run mode. To
-enable it, ItemEngine requires two parameters, fast_run=True/False and fast_run_base_filter which is a dictionary
-holding the properties to filter for as keys, and the item QIDs as dict values. If the value is not a QID but a literal,
-just provide an empty string. For the above example, the dictionary looks like this:
+* A property with a value: the entities must have this exact statement, e.g. `Item(prop_nr='P703', value='Q15978631')`
+  (found in taxon Homo sapiens).
+* A property without a value: the entities must have a statement with this property, whatever the value, e.g.
+  `ExternalID(prop_nr='P351')` (any NCBI Entrez Gene ID).
+* A property path, given as a list of two datatype instances: the value is reached through the first property followed
+  by any number of hops with the second one, e.g. `[Item(prop_nr='P31', value='Q11173'), Item(prop_nr='P279')]`
+  (instance of (P31) chemical compound (Q11173), directly or through a chain of subclass of (P279)).
 
 ```python
 from wikibaseintegrator.datatypes import ExternalID, Item
 
 fast_run_base_filter = [ExternalID(prop_nr='P351'), Item(prop_nr='P703', value='Q15978631')]
 ```
 
-The full example:
+## Checking if a write is required ##
+
+The entry point is `entity.write_required()`. It returns `True` if the local entity differs from the live data and an
+actual write is needed, `False` if the entity is already up to date and the write can be skipped. The full example, a
+modified version of the mass import bot from the normal mode example:
 
 ```python
 from wikibaseintegrator import WikibaseIntegrator, wbi_login
-from wikibaseintegrator.datatypes import ExternalID, Item, String, Time
+from wikibaseintegrator.datatypes import ExternalID, Item, Time
 from wikibaseintegrator.wbi_enums import WikibaseTimePrecision
 
 # login object
 login = wbi_login.OAuth2(consumer_token='<consumer_token>', consumer_secret='<consumer_secret>')
 
-fast_run_base_filter = [ExternalID(prop_nr='P351'), Item(prop_nr='P703', value='Q15978631')]
-fast_run = True
+wbi = WikibaseIntegrator(login=login)
 
-# We have raw data, which should be written to Wikidata, namely two human NCBI entrez gene IDs mapped to two Ensembl Gene IDs
+fast_run_base_filter = [ExternalID(prop_nr='P351'), ExternalID(prop_nr='P704'), Item(prop_nr='P703', value='Q15978631')]
+
+# We have raw data, which should be written to Wikidata, namely two human NCBI entrez gene IDs mapped to two Ensembl transcript IDs
 # You can iterate over any data source as long as you can map the values to Wikidata properties.
 raw_data = {
     '50943': 'ENST00000376197',
@@ -845,31 +870,75 @@ for entrez_id, ensembl in raw_data.items():
     # add some references
     references = [
         [
-            Item(value='Q20641742', prop_nr='P248')
-        ],
-        [
-            Time(time='+2020-02-08T00:00:00Z', prop_nr='P813', precision=WikibaseTimePrecision.DAY),
-            ExternalID(value='1017', prop_nr='P351')
+            Item(value='Q20641742', prop_nr='P248'),
+            Time(time='+2020-02-08T00:00:00Z', prop_nr='P813', precision=WikibaseTimePrecision.DAY)
         ]
     ]
 
-    # data type object
-    entrez_gene_id = String(value=entrez_id, prop_nr='P351', references=references)
-    ensembl_transcript_id = String(value=ensembl, prop_nr='P704', references=references)
+    # data type objects
+    entrez_gene_id = ExternalID(value=entrez_id, prop_nr='P351', references=references)
+    ensembl_transcript_id = ExternalID(value=ensembl, prop_nr='P704', references=references)
 
     # data goes into a list, because many data objects can be provided to
     data = [entrez_gene_id, ensembl_transcript_id]
 
-    # Search for and then edit/create new item
-    wb_item = WikibaseIntegrator(login=login).item.new()
-    wb_item.add_claims(claims=data)
-    wb_item.init_fastrun(base_filter=fast_run_base_filter)
-    wb_item.write()
+    item = wbi.item.new()
+    item.claims.add(data)
+
+    # Compare the local data with the live data and only write when something differs
+    if item.write_required(base_filter=fast_run_base_filter):
+        item.write()
 ```
 
-Note: Fastrun mode checks for equality of property/value pairs, qualifiers (not including qualifier attributes), labels,
-aliases and description, but it ignores references by default!
-References can be checked in fast run mode by setting `use_refs` to `True`.
+Note: only the claims whose property appears in the base filter are compared. In the example above, the P351 and P704
+claims are checked because both properties are part of the base filter; a claim on any other property would be ignored
+by the comparison.
+
+## Options ##
+
+`write_required()` accepts optional parameters:
+
+* `use_refs` (default `False`): also compare the references of the statements. By default, references are ignored.
+* `case_insensitive` (default `False`): compare string values case-insensitively.
+* `action_if_exists` (default `ActionIfExists.REPLACE_ALL`): how the local statements are compared with the live ones.
+  With `REPLACE_ALL`, the entity must hold exactly the provided statements for the filtered properties. With
+  `APPEND_OR_REPLACE`, the provided statements must already exist on the entity, but additional existing statements are
+  allowed. With `FORCE_APPEND`, a write is always reported as required.
+
+The preloaded data is shared: containers are cached at the module level and reused by every `write_required()` call
+using the same base filter and options, so the SPARQL queries are only executed once per property.
+
+## Checking labels, descriptions and aliases ##
+
+`write_required()` only compares claims. To check language data (labels, descriptions or aliases), use the fastrun
+container directly:
+
+```python
+from wikibaseintegrator import wbi_fastrun
+from wikibaseintegrator.datatypes import ExternalID, Item
+
+frc = wbi_fastrun.get_fastrun_container(base_filter=[ExternalID(prop_nr='P351'), Item(prop_nr='P703', value='Q15978631')])
+
+# Resolve the entity ID from a unique value, without any MediaWiki API call
+qid = frc.get_item(claims=[ExternalID(value='50943', prop_nr='P351')])
+
+# Returns True if the English label differs from 'CDK7' and a write is required
+frc.check_language_data(qid=qid, lang_data=['CDK7'], lang='en', lang_data_type='label')
+```
+
+`check_language_data()` accepts `'label'`, `'description'` or `'aliases'` as `lang_data_type`, and the same
+`action_if_exists` logic as above (`APPEND_OR_REPLACE` or `REPLACE_ALL`).
+
+## Limitations ##
+
+* The comparison is based on the SPARQL endpoint, which can lag behind the live data (e.g. the Wikidata Query Service
+  replication lag). A decision taken on stale data results, at worst, in one unnecessary write or one missed update
+  until the next run.
+* The whole data corpus selected by the base filter is loaded in memory. Choose a base filter that selects a reasonably
+  sized subset of the instance.
+* The statement attributes that are not exposed in the SPARQL simple values (time precision, globe coordinate
+  precision, quantity bounds...) are reconstructed with their default values: statements using non-default values for
+  these attributes are always reported as requiring a write.
 
 # Debugging #
 

test/test_datatypes.py

@@ -87,6 +87,22 @@ def test_large_year_parsing(self):
         # Ordering keeps working across the 4/5-digit boundary
         assert Time(time='+9999-01-01T00:00:00Z', prop_nr='P5') < time
 
+    def test_parse_sparql_value(self):
+        # A bare RDF timestamp: the missing '+' is added and the precision is inferred
+        time = Time(prop_nr='P5')
+        assert time.parse_sparql_value('2020-02-08T00:00:00Z') is True
+        assert time.mainsnak.datavalue['value']['time'] == '+2020-02-08T00:00:00Z'
+        assert time.mainsnak.datavalue['value']['precision'] == WikibaseTimePrecision.DAY.value
+
+        # A quoted literal with its datatype suffix
+        time = Time(prop_nr='P5')
+        assert time.parse_sparql_value('"+2020-02-00T00:00:00Z"^^xsd:dateTime') is True
+        assert time.mainsnak.datavalue['value']['precision'] == WikibaseTimePrecision.MONTH.value
+
+        # A timestamp with a time part can't be represented, its precision can't be inferred
+        assert Time(prop_nr='P5').parse_sparql_value('2020-02-08T12:34:56Z') is False
+        assert Time(prop_nr='P5').parse_sparql_value('not a timestamp') is False
+
 
 class TestRank:
     def test_rank_parsing(self):