Provenance of IDs: Incorrect or Missing Values for Primary Source

[eliarizzetto]

The same analysis made for issue #39, studying the primary sources mentioned in the provenance of Bibliographic Resources, was also performed on the provenance files for entities of Identifier type (datacite:Identifier). Similarly to #39, the analysis revealed that out of 216,914,486 processed provenance graphs, each corresponding to the provenance of a datacite:Identifier entity, 57,127,149 graphs do not store any information about the primary source (with 62,839,929 snapshots missing the prov:hadPrimarySource property). Moreover, the only primary sources that could be found in the whole dataset were OpenAlex and Crossref (https://api.crossref.org/snapshots/monthly/2024/03/all.json.tar.gz). As for #39, these results refer exclusively to version 7 of OC Meta RDF dataset (https://doi.org/10.6084/m9.figshare.21747536.v7).

Below, the script used for the analysis and its output. For further information refer to #39.

from collections import defaultdict
from tqdm import tqdm
import os
from zipfile import ZipFile
import json

def study_id_prov_attributes(data_dir):
    source_count = defaultdict(int)
    no_primsource_snaphots_count = 0
    no_primsource_id_count = 0
    total_id_count = 0
    fpaths = set()
    for dirpath, _, filenames in os.walk(data_dir):
        if os.path.basename(dirpath) == 'prov':
            for fn in filenames:
                fpaths.add(os.path.join(dirpath,fn))

    for fp in tqdm(fpaths):
        with ZipFile(fp) as archive:
            with archive.open('se.json') as f:
                data: list = json.load(f)

                for id_prov_g in data:
                    total_id_count += 1
                    no_primsource_id = True
                    for snapshot_g in id_prov_g['@graph']:
                        if snapshot_g.get('http://www.w3.org/ns/prov#hadPrimarySource'):
                            no_primsource_id = False  # the Id provenance graph has at least one snapshot specifying the primary source
                            for primary_source_g in snapshot_g['http://www.w3.org/ns/prov#hadPrimarySource']:
                                source_for_snapshot = primary_source_g['@id']
                                source_count[source_for_snapshot] += 1
                        else:
                            no_primsource_snaphots_count += 1
                    if no_primsource_id:
                        no_primsource_id_count +=1
        
    print(f'Primary sources distribution: {dict(source_count)}\n\n', 
          f'Snapshots without primary source: {no_primsource_snaphots_count}\n\n', 
          f'IDs without primary source: {no_primsource_id_count}\n\n', 
          f'Total number of Id-related provenance graphs: {total_id_count}')
    return dict(source_count), no_primsource_snaphots_count, no_primsource_id_count, total_id_count


print(study_id_prov_attributes('E:/id_v7/id'))

100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 218724/218724 [54:14<00:00, 67.21it/s]

Primary sources distribution: {'https://openalex.s3.amazonaws.com/browse.html': 157021382, 'https://api.crossref.org/snapshots/monthly/2024/03/all.json.tar.gz': 3904022}

 Snapshots without primary source: 62839929

 IDs without primary source: 57127149

 Total number of Id-related provenance graphs: 216914486