Local Reviews (via API and Croissant)

doc/release-notes/12313-local-reviews.md

@@ -0,0 +1,13 @@
+### Local Reviews
+
+Datasets can have local reviews, listable via API. A local review is a review dataset ("review" for short) that points at the URL form of a persistent ID of a dataset (e.g. itemReviewedUrl:https://doi.org/10.5072/FK2/ABCDEF) that is in the same Dataverse installation. Local reviews of a dataset can be listed via API (and we plan to build a UI for it some day).
+
+A new metadata block called "Trusted Data Dimensions and Intensities" has been added for testing. This is described in the setup instructions for review datasets.
+
+If you set `dataverse.feature.croissant-with-local-reviews` to true, local reviews will appear in the croissant and croissantSlim metadata export formats for any dataset that has local reviews. This feature is experiemental, which is why it is hidden behind a feature flag.
+
+See the guides for the new [list reviews](https://preview.guides.gdcc.io/en/develop/api/native-api.html#list-reviews) API endpoint, #12313, and #12314.
+
+## New Settings
+
+- dataverse.feature.croissant-with-local-reviews

doc/sphinx-guides/source/_static/api/list-reviews.json

@@ -0,0 +1,52 @@
+{
+    "status": "OK",
+    "data": {
+        "reviews": [
+            {
+                "title": "Review of Pediatric Asthma",
+                "authors": [
+                    "Wazowski, Mike"
+                ],
+                "persistentId": "doi:10.5072/FK2/1WD6BX",
+                "persistentIdUrl": "https://doi.org/10.5072/FK2/1WD6BX",
+                "id": 13,
+                "citation": "Wazowski, Mike, 2026, \"Review of Pediatric Asthma\", https://doi.org/10.5072/FK2/1WD6BX, Root, DRAFT VERSION",
+                "citationHtml": "Wazowski, Mike, 2026, \"Review of Pediatric Asthma\", <a href=\"https://doi.org/10.5072/FK2/1WD6BX\" target=\"_blank\">https://doi.org/10.5072/FK2/1WD6BX</a>, Root, DRAFT VERSION",
+                "datePublished": "",
+                "description": "This is a review of a dataset.",
+                "rubricMetadataBlocks": [
+                    {
+                        "name": "rubric_trusteddatadimensionsintensities",
+                        "displayName": "Trusted Data Dimensions and Intensities",
+                        "fields": [
+                            {
+                                "typeName": "licensingAndLegalClarity",
+                                "value": "High"
+                            },
+                            {
+                                "typeName": "authorAndProvenance",
+                                "value": "Medium"
+                            },
+                            {
+                                "typeName": "biasEquityAndRepresentativeness",
+                                "value": "Low"
+                            },
+                            {
+                                "typeName": "integrityAndUsability",
+                                "value": "High"
+                            },
+                            {
+                                "typeName": "fitnessForScopeAndContextualRelevance",
+                                "value": "Medium"
+                            },
+                            {
+                                "typeName": "transparencyOfMethodsAndDocumentation",
+                                "value": "Low"
+                            }
+                        ]
+                    }
+                ]
+            }
+        ]
+    }
+}

doc/sphinx-guides/source/admin/dataverses-datasets.rst

@@ -135,10 +135,15 @@ The Review metadata block gives you a few basic fields common to all reviews suc
 
 You probably will want to create your own metadata blocks specific to the resources you are reviewing, your own "rubric". See :doc:`metadatacustomization` for details on creating and enabling custom metadata blocks.
 
-Instead of creating a new custom metadata block from scratch (if you simply want to evaluate the feature, for example), you can use the metadata blocks at https://github.com/IQSS/dataverse.harvard.edu
+Instead of creating a new custom metadata block from scratch (if you simply want to evaluate the feature, for example), in a test environment, you can use the "Trusted Data Dimensions and Intensities" for testing. (A test environment is advised because metadata blocks cannot be deleted once they are loaded (https://github.com/IQSS/dataverse/issues/9628).) These are the files to download:
+
+- :download:`rubric_trusteddatadimensionsintensities.tsv <../../../../scripts/api/data/metadatablocks/rubric_trusteddatadimensionsintensities.tsv>`
+- :download:`rubric_trusteddatadimensionsintensities.properties <../../../../src/main/java/propertyFiles/rubric_trusteddatadimensionsintensities.properties>` (optional)
 
 After loading the block, don't forget to update the Solr schema!
 
+As in the example above, the metadata block must start with ``rubric_`` (the "metadataBlock name" in the tsv itself) to be included in the output of the :ref:`api-list-reviews` API endpoint.
+
 Create a Review Dataset Type
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

doc/sphinx-guides/source/admin/discoverability.rst

@@ -47,6 +47,8 @@ We include Croissant in the ``<head>`` because it's `recommended <https://github
 
 Before Croissant was invented, Google recommended a different format that Dataverse refers to as "Schema.org JSON-LD" in the user interface (and ``schema.org`` in the API). If you prefer to put that older format in the ``<head>``, which was the behavior in older versions of Dataverse, see :ref:`dataverse.legacy.schemaorg-in-html-head`.
 
+See also the :ref:`dataverse.feature.croissant-with-local-reviews` feature flag.
+
 .. _discovery-sign-posting:
 
 Signposting

doc/sphinx-guides/source/api/native-api.rst

@@ -4835,6 +4835,31 @@ The fully expanded example above (without environment variables) looks like this
 
   curl -H "X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" -X PUT "https://demo.dataverse.org/api/datasets/3/license" -H "Content-type:application/json" --upload-file license.json
 
+.. _api-list-reviews:
+
+List Reviews
+~~~~~~~~~~~~
+
+Datasets can have reviews. Specifically, if a :ref:`review dataset <review-datasets-user>` points at (using the ``itemReviewedUrl`` field) the URL form of a persistent ID of a dataset (e.g. https://doi.org/10.5072/FK2/ABCDEF) that is in the same Dataverse installation as the review dataset, the review dataset will be included in the list of reviews for the dataset. It is considered a local review. If additional "rubric" metadata blocks are enabled (see :ref:`review-datasets-setup`) the "metadataBlock name" must start with ``rubric_`` for the fields to be included in the output of this API endpoint.
+
+An API token is optional if the review dataset has been published.
+
+.. code-block:: bash
+
+  export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+  export SERVER_URL=https://demo.dataverse.org
+  export PERSISTENT_IDENTIFIER=doi:10.5072/FK2/ABCDEF
+
+  curl -H "X-Dataverse-key:$API_TOKEN" "$SERVER_URL/api/datasets/:persistentId/reviews?persistentId=$PERSISTENT_IDENTIFIER"
+
+The fully expanded example above (without environment variables) looks like this:
+
+.. code-block:: bash
+
+  curl -H "X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" "https://demo.dataverse.org/api/datasets/:persistentId/reviews?persistentId=doi:10.5072/FK2/ABCDEF"
+
+:download:`list-reviews.json <../_static/api/list-reviews.json>` contains sample output of how the API response might look.
+
 Files
 -----
 

doc/sphinx-guides/source/installation/config.rst

@@ -4127,6 +4127,13 @@ dataverse.feature.require-embargo-reason
 
 Require an embargo reason when a user creates an embargo on one or more files. See :ref:`embargoes`.
 
+.. _dataverse.feature.croissant-with-local-reviews:
+
+dataverse.feature.croissant-with-local-reviews
+++++++++++++++++++++++++++++++++++++++++++++++
+
+Have the croissant and croissantSlim metadata export formats include an extra "reviews" array if local reviews exist. See :ref:`croissant-head`, :ref:`review-datasets-user`, :ref:`creating-a-review-dataset`, and :ref:`api-list-reviews`.
+
 .. _:ApplicationServerSettings:
 
 Application Server Settings

doc/sphinx-guides/source/user/dataset-management.rst

@@ -936,6 +936,8 @@ Review Datasets can only be created via API. You have the following options:
 
 When creating a review dataset you will likely need to fill in required fields like ``itemReviewedUrl`` as well as fields from one or more "rubric" metadata blocks, as described above under :ref:`review-datasets-overview`.
 
+If you point ``itemReviewedUrl`` at the URL form of a dataset (e.g. https://doi.org/10.5072/FK2/ABCDEF) that is in the same Dataverse installation as the review dataset, the review dataset is considered a local review and can be listed using the :ref:`api-list-reviews` API endpoint. These reviews appear in the Croissant metadata export if you enable the :ref:`dataverse.feature.croissant-with-local-reviews` feature flag.
+
 .. _dataset-types-datacite:
 
 Dataset Types and DataCite

scripts/api/data/metadatablocks/rubric_trusteddatadimensionsintensities.tsv

@@ -0,0 +1,28 @@
+#metadataBlock	name	dataverseAlias	displayName	blockURI												
+	rubric_trusteddatadimensionsintensities		Trusted Data Dimensions and Intensities													
+#datasetField	name	title	description	watermark	fieldType	displayOrder	displayFormat	advancedSearchField	allowControlledVocabulary	allowmultiples	facetable	displayoncreate	required	parent	metadatablock_id	termURI
+	authorAndProvenance	Author and Provenance	The level of trust in the data creators and in other provenance information		text	1		TRUE	TRUE	FALSE	TRUE	FALSE	FALSE		rubric_trusteddatadimensionsintensities	
+	integrityAndUsability	Integrity and Usability	The level of trust in the accuracy, completeness, and ease of use of the data		text	2		TRUE	TRUE	FALSE	TRUE	FALSE	FALSE		rubric_trusteddatadimensionsintensities	
+	fitnessForScopeAndContextualRelevance	Fitness for Scope and Contextual Relevance	The level of trust in the suitability of the data for specific contexts, questions, or policy applications		text	3		TRUE	TRUE	FALSE	TRUE	FALSE	FALSE		rubric_trusteddatadimensionsintensities	
+	licensingAndLegalClarity	Licensing and Legal Clarity	The level of trust in the explicitness of the data’s usage rights and their compliance with relevant laws and regulations		text	4		TRUE	TRUE	FALSE	TRUE	FALSE	FALSE		rubric_trusteddatadimensionsintensities	
+	transparencyOfMethodsAndDocumentation	Transparency of Methods and Documentation	The level of trust in the clarity of the descriptions of data collection and processing methods		text	5		TRUE	TRUE	FALSE	TRUE	FALSE	FALSE		rubric_trusteddatadimensionsintensities	
+	biasEquityAndRepresentativeness	Bias, Equity, and Representativeness	The level of trust in the inclusivity and fairness of the coverage of the data		text	6		TRUE	TRUE	FALSE	TRUE	FALSE	FALSE		rubric_trusteddatadimensionsintensities	
+#controlledVocabulary	DatasetField	Value	identifier	displayOrder												
+	authorAndProvenance	Low		0												
+	authorAndProvenance	Medium		1												
+	authorAndProvenance	High		2												
+	integrityAndUsability	Low		0												
+	integrityAndUsability	Medium		1												
+	integrityAndUsability	High		2												
+	fitnessForScopeAndContextualRelevance	Low		0												
+	fitnessForScopeAndContextualRelevance	Medium		1												
+	fitnessForScopeAndContextualRelevance	High		2												
+	licensingAndLegalClarity	Low		0												
+	licensingAndLegalClarity	Medium		1												
+	licensingAndLegalClarity	High		2												
+	transparencyOfMethodsAndDocumentation	Low		0												
+	transparencyOfMethodsAndDocumentation	Medium		1												
+	transparencyOfMethodsAndDocumentation	High		2												
+	biasEquityAndRepresentativeness	Low		0												
+	biasEquityAndRepresentativeness	Medium		1												
+	biasEquityAndRepresentativeness	High		2												

src/main/java/edu/harvard/iq/dataverse/DatasetPage.java

@@ -32,6 +32,7 @@
 import edu.harvard.iq.dataverse.engine.command.impl.DeleteDatasetVersionCommand;
 import edu.harvard.iq.dataverse.engine.command.impl.DeletePrivateUrlCommand;
 import edu.harvard.iq.dataverse.engine.command.impl.DestroyDatasetCommand;
+import edu.harvard.iq.dataverse.engine.command.impl.GetDatasetReviewsCommand;
 import edu.harvard.iq.dataverse.engine.command.impl.GetPrivateUrlCommand;
 import edu.harvard.iq.dataverse.engine.command.impl.LinkDatasetCommand;
 import edu.harvard.iq.dataverse.engine.command.impl.PublishDatasetCommand;
@@ -104,6 +105,7 @@
 import jakarta.inject.Inject;
 import jakarta.inject.Named;
 import jakarta.json.Json;
+import jakarta.json.JsonObject;
 import jakarta.json.JsonObjectBuilder;
 import jakarta.persistence.OptimisticLockException;
 
@@ -133,6 +135,7 @@
 import edu.harvard.iq.dataverse.externaltools.ExternalToolServiceBean;
 import edu.harvard.iq.dataverse.globus.GlobusServiceBean;
 import edu.harvard.iq.dataverse.export.SchemaDotOrgExporter;
+import edu.harvard.iq.dataverse.export.croissant.CroissantExportUtil;
 import edu.harvard.iq.dataverse.externaltools.ExternalToolHandler;
 import edu.harvard.iq.dataverse.license.License;
 import edu.harvard.iq.dataverse.makedatacount.MakeDataCountLoggingServiceBean;
@@ -6110,6 +6113,20 @@ public String getCroissant() {
             final String CROISSANT_SCHEMA_NAME = "croissantSlim";
             ExportService instance = ExportService.getInstance();
             String croissant = instance.getLatestPublishedAsString(dataset, CROISSANT_SCHEMA_NAME);
+            if (FeatureFlags.CROISSANT_WITH_LOCAL_REVIEWS.enabled()) {
+                // Rewrite the export on the fly and insert local reviews until we have a solution for https://github.com/gdcc/dataverse-spi/issues/5
+                JsonObjectBuilder reviewsJsonObj = null;
+                try {
+                    reviewsJsonObj = commandEngine.submit(new GetDatasetReviewsCommand(dvRequestService.getDataverseRequest(), dataset));
+                    JsonObjectBuilder reviews = CroissantExportUtil.getReviews(reviewsJsonObj);
+                    JsonObject croissantJson = JsonUtil.getJsonObject(croissant);
+                    String updatedContent = Json.createObjectBuilder(croissantJson)
+                        .add("reviews", reviews.build().getJsonArray("reviews")).build().toString();
+                    return updatedContent;
+                } catch (CommandException e) {
+                    logger.fine("Couldn't get reviews");
+                }
+            }
             if (croissant != null && !croissant.isEmpty()) {
                 logger.fine("Returning cached CROISSANT.");
                 return croissant;

src/main/java/edu/harvard/iq/dataverse/api/Datasets.java

@@ -36,6 +36,7 @@
 import edu.harvard.iq.dataverse.engine.command.exception.UnforcedCommandException;
 import edu.harvard.iq.dataverse.engine.command.impl.*;
 import edu.harvard.iq.dataverse.export.ExportService;
+import edu.harvard.iq.dataverse.export.croissant.CroissantExportUtil;
 import edu.harvard.iq.dataverse.externaltools.ExternalTool;
 import edu.harvard.iq.dataverse.externaltools.ExternalToolHandler;
 import edu.harvard.iq.dataverse.globus.GlobusServiceBean;
@@ -87,9 +88,11 @@
 import org.glassfish.jersey.media.multipart.FormDataParam;
 import software.amazon.awssdk.services.s3.model.CompletedPart;
 
+import java.io.ByteArrayInputStream;
 import java.io.IOException;
 import java.io.InputStream;
 import java.net.URI;
+import java.nio.charset.StandardCharsets;
 import java.sql.Timestamp;
 import java.text.MessageFormat;
 import java.text.SimpleDateFormat;
@@ -275,6 +278,17 @@ public Response exportDataset(@Context ContainerRequestContext crc, @QueryParam(
             ExportService instance = ExportService.getInstance();
 
             InputStream is = instance.getExport(datasetVersion, exporter);
+            if (FeatureFlags.CROISSANT_WITH_LOCAL_REVIEWS.enabled()
+                    && (exporter.equals("croissant") || exporter.equals("croissantSlim"))) {
+                // Rewrite the export on the fly and insert local reviews until we have a solution for https://github.com/gdcc/dataverse-spi/issues/5
+                JsonObjectBuilder reviews = CroissantExportUtil
+                        .getReviews(commandEngine.submit(new GetDatasetReviewsCommand(req, dataset)));
+                String content = new String(is.readAllBytes(), StandardCharsets.UTF_8);
+                JsonObject croissantJson = JsonUtil.getJsonObject(content);
+                String updatedContent = Json.createObjectBuilder(croissantJson)
+                        .add("reviews", reviews.build().getJsonArray("reviews")).build().toString();
+                is = new ByteArrayInputStream(updatedContent.getBytes(StandardCharsets.UTF_8));
+            }
 
             String mediaType = instance.getMediaType(exporter);
 
@@ -6261,7 +6275,23 @@ public Response updateLicense(@Context ContainerRequestContext crc,
             }
         }, getRequestUser(crc));
     }
-
+
+    @GET
+    @AuthRequired
+    @Path("{identifier}/reviews")
+    @Produces(MediaType.APPLICATION_JSON)
+    public Response getReviews(@Context ContainerRequestContext crc, @PathParam("identifier") String id) {
+        return response(req -> {
+            Dataset dataset = findDatasetOrDie(id);
+            try {
+                JsonObjectBuilder job = execCommand(new GetDatasetReviewsCommand(req, dataset));
+                return ok(job);
+            } catch (Exception ex) {
+                return error(BAD_REQUEST, ex.getMessage());
+            }
+        }, getRequestUser(crc));
+    }
+
     /**
      * Storage quotas and use. Note that these methods replicate the
      * collection-level equivalents 1:1. Both the quotas and the system for