[KYUUBI #7593][FEATURE][AUTHZ] "Paranoid mode", try to catch missing authz checks

.rat-excludes

@@ -37,6 +37,8 @@ build/scala-*/**
 **/**/server_operation_logs/**/**
 **/**/engine_operation_logs/**/**
 **/*.output.schema
+**/classification_backlog_spark_*.txt
+**/spec_verified_spark_versions.txt
 **/apache-kyuubi-*-bin*/**
 **/benchmarks/**
 **/org.apache.spark.status.AppHistoryServerPlugin

docs/security/authorization/spark/install.md

@@ -153,3 +153,38 @@ Add `org.apache.kyuubi.plugin.spark.authz.ranger.RangerSparkExtension` to the sp
 spark.sql.extensions=org.apache.kyuubi.plugin.spark.authz.ranger.RangerSparkExtension
 ```
 
+### Handling of unclassified plan nodes
+
+The plugin builds access requests by recognizing Spark logical plan nodes. A plan node it
+does not recognize — for example a command introduced by a newer Spark version or by a
+third-party catalog plugin — carries no access request, so by default it would execute
+without any authorization check. The `spark.kyuubi.authz.unclassifiedNode.behavior`
+configuration controls what happens when such a node is encountered:
+
+```properties
+# allow | warn | deny (default: warn)
+spark.kyuubi.authz.unclassifiedNode.behavior=deny
+```
+
+This must be set on the engine application itself (`spark-defaults.conf`, `--conf`, or the
+Kyuubi engine configuration), because it is read from the application's `SparkConf`. A
+session-level override — SQL `SET`, the Spark Connect configuration API, `spark.conf.set` —
+does not change it, so an end user cannot relax `deny` for their own queries.
+
+- `allow`: legacy behavior; the node is silently treated as not authorization-relevant.
+- `warn` (default): the query proceeds, but a warning naming the unclassified plan node
+  class is logged once per class per JVM.
+- `deny`: the query fails with an `AccessControlException` naming the unclassified class.
+  This makes the plugin fail closed and is the recommended setting for security-sensitive
+  deployments.
+
+The same setting governs extraction failures against recognized commands (e.g. after a
+Spark upgrade changes a plan node's shape): `warn` logs them, `deny` fails the query.
+
+Plan nodes that are genuinely not authorization-relevant (e.g. `SELECT` without a table,
+session `SET` commands) are declared in the plugin's `known_harmless_spec.json` resource.
+Each entry carries a human-reviewed reason and names the exact Spark `major.minor`
+versions that review applies to; on any other Spark version the entry is inert and the
+node is treated as unclassified, so upgrading Spark past the reviewed versions surfaces
+each entry for re-review instead of silently trusting it.
+

extensions/spark/kyuubi-spark-authz/README.md

@@ -22,6 +22,15 @@
 - [x] Column-level fine-grained authorization
 - [x] Row-level fine-grained authorization, a.k.a. Row-level filtering
 - [x] Data masking
+- [x] Fail-closed handling of unclassified plan nodes ("paranoid mode"),
+      via `spark.kyuubi.authz.unclassifiedNode.behavior=allow|warn|deny`
+
+## Design Notes
+
+- [Paranoid mode](docs/paranoid-mode.md) — why non-recognition of a plan node must not
+  silently authorize it, the runtime `allow|warn|deny` mechanism, the
+  `known_harmless_spec.json` allowlist policy, and the per-Spark-profile build-time
+  coverage checks (`ClassificationCoverageSuite`).
 
 ## Build
 

extensions/spark/kyuubi-spark-authz/src/main/resources/database_command_spec.json

@@ -8,7 +8,8 @@
     "comment" : ""
   } ],
   "opType" : "ALTERDATABASE",
-  "uriDescs" : [ ]
+  "uriDescs" : [ ],
+  "verifiedSparkVersions" : [ "3.5" ]
 }, {
   "classname" : "org.apache.spark.sql.catalyst.plans.logical.CreateNamespace",
   "databaseDescs" : [ {
@@ -40,7 +41,8 @@
     "fieldExtractor" : "PropertiesLocationUriExtractor",
     "isInput" : false,
     "comment" : ""
-  } ]
+  } ],
+  "verifiedSparkVersions" : [ "3.5" ]
 }, {
   "classname" : "org.apache.spark.sql.catalyst.plans.logical.DescribeNamespace",
   "databaseDescs" : [ {
@@ -51,7 +53,8 @@
     "comment" : ""
   } ],
   "opType" : "DESCDATABASE",
-  "uriDescs" : [ ]
+  "uriDescs" : [ ],
+  "verifiedSparkVersions" : [ "3.5" ]
 }, {
   "classname" : "org.apache.spark.sql.catalyst.plans.logical.DropNamespace",
   "databaseDescs" : [ {
@@ -62,7 +65,8 @@
     "comment" : ""
   } ],
   "opType" : "DROPDATABASE",
-  "uriDescs" : [ ]
+  "uriDescs" : [ ],
+  "verifiedSparkVersions" : [ "3.5" ]
 }, {
   "classname" : "org.apache.spark.sql.catalyst.plans.logical.SetCatalogAndNamespace",
   "databaseDescs" : [ {
@@ -89,7 +93,8 @@
     "comment" : ""
   } ],
   "opType" : "SWITCHDATABASE",
-  "uriDescs" : [ ]
+  "uriDescs" : [ ],
+  "verifiedSparkVersions" : [ "3.5" ]
 }, {
   "classname" : "org.apache.spark.sql.catalyst.plans.logical.SetNamespaceLocation",
   "databaseDescs" : [ {
@@ -105,7 +110,8 @@
     "fieldExtractor" : "StringURIExtractor",
     "isInput" : false,
     "comment" : ""
-  } ]
+  } ],
+  "verifiedSparkVersions" : [ "3.5" ]
 }, {
   "classname" : "org.apache.spark.sql.catalyst.plans.logical.SetNamespaceProperties",
   "databaseDescs" : [ {
@@ -116,7 +122,8 @@
     "comment" : ""
   } ],
   "opType" : "ALTERDATABASE",
-  "uriDescs" : [ ]
+  "uriDescs" : [ ],
+  "verifiedSparkVersions" : [ "3.5" ]
 }, {
   "classname" : "org.apache.spark.sql.execution.command.AlterDatabasePropertiesCommand",
   "databaseDescs" : [ {
@@ -127,7 +134,8 @@
     "comment" : ""
   } ],
   "opType" : "ALTERDATABASE",
-  "uriDescs" : [ ]
+  "uriDescs" : [ ],
+  "verifiedSparkVersions" : [ "3.5" ]
 }, {
   "classname" : "org.apache.spark.sql.execution.command.AlterDatabaseSetLocationCommand",
   "databaseDescs" : [ {
@@ -143,7 +151,8 @@
     "fieldExtractor" : "StringURIExtractor",
     "isInput" : false,
     "comment" : ""
-  } ]
+  } ],
+  "verifiedSparkVersions" : [ "3.5" ]
 }, {
   "classname" : "org.apache.spark.sql.execution.command.AnalyzeTablesCommand",
   "databaseDescs" : [ {
@@ -154,7 +163,8 @@
     "comment" : ""
   } ],
   "opType" : "ANALYZE_TABLE",
-  "uriDescs" : [ ]
+  "uriDescs" : [ ],
+  "verifiedSparkVersions" : [ "3.5" ]
 }, {
   "classname" : "org.apache.spark.sql.execution.command.CreateDatabaseCommand",
   "databaseDescs" : [ {
@@ -170,7 +180,8 @@
     "fieldExtractor" : "StringURIExtractor",
     "isInput" : false,
     "comment" : ""
-  } ]
+  } ],
+  "verifiedSparkVersions" : [ "3.5" ]
 }, {
   "classname" : "org.apache.spark.sql.execution.command.DescribeDatabaseCommand",
   "databaseDescs" : [ {
@@ -181,7 +192,8 @@
     "comment" : ""
   } ],
   "opType" : "DESCDATABASE",
-  "uriDescs" : [ ]
+  "uriDescs" : [ ],
+  "verifiedSparkVersions" : [ "3.5" ]
 }, {
   "classname" : "org.apache.spark.sql.execution.command.DropDatabaseCommand",
   "databaseDescs" : [ {
@@ -192,7 +204,8 @@
     "comment" : ""
   } ],
   "opType" : "DROPDATABASE",
-  "uriDescs" : [ ]
+  "uriDescs" : [ ],
+  "verifiedSparkVersions" : [ "3.5" ]
 }, {
   "classname" : "org.apache.spark.sql.execution.command.SetDatabaseCommand",
   "databaseDescs" : [ {
@@ -203,7 +216,8 @@
     "comment" : ""
   } ],
   "opType" : "SWITCHDATABASE",
-  "uriDescs" : [ ]
+  "uriDescs" : [ ],
+  "verifiedSparkVersions" : [ "3.5" ]
 }, {
   "classname" : "org.apache.spark.sql.execution.command.SetNamespaceCommand",
   "databaseDescs" : [ {
@@ -214,5 +228,6 @@
     "comment" : ""
   } ],
   "opType" : "SWITCHDATABASE",
-  "uriDescs" : [ ]
+  "uriDescs" : [ ],
+  "verifiedSparkVersions" : [ "3.5" ]
 } ]

extensions/spark/kyuubi-spark-authz/src/main/resources/function_command_spec.json

@@ -31,7 +31,8 @@
     "isInput" : false,
     "comment" : ""
   } ],
-  "opType" : "CREATEFUNCTION"
+  "opType" : "CREATEFUNCTION",
+  "verifiedSparkVersions" : [ "3.5" ]
 }, {
   "classname" : "org.apache.spark.sql.execution.command.DescribeFunctionCommand",
   "functionDescs" : [ {
@@ -59,7 +60,8 @@
     "isInput" : true,
     "comment" : ""
   } ],
-  "opType" : "DESCFUNCTION"
+  "opType" : "DESCFUNCTION",
+  "verifiedSparkVersions" : [ "3.5" ]
 }, {
   "classname" : "org.apache.spark.sql.execution.command.DropFunctionCommand",
   "functionDescs" : [ {
@@ -93,7 +95,8 @@
     "isInput" : false,
     "comment" : ""
   } ],
-  "opType" : "DROPFUNCTION"
+  "opType" : "DROPFUNCTION",
+  "verifiedSparkVersions" : [ "3.5" ]
 }, {
   "classname" : "org.apache.spark.sql.execution.command.RefreshFunctionCommand",
   "functionDescs" : [ {
@@ -117,5 +120,6 @@
     "isInput" : false,
     "comment" : ""
   } ],
-  "opType" : "RELOADFUNCTION"
+  "opType" : "RELOADFUNCTION",
+  "verifiedSparkVersions" : [ "3.5" ]
 } ]

extensions/spark/kyuubi-spark-authz/src/main/resources/known_harmless_spec.json

@@ -0,0 +1,73 @@
+[ {
+  "classname" : "org.apache.kyuubi.plugin.spark.authz.rule.rowfilter.FilteredShowColumnsCommand",
+  "reason" : "This plugin's own row-filtering replacement for ShowColumnsCommand (installed by RuleReplaceShowObjectCommands); PrivilegesBuilder.build handles it with a dedicated dispatch arm and every result row is checked for SHOWCOLUMNS access",
+  "verifiedSparkVersions" : [ "3.5", "4.0", "4.1", "4.2" ]
+}, {
+  "classname" : "org.apache.kyuubi.plugin.spark.authz.rule.rowfilter.FilteredShowFunctionsCommand",
+  "reason" : "This plugin's own row-filtering replacement for ShowFunctionsCommand (installed by RuleReplaceShowObjectCommands); PrivilegesBuilder.build handles it with a dedicated dispatch arm and every result row is checked for SHOWFUNCTIONS access",
+  "verifiedSparkVersions" : [ "3.5", "4.0", "4.1", "4.2" ]
+}, {
+  "classname" : "org.apache.kyuubi.plugin.spark.authz.rule.rowfilter.FilteredShowTablesCommand",
+  "reason" : "This plugin's own row-filtering replacement for ShowTablesCommand (installed by RuleReplaceShowObjectCommands); PrivilegesBuilder.build handles it with a dedicated dispatch arm and every result row is checked for SHOWTABLES access",
+  "verifiedSparkVersions" : [ "3.5", "4.0", "4.1", "4.2" ]
+}, {
+  "classname" : "org.apache.spark.sql.catalyst.analysis.ResolvedNamespace",
+  "reason" : "Analysis-time resolution artifact naming a namespace; reads no data itself, and the commands resolved over it are classified in their own right",
+  "verifiedSparkVersions" : [ "3.5", "4.0", "4.1", "4.2" ]
+}, {
+  "classname" : "org.apache.spark.sql.catalyst.plans.logical.CTERelationRef",
+  "reason" : "Leaf reference to a CTE definition; the definition's own plan appears under WithCTE in the same tree and is authorized there",
+  "verifiedSparkVersions" : [ "3.5", "4.0", "4.1", "4.2" ]
+}, {
+  "classname" : "org.apache.spark.sql.catalyst.plans.logical.CommandResult",
+  "reason" : "Holds rows already produced by an eagerly executed command; that command was authorized when it executed",
+  "verifiedSparkVersions" : [ "3.5", "4.0", "4.1", "4.2" ]
+}, {
+  "classname" : "org.apache.spark.sql.catalyst.plans.logical.LocalRelation",
+  "reason" : "Holds in-memory literal rows (VALUES lists, createDataFrame); reads no stored data",
+  "verifiedSparkVersions" : [ "3.5", "4.0", "4.1", "4.2" ]
+}, {
+  "classname" : "org.apache.spark.sql.catalyst.plans.logical.NoopCommand",
+  "reason" : "Spark's placeholder for commands with nothing to do (e.g. IF EXISTS / IF NOT EXISTS variants when the object is absent); executes nothing",
+  "verifiedSparkVersions" : [ "3.5", "4.0", "4.1", "4.2" ]
+}, {
+  "classname" : "org.apache.spark.sql.catalyst.plans.logical.OneRowRelation",
+  "reason" : "The implicit single-row relation backing SELECT without FROM; reads no stored data",
+  "verifiedSparkVersions" : [ "3.5", "4.0", "4.1", "4.2" ]
+}, {
+  "classname" : "org.apache.spark.sql.catalyst.plans.logical.Range",
+  "reason" : "Generates rows from a numeric range (e.g. spark.range); reads no stored data",
+  "verifiedSparkVersions" : [ "3.5", "4.0", "4.1", "4.2" ]
+}, {
+  "classname" : "org.apache.spark.sql.catalyst.plans.logical.ShowNamespaces",
+  "reason" : "Enforced elsewhere: results are row-filtered per namespace by ObjectFilterPlaceHolder + FilterDataSourceV2Strategy; Spark eagerly executes the bare command in a nested QueryExecution whose unfiltered result the placeholder discards",
+  "verifiedSparkVersions" : [ "3.5", "4.0", "4.1", "4.2" ]
+}, {
+  "classname" : "org.apache.spark.sql.catalyst.plans.logical.ShowTables",
+  "reason" : "Enforced elsewhere: results are row-filtered per table by ObjectFilterPlaceHolder + FilterDataSourceV2Strategy; Spark eagerly executes the bare command in a nested QueryExecution whose unfiltered result the placeholder discards",
+  "verifiedSparkVersions" : [ "3.5", "4.0", "4.1", "4.2" ]
+}, {
+  "classname" : "org.apache.spark.sql.execution.ExternalRDD",
+  "reason" : "Wraps a session-created RDD/local collection (e.g. spark.createDataset, and Delta's internal VACUUM plumbing); RDD-level access is outside the plugin's scope and is an existing, separate concern",
+  "verifiedSparkVersions" : [ "3.5", "4.0", "4.1", "4.2" ]
+}, {
+  "classname" : "org.apache.spark.sql.execution.LogicalRDD",
+  "reason" : "Wraps a pre-existing RDD; RDD-level access is outside the plugin's scope and is an existing, separate concern",
+  "verifiedSparkVersions" : [ "3.5", "4.0", "4.1", "4.2" ]
+}, {
+  "classname" : "org.apache.spark.sql.execution.command.DropTempViewCommand",
+  "reason" : "Operates only on session-local temporary views, which are deliberately not authz resources (their reads are authorized against the underlying tables); see KYUUBI #3426",
+  "verifiedSparkVersions" : [ "3.5", "4.0", "4.1", "4.2" ]
+}, {
+  "classname" : "org.apache.spark.sql.execution.command.ResetCommand",
+  "reason" : "Resets session configuration only; sensitive configs are separately guarded by AuthzConfigurationChecker",
+  "verifiedSparkVersions" : [ "3.5", "4.0", "4.1", "4.2" ]
+}, {
+  "classname" : "org.apache.spark.sql.execution.command.SetCommand",
+  "reason" : "Sets session configuration only; sensitive configs are separately guarded by AuthzConfigurationChecker",
+  "verifiedSparkVersions" : [ "3.5", "4.0", "4.1", "4.2" ]
+}, {
+  "classname" : "org.apache.spark.sql.execution.command.ShowNamespacesCommand",
+  "reason" : "Enforced elsewhere: Spark 4.x's v1 SHOW DATABASES/NAMESPACES command; results are row-filtered per namespace by ObjectFilterPlaceHolder + FilterDataSourceV2Strategy exactly like v2 ShowNamespaces, and the bare command Spark eagerly executes in a nested QueryExecution has its unfiltered result discarded by the placeholder",
+  "verifiedSparkVersions" : [ "4.0", "4.1", "4.2" ]
+} ]