[bug][COM][hadoop-common] cached hdfs filesystem fails after kerberos tgt expires when cache is enabled

[v-kkhuang]

Search before asking

• I had searched in the issues and found no similar issues.

Linkis Component

• linkis-commons
• linkis-computation-governance
• linkis-dist
• linkis-engineconn-plugin
• linkis-extensions
• linkis-orchestrator
• linkis-public-enhancements
• linkis-spring-cloud-services
• linkis-web

Description

When the HDFS FileSystem cache is enabled (linkis.hadoop.hdfs.cache.enable=true) in a Kerberos-secured cluster, the cached HDFS FileSystem holds a Kerberos TGT (ticket-granting ticket) that can expire over time. Once the TGT expires, reusing the cached FileSystem leads to authentication errors (e.g. InvalidTicket / Kerberos credentials are no longer valid), because the cache returns the stale FileSystem without checking TGT validity.

Steps to reproduce

1. Enable HDFS FileSystem cache (linkis.hadoop.hdfs.cache.enable=true) in a Kerberos-secured Hadoop cluster.
2. Obtain an HDFS FileSystem for a user and let it sit idle in the cache longer than the Kerberos ticket lifetime (typically 10-24h).
3. Trigger another operation that retrieves the same cached FileSystem.
4. Observe that the operation fails with a Kerberos authentication error because the cached FileSystem's TGT has expired.

Expected behavior

The cached HDFS FileSystem should not be reused once its Kerberos TGT has expired. The system should proactively detect the expired TGT, remove the stale entry from the cache, and recreate a FileSystem with a fresh TGT (via reloginFromKeytab / re-authentication from the keytab).

Your environment

• Linkis version used: 2.1.0
• Environment name and version:
  • hadoop-2.7.2 / 3.3.1
  • kerberos enabled cluster
  • scala-2.11.12
  • jdk 1.8

Anything else

The fix introduces a feature switch linkis.hadoop.hdfs.tgt.proactive.check.enable (default off). When enabled, before returning a cached FileSystem the code checks the UGI's TGT validity via a new KerberosTgtUtils; if the TGT is expired, the cached container is removed and a new FileSystem with a fresh TGT is created. The switch defaults to off so existing behavior is unchanged.

Are you willing to submit a PR?

• Yes I am willing to submit a PR!