Skip to content

[fix][COM][hadoop-common] resolve kerberos tgt expiry issue for cached hdfs filesystem - #1093

Merged
casionone merged 1 commit into
dev-2.1.0-webankfrom
dev-2.1.0-keytab-error-fix
Aug 12, 2026
Merged

[fix][COM][hadoop-common] resolve kerberos tgt expiry issue for cached hdfs filesystem#1093
casionone merged 1 commit into
dev-2.1.0-webankfrom
dev-2.1.0-keytab-error-fix

Conversation

@v-kkhuang

Copy link
Copy Markdown

What is the purpose of the change

Background/Problem:
When the HDFS FileSystem cache is enabled in a Kerberos-secured cluster, the cached FileSystem holds a Kerberos TGT that can expire over time. Once expired, reusing the cached FileSystem leads to Kerberos authentication errors, because the cache returns the stale FileSystem without checking TGT validity.

Purpose of Change:
To address this problem, this PR adds a feature switch linkis.hadoop.hdfs.tgt.proactive.check.enable (default off). When enabled, before returning a cached FileSystem the code checks the UGI's TGT validity via a new KerberosTgtUtils; if the TGT has expired, the stale cached container is removed and a new FileSystem with a fresh TGT is created.

Value/Impact:
After the change, cached HDFS FileSystems are no longer reused once their Kerberos TGT expires, eliminating authentication failures in long-running cached scenarios. The switch defaults to off, so existing behavior is unchanged when disabled.

Related issues/PRs

Related issues: close #1092
Related pr:none

Brief change log

  • Add linkis.hadoop.hdfs.tgt.proactive.check.enable switch in HadoopConf
  • Add KerberosTgtUtils to check Kerberos TGT validity in a UGI via reflection (getSubject is protected in Hadoop 2.7.x)
  • Add UGI field and isTgtValid() method to HDFSFileSystemContainer
  • Add proactive TGT expiry check in HDFSUtils: remove stale cached FileSystem and recreate with fresh TGT when expired
  • Add test assertion for the new switch default value

Checklist

  • I have read the Contributing Guidelines on pull requests.
  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change
  • I have updated the documentation to reflect this change
  • I have verified that this change is backward compatible
  • If this is a code change: I have written unit tests to fully verify the new behavior.

@v-kkhuang v-kkhuang added the bug Something isn't working label Aug 12, 2026

@casionone casionone left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@casionone
casionone merged commit c62b7bf into dev-2.1.0-webank Aug 12, 2026
10 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants