V6.0: 指纹库扩充至 16,858 条 + 置信度引擎重写 + 架构统一 + 多项增强 - #58
Open
sangnigege wants to merge 71 commits into
Open
Conversation
## 指纹库 - 合并 16 个来源，254K 原始规则去重清洗 → 16,277 条，覆盖 11,600 产品 - 支持 keyword (AND/OR) / faviconhash / regula 三种匹配方式 - 自动降权 body 单关键词规则，过滤置信度 <50 的匹配 - CMS 名归一化：统一大小写/空格/横线变体 - 同名 CMS 去重 + 版本号提取 (version_regex) ## CDN 检测 - 三层检测：DNS 多IP + CIDR 匹配 (549条) + 响应头特征 (16种) - 零额外发包，Header 兼容 CaseInsensitiveDict ## WAF 隐匿 - 添加 Sec-Fetch-* 浏览器头，favicon 请求修正 Accept - 移除 rememberMe=test Shiro Cookie (自曝扫描器特征) - Chrome 131 / Firefox 135 UA 池 ## ip2region v3 - v1 (11MB, 已停更) → v3.16.0 xdb (7MB, 2025年数据) - 新 lib/ip2region/ 包，Python v3 查询引擎 ## 库模式 - from lib.finger import Finger → 可被其他工具直接引用 - scan() / scan_and_save() 接口，返回结构化结果 ## 代码质量 - Python 3.10-3.13 兼容 (修复 SyntaxWarning/ResourceWarning) - 依赖精简：12→5 个直接依赖，全部活跃维护 - CRLF→LF，命名规范化，config→settings - XLSX 输出按置信度红/橙着色 - .gitignore 覆盖 venv/output/__pycache__ ## 版本提取与回归测试 - 修复 CaseInsensitiveDict 导致 header 规则全部失效的关键 bug - 清洗通用关键词 (set-cookie/login/copyright 等) - 修复误分类的 faviconhash/keyword 规则 Co-Authored-By: Claude <noreply@anthropic.com>
Removed outdated sections including fingerprint library update records, TODO list, and acknowledgments. Updated project introduction and core capabilities.
Glass, WebAliveScan, OneForAll have zero code references in Finger. Only EHole (fingerprint format) and ip2region (IP geolocation) are actually used. Keep acknowledgments honest. Co-Authored-By: Claude <noreply@anthropic.com>
- Author: EASY -> sangnigege - Remove website line from banner - Version: V5.2 -> V6.0 - Clean up argparse description Co-Authored-By: Claude <noreply@anthropic.com>
Remove [:8] truncation in both run() and match() methods. All matches with confidence >= 50 and unique CMS names are now shown in output, sorted by confidence descending. Co-Authored-By: Claude <noreply@anthropic.com>
Before: "2.4.57" -> After: "Apache 2.4.57" Co-Authored-By: Claude <noreply@anthropic.com>
77 new version_regex patterns added (33->110 total, 80 products), prioritizing products where version directly determines exploitability: Tier 1 (exploit-critical): Spring Boot, WordPress, Laravel, Drupal, ActiveMQ, 通达OA, 泛微全系列, 用友系列, 蓝凌OA Tier 2 (intelligence): 金蝶系列, 万户OA, Jumpserver, Elasticsearch, Kibana, Zabbix Tier 3 (free): Gunicorn, Dubbo, Hadoop, Rancher, RabbitMQ Co-Authored-By: Claude <noreply@anthropic.com>
Each favicon now produces two MurmurHash3 hashes: - ehole: base64.encodebytes(raw) + mmh3 (existing rules, 872 hashes) - fofa: base64.b64encode(raw) + mmh3 (FOFA/Mmh3Hash32 compatible) Matching logic checks both formats. Existing fingerprint DB rules continue to work unchanged. New rules can use FOFA icon_hash values directly from FOFA search results. Co-Authored-By: Claude <noreply@anthropic.com>
All 8 HTTP request paths now route through proxy: - Main page GET (req.py, finger.py) - Favicon GET (req.py, finger.py) - Fingerprint update (checkenv.py) - FOFA API search + auth check (fofa.py) - Quake API (quake.py) Usage: python Finger.py -u http://target --proxy socks5://127.0.0.1:1080 Proxy is validated by requests library (supports http, https, socks5). If pysocks is not installed, SOCKS5 will fail with a clear error. Co-Authored-By: Claude <noreply@anthropic.com>
MUKI (yingfff123/MUKI) contributed 24,981 passive fingerprint rules. After dedup against existing database: - 12,663 rules already covered - 11,731 genuinely new rules added (312 faviconhash + 11,419 keyword) Final: 16,918 rules covering 12,138 products. Co-Authored-By: Claude <noreply@anthropic.com>
Removed ultra-generic keywords from OR-logic rules: - login (matched every login link on every page) - password (matched every password field) - 登录/注册/忘记密码/Login/Sign up/Forgot password Deleted 2 rules entirely: - 登录页面 (all 4 keywords were generic, no specific ones left) Affected products: H3CER6300G2, Hue大数据框架, 中新金盾, 78oa, 天融信WAF, 微擎, Weaver-e-Bridge, CAS单点登录, layer.js, CanalAdmin etc. These rules now only match on their specific product keywords, not on every website with a login form. Co-Authored-By: Claude <noreply@anthropic.com>
Round 2 cleanup - removed from OR-logic header/body rules: - server (every HTTP response has Server header) - set-cookie:/set-cookie (every response has Set-Cookie) - x-cache (Varnish and many CDNs set this, not just Alibaba) - copyright (every page footer) - location.href (common JS pattern) Affected: DLink, HUAWEI-S5730/7700, Eudemon8000E-X8, HUAWEI-Secoway, Ali-CDN, Aruba-Device, H3C-Company, viewgood-streammedia Co-Authored-By: Claude <noreply@anthropic.com>
CDN detection (--cdn) and IP geolocation (--geo) are now disabled by default. This improves scanning speed significantly when these features are not needed. Usage: python Finger.py -f targets.txt # fast mode python Finger.py -f targets.txt --cdn # + CDN detection python Finger.py -f targets.txt --cdn --geo # + both Affects both CLI mode and library mode (lib/finger.py). Co-Authored-By: Claude <noreply@anthropic.com>
Removed beautifulsoup4 dependency (and its transitive deps: soupsieve, typing-extensions). Title and favicon extraction now use regex, which is faster for simple tag extraction. - req.py: get_title(), _find_favicon_href() → regex - finger.py: _get_title(), _find_favicon_href() → regex - requirements.txt: removed beautifulsoup4 - Finger.py: removed bs4 deprecation warning filter Dependencies reduced from 5 to 4 (mmh3, requests, XlsxWriter, colorama). Co-Authored-By: Claude <noreply@anthropic.com>
Cleaned generic keywords from OR-logic rules: - must-revalidate (HTTP standard Cache-Control directive) - powered by (every CMS footer) - refresh, created by intellij idea. (generic HTML/IDE) - window.location (common JS pattern) - Application (too generic standalone) - workflow, web service, 欢迎登陆系统 (generic terms) - <div class="copyright"> (every website footer) Deleted: - Alt-Svc (HTTP standard header, not a product) Restored (false fix): - Synology, DiskStation (legitimate brand keywords) Co-Authored-By: Claude <noreply@anthropic.com>
Update all numbers to reflect current state: - Rules: 16,869 (from 16 database merge + MUKI) - Products: 12,100 - Dependencies: 4 (removed beautifulsoup4) - Proxy support: --proxy flag - CDN/Geo opt-in: --cdn, --geo flags Co-Authored-By: Claude <noreply@anthropic.com>
Replace pool.map() with submit()+as_completed() pattern. On KeyboardInterrupt, call pool.shutdown(wait=False, cancel_futures=True) to immediately stop all running threads. Affects: lib/req.py, lib/finger.py Co-Authored-By: Claude <noreply@anthropic.com>
- Kyan监控设备: platform (every page uses it) - ARRIS-Router: copyright (every footer has it) - DouPHP: theme, indexLeft (generic CSS/template terms) Co-Authored-By: Claude <noreply@anthropic.com>
- login.php, language (Igenus): every PHP site has login.php - shortcut icon (JEECMS): every HTML page with favicon has this - /js/app (锐捷RG): extremely generic path - content="webkit" (Weaver-e-Bridge): common responsive meta tag - 取消此条规则 (layer.js): annotation text, not a keyword - <title></title> (byzoro-NFW): empty title, common on redirect pages - js/app (CanalAdmin): generic path - DELETE 登陆页面 body rule (type="password"): every login form Co-Authored-By: Claude <noreply@anthropic.com>
Filtered 29,080 EHole rules: - Excluded empty CMS names, short keywords (≤4 chars) - Excluded known generic keywords (login, server, copyright etc.) - Added 8,719 genuinely new rules (including 526 faviconhash) After OR consolidation: 16,913 rules, 12,135 products Favicon: 877 unique hashes (+2 new) Co-Authored-By: Claude <noreply@anthropic.com>
- Unified 530 rules with same normalized CMS name but different display names (spacing/hyphens/case variants) - Intentionally kept short product names (AKCMS, dubbo, wdcp, etc.) These are legitimate product identifiers, not generic words - 0 generic-word-only rules found to delete Final: 16,347 rules, 12,135 products, 0 unmerged variants Co-Authored-By: Claude <noreply@anthropic.com>
- Remove img/8.png (no longer referenced in README) - Add *.xlsx to .gitignore (scan output files) - Remove stale library/archive/ from .gitignore Co-Authored-By: Claude <noreply@anthropic.com>
Confidence-based auto-downgrade was penalizing short-but-specific keywords (e.g. AKCMS 5chars -60). Since generic keywords are already filtered during fingerprint import, runtime guessing is redundant. - Commented out auto-downgrade logic (150-164 lines) - Changed confidence>=50 filter to show all matches - Rules now output with their original rule-defined confidence Co-Authored-By: Claude <noreply@anthropic.com>
- Updated rule counts (16,347 / 11,617 products) - Added --cdn, --geo, --proxy to parameter table - Removed auto-downgrade descriptions (disabled) - Updated matching logic section - Fixed changelog and known limitations Co-Authored-By: Claude <noreply@anthropic.com>
- eladmin, evoa, FastAdmin, 勾股OA, J2eeFAST, JFinalOA - jpress, ktg-mes, oasys, rebuild, SSCMS, YuebonNetCore - catchAdmin (4 hashes ORed) - FastAdmin hash already existed, skipped Co-Authored-By: Claude <noreply@anthropic.com>
Updated fingerprint recognition rules description and removed project structure and dependencies sections.
Rule Library (finger.json: 16,359→16,858 rules, 11,945 products): - Remove 32 generic FP keywords (body, /login, download, self.location, username, etc.) - Add ~500 new rules: OA/ERP/printers/security/AI platforms (60+ categories) - Add 130 url-location rules for path-based matching (dirsearch-friendly) - Add model_regex for hardware (HP/Hikvision/Dell/Canon/Lenovo) - Add expected_server cross-validation for 17 FP-prone products - Add 34-product default credentials knowledge base - Deduplicate 67 groups of identical keyword rules Engine (lib/identify.py): - Rewrite confidence scoring: algorithmic (location/method/hit-ratio/keyword-quality) - Add location:url matching with trailing-slash bidirectional compatibility - Add model extraction (model_regex support) - Add Server header cross-validation (expected_server) - Remove static per-rule confidence field HTTP & Output (lib/finger.py, lib/output.py): - Fix favicon SSL errors (add verify=False to favicon requests) - Add Server header version extraction (nginx/Apache/IIS/PHP/etc.) - Add DefaultCreds column to xlsx output - Fix confidence coloring threshold (>=80 red, >=50 yellow) - Fix JSON output Chinese encoding (ensure_ascii=False) - Fix favicon href regex (handle rel/href both orderings) - Fix response size (use len(html) instead of len(response.text)) - Performance: default timeout 10s->5s Architecture: - Unify CLI + library mode to single Finger class (lib/finger.py) - Deprecate lib/req.py (duplicate scanning engine) - Add --audit flag and lib/audit.py (4-dimension rule quality audit) Code Quality: - Remove dead imports (base64, duplicate json) - Fix inconsistent confidence logic between finger.py and output.py - Add comprehensive error handling for edge cases - Add concurrent safety verification - Fix model_regex patterns (HP MFP prefix, Hikvision /SL suffix, Canon ADV)
…id/tomcat/ruoyi/grafana/activemq/seeyon/etc)
…hange" This reverts commit 1ca3a4d.
Updated README to reflect new version details and fixed badge information.
…er dead host detection
Author
|
您好，经过多天的迭代，目前已经形成了稳定版本。经实践验证，使用效果也不错，而且可以在高版本python下使用，如python 3.12、python 3.13。 |
Author
|
6.1版本已完成更新 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
结合AI进行了重构与二开，主要是优化结构、精简依赖、对高版本python适配、升级Ip2region等组件、增加功能。
前四部分比较容易理解，这里重点讲一下功能上的强化：先收集了GitHub上几乎所有ehole系列指纹，去重合并，但指纹数量上来误报问题就很严重；通过实践筛去了一些低质量指纹，删多了边际效应很明显，且会提高漏报率；于是对指纹库的结构进行了优化，增加了and、or这种逻辑，顺便增加了location的种类，还有version_regex、model_regex（方便提取版本，诸如nginx、IIS的版本），增加expected_server等等，从而优化指纹匹配的逻辑；但误报还是不可避免，所以写了置信度算法，区分指纹的可信程度，并提供规则质量审计。
还有一些功能的优化，比如增加了默认口令，每次匹配到产品去搜索默认口令比较麻烦，所以直接内置了大量口令，扫到某个产品会输出所有常见口令。再就是cdn也进行了优化，而且和Ip2region这两个功能默认是不开启的，指定参数才开启（功能太多了会降低速率）。
虽然代码是结合AI进行重构，但主要功能基本都进行了手工测试，确保可用性，下面是AI的总结：
V6.0 主要变更
基于 V5.1 进行了全面优化，核心目标：降低误报、提升覆盖、增强可用性。
📊 规则库
body、/login、download、self.location、username等），消除 VMware-ESX 83% 误报率等问题location: url规则model_regex硬件型号提取（HP/海康/Dell/Canon 等）expected_server交叉校验（17 个易误报产品）🧠 置信度引擎
confidence: 100→ 算法自动计算🏗️ 架构统一
Finger类（lib/finger.py）lib/req.py废弃（原 CLI 独立引擎，功能缺失）⚡ 引擎增强
verify=False，失败率从 ~15% → 1.6%）--audit规则质量审计（四维检测：Server 多样性/404命中率/命中率异常/关键词质量）📋 输出增强
DefaultCreds列ensure_ascii=False）scan_and_save()库模式输出完整🔧 代码质量
finger.py和req.py之间 ~80% 重复代码size = len(response.text)Unicode 潜在异常Urlerror.result未初始化崩溃FingerPrint_Update默认关闭（避免覆盖自定义规则库）config/settings.py新增timeout = 5配置项