-
Notifications
You must be signed in to change notification settings - Fork 90
feat: add Venus IPS log query service package #258
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
cc06
wants to merge
7
commits into
chaitin:main
Choose a base branch
from
cc06:add-venus-ips
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
d3738a3
Add Venus IPS log query service package
cc06 da4ecbf
Harden Venus IPS service for L2
kingfs 53b2345
Fix Venus IPS transport controls
kingfs 52cf323
Align Venus IPS response handling
kingfs 0bcc457
Harden Venus IPS log parsing
kingfs f343548
Fail closed on malformed IPS logs
kingfs 5450659
fix venus IPS strict log parsing
kingfs File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| #!/usr/bin/env node | ||
|
|
||
| import { fileURLToPath } from "node:url"; | ||
| import { runServiceMain } from "@chaitin-ai/octobus-sdk"; | ||
|
|
||
| import { service } from "../venus__ips/src/service.js"; | ||
|
|
||
| runServiceMain(service, { | ||
| entryFile: fileURLToPath(new URL("../venus__ips/bin/venus-ips.js", import.meta.url)), | ||
| }); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,74 @@ | ||
| # Venustech IPS | ||
|
|
||
| 启明星辰 IPS(入侵防御系统)攻击日志查询的 OctoBus service package。 | ||
| 属于「流量检测 / NIPS」类。设备日志页返回 HTML,本包解析其中的日志表行为结构化条目。 | ||
|
|
||
| ## 支持版本 | ||
|
|
||
| 启明星辰 IPS(web 控制台,日志页 `/log/memorylog/ipslog.php`)。请求/响应按真机抓包对齐。 | ||
|
|
||
| ## 认证方式(web 会话 Cookie) | ||
|
|
||
| 控制台以 **会话 Cookie** 鉴权(浏览器 `credentials: include`)。请求头 `Cookie: <会话cookie>`, | ||
| cookie 经 `secret.cookie` 外部传入。 | ||
|
|
||
| > ⚠️ cookie 有时效,过期需更换。会话失效时设备会以 200 返回登录页;本包用日志页标记 | ||
| > (`ips_log_filter`)识别,识别失败时报 `FAILED_PRECONDITION`,避免把登录页当成空结果。 | ||
|
|
||
| ## 配置 | ||
|
|
||
| ```json | ||
| // config | ||
| { "host": "https://192.168.1.10", "timeoutMs": 5000, "maxResponseBytes": 2097152, "skipTlsVerify": true } | ||
| // secret | ||
| { "cookie": "PHPSESSID=<会话id>" } | ||
| ``` | ||
|
|
||
| ## 方法 | ||
|
|
||
| | RPC | 上游接口 | | ||
| | --- | --- | | ||
| | `ProbeConnectivity` | `GET /log/memorylog/ipslog.php`(仅报告 HTTP 可达性,不读取正文) | | ||
| | `QueryIpsLog` | `GET /log/memorylog/ipslog.php` | | ||
|
|
||
| ### 请求 / 响应 | ||
|
|
||
| - 请求:`limit`(返回条目上限,客户端侧截断;`0` 表示全部,负值返回 `INVALID_ARGUMENT`)。 | ||
| - 响应:`http_status`、`total`(解析到的条目数)、`entries[]`。每条 `entries` 含: | ||
| `name`(名称)、`src_ip`/`src_port`、`dst_ip`/`dst_port`、`protocol`、`time`、`type`(类型)、 | ||
| `severity`(事件级别)、`priority`(优先级)、`action`(动作)、`policy_id`(策略ID)、`count`(发生次数)、`content`(内容)。 | ||
|
|
||
| > 注:当前抓包为不带过滤的全量 GET;按源/目的 IP/时间过滤(设备 `ips_log_filter` 表单)未实现, | ||
| > 待补对应抓包后扩展。响应原始 HTML(较大且含内网地址)不回传,仅返回结构化条目。 | ||
|
|
||
| ## 风险边界 | ||
|
|
||
| - 本方法为**只读查询**,无写操作,风险面低。 | ||
| - 会话 cookie 等同登录态,泄露即会话失陷;仅放 `secret`,勿写入 `config`、日志或截图。 | ||
| - 默认校验 TLS;私有自签部署需 `skipTlsVerify: true`。 | ||
| - 禁止跨站重定向,响应体默认最多 2 MiB;上游正文、网络异常详情和 cookie 不写入错误信息。 | ||
|
|
||
| ## 错误映射 | ||
|
|
||
| | 场景 | gRPC code | | ||
| | --- | --- | | ||
| | 缺 host/cookie | `INVALID_ARGUMENT` | | ||
| | 上游 401/403 | `PERMISSION_DENIED` | | ||
| | 其它 4xx / 会话失效(返回登录页) | `FAILED_PRECONDITION` | | ||
| | 网络错误/超时/5xx | `UNAVAILABLE` | | ||
|
|
||
| ## 建议 capset | ||
|
|
||
| `query-ips-log`(只读),可直接授权给 AI SOC / 工作流做告警拉取与研判。 | ||
|
|
||
| ## 验证方式 | ||
|
|
||
| ```bash | ||
| cd services | ||
| npm run validate -- --service-dir venus__ips | ||
| npm test -- --service-dir venus__ips --coverage | ||
| npm run pack:check | ||
| ``` | ||
|
|
||
| 真机验证:用一个有效会话 cookie 调 `query-ips-log`,确认返回 IPS 告警条目(名称/源IP/目的IP/时间/级别/动作 等)。 | ||
| 原作者说明实现按设备抓包对齐,但本 PR 当前没有可独立审计的脱敏真机截图;合并前仍需维护者核验实际设备兼容性。**代码/测试/截图里不得出现真实 cookie、内网地址或业务数据。** |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| #!/usr/bin/env node | ||
| import { runServiceMain } from '@chaitin-ai/octobus-sdk'; | ||
|
|
||
| import { service } from '../src/service.js'; | ||
|
|
||
| runServiceMain(service); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| { | ||
| "$schema": "https://json-schema.org/draft/2020-12/schema", | ||
| "type": "object", | ||
| "additionalProperties": true, | ||
| "properties": { | ||
| "host": { "type": "string", "description": "Venus IPS console base URL with scheme, e.g. https://192.168.1.10." }, | ||
| "restBaseUrl": { "type": "string", "description": "Alias for host." }, | ||
| "baseUrl": { "type": "string", "description": "Alias for host." }, | ||
| "timeoutMs": { "type": "integer", "minimum": 1, "default": 5000, "description": "HTTP timeout in milliseconds." }, | ||
| "maxResponseBytes": { "type": "integer", "minimum": 1024, "maximum": 8388608, "default": 2097152, "description": "Maximum upstream response size in bytes." }, | ||
| "skipTlsVerify": { "type": "boolean", "default": false, "description": "Skip TLS certificate verification for private deployments." }, | ||
| "tlsInsecureSkipVerify": { "type": "boolean", "default": false, "description": "Legacy alias for skipTlsVerify." }, | ||
| "insecureSkipVerify": { "type": "boolean", "default": false, "description": "Alias for skipTlsVerify." }, | ||
| "headers": { "type": "object", "additionalProperties": { "type": "string" }, "description": "Optional non-sensitive HTTP headers. Authentication and hop-by-hop headers are ignored." } | ||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| { | ||
| "name": "venus-ips", | ||
| "version": "0.0.0", | ||
| "private": true, | ||
| "type": "module", | ||
| "bin": { | ||
| "venus-ips": "bin/venus-ips.js" | ||
| }, | ||
| "dependencies": { | ||
| "@chaitin-ai/octobus-sdk": "^0.6.0" | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,50 @@ | ||
| syntax = "proto3"; | ||
|
|
||
| package VENUS_IPS; | ||
|
|
||
| option go_package = "miner/grpc-service/VENUS_IPS"; | ||
|
|
||
| // 启明星辰 IPS(入侵防御系统)攻击日志查询。 | ||
| // 认证为 web 会话 Cookie:GET /log/memorylog/ipslog.php,浏览器 credentials=include。 | ||
| // 设备返回 HTML 日志页,本 service 解析其中的日志表行,映射为结构化条目。 | ||
| service VENUS_IPS { | ||
| // 探测控制台 HTTP 可达性;不读取或返回响应正文,也不代表会话一定有效。 | ||
| rpc ProbeConnectivity(ProbeConnectivityRequest) returns (ProbeConnectivityResponse) {} | ||
| // 查询 IPS 攻击日志(内存日志): GET /log/memorylog/ipslog.php | ||
| rpc QueryIpsLog(QueryIpsLogRequest) returns (QueryIpsLogResponse) {} | ||
| } | ||
|
|
||
| message ProbeConnectivityRequest {} | ||
|
|
||
| message ProbeConnectivityResponse { | ||
| bool reachable = 1; | ||
| int32 http_status = 2; | ||
| } | ||
|
|
||
| message QueryIpsLogRequest { | ||
| int32 limit = 1; // 返回条目上限(客户端侧截断),0 表示全部;负值返回 INVALID_ARGUMENT | ||
| } | ||
|
|
||
| // 对应日志表列:名称/源IP/源端口/目的IP/目的端口/协议/时间/类型/级别/优先级/动作/策略ID/次数/内容。 | ||
| message IpsLogEntry { | ||
| string name = 1; // 名称 | ||
| string src_ip = 2; // 源IP | ||
| string src_port = 3; // 源端口 | ||
| string dst_ip = 4; // 目的IP | ||
| string dst_port = 5; // 目的端口 | ||
| string protocol = 6; // 协议类型 | ||
| string time = 7; // 时间 | ||
| string type = 8; // 类型 | ||
| string severity = 9; // 事件级别 | ||
| string priority = 10; // 优先级 | ||
| string action = 11; // 动作 | ||
| string policy_id = 12;// 入侵防御策略ID | ||
| string count = 13; // 发生次数 | ||
| string content = 14; // 内容 | ||
| } | ||
|
|
||
| message QueryIpsLogResponse { | ||
| int32 http_status = 1; // 上游 HTTP 状态码 | ||
| int32 total = 2; // 解析到的条目数 | ||
| repeated IpsLogEntry entries = 3; // 结构化日志条目 | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| { | ||
| "$schema": "https://json-schema.org/draft/2020-12/schema", | ||
| "type": "object", | ||
| "additionalProperties": true, | ||
| "required": ["cookie"], | ||
| "properties": { | ||
| "cookie": { "type": "string", "description": "Venus IPS web session cookie header value (e.g. 'PHPSESSID=...'). Time-limited; refresh when expired." }, | ||
| "sessionCookie": { "type": "string", "description": "Alias for cookie." }, | ||
| "session_cookie": { "type": "string", "description": "Alias for cookie." } | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| { | ||
| "schema": "chaitin.octobus.service.v1", | ||
| "name": "venus-ips", | ||
| "displayName": "Venustech IPS", | ||
| "description": "OctoBus package for Venustech IPS attack-log query (GET /log/memorylog/ipslog.php, web session cookie, HTML log page parsed into structured entries).", | ||
| "runtime": { "mode": "long-running" }, | ||
| "proto": { "roots": ["proto"], "files": ["proto/venus_ips.proto"] }, | ||
| "configSchema": "config.schema.json", | ||
| "secretSchema": "secret.schema.json", | ||
| "sdk": { | ||
| "cli": { | ||
| "commands": { | ||
| "VENUS_IPS.VENUS_IPS/ProbeConnectivity": { | ||
| "name": "probe-connectivity", | ||
| "description": "Probe Venus IPS console HTTP connectivity without returning response content." | ||
| }, | ||
| "VENUS_IPS.VENUS_IPS/QueryIpsLog": { | ||
| "name": "query-ips-log", | ||
| "description": "Query Venus IPS attack logs (memory log)." | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| import { defineService } from '@chaitin-ai/octobus-sdk'; | ||
|
|
||
| import { handlers } from './venus-ips.js'; | ||
|
|
||
| export { handlers } from './venus-ips.js'; | ||
|
|
||
| export const service = defineService({ handlers }); |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
config schema 未禁止把会话 cookie 放入普通 config,绕过 secret 安全边界
config.schema.json 使用 additionalProperties: true 且未禁止 cookie 字段,而 resolveCallContext 会把 config 与 secret 合并进 bindings,resolveCookie 从合并后的 bindings 中读取 cookie/sessionCookie/session_cookie。因此运维人员若按 README 警告将 cookie 放在 config 中,服务仍会静默接受并以该 cookie 请求上游——config 与 secret 在 octobus 中通常有不同的存取/展示保护级别,会话 cookie 落入普通 config 会增加泄露面。README 明确要求“仅放 secret,勿写入 config”,但 schema 与代码都没有强制执行这一安全边界。
Problem code:
Recommendation:
在 config.schema.json 中显式禁止 cookie 相关字段(例如增加 "not": { "required": ["cookie", "sessionCookie", "session_cookie"] }),或让 resolveCookie 仅从 secret(ctx.secret)读取、不接受来自 config 的 cookie,从而强制 cookie 只能经 secret 注入。
Suggested diff:
"headers": { "type": "object", "additionalProperties": { "type": "string" }, "description": "Optional non-sensitive HTTP headers. Authentication and hop-by-hop headers are ignored." }, - } + }, + "not": { "required": ["cookie", "sessionCookie", "session_cookie"] } }