Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 16 additions & 29 deletions .github/workflows/on-rc-tag.yml
Original file line number Diff line number Diff line change
@@ -1,17 +1,16 @@
name: SDK rc tag — dispatch Tier 2

# Per RELEASE_GATE.md §2.4 + §4.4. Triggered by:
# Triggered by:
# 1. Pushing an rc tag matching v-*-rc.* on this repo.
# 2. workflow_dispatch with the rc_tag input (lets you re-run the gate
# after a manual dev-pin without cutting a new tag).
# without cutting a new tag).
#
# Body:
# 1. Read fr-backend's env/targets.yaml via the GitHub API.
# 2. Pre-flight: dev.image_tag must equal prod.image_tag (so the rc SDK
# is validated against the backend version customers actually run).
# 1. Read the backend release target manifest via the GitHub API.
# 2. Pre-flight: dev.image_tag must equal prod.image_tag.
# If they differ, FAIL FAST with an actionable error message.
# 3. If they match (or prod is in `bootstrap` mode), dispatch
# event_type=sdk_rc to fr-system-tests with rc_tag in the payload.
# 3. If they match (or prod is in `bootstrap` mode), dispatch the SDK rc
# validation event with rc_tag in the payload.

on:
push:
Expand Down Expand Up @@ -51,24 +50,20 @@ jobs:
fi
echo "rc_tag=$RC_TAG" >> "$GITHUB_OUTPUT"

- name: Read fr-backend env/targets.yaml
- name: Read backend release target manifest
id: targets
env:
# fr-backend is private. Keep this read-only token separate from
# the write-capable repository_dispatch token below.
GH_TOKEN: ${{ secrets.BACKEND_TARGETS_READ_TOKEN }}
run: |
set -euo pipefail
if [[ -z "${GH_TOKEN:-}" ]]; then
echo "::error::BACKEND_TARGETS_READ_TOKEN secret is required to read fr-backend env/targets.yaml."
echo "::error::Add a fine-grained PAT with Contents: Read-only on FortifyRoot/fr-backend."
echo "::error::See fr-meta/docs/CICD_SETUP.md §5.3."
echo "::error::Repository secret BACKEND_TARGETS_READ_TOKEN is not configured."
exit 1
fi
# Read from `main` because that is the GitOps source of truth.
# Read from `main` because that is the release target source of truth.
if ! gh api --method GET repos/FortifyRoot/fr-backend/contents/env/targets.yaml \
-f ref=main --jq '.content' | base64 -d > targets.yaml; then
echo "::error::Could not fetch FortifyRoot/fr-backend env/targets.yaml from main"
echo "::error::Could not fetch the backend release target manifest from main."
exit 1
fi
# `yq` (mikefarah/yq) is preinstalled on GitHub-hosted runners.
Expand Down Expand Up @@ -97,35 +92,27 @@ jobs:
dev.image_tag = $DEV
prod.image_tag = $PROD
Dev must match prod for SDK rc validation. To fix:
1. Open a PR on fr-backend that sets env/targets.yaml:
dev.image_tag = $PROD
2. Merge it. deploy-dev.yml will roll dev-api back to '$PROD'
and run Tier 1.
3. Wait for Tier 1 green, then re-run THIS workflow via
workflow_dispatch with rc_tag = ${{ steps.vars.outputs.rc_tag }}.
1. Align the backend dev target with prod.image_tag ($PROD).
2. Wait for the dev deployment and validation gate to complete.
3. Re-run this workflow with rc_tag = ${{ steps.vars.outputs.rc_tag }}.
EOF
exit 1
fi
echo "Pre-flight OK — dev == prod == $DEV"

- name: Dispatch sdk_rc to fr-system-tests
- name: Dispatch SDK rc validation
env:
# SYSTEM_TEST_DISPATCH_TOKEN is REQUIRED for cross-repo dispatch.
# GITHUB_TOKEN cannot fire `repository_dispatch` on a foreign repo.
# See CICD_SETUP.md §5.3.
GH_TOKEN: ${{ secrets.SYSTEM_TEST_DISPATCH_TOKEN }}
RC_TAG: ${{ steps.vars.outputs.rc_tag }}
run: |
set -euo pipefail
if [[ -z "${GH_TOKEN:-}" ]]; then
echo "::error::SYSTEM_TEST_DISPATCH_TOKEN secret is required to dispatch sdk_rc to fr-system-tests."
echo "::error::Add a fine-grained PAT with Contents: Read and write on FortifyRoot/fr-system-tests."
echo "::error::See fr-meta/docs/CICD_SETUP.md §5.3."
echo "::error::Repository secret SYSTEM_TEST_DISPATCH_TOKEN is not configured."
exit 1
fi
gh api -X POST \
repos/FortifyRoot/fr-system-tests/dispatches \
-f event_type=sdk_rc \
-F "client_payload[rc_tag]=${RC_TAG}" \
-F "client_payload[repo]=${{ github.repository }}"
echo "sdk_rc dispatched. Watch fr-system-tests Actions for result."
echo "SDK rc validation dispatched."
8 changes: 3 additions & 5 deletions .github/workflows/publish-release.yml
Original file line number Diff line number Diff line change
@@ -1,9 +1,6 @@
name: Publish to PyPI

# Manual workflow_dispatch only — see RELEASE_GATE.md §9.3.
#
# Hardening status: Tier 1 items 1-3 implemented; T1.4 + Tier 2 + Tier 3
# tracked in fr-system-tests/docs/SDK_PUBLISH_HARDENING.md.
# Manual workflow_dispatch only.
#
# Body (in safe order — no user-supplied code runs until ALL pre-flight
# checks pass):
Expand Down Expand Up @@ -37,7 +34,8 @@ jobs:
name: Publish ${{ inputs.rc_tag }} to ${{ inputs.pypi_repository }}
runs-on: ubuntu-latest
timeout-minutes: 30
environment: pypi # 1 reviewer (the releaser) per RELEASE_GATE.md §7 L2-15
# Requires the configured PyPI release environment approval.
environment: pypi
steps:
# ---------------------------------------------------------------
# T1.2 — Validate the rc_tag input matches a strict pattern.
Expand Down
2 changes: 1 addition & 1 deletion DEV.MD
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ It describes how to set up a local development environment, manage the
FortifyRoot fork of OpenLLMetry (Traceloop), vendor OpenLLMetry into the
FortifyRoot Ocelle SDK, and work effectively with Poetry and VS Code.

This file is internal-facing and intentionally separate from README.md.
This file is maintainer-facing and intentionally separate from README.md.

---

Expand Down
143 changes: 68 additions & 75 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,16 @@
# FortifyRoot Ocelle

FortifyRoot Ocelle is a Python SDK for LLM observability, safety, and auditability. With a single initialization call, Ocelle instruments supported LLM providers and frameworks, captures traces, records token/latency metadata, and applies FortifyRoot safety callbacks for prompt and completion content.
FortifyRoot Ocelle is the Python SDK for FortifyRoot LLM observability and runtime safety. Add one initialization call to your application and Ocelle will instrument supported LLM providers and frameworks, emit OpenTelemetry traces/metrics/logs to FortifyRoot, and apply configured prompt/completion safety rules before sensitive content leaves or enters your application flow.

Ocelle is built on a FortifyRoot-maintained OpenLLMetry fork, vendored into this repository for dependency isolation, safety extensions, and stable FortifyRoot packaging.

## What Ocelle Captures

- LLM request/response traces with provider, model, span hierarchy, token usage, latency, retry-attempt, and streaming timing metadata.
- Framework spans for workflows, tasks, agents, and tools through decorators and supported framework integrations.
- Optional prompt and completion content when content tracing is enabled.
- Safety findings for prompt and completion content, including masking decisions and rule metadata.
- OTLP traces, metrics, and correlated logs for FortifyRoot ingestion.

## Installation

Expand All @@ -10,11 +20,12 @@ Ocelle supports Python 3.10 and newer.
pip install fortifyroot-ocelle
```

Install provider extras as needed:
Install only the provider/framework extras your application uses:

```bash
pip install "fortifyroot-ocelle[openai]"
pip install "fortifyroot-ocelle[openai,anthropic,langchain]"
pip install "fortifyroot-ocelle[bedrock,litellm,llamaindex]"
```

## Quick Start
Expand All @@ -23,20 +34,21 @@ pip install "fortifyroot-ocelle[openai,anthropic,langchain]"
import fortifyroot.ocelle as ocelle

ocelle.init(
app_name="my-llm-app", # any name you choose for this service
app_name="my-llm-app",
api_key="fr_sk_...",
resource_attributes={"environment": "dev"}, # dev / prod / testing — or any label you choose
resource_attributes={"environment": "prod"},
)

import openai
from openai import OpenAI

response = openai.chat.completions.create(
model="gpt-4",
client = OpenAI()
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Hello!"}],
)
```

The canonical import is:
The canonical SDK import is:

```python
import fortifyroot.ocelle as ocelle
Expand All @@ -48,55 +60,49 @@ The package also exposes a convenience alias:
import ocelle
```

The root `fortifyroot` package is reserved for internal namespaces such as vendored instrumentation. Public SDK code should use `fortifyroot.ocelle` or the `ocelle` convenience alias.
## Supported Providers And Frameworks

## Network Requirements
This table lists the launch-supported instrumentation exposed through `fortifyroot.ocelle.Instruments` and SDK extras. Package ranges are the ranges declared by this SDK; they are not a claim about the latest upstream release.

If your app runs in a private subnet, VPC, Kubernetes cluster, or locked-down CI/runtime, allow outbound HTTPS egress on TCP 443 to `api.fortifyroot.com`.
| Library / framework | Instrument | Extra | Declared package range | Sync | Async | Streaming | Safety |
|---|---|---|---|---:|---:|---:|---|
| OpenAI | `Instruments.OPENAI` | `openai` | `openai >=1.31.1,<3` | Yes | Yes | Yes | Prompt + completion, including streaming paths |
| Anthropic | `Instruments.ANTHROPIC` | `anthropic` | `anthropic >=0.49,<1.0.0` | Yes | Yes | Yes | Prompt + completion, including streaming paths |
| Google GenAI / Gemini | `Instruments.GOOGLE_GENERATIVEAI` | `google-generativeai` | `google-genai >=1.0.0,<2` | Yes | Yes | Yes | Prompt + completion, including streaming paths |
| AWS Bedrock Runtime | `Instruments.BEDROCK` | `bedrock` | `boto3 >=1.34.120,<2` | Yes | No native async client path | Yes | Prompt + completion for invoke/converse paths, including stream wrappers |
| LiteLLM | `Instruments.LITELLM` | `litellm` | `litellm >=1.71.2,<2,!=1.82.7,!=1.82.8` | Yes | Yes | Yes | Prompt + completion, including streaming paths |
| LangChain | `Instruments.LANGCHAIN` | `langchain` | `langchain >=0.2.5,<2.0.0`, `langchain-openai >=0.1.15,<2.0.0` | Yes | Yes | Provider-dependent | Prompt + completion for supported chat/LLM paths |
| LangGraph | via `Instruments.LANGCHAIN` | install with app | Covered through LangChain/OpenAI launch path | Yes | Yes | Provider-dependent | Same supported path as LangChain |
| LlamaIndex | `Instruments.LLAMA_INDEX` | `llamaindex` | `llama-index >=0.14.12,<0.15.0` | Yes | Yes | Yes | Prompt + completion, including streaming paths |

Ocelle exports telemetry over OTLP/HTTP to:
For provider-role behavior, routed providers such as OpenRouter, LiteLLM, Bedrock, Azure OpenAI, and planned/mapper-supported providers, see [Provider Support](docs/PROVIDERS.md). That document is the source of truth for what is launch-certified versus planned.

- `https://api.fortifyroot.com/v1/traces`
- `https://api.fortifyroot.com/v1/metrics`
- `https://api.fortifyroot.com/v1/logs`

If safety enforcement is enabled with `config_profile_id`, the SDK also polls:

- `https://api.fortifyroot.com/v1/sdk/config/{config_profile_id}`
## Runtime Safety

No inbound firewall rule is required. Hosted FortifyRoot usage does not require opening OTLP ports `4317` or `4318`; those are local/internal listener ports. Your workload still needs separate egress to whichever LLM providers it calls.
Ocelle can poll a FortifyRoot SDK config profile and apply configured safety rules locally in the SDK. Rules can inspect prompt and completion text and currently resolve to `ALLOW` or `MASK`.

## Auto-Instrumented LLM Libraries
Supported safety categories are:

The MVP SDK vendors and exposes the following supported instrumentation packages:
`PII`, `PCI`, `PHI`, `API_KEY`, `SECRET`, `PROMPT_INJECTION`, `PROFANITY`, `TOXICITY`, `VIOLENCE`, `SELF_HARM`, `CONFIDENTIAL`, and `CUSTOM`.

- **LLM providers**: OpenAI, Anthropic, Google Generative AI, AWS Bedrock, LiteLLM
- **Frameworks**: LangChain, LlamaIndex
Rules can be backed by regex matchers, string-list matchers, or approved user-defined detectors. Masking is applied before the instrumented provider/framework returns the text to application code where the integration can safely mutate the response object or stream chunk.

For the current launch-certified provider-role matrix and support tiers, see [Provider Support](docs/PROVIDERS.md).
Because Ocelle is open source, the SDK's enforcement flow is visible by design. Organization-specific safety policy is fetched at runtime from your FortifyRoot SDK config profile, so public code review exposes the engine and built-in defaults, not customer-specific rules.

## Configuration

### Environment Variables

Ocelle keeps the FortifyRoot environment variable namespace stable:

| Environment Variable | Description | Default |
|---------------------|-------------|---------|
| Environment variable | Description | Default |
|---|---|---|
| `FORTIFYROOT_API_KEY` | FortifyRoot API key | None |
| `FORTIFYROOT_BASE_URL` | API endpoint URL | `https://api.fortifyroot.com` |
| `FORTIFYROOT_BASE_URL` | FortifyRoot API endpoint | `https://api.fortifyroot.com` |
| `FORTIFYROOT_TRACE_CONTENT` | Capture prompt/response content | `true` |
| `FORTIFYROOT_TRACING_ENABLED` | Enable/disable tracing | `true` |
| `FORTIFYROOT_METRICS_ENABLED` | Enable/disable metrics | `true` |
| `FORTIFYROOT_TRACING_ENABLED` | Enable trace export | `true` |
| `FORTIFYROOT_METRICS_ENABLED` | Enable metric export | `true` |
| `FORTIFYROOT_LOGGING_ENABLED` | Enable OTLP log export and synthetic span-end logs | `false` |

When `FORTIFYROOT_LOGGING_ENABLED=true`:

- stdlib Python `logging` records keep using the app's existing handlers and formatting
- stdlib Python `logging` records emitted inside active spans are exported with trace/span correlation
- stdlib Python `logging` records emitted outside active spans can still be exported, but they remain uncorrelated
- Ocelle emits one synthetic correlated log for each completed instrumented span
- `print(...)`, stdout, and stderr capture are not included in MVP; use Python `logging` for application logs
When `FORTIFYROOT_LOGGING_ENABLED=true`, Python `logging` records emitted inside active spans are exported with trace/span correlation. `print(...)`, stdout, and stderr capture are not included; use Python `logging` for application logs.

### Programmatic Configuration

Expand Down Expand Up @@ -127,38 +133,9 @@ ocelle.configure() \
.init()
```

### Advanced Configuration

```python
import fortifyroot.ocelle as ocelle
from opentelemetry.sdk.trace.export import ConsoleSpanExporter, SimpleSpanProcessor
from opentelemetry.sdk.trace.sampling import TraceIdRatioBased

ocelle.init(
app_name="my-app",
api_key="fr_sk_...",
sampler=TraceIdRatioBased(0.1),
)

ocelle.init(
app_name="my-app",
processors=[SimpleSpanProcessor(ConsoleSpanExporter())],
)

def span_callback(span):
# Inspect span attributes, log alerts, etc.
pass

ocelle.init(
app_name="my-app",
api_key="fr_sk_...",
span_postprocess_callback=span_callback,
)
```

## Decorators

Use decorators to trace custom functions and create hierarchical traces:
Use decorators to add trace structure around your own application logic:

```python
from fortifyroot.ocelle import agent, task, tool, workflow
Expand All @@ -184,7 +161,7 @@ def generate_answer(context, question):

## Association Properties

Attach custom properties to traces for filtering and correlation:
Attach properties to traces for filtering and correlation:

```python
import fortifyroot.ocelle as ocelle
Expand All @@ -198,9 +175,25 @@ ocelle.set_association_properties({
})
```

## Network Requirements

If your app runs in a private subnet, VPC, Kubernetes cluster, or locked-down runtime, allow outbound HTTPS egress on TCP 443 to `api.fortifyroot.com`.

Ocelle exports telemetry over OTLP/HTTP to:

- `https://api.fortifyroot.com/v1/traces`
- `https://api.fortifyroot.com/v1/metrics`
- `https://api.fortifyroot.com/v1/logs`

If safety enforcement is enabled with `config_profile_id`, the SDK also polls:

- `https://api.fortifyroot.com/v1/sdk/config/{config_profile_id}`

No inbound firewall rule is required. Hosted FortifyRoot usage does not require opening OTLP ports `4317` or `4318`. Your workload still needs separate egress to whichever LLM providers it calls.

## Privacy And Content Tracing

To disable prompt and response content capture:
Disable prompt and response content capture with:

```bash
export FORTIFYROOT_TRACE_CONTENT=false
Expand All @@ -218,10 +211,10 @@ ocelle.init(
)
```

## Attribution

FortifyRoot Ocelle includes code derived from [OpenLLMetry](https://github.com/traceloop/openllmetry) and `traceloop-sdk` by Traceloop, licensed under the Apache License, Version 2.0. The SDK retains the Apache 2.0 license text and attribution in [LICENSE](LICENSE).
Safety rules can still run when configured; content tracing controls what is exported as telemetry content.

## License
## License And Attribution

Apache License, Version 2.0.

FortifyRoot Ocelle includes code derived from [OpenLLMetry](https://github.com/traceloop/openllmetry) and `traceloop-sdk` by Traceloop, licensed under the Apache License, Version 2.0. The license text and attribution are retained in [LICENSE](LICENSE).
Loading
Loading