Skip to content

iPad port with Apple Pencil support #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
18631665217 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

iPad port with Apple Pencil support #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 26 additions & 9 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,26 @@
riddle/target/
quill/build/
quill/vendor/
*.log

# never commit a real API key
riddle/oracle.env
oracle.env
riddle/dist/
# Xcode
build/
DerivedData/
*.xcuserstate
*.xcuserdatad/
xcuserdata/
*.xcscmblueprint
*.xccheckout

# Swift Package Manager
.swiftpm/
.build/
Package.resolved

# macOS
.DS_Store
*.swp
*~

# Secrets — NEVER commit API keys
*.env
.env
secrets.plist

# Old experiment
RiddlePlayground.swiftpm/
42 changes: 42 additions & 0 deletions Info.plist
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>CFBundleDevelopmentRegion</key>
<string>$(DEVELOPMENT_LANGUAGE)</string>
<key>CFBundleDisplayName</key>
<string>The Diary</string>
<key>CFBundleExecutable</key>
<string>$(EXECUTABLE_NAME)</string>
<key>CFBundleIdentifier</key>
<string>$(PRODUCT_BUNDLE_IDENTIFIER)</string>
<key>CFBundleInfoDictionaryVersion</key>
<string>6.0</string>
<key>CFBundleName</key>
<string>Riddle</string>
<key>CFBundlePackageType</key>
<string>APPL</string>
<key>CFBundleShortVersionString</key>
<string>1.0</string>
<key>CFBundleVersion</key>
<string>1</string>
<key>ITSAppUsesNonExemptEncryption</key>
<false/>
<key>NSApplePencilUsageDescription</key>
<string>The diary needs Apple Pencil input for handwriting.</string>
<key>UILaunchScreen</key>
<dict>
<key>UIColorName</key>
<string></string>
</dict>
<key>UIRequiresPersistentWiFi</key>
<false/>
<key>UISupportedInterfaceOrientations</key>
<array>
<string>UIInterfaceOrientationPortrait</string>
<string>UIInterfaceOrientationPortraitUpsideDown</string>
<string>UIInterfaceOrientationLandscapeLeft</string>
<string>UIInterfaceOrientationLandscapeRight</string>
</array>
</dict>
</plist>
190 changes: 55 additions & 135 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,159 +1,79 @@
# riddle — the diary of Tom Riddle, for the reMarkable Paper Pro

Write on the page with your pen. After a pause, the diary **drinks your ink** —
your words fade into the paper — the page thinks for a moment, and an answer
writes itself back in a flowing hand, stroke by stroke, then fades away.
# Riddle for iPad

No screen glow, no keyboard, no chat UI. Just ink appearing on paper.

_This is the diary from [the demo](https://x.com/MaximeRivest)._

### 🪄 New to this? Start here

You need a **reMarkable Paper Pro** in developer mode with a launcher installed.
If that sounds like a lot, it isn't — **[remagic](https://github.com/maximerivest/remagic)**
walks you through turning on developer mode and sets up everything with one
command. Come back here, drop riddle in, and start writing to Tom.
An iPad port of [MaximeRivest/riddle](https://github.com/MaximeRivest/riddle) — Tom Riddle's enchanted diary, reimagined for Apple Pencil.

Already have xovi + AppLoad? **[Download the latest release](https://github.com/MaximeRivest/riddle/releases/latest)** — a ready-to-drop bundle, no compiler needed — or [build from source](#building).
Write with Apple Pencil → the diary drinks your ink → Tom replies in handwriting that appears stroke by stroke.

### Install the prebuilt bundle
## Screenshots

1. Grab `riddle-appload-aarch64.zip` from the [latest release](https://github.com/MaximeRivest/riddle/releases/latest) and unzip it.
2. Copy the folder to your tablet:
`scp -O -r riddle root@10.11.99.1:/home/root/xovi/exthome/appload/`
3. Add an API key: `cp oracle.env.example oracle.env` in that folder and put your `RIDDLE_OPENAI_KEY` in it (any OpenAI-compatible key). Or skip it to use [pi](#option-b--pi-the-power-path).
4. In **AppLoad**: tap **Reload**, then **The Diary**. Write, and rest your pen.
(Coming soon)

> ⚠️ **This modifies your device.** It runs as root, stops the vendor UI
> (in takeover mode), and drives the e-ink engine directly. It has only been
> tested on a **reMarkable Paper Pro** (ferrari, aarch64, OS 3.26–3.27). It may
> not work on other models or OS versions, and you use it entirely at your own
> risk. Not affiliated with reMarkable AS. Keep SSH access working before you
> install anything — that is your escape hatch.
## Requirements

## How it works
- iPad with Apple Pencil support
- iOS 17.0+
- Xcode 26.6+
- An API key for any OpenAI-compatible, vision-capable LLM

```
pen (raw evdev, full 4096-level pressure, hardware event rate)
│ strokes
▼
riddle ── idle 2.8s → commit page → PNG ──► oracle (resident LLM process,
│ streams reply sentence-by-sentence)
▼ strokes (Dancing Script → skeletonized to single-pixel pen paths)
display backend
├── qtfb — windowed, inside xochitl (AppLoad app)
└── quill — full takeover: xochitl stopped, vendor e-ink engine
driven directly for instant ink (lowest latency there is)
```
## Setup

- **`riddle/`** — the app (Rust). Pen input, ink surface, handwriting
synthesis (rasterize → Zhang-Suen thinning → stroke tracing → animated
replay), the oracle process manager, and both display backends.
- **`quill/`** — the takeover display host (C/C++). An
[epfb-re](https://github.com/asivery)-style QImage-constructor interposition
shim over the vendor `libqsgepaper.so` waveform engine, exposed as a small
C ABI (`quill_init` / `quill_buffer` / `quill_swap`) that riddle links
against with `--features takeover`. Includes `scribble`, a minimal
pen-to-glass latency demo.
1. Clone the repo
2. Open `Riddle.xcodeproj` in Xcode (use [XcodeGen](https://github.com/yonaskolb/XcodeGen) to regenerate from `project.yml` if needed)
3. Build and run on your iPad
4. On first launch, the Settings screen will appear — enter your API configuration:
- **API Key**: Your API key
- **Base URL**: The API endpoint (defaults to OpenAI)
- **Model**: A vision-capable model name

## Gestures
## Supported APIs

| Do this | And |
|---------|-----|
| Write, then rest the pen | The diary drinks your ink and Tom replies |
| Flip the marker | Erase |
| Draw a large **?** | Summon the built-in guide |
| Tap five fingers at once | Leave the diary |
| Power button | The page turns to *"The diary sleeps."*, then the tablet suspends; press again to wake exactly where you were |
Any OpenAI-compatible `/chat/completions` endpoint that supports image input works:

## The oracle (the "spirit" in the diary)
| Provider | Base URL | Model |
|----------|----------|-------|
| OpenAI | `https://api.openai.com/v1` | `gpt-4o-mini` |
| OpenRouter | `https://openrouter.ai/api/v1` | `openai/gpt-4o-mini` |
| Volcano Ark | `https://ark.cn-beijing.volces.com/api/plan/v3` | `doubao-seed-2-0-pro` |
| Groq | `https://api.groq.com/openai/v1` | `llama-3.2-90b-vision-preview` |

The diary's replies come from a vision LLM that reads your handwriting from the
committed page (sent as an inline PNG). There are **two backends**, chosen at
startup — pick whichever you have:
Presets are available in the Settings screen.

### Option A — any OpenAI-compatible API (easiest, zero setup)
## How It Works

Set an API key and riddle talks straight to an OpenAI-compatible
`/chat/completions` endpoint. Works with OpenAI, OpenRouter, Groq, a local
server — anything that speaks the format. No extra software on the tablet.
1. **Write** with Apple Pencil on the white page
2. After 1.5s idle, the diary **drinks the ink** — pixels dissolve using a per-pixel hash
3. The ink snapshot is sent as a PNG to your configured LLM **during** the dissolve (hides network latency)
4. The model **reads your handwriting** from the image and replies as Tom Riddle
5. The reply is rendered using **handwriting synthesis**: text is rasterized → Zhang-Suen skeletonized → stroke-traced → animated point by point
6. After lingering, the reply **dissolves** the same way the ink did

```sh
export RIDDLE_OPENAI_KEY="sk-..." # required
export RIDDLE_OPENAI_BASE="https://api.openai.com/v1" # optional (default)
export RIDDLE_OPENAI_MODEL="gpt-4o-mini" # optional; must see images
```
## Architecture

Any vision-capable model works. Example with OpenRouter:
Direct port of the original project's approach:

```sh
export RIDDLE_OPENAI_KEY="$OPENROUTER_API_KEY"
export RIDDLE_OPENAI_BASE="https://openrouter.ai/api/v1"
export RIDDLE_OPENAI_MODEL="openai/gpt-4o-mini"
```
- **Surface.swift** — Pixel buffer (like reMarkable's framebuffer). Direct `putPx`/`brushLine`/`stamp` operations.
- **Ink** — Stroke capture, PNG export, pixel dissolve (`dissolve_pass` with `px_hash`)
- **Oracle.swift** — HTTP client for OpenAI-compatible `/chat/completions`
- **HandwritingSynthesis.swift** — Zhang-Suen thinning + stroke tracing (port of `script.rs`)
- **DiaryCanvasView.swift** — Apple Pencil input via UIKit touches, draws directly to Surface
- **DiaryViewModel.swift** — State machine: Listening → Drinking → Thinking → Replying → Lingering → Fading

Verify your setup before launching the diary:
## Differences from Original

```sh
riddle --oracle-test path/to/handwriting.png # prints the streamed reply
```
| | Original (reMarkable) | iPad Port |
|---|---|---|
| Platform | reMarkable Paper Pro | iPad + Apple Pencil |
| Language | Rust | Swift / SwiftUI |
| Pen Input | evdev `/dev/input/eventN` | UIKit `UITouch` (`.pencil` type only) |
| Display | Linux framebuffer | UIView `draw()` + Surface bitmap |
| API Config | `oracle.env` file | In-app Settings screen |
| LLM | OpenAI-compatible (env vars) | OpenAI-compatible (UserDefaults) |

Measured ~0.9–1.1 s to first ink on-device. The HTTPS is built into riddle
(pure-Rust, no extra libraries).
## Credits

### Option B — pi (the power path)

If you already run [`pi`](https://github.com/badlogic/pi-mono), riddle will use
a resident `pi --mode rpc` process kept warm (Node + your subscription auth
loaded once), so each turn pays only model latency. Used automatically when
`RIDDLE_OPENAI_KEY` is **not** set.

Both stream the reply sentence-by-sentence, so the quill starts writing seconds
before the model finishes. The persona prompt lives in `riddle/src/oracle.rs`.

## Building

Cross-compiled from x86_64. Two flavours:

### Windowed (AppLoad/qtfb) — easiest

Requires [xovi + AppLoad](https://github.com/asivery/rm-appload) on the device.

```sh
cd riddle
cargo build --release --target aarch64-unknown-linux-gnu
```

Install to `/home/root/xovi/exthome/appload/riddle/` with
`external.manifest.json`, `appload-launch.sh`, and the binary.

### Takeover (instant ink) — the one from the demo

Requires the reMarkable SDK toolchain (`~/rm-sdk-3.26`) because the linked
vendor Qt libs need its glibc, **and** `libqsgepaper.so` pulled from *your own
device* (it is proprietary and not distributed here):

```sh
cd quill && ./build.sh # pulls libqsgepaper.so from the device over ssh,
# builds libquill.so + scribble
cd ../riddle && ./build-takeover.sh
```

Deploy `libquill.so` to `/home/root/quill/` and `riddle-takeover` to
`/home/root/riddle/riddle`, plus `scripts/riddle-takeover.sh`. Launching via
AppLoad (`appload-launch.sh`) detaches into a transient systemd unit, stops
xochitl, runs the diary, and **always restores xochitl on exit** — exit with
the power button, a 5-finger tap, or SIGTERM. If anything wedges:
`ssh root@10.11.99.1 'systemctl start xochitl'`.

## Fonts

The reply hand is [Dancing Script](https://github.com/googlefonts/DancingScript)
(SIL OFL 1.1 — see `riddle/fonts/OFL.txt`).
- Original project: [MaximeRivest/riddle](https://github.com/MaximeRivest/riddle)
- Persona prompt, state machine, handwriting synthesis, and dissolve algorithm are direct ports from the original Rust source

## License

MIT for everything in this repository (see `LICENSE`). The vendor libraries it
interposes (`libqsgepaper.so`, Qt) are **not** included and must come from
your own device/SDK.
Same license as the original project.
Loading