Skip to content

[VIndex] Update docs on MapFn #35

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mhutchinson merged 1 commit into from
Sep 4, 2025
Merged

[VIndex] Update docs on MapFn #35

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 17 additions & 23 deletions vindex/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,40 +104,34 @@ The verifier builds a map at the same size as the verifiable index and if the ma

### MapFn Specified in Universal Language

Being able to specify which keys are relevant to any particular entry in the log is critical to allow a verifier to check for correct construction of the map. Ideally this MapFn would be specified in a universal way, to allow the verifier to be running a different technology stack than the core map operator. i.e. having the Go implementation be the specification is undesirable, as it puts a lot of tax on the verifier to reproduce this behaviour identically in another environment.

Some options:

* WASM ([go docs](https://go.dev/wiki/WebAssembly))
* Formal spec
* Functional language that can be transpiled
The verifiable index is constructed by taking each leaf in turn from the Input Log.
Each leaf entry is "mapped" (the terminology is borrowed from [MapReduce](https://en.wikipedia.org/wiki/MapReduce#Map_function)), which outputs all of the locations in the index that this leaf should be found.
For example, in CT this would take a `[]byte`, parse it as a certificate, and output the domains that this cert covers.

In any case, the MapFn functionally needs to be of the form:
Being able to specify which keys are relevant to any particular entry in the log is critical to allow a verifier to check for correct construction of the map. Ideally this MapFn would be specified in a universal way, to allow the verifier to be running a different technology stack than the core map operator. i.e. having the Go implementation be the specification is undesirable, as it puts a lot of tax on the verifier to reproduce this behaviour identically in another environment.

```
type MapFn func([]byte) [][]byte
```
The current plan is to use WASM ([go docs](https://go.dev/wiki/WebAssembly)), however other options could be considered (e.g. a formal spec, or a functional language that can be transpiled).

This would be used similarly to:
The current implementation in [map.go](./map.go) takes a `MapFn` interface:

```
var i uint64 // the index currently being processed. Needs to be set to non-zero if log started mid-way through.
var log chan []byte // channel on which leaves from the log will be written
var mapFn MapFn // initialized somehow (maybe loading wasm)
var output func(i uint64, mapKeys ...[][]byte) // probably just writes the write ahead log

for leaf := <- log {
mapKeys := mapFn(leaf)
output(i, mapKeys...)
i++
}
// MapFn takes the raw leaf data from a log entry and outputs the SHA256 hashes
// of the keys at which this leaf should be indexed under.
// A leaf can be recorded at any number of entries, including no entries (in which case an empty slice must be returned).
//
// MapFn is expected to consume any error states that it encounters in some way that
// makes sense to the particular ecosystem. This might mean outputting any invalid leaves
// at a known locations (e.g. all 0s), or not outputting any entry. Any panics will cause
// the mapping process to terminate.
type MapFn func([]byte) [][sha256.Size]byte
```

i.e. consume each entry in the log, apply the map function in order to determine the keys to update, and then output this operation to the next stage of the pipeline.
Clients currently pass in implementations purely written in Go, however the [Milestones](#milestones) tracks adding WASM support.

> [!IMPORTANT]
> This describes the MapFn as returning key hashes.
> We _may_ want to have the map return the raw key (e.g. `maps.google.com`) so that a prefix trie can be constructed.
> See https://github.com/transparency-dev/incubator/issues/33 for discussion and to provide feedback.

> [!IMPORTANT]
> The `MapFn` is fixed for the life of the Verifiable Index.
Expand Down
Loading