Add SQLite3MultipleCiphers 2.3.3 support (SQLite 3.53.0) #1
Open
gainskills wants to merge 130 commits into
Open
Add SQLite3MultipleCiphers 2.3.3 support (SQLite 3.53.0) #1gainskills wants to merge 130 commits into
gainskills wants to merge 130 commits into
Conversation
use same workaround as authelia/authelia#6404 before mattn#1177 fixes the build otherwise
action/upload-artifact@v1 has been deprecated for a while. It seems like GitHub Actions will now cancel workflows if it is still using v1 of the action. This upgrades to the latest v4 of the action. See https://github.blog/news-insights/product-news/get-started-with-v4-of-github-actions-artifacts/ for details.
The commit removes the use of runtime.SetFinalizer to finalize
SQLiteRows since only serves to close the associated SQLiteStmt which
already has a registered finalizer.
It also fixes a race and potential panic in SQLiteRows.Close around the
SQLiteRows.s field (*SQLiteStmt) which is accessed without a mutex being
held, but modified with it held (null'd out). Further the mutex we are
holding is that of the SQLiteStmt so a subsequent call to Close will
cause a panic sine it'll attempt to dereference a nil field. The fix
here is to add a mutex for closing to SQLiteRows.
Since we now also set the s field to nil when closing this commit
removes the "closed" field (since checking if s is nil is the same) and
also changes the type of "nc" (number of columns) to an int32 so that we
can pack the nc and cls fields, and add the close mutex without making
the struct any bigger.
```
goos: darwin
goarch: arm64
pkg: github.com/charlievieth/go-sqlite3
cpu: Apple M4 Pro
│ x1.txt │ x4.txt │
│ sec/op │ sec/op vs base │
Suite/BenchmarkExec/Params-14 719.2n ± 2% 716.9n ± 1% ~ (p=0.897 n=10)
Suite/BenchmarkExec/NoParams-14 506.5n ± 3% 500.1n ± 0% -1.25% (p=0.002 n=10)
Suite/BenchmarkExecContext/Params-14 1.584µ ± 0% 1.567µ ± 1% -1.07% (p=0.007 n=10)
Suite/BenchmarkExecContext/NoParams-14 1.524µ ± 1% 1.524µ ± 1% ~ (p=0.539 n=10)
Suite/BenchmarkExecStep-14 443.9µ ± 3% 441.4µ ± 0% -0.55% (p=0.011 n=10)
Suite/BenchmarkExecContextStep-14 447.8µ ± 1% 442.9µ ± 0% -1.10% (p=0.000 n=10)
Suite/BenchmarkExecTx-14 1.643µ ± 1% 1.640µ ± 0% ~ (p=0.642 n=10)
Suite/BenchmarkQuery-14 1.968µ ± 3% 1.821µ ± 1% -7.52% (p=0.000 n=10)
Suite/BenchmarkQuerySimple-14 1.207µ ± 2% 1.040µ ± 1% -13.84% (p=0.000 n=10)
Suite/BenchmarkQueryContext/Background-14 2.400µ ± 1% 2.320µ ± 0% -3.31% (p=0.000 n=10)
Suite/BenchmarkQueryContext/WithCancel-14 8.847µ ± 5% 8.512µ ± 4% -3.79% (p=0.007 n=10)
Suite/BenchmarkParams-14 2.131µ ± 2% 1.967µ ± 1% -7.70% (p=0.000 n=10)
Suite/BenchmarkStmt-14 1.444µ ± 1% 1.359µ ± 1% -5.89% (p=0.000 n=10)
Suite/BenchmarkRows-14 61.57µ ± 1% 60.24µ ± 1% -2.16% (p=0.000 n=10)
Suite/BenchmarkStmtRows-14 60.15µ ± 1% 59.08µ ± 1% -1.78% (p=0.000 n=10)
Suite/BenchmarkQueryParallel-14 960.9n ± 1% 420.8n ± 2% -56.21% (p=0.000 n=10)
geomean 4.795µ 4.430µ -7.62%
```
A valid sqlite header must always be included (like in the other files) but sqlite3-binding.h explicitly guards against the system library case.
Because of sqlite's flexible typing, even though the column was declared as jsonb, the values are stored as the TEXT-typed json because they're converted to strings by Value/Scan. If the table is strict with a BLOB column, the example fails because of the type mismatch. This can be fixed by using the `jsonb()` function to convert incoming string-typed json and the `json()` function to convert outgoing binary-typed jsonb. The example is expanded to show both of these approaches. Note that both approaches use the same string-typed marshalling functions because the conversion to jsonb occurs within sqlite3, not within the Go code. SQLite docs state that the binary format is internal and applications shouldn't try to generate it: https://sqlite.org/json1.html#jsonb
GCP (at least Google Compute Engine) is a VM, and does not have restrictions relating to gcc. Also it appears that this whole section refers to general compiling, rather than compiling on GCP.
* Add ability to set an int64 file control * Update documentation * Remove duplicate err check in test * Update sqlite3.go Co-authored-by: rittneje <rittneje@gmail.com> --------- Co-authored-by: rittneje <rittneje@gmail.com>
Signed-off-by: Jakob Borg <jakob@kastelo.net>
add sqlite_dbstat tag for the DBSTAT virtual table
avoid out of bounds write in unlock_notify_wait on 64 bit platforms
modernise reflect.SliceHeader to unsafe.Slice
guard oversized string length in ResultText
bind via sqlite3_bind_text64/blob64 to avoid 32-bit length truncation
Upgrade SQLite to version 3053002
guard oversized blob length in callbackRetBlob
preserve embedded NUL bytes in custom function text values
Follow documented call order for sqlite3_value_blob in callbackArgString
Use atomic.Value for handle table and add concurrent lookup benchmark
Column names and declared types are invariant for the lifetime of a prepared statement, but Columns()/declTypes() called C.sqlite3_column_name and sqlite3_column_decltype on every query, once per column. On hot QueryRow paths reusing an explicit Prepare or a _stmt_cache_size statement, this is a fixed set of cgo crossings paid on every execution. Cache the names and decltypes on the SQLiteStmt the first time they are materialized and reuse them on subsequent executions. Caching is gated by cacheMetadata(): explicit prepared statements always cache, Query-created ephemeral statements cache only when they live in the connection stmt cache. One-shot statements keep the previous per-call behavior.
cache column metadata for prepared and cached statements
free leaked schema string in GetFilename
Fix race in SQLiteStmt.Close by holding conn lock across cache check
Add CodeRabbit as a sponsor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
aes128cbc,aes256cbc,chacha20,sqlcipher, andrc4._cipher,_key,_kdf_iter,_hmac_check,_hmac_use,_legacy,_legacy_page_size,_mc_legacy_wal,_plaintext_header_size. Cipher PRAGMAs run beforeany schema-touching SQL;
_keyruns last in the cipher block so key derivation triggers after configuration.sqlite3mc-2.2.7), upgraded to MC 2.3.3 (adds thread-safety fix insqlite3mc_cipher_name, symbol-conflict fix when statically linking withlibsodium, and secure clearing of cipher data structures on free).
mattn/masterso the cipher integration coexists with upstream's CGO/perf rewrite, opt-in statement cache (_stmt_cache_size), Go 1.21 floor, and recent bug fixes.Test plan
go build ./...succeedsgo test -run Cipher -v ./...passes (end-to-end cipher integration test insqlite3_cipher_test.go)go test ./...passes (full suite, no regressions on the merged tree)_stmt_cache_size_keylast in the cipher block)sqlite3ext.hentries (SQLite 3.52.0+ function pointers) compile cleanly with the existing build flags🤖 Generated with Claude Code