Skip to content

Add (semi)automated integration testing#47

Open
stan1t0 wants to merge 4 commits into
jdjaxon:mainfrom
stan1t0:testing
Open

Add (semi)automated integration testing#47
stan1t0 wants to merge 4 commits into
jdjaxon:mainfrom
stan1t0:testing

Conversation

@stan1t0

@stan1t0 stan1t0 commented May 3, 2026

Copy link
Copy Markdown
Contributor

CI testing would be ideal but I found most approaches to be cumbersome given the direct involvement of systemd. The Vagrant approach in this PR accomplishes more-or-less automated testing by running the script on Ubuntu 24.04/Debian 13 instances and calling BATS tests to determine that

  • the script ran successfully
  • pcscd socket is active
  • pkcs11 module registered in ff/chrome profiles
  • dod certs got imported

There is probably a better way to isolate re-running the script on a provisioned system's clean state in the Vagrantfile but I don't yet know enough ruby/vagrant to pull that off so a typical workflow is:

vagrant up
# box A spins up
# script runs
# test runs
# box B spins up
# script runs
# test runs
vagrant destroy

In case of a failure in a test, vagrant process quits but it's possible to

vagrant ssh <box-name>

because provisioned boxes continue running and determine exactly what happened but once fixed the entire procedure has to repeat to ensure a clean state.


On a separate note, during testing I found that pkcs11-register is really only useful for apt-installed Firefox (and even then there are annoying interface differences between versions in upstream Ubuntu and Debian repos) and decided to switch entirely to modutil for module registration (which also proved cleaner as it was already being used for snapped FF).

stan1t0 added 4 commits May 2, 2026 19:31
- version 0.26.1 defaults to skipping both firefox and chrome
- search paths for firefox profiles are hardcoded relative to the
  running user's home directory
Currently tests are executed on:

- Ubuntu 24.04
- Debian 13

targeting only Firefox.
Versions of the tool differ between Ubuntu 24.04 and Debian 13 with
slight modifications to the interface, necessitating conditional checks
to determine how to correctly call it. Additionally, it proved useless
for Chrome because of a hard-coded profile path which doesn't meet the
reality of where Chrome actually stores its profile.

In the end, it made sense to use modutil to register the cac module
across all identified profiles, to include Chrome and both versions of
Firefox.
@jdjaxon

jdjaxon commented May 3, 2026

Copy link
Copy Markdown
Owner

I completely agree. I really appreciate your work on this!

I'll start by saying that I'm happy to integrate this. I looked at using Vagrant in the past. The only thing that dissuaded me was that most user issues stem from some quirk of a service or browser, which has alway required interactive troubleshooting since you need a CAC. I've had multiple people successfully configure their machine with the script, but 1 out of 10 will have some issue where the browser doesn't properly query the cert DB or request the pin as expected. To remediate, we would run specific steps manually to rebuild where I thought the issue occurred. A CI test would indicate a successful configuration, but in those specific cases, the process failed.

That being said, some testing is always better than none, so thank you again.

pkcs11-register has previously had some odd behavior anyways (can be seen in older issues), so moving away from it is probably best.

I will review and merge unless I have a question. Please be patient with me as I have a newborn. I'll try to get it merged as soon as I have free time.

@stan1t0

stan1t0 commented May 3, 2026

Copy link
Copy Markdown
Contributor Author

Congrats! I completely understand and it's no rush at all :)

There is just too much variation in desktop configuration to definitively account for all edge cases. Perhaps for another weekend, I was thinking of adding options for dry run and specifying profile paths to modify explicitly if desirable. Also I'm curious if there would be any benefit in maintaining a deb package for the CA certs (e.g., following the example in /usr/share/doc/ca-certificates/examples/ca-certificates-local/README) to allow easy system-wide installation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants