Skip to content

fix Option<Self> in function/method parameters and returns#30

Merged
Tired-Fox merged 1 commit into
Tired-Fox:mainfrom
selaux:option-self
Apr 27, 2026
Merged

fix Option<Self> in function/method parameters and returns#30
Tired-Fox merged 1 commit into
Tired-Fox:mainfrom
selaux:option-self

Conversation

@selaux
Copy link
Copy Markdown
Contributor

@selaux selaux commented Apr 24, 2026

Currently when registering a type that has methods that take or return Option<Self> in a Definition, we get a stack overflow, because it will call Ty::ty() in an infinite recursion loop. This should fix this case.

I also added some tests, but they are a little convoluted, as I couldn't find a really good way to test this. Feel free to make suggestions how to adapt or edit directly.

@Tired-Fox
Copy link
Copy Markdown
Owner

@selaux

Good catch, this was a bit of an oversite on my part and I missed this happening when I added as_param and as_return. The recursion happens as follows:

Assume there is a struct Custom that implements Typed with the ty() function returning a class type and TypedUserData that adds some function that has a param or return type of Option<Self>

  1. Custom::ty()
  2. TypedUserData::add_methods(...)
  3. methods.add_function("test", |_, _: ()| { Ok(Some(Self)) })
  4. Option::<Self>::as_return() <- Recursion happens here based on the impl of this function
  5. Self::ty() <- Recursion

The test may seem like it has a lot of setup, but it seems correct.

Comment thread tests/recursive.rs
@Tired-Fox
Copy link
Copy Markdown
Owner

Tired-Fox commented Apr 24, 2026

I also think it could happen with Arc, Mutex, etc. This would be the case if userdata-wrapper was enabled for those wrapper types.

This PR can focus on fixing the Option case. I can create another PR for the rest

@Tired-Fox Tired-Fox merged commit 4202d91 into Tired-Fox:main Apr 27, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants