Document references / cross-collection population

[wbrickner]

Ever intrepid, I am attempting to use Wither in production (😄 ).

I'm familiar with NodeJS's Mongoose, in which you can define models like (pseudocode)

Product {
  creator { type: ObjectId, reference:  "User" }
}

Later, you can run a query, the result of which is either a product or a list of products, etc., and you can .populate("creator"), which transforms the creator field from an ObjectId into a proper document from the Users collection.

This seems tricky in the Rust type system, but I am hoping this feature is available.

Thank you.

[thedodd]

in production

Woot woot! Becoming more and more common place these days, but kudos on staying intrepid!

Wither does not currently offer any convenience mechanism for joins. Though doing so is still possible as you have direct access to the driver. The way you would normally do this mongo would be to perform two separate queries, and then join application side. First query to get your products, second query to get your users using the IDs of the referenced users and the $in query constraint.

So, short answer is no, we do not have a convenience mechanism for that. Long answer, you can do it by hand without too much trouble.

Keep me posted.

[simoneromano96]

Hi, maybe we could work on it via a new aggregation system?
This feature is also something that could be useful for me.

[thedodd]

I dig it. We will need to put a design together on what we would like to see for Wither aggregations. Glorified wrapper around $lookup, or something more sophisticated.

[wbrickner]

Thought: could the proc macro that derives Model trait accept information about document references? This could be used to solve the type system issue in a very efficient and ergonomic way, where the proc macro also defines new types that are returned from different methods.

For example in pseudocode

#derive Model
User { ... }

#derive Model
Product {
   #[wither(reference = User)]
   pub creator: ObjectId
}

Which then under the hood generates some traits and types something like

struct MaybePopulatedProduct<P: CreatorTrait> {
  pub creator: P
}

where P can be either ObjectId or User document or something. This is to avoid dynamic dispatch in trait objects, bad ergonomics in returning 1000 documents with an enum variant, etc. The idea is that type would be returned from a product.populate(&[Product::PopulatableFields]) method (a const associated enum would also be generated if that's possible in stable rust yet, I forget).

This is nice because everything is knowable at compile time via the type system, property names are not strings they are a few bytes etc, and everything tricky is handled by the proc macro system, abstracted away from the user.

Thoughts?

[thedodd]

@wbrickner that's an excellent idea, and I think we've got a lot of potential there. We can generate pretty much any code we want to from the derive macro. I refactored the macro a little over a year ago, and it is pretty approachable / pretty well factored right now (IMHO), so adding some new codegen shouldn't be too crazy.

We could just expose a library level enum, say RefOrObject generic over two types. Then in the codegen, we just materialize the types to be the reference type declared on the field, or the type declared in the #[wither(reference = T)] attr. We could add some additional customization as when/how Wither attempts to join the data. Perhaps we update some of the model methods to take some wither specific config to control this behavior.

Thoughts?

[wbrickner]

I don't like that each object now has to have an enum variant, because in reality the RefOrObject object variant will have to be an Option for instance, and actually using that data after will be less elegant than if we had baked the type of field into the return type (e.g. ObjectId for unpopulated references and Option for documents that have gone through the population machinery). This is a type system win as well, because you will call populate and get an Option back, guaranteed. Except for the case in which you have 2 referential fields and populate one of them.

I wonder, is it possible to accept a const generic that is an enum variant / collection of enum variants yet? It may be asking too much of the type system (yet) for it to be able to know that when I request population of ReferentialFields::Creator, that the creator field has been retyped to Option, when I use ReferentialFields::StoreLocation it knows the store field is an Option now, and when I use both they're both retyped etc.

In a perfect world we could find a way to accept a variadic set of constant generic enum values that then determine the field types, and the type system would catch you red handed if you were performing unsound database interactions.

A downside of this could be a loss of flexibility, let's say I have a user model which I pass into a function to mint a PASETO/JWT token or something. That function expects a User struct, not a ManHandledByTheTypeSystemUserStruct<X, Y, Z, ...> generic mess, so I couldn't use it directly if I had used population machinery. So now I can either duplicate the function, make it generic, or have it accept field values instead of a struct, etc. Kindof a mess.

[somehowchris]

Any news on that? :)

[Akronae]

Interested too, seems to me that population is a very basic feature for an ORM