Support more projection operators

[rozza]

Doc.objects(myList=1).fields(myList__S=1)

Refs: #525

[geuben]

+1

This would be very useful

[wojcikstefan]

Our support for projection operators is a bit of a mess right now... It's not cohesive nor documented well.

https://docs.mongodb.com/manual/reference/operator/projection/ specifies 4 projection operators as of v3.4: $, $elemMatch, $slice, $meta. $meta is only relevant when doing a $text search and doesn't have an easy fix, because we'd also have to support sorting by a field that doesn't exist on a document (e.g. an equivalent of db.articles.find({ $text: { $search: "cake" } }, { score: { $meta: "textScore" } }). That leaves us with $, $elemMatch and $slice...

$elemMatch and $slice can already be supplied as kwargs values to BaseQuerySet.fields(...). For example:

In [46]: class EmbeddedDoc(EmbeddedDocument):
    text = StringField()
   ....:

In [47]: class Doc(Document):
    nums = ListField(IntField())
    embedded_docs = ListField(EmbeddedDocumentField(EmbeddedDoc))
   ....:

In [48]: Doc.drop_collection()

In [49]: Doc.objects.create(nums=[1, 2, 3, 4, 5], embedded_docs=[EmbeddedDoc(text='aaa'), EmbeddedDoc(text='bbb')])
Out[49]: <Doc: Doc object>

In [50]: doc = Doc.objects.fields(nums={'$slice': 2}, embedded_docs={'$elemMatch': {'text': 'bbb'}}).first()

In [51]: doc.nums
Out[51]: [1, 2]

In [52]: doc.embedded_docs
Out[52]: [<EmbeddedDoc: EmbeddedDoc object>]

In [53]: doc.embedded_docs[0].text
Out[53]: u'bbb'

On top of that, there's already some custom code for dealing with slices which allows for a more MongoEngine-like notation:

In [54]: Doc.objects.fields(slice__nums=2).first().nums
Out[54]: [1, 2]

That being said, I don't like its implementation. It seems confusing that it's implemented as a prefix whereas in MongoDB/PyMongo $slice is used as a suffix (e.g. {comments: {$slice: 1}} and not {$slice: {comments: 1}}). This way it's less readable in e.g. this sample scenario:

In [73]: OtherDoc.objects.first().embedded_docs[0].letters
Out[73]: [u'a', u'b', u'c', u'd', u'e']

In [74]: OtherDoc.objects.fields(slice__embedded_docs__letters=2).first().embedded_docs[0].letters
Out[74]: [u'a', u'b']

I'd much rather either re-implement it as a suffix (e.g. .fields(embedded_docs__letters__slice=2)) or get rid of it completely and use the PyMongo way (e.g. .fields(embedded_docs__letters={'$slice': 2})).

I'm even more conflicted though because the PyMongo way doesn't allow us to use the $ projection easily (e.g. db.students.find({ semester: 1, grades: { $gte: 85 } }, { "grades.$": 1 })). We'd have to do a somewhat hacky-looking Student.objects.filter(grades__gte=85).fields(**{'grades.$': 1}).

Guess the alternative would be to re-implement slice as a suffix and support MongoEngine-like syntax for all operators (besides $meta for now):

Doc.objects.fields(embedded_docs__letters__slice=2)  # transforms it into {'embedded_docs.letters': {$slice: 2}}
Doc.objects.fields(embedded_docs__match={'name': 'aaa'})  # transforms it into {embedded_docs: {$elemMatch: {name: 'aaa'}}}
Doc.objects.filter(nums=2).fields(nums__S=1)  # transforms it into {'nums.$': 1}

I just hate this because it adds bloat that doesn't provide any extra functionality on top of what PyMongo already provides... It also has to be maintained, tested, etc., and when MongoDB adds an operator, we need to add it, too. We already do something like this in many other places (update operators, for instance), but is that strong enough of an argument to just keep doing it?

What do you think @JohnAD @lafrech @gukoff ?

[KeironO]

Was this ever implemented?

[StoneMoe]

Any update on this?