Dereferencing causing TypeError at first run

[ripperdoc]

This is a peculiar error that seem to have appeared since I upgraded to 0.10.6. I use a pagination library that calls select_related() on the QuerySet. The first time this happens at each execution, this causes a TypeError from PyMongo. The error disappears at subsequent requests, but re-appears if I restart the process.

Digging into the code, the problem occurs at mongoengine/dereference.py", line 150, in _fetch_objects. The check if hasattr(collection, 'objects') will return False even if it encounters a TopLevelDocumentMetaclass instance, that should have objects. Instead it goes to the else clause, where it assumes that the variable collection is a string. When it calls references = get_db()[collection].find({'_id': {'$in': refs}}) it throws a TypeError from PyMongo as it encounters a class of TopLevelDocumentMetaclass not basestring.

It feels like some race condition, where the attribute objects cannot be found on something that should have it. A safer approach could be to not assume collection variable can be sent to PyMongo without verifying / coercing it to a string.

See code below and stacktrace:

   def _fetch_objects(self, doc_type=None):
        """Fetch all references and convert to their document objects
        """
        object_map = {}
        for collection, dbrefs in self.reference_map.iteritems():
            if hasattr(collection, 'objects'):  # We have a document class for the refs
                col_name = collection._get_collection_name()
                refs = [dbref for dbref in dbrefs
                        if (col_name, dbref) not in object_map]
                references = collection.objects.in_bulk(refs)
                for key, doc in references.iteritems():
                    object_map[(col_name, key)] = doc
            else:  # Generic reference: use the refs data to convert to document
                if isinstance(doc_type, (ListField, DictField, MapField,)):
                    continue

                refs = [dbref for dbref in dbrefs
                        if (collection, dbref) not in object_map]

                if doc_type:
                    references = doc_type._get_db()[collection].find({'_id': {'$in': refs}})
                    for ref in references:
                        doc = doc_type._from_son(ref)
                        object_map[(collection, doc.id)] = doc
                else:
                    references = get_db()[collection].find({'_id': {'$in': refs}})
                    for ref in references:
                        if '_cls' in ref:
                            doc = get_document(ref["_cls"])._from_son(ref)
                        elif doc_type is None:
                            doc = get_document(
                                ''.join(x.capitalize()
                                        for x in collection.split('_')))._from_son(ref)
                        else:
                            doc = doc_type._from_son(ref)
                        object_map[(collection, doc.id)] = doc
        return object_map

Traceback (most recent call last):
 [...]
  File ".../lib/python2.7/site-packages/flask_mongoengine/pagination.py", line 33, in __init__
    self.items = self.items.select_related()
  File ".../lib/python2.7/site-packages/mongoengine/queryset/base.py", line 705, in select_related
    return queryset._dereference(queryset, max_depth=max_depth)
  File ".../lib/python2.7/site-packages/mongoengine/dereference.py", line 77, in __call__
    self.object_map = self._fetch_objects(doc_type=doc_type)
  File ".../lib/python2.7/site-packages/mongoengine/dereference.py", line 150, in _fetch_objects
    references = get_db()[collection].find({'_id': {'$in': refs}})
  File ".../lib/python2.7/site-packages/pymongo/database.py", line 233, in __getitem__
    return Collection(self, name)
  File ".../lib/python2.7/site-packages/pymongo/collection.py", line 144, in __init__
    "of %s" % (string_type.__name__,))
TypeError: name must be an instance of basestring

[wojcikstefan]

Hi @ripperdoc! Thank you for reporting this issue. Could you share what your document's schema is? And does calling select_related() on its query set always fail the first time? If so, are you able to reproduce the problem in a unit test?

[bagerard]

Hi, I've improved something in that part of the code which should allow you to get the actual root cause. In fact under python2 hasattr is swallowing any exceptions that occurs during the hasattr call and return False in case one is thrown. So using getattr is a safer approach since we must be python2/3.

See below:

In [6]: class Test(object):
   ...:     @property
   ...:     def a(self):
   ...:         raise Exception('Do you see me?')
   ...:     

In [7]: t = Test()

In [8]: hasattr(t, 'a')
Out[8]: False
In [9]: getattr(t, 'a')
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
<ipython-input-9-8345b7adbde9> in <module>()
----> 1 getattr(t, 'a')

<ipython-input-6-50dd9f69bde5> in a(self)
      2     @property
      3     def a(self):
----> 4         raise Exception('Do you see me?')
      5 

Exception: Do you see me?

Issue #1688 had a comparable problem (first call to to .select_related() was failing but second was passing), It turned out to be a problem with an index creation error that was swallowed in that case. Please retry your code with the latest master, most likely there is an exception being swallowed and you should now see it

[bagerard]

I'm closing this since it spent a lot of time waiting for a feedback from the author. @ripperdoc Please note that the latest release (0.16.0) includes the fix I mentioned in my last post which should allow you to see the actual error. In case you are still experiencing this with the latest version, provide us additional details and I'll be happy to re-open this ticket