Limit Request Size?

[cristianocoelho]

Django uses the settings DATA_UPLOAD_MAX_NUMBER_FIELDS and DATA_UPLOAD_MAX_MEMORY_SIZE to help against denial of service of large suspicious requests.

Django-Rest-Framework's parsers do not honor DATA_UPLOAD_MAX_MEMORY_SIZE setting in any way since it never uses request.body directly but instead request.read(). DATA_UPLOAD_MAX_NUMBER_FIELDS is probably not honored as well but this one is not that important.

Since the setting is not honored, a valid huge json request can cause an unhandled MemoryError on the parsers which is quite ugly.

I'm wondering if there's an easy way to honor this flag, or should we manually write/override the existing parsers to check this flag?

[tomchristie]

We might be able to wrap the stream reader in an interface that ensures DATA_UPLOAD_MAX_MEMORY_SIZE is honoured.

[cristianocoelho]

The issue is that the checks depend on the content-type/parser used, for example, for file uploads we don't want to use this flag, this involves different checks on the multi part and file uploads parsers.

A very easy fix however, since the out of memory and high cpu usage most of the time will come from the most used parser, the json parser, and at some extent the form parser, is adding a check there, such as:

def check_content_length(parser_context):

    if parser_context and DATA_UPLOAD_MAX_MEMORY_SIZE and 'request' in parser_context:
        
        try:
            content_length = int(parser_context['request'].META.get('CONTENT_LENGTH', 0))
        except (ValueError, TypeError):
            content_length = 0

        if content_length and content_length > DATA_UPLOAD_MAX_MEMORY_SIZE or content_length < 0:
            raise ParseError('Form parse error - Invalid Content')

class LimitedJSONParser(parsers.JSONParser):
    """
    Parses JSON-serialized data.
    This json parser won't allow large file uploads through base64 encoded strings. Use multipart instead
    """
    media_type = 'application/json'
    renderer_class = renderers.JSONRenderer

    def parse(self, stream, media_type=None, parser_context=None):
        """
        Parses the incoming bytestream as JSON and returns the resulting data.
        """
        
        check_content_length(parser_context)     

        return super(LimitedJSONParser, self).parse(stream, media_type, parser_context)

class LimitedFormParser(parsers.FormParser):
    """
    Parser for form data.
    This parser won't allow large file uploads through urlencoded encoded strings. Use multipart instead
    """
    media_type = 'application/x-www-form-urlencoded'

    def parse(self, stream, media_type=None, parser_context=None):
        """
        Parses the incoming bytestream as a URL encoded form,
        and returns the resulting QueryDict.
        """
        
        check_content_length(parser_context)

        return super(LimitedFormParser, self).parse(stream, media_type, parser_context)

At least for now, those are the checks I'm thinking that would prevent 90% of possible attackers attempting to send large requests to crash the server. Multi-part requests would still fail, or maybe not, that depends if the call to the django multi-part parser honor the settings values.
Also of course this wouldn't work for chunked / forged content-type requests but not even django solves that, this is just a mimic of what django does. Thoughts?

[kevin-brown]

It looks like DATA_UPLOAD_MAX_MEMORY_SIZE is handled on the request level when HttpRequest.body is used. The only issue right now is when the request body has not been read in, in which case we pass the raw request to the parser to be used, and the checks get skipped.

Multi-part requests would still fail, or maybe not, that depends if the call to the django multi-part parser honor the settings values.

I'm pretty sure Django still handles parsing those, so they should already be honoring the DATA_UPLOAD_MAX_NUMBER_FIELDS setting. Same thing with the DATA_UPLOAD_MAX_MEMORY_SIZE when using multipart requests, as Django also handles validation within there.

[jsj14]

I am confused as to why when I was using django 1.11.23 , I was able to handle payloads of size > 2.5MB without adding any such limit in settings.py but when I upgraded to 2.2.6 I am forced to add a
DATA_UPLOAD_MAX_MEMORY_SIZE = 1024*1024*15 to handle the large payloads.