feat: Add request rate limiter based on IP address

[dblythy]

New Pull Request Checklist

• I am not disclosing a vulnerability.
• I am creating this PR in reference to an issue.

Issue Description

Parse Server leaves rate limiting to the developer. As Parse is built for ease of use, internal rate limiting should be optional.

Related issue: #8170
Closes: #8170

Approach

Adds rate limiting via Parse Server config option:

rateLimit: [
        {
          path: '/functions/*',
          windowMs: 10000,
          max: 1,
          message: 'Too many requests. Please try again later',
        },
      ],

However, I thought this still might be complex for new users who aren't sure how Parse Server mounts it's routes. So, I also added the ability to rate limit via a cloud validator:

Parse.Cloud.define('test', () => 'Abc', {
    rateLimit: {
    windowMs: 10000,
    max: 1,
    message: 'Too many requests. Please try again later',
  },
});

TODOs before merging

• Add tests
• A changelog entry is created automatically using the pull request title (do not manually add a changelog entry)

[parse-github-assistant]

Thanks for opening this pull request!

• 🎉 We are excited about your hands-on contribution!

[codecov]

Codecov Report

Base: 94.32% // Head: 87.41% // Decreases project coverage by -6.91% ⚠️

Coverage data is based on head (56bcbe7) compared to base (a4990dc).
Patch coverage: 95.53% of modified lines in pull request are covered.

❗ Current head 56bcbe7 differs from pull request most recent head 160706c. Consider uploading reports for the commit 160706c to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##            alpha    #8174      +/-   ##
==========================================
- Coverage   94.32%   87.41%   -6.92%     
==========================================
  Files         181      181              
  Lines       14266    14357      +91     
==========================================
- Hits        13457    12550     -907     
- Misses        809     1807     +998     

Impacted Files	Coverage Δ
src/Options/index.js	100.00% <ø> (ø)
src/Routers/FilesRouter.js	87.07% <ø> (-5.45%)	⬇️
src/middlewares.js	96.69% <93.44%> (-0.26%)	⬇️
src/Config.js	90.32% <96.00%> (+0.49%)	⬆️
src/GraphQL/ParseGraphQLServer.js	93.87% <100.00%> (+0.12%)	⬆️
src/Options/Definitions.js	100.00% <100.00%> (ø)
src/ParseServer.js	91.85% <100.00%> (-0.32%)	⬇️
src/cloud-code/Parse.Cloud.js	99.28% <100.00%> (+0.10%)	⬆️
src/Adapters/Storage/Mongo/MongoCollection.js	6.97% <0.00%> (-90.70%)	⬇️
src/Adapters/Files/GridFSBucketAdapter.js	10.14% <0.00%> (-84.06%)	⬇️
... and 26 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[mtrezza]

Well thought to add the ability to exclude the master key form rate limiting.

I think we need to document this feature well, either in the README or in a chapter in the docs:

• Does rate limiting also apply to Parse Server internal calls?
• Does rate limiting also apply to operations executed in Cloud Code, or only external requests by a client?
• Give config examples how to use this
• It seems the default values for the config options are missing in the docs

Is it possible to specify different values for different routes? That may be a very practical scenario, because not every route has the same performance impact. Currently is seems that only a single rate limit can be applies to a set of routes. That makes the effective rate limit a function of the number of routes, which seems unintuitive because the server ressources don't scale with the number of routes.

[dblythy]

Is it possible to specify different values for different routes

Yes, the option is an array and can specify specific routes if required.

rateLimitOptions: [
 {
    path: '/functions/*',
    windowMs: 100000,
    max: 50,
    message: 'Too many requests. Please try again later',
  },
  {
    path: '/functions/test',
    windowMs: 1000,
    max: 3,
    message: 'Rate limit specific to test,
  },
],

In this case all cloud functions will be restricted to 50 calls, whereas the function test will be restricted to 3.

[dblythy]

This also moves session lookup into its own middleware so that the rate limiter is performed before any database ops are ran

[dblythy]

Thank you for the thorough review @mtrezza :)

[dblythy]

I've just read that express-rate-limit doesn't support clusters either.

Perhaps rate limit flexible is a better option? Also supports using mongo/postgres as a backing store, and cluster mode.

Also has a block strategy against DDOS

[mtrezza]

I've just read that express-rate-limit doesn't support clusters either.

should also be added to the option docs

[dblythy]

Not really sure why the test to validate the config options is failing on Postgres with

Unhandled promise rejection: error: duplicate key value violates unique constraint "pg_class_relname_nsp_index" (line 534)

[dblythy]

Future evolution:

• add more options to rate limit options
• add redis memory store to allow for clusters / load balanced environments
• add "zone" where rate limit can be restricted by sessionToken (easy to implement, inexpensive) or user Id (requires a DB operation), or IP

[mtrezza]

Is this ready for review or still WIP?

[mtrezza]

Looks good!

[parseplatformorg]

🎉 This change has been released in version 6.0.0-alpha.21

[cbaker6]

@dblythy I attempted to enable the rate limiter, but I couldn't get it to allow any connections. Definitely possible I'm configuring or understanding how to use it incorrectly, though the server seems to like my configuration. I didn't want to open an issue yet, as it could be a config problem on my side. I attempted to try similar configurations to your tests. Below is what I try (you can see the whole file here:

// Use default rate limiter values for PARSE_SERVER_RATE_LIMIT_REQUEST_PATH = '*'
// Configure Parse Server using startApp
const parseServer = await ParseServer.startApp(configuration);

Have you gotten this to work on docker or another setup outside of the test environment? If so, can you share relevant parts of your config?

[mtrezza]

@cbaker6 Please use the community forum or slack for code level support to prevent fragmented discussions.

[dblythy]

@cbaker6 are you saying that the rate limiter is rejecting all requests? It could possibly be related to trust proxy:

Troubleshooting Proxy Issues

If you are behind a proxy/load balancer (usually the case with most hosting services, e.g. Heroku, Bluemix, AWS ELB, Nginx, Cloudflare, Akamai, Fastly, Firebase Hosting, Rackspace LB, Riverbed Stingray, etc.), the IP address of the request might be the IP of the load balancer/reverse proxy (making the rate limiter effectively a global one and blocking all requests once the limit is reached) or undefined. To solve this issue, add the following line to your code (right after you create the express application):

app.set('trust proxy', numberOfProxies)
Where numberOfProxies is the number of proxies between the user and the server. To find the correct number, create a test endpoint that returns the client IP:

app.set('trust proxy', 1)
app.get('/ip', (request, response) => response.send(request.ip))
Go to /ip and see the IP address returned in the response. If it matches your public IP address, then the number of proxies is correct and the rate limiter should now work correctly. If not, then keep increasing the number until it does.

https://github.com/express-rate-limit/express-rate-limit#troubleshooting-proxy-issues

[dblythy]

@cbaker6 I'm thinking you may have observed #8399

[cbaker6]

Definitely possible, I'll test your new PR once it's merged and let you know if it fixed the problem

[parseplatformorg]

🎉 This change has been released in version 6.0.0-beta.1

[parseplatformorg]

🎉 This change has been released in version 6.0.0