App Security Score inconsistencies

[Prehistoic]

ENVIRONMENT

OS and Version: all
Python Version: all
MobSF Version: 3.5.0

EXPLANATION OF THE ISSUE

I'm noticing some inconsistencies in security scores since the release of 3.5.0.

I ran a static analysis on 2 applications and got these results :

With this new security_score formula

the security scores obtained are 39/100 and 36/100. So basically the first one which has way more findings receives a better score ?

I understand the intention to ponder the score based on the ratio of the findings' criticities but that case seems to be a bit extreme.

EDIT :

Another example that shows the pitfalls of this formula :

• if I got just 1 HIGH it means that my security score is 100 - (1 + 0 - 0) * 100 = 100 - 1 * 100 = 0

[github-actions]

👋 @mat42290
Issues is only for reporting a bug/feature request. For limited support, questions, and discussions, please join MobSF Slack channel
Please include all the requested and relevant information when opening a bug report. Improper reports will be closed without any response.

[ajinabraham]

Thanks for brining this up. Do you have any thoughts on alternative implementations?

[Prehistoic]

Honestly I thought that the previous implementation wasn't so bad, that is to say : 100 - X * number_of_highs - Y * number_of_mediums + Z * number_of_secures with some arbitrary X, Y and Z values.

You then capped the score to 10 if it went below that. What about lowering that cap value to 0 or 1 ? I think using the whole range between 0 and 100 would be beneficial and allow to better compare apps with low scores.

I guess you were not satisfied with this implementation because you changed it but in my opinion even if it is pretty basic that's the way to go. I think that any pondering based on criticities will always result in inconsistencies between apps with just a few findings and apps with a lot of findings.

Actually a big selling point of this implementation (imo) is that it is very easy to give a goal to the developers of an app. For example their app got a 20/100. Let's say they aim at getting 70/100 before releasing any app. If they know that fixing 1 High will grant them X points, they can easy plan and prioritize what needs to be done. Following the same logic I think it could be great to advertize about how the score is calculated somewhere in the documentation (or even in the webpage with the scan results) to make it clear for everyone.

EDIT :

There might also be something to do with CVSS scores there. Maybe you could adapt the X and Y values based on the average CVSS scores of the findings of the corresponding criticity.

For example :
Security_score = 100 - X * (average_cvss_for_high_findings/A) * number_of_highs - Y * (average_cvss_for_medium_findings/B) * number_of_mediums + Z * number_of_secures

Where A is the maximum CVSS score possible for High findings (should be 10) and B is the maximum CVSS score possible for Medium findings (depends on where you placed the barrier between high and medium, I think it's 6.9 but haven't checked)

But that might make the formula more complex for no real gain.

[ajinabraham]

Let me take a look at this.

[michaelkyawsan]

May I know, Is this issues solved in latest version?

[devtty1er]

@michaelkyawsan no, unfortunately the scoring system is really just a weighted average. E.g. if you have one high finding, it is "worse" than any number of high findings and some additional medium findings.

@ajinabraham please advise. Would you accept a PR that reverts the scoring to pre-#1881?

mobsf/StaticAnalyzer/views/common/appsec.py

     high = len(findings.get('high'))
     warn = len(findings.get('warning'))
     sec = len(findings.get('secure'))
-    total = high + warn + sec
-    score = 0
-    if total > 0:
-        score = int(100 - (
-            ((high * 1) + (warn * .5) - (sec * .2)) / total) * 100)
+    score = 100 - (high * 15) - (warn * 10) + (sec * 5)
     if score > 100:
         score = 100
+    elif score < 0:
+        score = 10

[johnxguo]

how about this algorithom
first, calculate a "loss score"，e.g. , loss_score = high * 10 + warning * 5 - sec * 2，it‘s value range is (-∞,+∞), but in most case, the range is [0, +∞)
then, normalize the loss score to [-1, 1] or some other range, you can use softmax functions，e.g., sigmoid like functions
at last，map to your score interval
example code:

def sig_like(x):
    e = 2.71828
    return 2 / (1 + pow(e, x / 30))

def loss_score(high, warning, sec):
    return high * 10 + warning * 5  - sec * 2

def score(high, warning, sec):
    loss = loss_score(high, warning, sec)
    return min(sig_like(loss), 1) * 100

print(score(4, 3, 1))

[johnxguo]

how about this algorithom first, calculate a "loss score"，e.g. , loss_score = high * 10 + warning * 5 - sec * 2，it‘s value range is (-∞,+∞), but in most case, the range is [0, +∞) then, normalize the loss score to [-1, 1] or some other range, you can use softmax functions，e.g., sigmoid like functions at last，map to your score interval example code:

def sig_like(x):
    e = 2.71828
    return 2 / (1 + pow(e, x / 30))

def loss_score(high, warning, sec):
    return high * 10 + warning * 5  - sec * 2

def score(high, warning, sec):
    loss = loss_score(high, warning, sec)
    return min(sig_like(loss), 1) * 100

print(score(4, 3, 1))

@ajinabraham

[ajinabraham]

Do you want to send a PR?

[johnxguo]

Do you want to send a PR?

#2311 @ajinabraham

[ajinabraham]

Thanks, I will test this out.

[sumit-jha-Pw]

I am also facing a similar issue where the score decreases even after improved results.