Investigate the reason for performance drop in the performance tests

[ljianghedera]

Summary of the defect
All our performance tests are named to do a TPS of 10k. But in the recent days, performance has dropped and could not reach above 8K TPS.

From the stats, secTransH is < 100 micro secs, but as q2 fills up one node falls behind. Need to understand the root cause for this behavior, by profiling the test locally or using 1N_1C in cloud.

How to reproduce (if possible)
Refer to the summary link: https://hedera-hashgraph.slack.com/archives/C018Y4E6ADT/p1618739867028600
and https://hedera-hashgraph.slack.com/archives/C018CBQBTGB/p1615317135306900

or directly: https://hedera-hashgraph.slack.com/archives/CKWHL8R9A/p1618733310331400,
https://hedera-hashgraph.slack.com/archives/CKWHL8R9A/p1618737295333500,
and https://hedera-hashgraph.slack.com/archives/C018CBQBTGB/p1613038049337600

To solve/alleviate this issue, we will need to solve the following (sub) issues and probably more:

• Construct ExpirableTxnRecord directly in transaction context issue Construct ExpirableTxnRecord directly in transaction context #1393;
• Reuse the SignedTxnAccessor from expandSignatures in handleTransaction Reuse the SignedTxnAccessor from expandSignatures in handleTransaction #1421;
• Expire records directly from FCM, not thru transactional proxy Expire records directly from FCM, not thru transactional proxy #1422 ;
• Avoid expensive stream operations when sourcing public key's sig from sigMap Avoid expensive stream operations when sourcing public key's sig from sigMap #1423 ;
• Simplify updates to recent history of a transaction id Simplify updates to recent history of a transaction id #1424;
• Avoid repeated interconversion between Instant and RichInstant Avoid repeated interconversion between Instant and RichInstant #1425 ;
• Consider use of caching factories for gRPC <-> SelfSerializable ids Consider use of caching factories for gRPC <-> SelfSerializable ids #1426

[Neeharika-Sompalli]

While profiling on master (with ledger.keepRecordsInState=true) and release/0.13 branch, one data point observed is heap memory grows more than 90GB.

[Neeharika-Sompalli]

Regarding the path to resolve this issue,

1. All the issues with perf tag needs to be merged.
2. After those are merged, need to profile and investigate if there is still performance improvement that can be done.