Highstate from salt-master is very slow unlike local salt-call

[dosercz]

I'm running state.apply from salt-master and it's very slow unlike local salt-call after upgrade to v 2016.11.1. I have there 495 states, 185 sec on salt-master but only 14 sec on salt minion. Why so big difference, where could be a problem? Any ideas? Both servers are in the same network, so there should not be a problem with slow connection.

Salt-master

Salt Version:
           Salt: 2016.11.1

Dependency Versions:
           cffi: 0.8.6
       cherrypy: Not Installed
       dateutil: 2.2
          gitdb: 0.5.4
      gitpython: 0.3.2 RC1
          ioflo: Not Installed
         Jinja2: 2.7.3
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.2
   mysql-python: 1.2.3
      pycparser: 2.10
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.9 (default, Mar  1 2015, 12:57:24)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 14.4.0
           RAET: Not Installed
          smmap: 0.8.2
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5

System Versions:
           dist: debian 8.5
        machine: x86_64
        release: 3.2.0-4-amd64
         system: Linux
        version: debian 8.5

Salt-minion

Salt Version:
           Salt: 2016.11.1

Dependency Versions:
           cffi: 0.8.6
       cherrypy: Not Installed
       dateutil: 2.2
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.3
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.2
   mysql-python: 1.2.3
      pycparser: 2.10
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.9 (default, Mar  1 2015, 12:57:24)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 14.4.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5

System Versions:
           dist: debian 8.5
        machine: x86_64
        release: 3.2.0-4-amd64
         system: Linux
        version: debian 8.5

[Ch3LL]

@dosercz you might be running into this same issue: #38071
If you apply the PR in #38167 does it help with performance?

[dosercz]

I can try it, but do you have some manual, how to apply changes in PR to installed salt? I'm not familiar with python. I have salt installed as debian package.

[bobrik]

I think this is the same as #38758.

[Ch3LL]

@dosercz yes you might be running into #38758 as @bobrik pointed out. Looks like there was a performance hit with file.managed and setting a source in 2016.11. Can you confirm that this is the state thats causing issues. If you don't call a file.managed state do you see an okay performance result?

The PR I did mention might help increase execution time. It helps improve the time it takes the master to publish a command. I would be interested to see if its the file.managed state performance hit that is causing the issue first though.

But to answer your question on how to apply the fix. Its simply a master side edit. So on the master you would just need to add that fix from teh PR. But as stated above I think you actually miht be running into that other issue. Let me know. THanks

[dosercz]

We are using file.managed so much. For minions with hundreds states is the performance difference between local/remote call big. I have tested it on small set of states with/without file.managed, and it could be a same problem probably. Set with file.managed is much slower with remote call than with local call. Set without it has similar times for remote/local call.

[Ch3LL]

@dosercz are you okay closing this issue and tracking it on #38758 ? Since you are using file.managed I bet when this is fixed performance will increase for you

[dosercz]

ok