remote execute is very slow

[luochun-95]

Description of Issue/Question

master : mass_server10
minion : mass_server13

execute test,ping in mass_server10, very very slow

[root@mass_server10 ~]# time salt "mass_server13" test.ping
mass_server13:
    True

real    0m1.088s
user    0m0.763s
sys     0m0.103s

[root@mass_server10 ~]# time salt "mass_server13" test.ping
mass_server13:
    True

real    0m1.071s
user    0m0.770s
sys     0m0.089s
[root@mass_server10 ~]# 

[root@mass_server10 ~]# ping mass_server13
PING mass_server13 (192.168.100.209) 56(84) bytes of data.
64 bytes from mass_server13 (192.168.100.209): icmp_seq=1 ttl=64 time=0.257 ms
64 bytes from mass_server13 (192.168.100.209): icmp_seq=2 ttl=64 time=0.171 ms
64 bytes from mass_server13 (192.168.100.209): icmp_seq=3 ttl=64 time=0.121 ms
64 bytes from mass_server13 (192.168.100.209): icmp_seq=4 ttl=64 time=0.238 ms
^C
--- mass_server13 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3000ms
rtt min/avg/max/mdev = 0.121/0.196/0.257/0.056 ms
[root@mass_server10 ~]# 

Setup

(Please provide relevant configs and/or SLS files (Be sure to remove sensitive info).)

master:
[root@mass_server10 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.100.209 mass_server13
192.168.100.123 mass_server10
[root@mass_server10 ~]# 

minion:

[root@localhost network-scripts]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.100.209 mass_server13
192.168.100.123 mass_server10
[root@localhost network-scripts]# 

[root@localhost salt]# cat minion_id 
mass_server13
[root@localhost salt]# 

the minion config

# Set the location of the salt master server. If the master server cannot be
# resolved, then the minion will fail to start.
#master: salt
master: 192.168.100.123

Steps to Reproduce Issue

(Include debug logs if possible and relevant.)

Versions Report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

master:
[root@mass_server10 ~]# salt --versions-report
Salt Version:
           Salt: 2016.11.0
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: 3.2.2
       dateutil: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: 1.2.5
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.5 (default, Sep 15 2016, 22:37:39)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4
 
System Versions:
           dist: centos 7.2.1511 Core
        machine: x86_64
        release: 3.10.0-327.el7.x86_64
         system: Linux
        version: CentOS Linux 7.2.1511 Core
 
[root@mass_server10 ~]# 

minion:

[root@localhost salt]# salt --versions-report
Salt Version:
           Salt: 2016.11.0
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: 3.2.2
       dateutil: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: 1.2.3
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.5 (default, Nov 20 2015, 02:00:19)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 14.3.1
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 3.2.5
 
System Versions:
           dist: centos 7.2.1511 Core
        machine: x86_64
        release: 3.10.0-327.el7.x86_64
         system: Linux
        version: CentOS Linux 7.2.1511 Core
 
[root@localhost salt]# 

[Ch3LL]

@luochun-95 okay so what I found is 2016.3.4/2016.11.0 have slowed down since 2015.8.10 is where i tested from. The test I setup is just a local minion to the box so a little different from your setup.

[root@7f560ab1f270 testing]# time salt '*' test.ping; salt --version
7f560ab1f270:
    True

real    0m0.427s
user    0m0.324s
sys     0m0.072s
salt 2015.8.10 (Beryllium)

[root@7f560ab1f270 testing]# time salt '*' test.ping; salt --version
7f560ab1f270:
    True

real    0m0.523s
user    0m0.355s
sys     0m0.064s
salt 2016.3.4 (Boron)

[root@7f560ab1f270 testing]# time salt '*' test.ping; salt --version
7f560ab1f270:
    True

real    0m0.527s
user    0m0.360s
sys     0m0.072s
salt 2016.11.0 (Carbon)

As shown above 2015 branch is indeed faster. We will need to get this fixed up . Thanks!

[cachedout]

@luochun-95 If you apply the fix in #38167 do things improve for you?

[damntoken]

Win2016 and salt 2018.3, still having same issue.

[Ch3LL]

@damntoken can you open a new issue with the details?

[damntoken]

Sorry, done.
#48882