Vault does not always respond to SIGTERM

[astorath]

When vault binary is making an http request or vault is running in agent mode and performs server call it seems to ignore SIGTERM signal.

Steps to reproduce the behaviour:

1. Run vault status on some firewalled port
2. Press Ctrl+C
3. Program does not quit
   OR
4. Run nohup vault agent -config ...
5. Run kill -SIGTERM <vault_pid>
6. Vault agent is still running

Expected behavior
Vault should abort http request and exit

Environment:

• Vault Server Version (retrieve with vault status): 0.10.4
• Vault CLI Version (retrieve with vault version): 0.10.4
• Server Operating System/Architecture: CentOS 7.4 x64, MacOS 10.13.6

[jefferai]

How long are you waiting after sending the signal? How can one reliably reproduce this?

[vreon]

I can reproduce it with these steps using v0.10.1:

1. In one terminal, mimic an unresponsive Vault server with

   nc -l -p 8080
   

2. In another, run

   VAULT_CLIENT_TIMEOUT=30 VAULT_ADDR=http://localhost:8080 vault status
   

3. In a third terminal, try to kill the vault client:

   ps a | grep vault
   kill -SIGTERM <pid>
   

   (Alternatively, and more realistically, mash Ctrl+C in terminal 2 to send SIGINT)

4. The client will ignore these signals and times out after 30 seconds.

[astorath]

@jefferai,

1. See @vreon answer for vault client,
2. as of vault agent - I'm unable to reproduce it reliably, it happens accidentally (you have to issue SIGTERM while vault agent is performing an http call I think). I've caught this one while testing my ansible deployment script for vault agent in kitchen-test: test run is about ~8 sec normally, but sometimes it takes 3+ minutes. Then I've noticed that systemd is waiting for service to stop (in deactivating) state for 1.5 minutes.

[jefferai]

I thought you were referencing the Vault server initially; right now the CLI doesn't respond to Ctrl-C, only agent and server do.

[astorath]

@jefferai, CLI should respond to Ctrl-C, this is default *nix utils behaviour. But the issue is primary about vault agent - right now we have to restart service with kill -9 <pid> instead of kill -SIGTERM <pid>.

[jefferai]

CLI should respond to Ctrl-C, this is default *nix utils behaviour

I'm not disagreeing, I'm just telling you that currently it does not.

Next time agent is hung, please send a SIGQUIT to get a stack trace and I can see where it's hung. With agent we do cancel a context that the HTTP library uses when you hit Ctrl+C, so if the issue is that context being canceled isn't working immediately there may be nothing we can do as the Go library may still be in the middle of something.

It would also help to know your agent config.

[jefferai]

Please respond once you have a stack trace and I'll reopen.

[astorath]

@jefferai Sorry for late response.

Please see log in the attachment:
vault-agent.v10.4.log

Service shutdown triggered at 09:56:48 (with SIGTERM), and Vault agent responded in log accordingly. SIGQUIT sent at 09:57:13.

Another log (from 0.11.1 vault):
vault-agent.v11.1.log

My config:

pid_file = "/var/run/vault-agent.pid"

auto_auth {
  method "gcp" {
    config = {
      type = "iam"
      role = "test-role"
      // credentials = ""
      service_account = "[email protected]"
      project = "kitchen-test-211905"
    }
  }

  sink "file" {
    config = {
      path = "/tmp/vault/.token"
    }
  }
}

[astorath]

@jefferai Hi, would you reopen the issue?

[jefferai]

Probably fixed via #5507

[astorath]

I see, thanks