Help need: aws_s3 sink can't read the credential file.

[waltzbucks]

Hello, I've leveraging vectordotdev in order to syslog collecting then putting in the S3 Bucket.

Vector installed version

vector 0.42.0 (x86_64-unknown-linux-gnu 3d16e34 2024-10-21 14:10:14.375255220)

But. I ran into problem regard of file read permission error.

I've configuration like this:

sinks:
  s3-syslog:
    healthcheck:
      enable: false
    inputs:
      - "syslog-server"
    type: "aws_s3"
    auth:
      credentials_file: "/home/ubuntu/.aws/credentials"
      profile: "default"

    bucket: "syslog-bucket"
    region: "ap-northeast-2"
    batch:
      timeout_secs: 30
    buffer:
      type: "memory"
      max_events: 1000
      when_full: "block"
    compression: "gzip"
    content_encoding: "gzip"
    encoding:
      codec: "raw_message"
    server_side_encryption: "AES256"
    timezone: "Asia/Seoul"
    tls:
      verify_certificate: false

It works when I testing to this config through the CMD vector -v. But has not working when I let run with systemctl in order to want to be work in background.

× vector.service - Vector
     Loaded: loaded (/lib/systemd/system/vector.service; disabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Tue 2024-11-26 16:23:44 KST; 5s ago
       Docs: https://vector.dev
    Process: 10308 ExecStartPre=/usr/bin/vector validate (code=exited, status=78)
        CPU: 110ms

Nov 26 16:23:44 HOSTNAME systemd[1]: vector.service: Scheduled restart job, restart counter is at 5.
Nov 26 16:23:44 HOSTNAME systemd[1]: Stopped Vector.
Nov 26 16:23:44 HOSTNAME systemd[1]: vector.service: Start request repeated too quickly.
Nov 26 16:23:44 HOSTNAME systemd[1]: vector.service: Failed with result 'exit-code'.
Nov 26 16:23:44 HOSTNAME systemd[1]: Failed to start Vector.

And I got this system log:

Nov 26 16:25:29 HOSTNAME systemd[1]: Starting Vector...
Nov 26 16:25:29 HOSTNAME vector[10417]: √ Loaded ["/etc/vector/vector.yaml"]
Nov 26 16:25:29 HOSTNAME vector[10417]: 2024-11-26T07:25:29.487089Z  WARN sink{component_kind="sink" component_id=s3-syslog component_type=aws_s3}: vector_core::tls::settings: The `verify_certificate` option is DISABLED, this may lead to security vulnerabilities.
Nov 26 16:25:29 HOSTNAME vector[10417]: 2024-11-26T07:25:29.521580Z  WARN sink{component_kind="sink" component_id=s3-syslog component_type=aws_s3}: vector_core::tls::settings: The `verify_certificate` option is DISABLED, this may lead to security vulnerabilities.
Nov 26 16:25:29 HOSTNAME vector[10417]: √ Component configuration
Nov 26 16:25:29 HOSTNAME vector[10417]: 2024-11-26T07:25:29.557294Z  WARN aws_config::provider_config: failed to parse profile err=could not read file `/home/ubuntu/.aws/credentials`: Permission denied (os error 13) (CouldNotReadFile(CouldNotReadProfileFile { path: "/home/ubuntu/.aws/credentials", cause: Os { code: 13, kind: PermissionDenied, message: "Permission denied" } }))
Nov 26 16:25:29 HOSTNAME vector[10417]: 2024-11-26T07:25:29.557366Z ERROR vector::topology::builder: msg="Healthcheck failed." error=dispatch failure component_kind="sink" component_type="aws_s3" component_id=s3-syslog
Nov 26 16:25:29 HOSTNAME vector[10417]: x Health check for "s3-syslog" failed: dispatch failure
Nov 26 16:25:29 HOSTNAME vector[10417]: √ Health check "console-out"
Nov 26 16:25:29 HOSTNAME systemd[1]: vector.service: Control process exited, code=exited, status=78/CONFIG
Nov 26 16:25:29 HOSTNAME systemd[1]: vector.service: Failed with result 'exit-code'.
Nov 26 16:25:29 HOSTNAME systemd[1]: Failed to start Vector.
Nov 26 16:25:29 HOSTNAME systemd[1]: vector.service: Scheduled restart job, restart counter is at 1.
Nov 26 16:25:29 HOSTNAME systemd[1]: Stopped Vector.

I though that just adjust permission of credential file. Thus changed permission to maximum 777. But still not working with same error.
yes! tried that changed to the file ownership perform as the vector user. but same this occur.

[pront]

changed permission to maximum 777. But still not working with same error.

This is strange. Maybe worth trying copying the file over to /etc/vector/.aws/credentials.

Did you change the User and/or Group in your service file? You can also do a ps aux | grep vector.

[jszwedko]

This does indeed appear to be a filesystem permissions issue. Make sure that the vector user is able to read the file. You can test with something like sudo -u vector cat /home/ubuntu/.aws/credentials.

[waltzbucks]

@pront @jszwedko

Thanks for help.

In my linux systems has been configured that home directories to restrict drwx------ each other by any others.
Although, more loosen the permission to credential file owned home directory, vector works fine.
literally, this is my system permission problem.

Very thanks y'all did effort to help me