podman + sam locally stuck on "Testing application behaviour against authorizers..." step #24839

gminova · 2024-12-13T19:28:03Z

Issue Description

Hello,

I have been trying to set up podman and sam to run a serverless app backend locally. When I use docker it works normally and my lambda functions mount, however when running podman sam gets stuck on this "Testing application behaviour..."step without proceeding to mount my functions. The same issue was observed by other members of our team.

My device: Apple M1 Pro
sam version: SAM CLI, version 1.132.0
podman: podman version 5.3.1
Docker cli: Docker version 27.3.1, build ce1223035a
Docker-compose: Docker Compose version 2.31.0

For context,
My aws config is set up correctly.
Podman is running and my containers are mapping to the right ports, I am able to ping them with curl.
I have also tried setting my DOCKER_HOST path as here

docker context create podman --docker "host=unix://$HOME.local/share/containers/podman/machine/podman.sock"

and then running my api after sam build has been successful:

SAM_CLI_CONTAINER_RUNTIME=podman sam local start-api --env-vars .env.json --parameter-overrides EnvironmentClientIdKey=local

However, sam remains hanging indefinitely not mounting functions

sam local start-api --env-vars .env.json --parameter-overrides EnvironmentClientIdKey=local 
No current session found, using default AWS::AccountId                                                 
                                                                                                       
AWS SAM CLI does not guarantee 100% fidelity between authorizers locally                               
and authorizers deployed on AWS. Any application critical behavior should                              
be validated thoroughly before deploying to production.                                                
                                                                                                       
Testing application behaviour against authorizers deployed on AWS can be done using the sam sync       
command.

Running simply sam local start-api or sam local start-api --env-vars .env.json --parameter-overrides EnvironmentClientIdKey=local also gets stuck without mounting lambda functions.

I originally reported it to the sam repo here and then found another person reporting the issue on the sam repo here and a workaround here

However, since sam works seamlessly with docker, I wonder if there is anything that could be changed on podman to avoid this issue. Thank you.

Steps to reproduce the issue

sam build
podman compose up
sam local start-api --env-vars .env.json

Describe the results you received

sam hangs without mounting functions

sam local start-api --env-vars .env.json --parameter-overrides EnvironmentClientIdKey=local 
No current session found, using default AWS::AccountId                                                 
                                                                                                       
AWS SAM CLI does not guarantee 100% fidelity between authorizers locally                               
and authorizers deployed on AWS. Any application critical behavior should                              
be validated thoroughly before deploying to production.                                                
                                                                                                       
Testing application behaviour against authorizers deployed on AWS can be done using the sam sync       
command.

Describe the results you expected

I would expect functions to mount and to be able to ping them

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

The text was updated successfully, but these errors were encountered:

baude · 2024-12-13T19:32:27Z

does it work rootfully with podman?

gminova · 2024-12-13T19:34:53Z

@baude no it does not work, I am admin on my machine

gminova · 2024-12-13T19:40:17Z

@baude Since I am admin on my machine I assume it runs as root?
Also tried using the explicit flag suggested by copilot

podman machine start --privileged
Error: unknown flag: --privileged
See 'podman machine start --help'

but got an error.
I tried this too

podman compose up --privileged
>>>> Executing external compose provider "/opt/homebrew/bin/docker-compose". Please see podman-compose(1) for how to disable this message. <<<<

unknown flag: --privileged
Error: executing /opt/homebrew/bin/docker-compose up --privileged: exit status 16

mheon · 2024-12-13T19:43:47Z

Not knowing what the AWS CLI is doing exactly, it's hard to say what's going wrong. Switching the machine to rootful with podman machine set --rootful as @baude suggested might help. If that doesn't work, you could try pulling trace-level logs by SSHing into the machine VM, stopping the podman.service API, and running podman system service -t 0 --log-level=trace to see if anything pops out in the logs, but that's going to be looking for a needle in a haystack - we really need to know what exactly the SAM tool is doing that fails to debug effectively.

gminova · 2024-12-13T19:50:04Z

@baude @mheon Thank you for your suggestions I have tried them out but the same issue remains:

podman machine set --rootful
podman machine start        
Starting machine "podman-machine-default"
API forwarding listening on: /var/run/docker.sock
Docker API clients default to this address. You do not need to set DOCKER_HOST.

Machine "podman-machine-default" started successfully
podman compose up           
>>>> Executing external compose provider "/opt/homebrew/bin/docker-compose". Please see podman-compose(1) for how to disable this message. <<<<

[+] Running 14/14
................
...............

w Enable Watch

Then when running sam it remains hanging.
Is there a reason why this workaround solution solves it and why sam works seamlessly when using docker?

gminova · 2024-12-13T19:53:31Z

@mheon Here is the open fix for sam which seems to have worked for 2 people aws/aws-sam-cli#7760
but since sam works with docker I thought the fix might be needed within podman rather than sam?
I hope this makes sense thank you

mheon · 2024-12-13T20:17:08Z

Oh, that helps a lot. Looks like a difference in how we handle events based on the pull request.

Luap99 · 2024-12-13T21:00:16Z

Can you provide the the full podman info output, is 5.3.1 only the client version or does it actually use the same version on the server.
I already fixed an issue that was blocking the event response until an event was generated in 5.3.0: 768ad86

gminova · 2024-12-13T22:51:38Z

@Luap99 Is there anything unusual? I have Podman Desktop 1.14.2 release and podman --version podman version 5.3.1

podman info
host:
  arch: arm64
  buildahVersion: 1.37.5
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.12-2.fc40.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: '
  cpuUtilization:
    idlePercent: 99.32
    systemPercent: 0.19
    userPercent: 0.49
  cpus: 4
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: coreos
    version: "40"
  eventLogger: journald
  freeLocks: 2045
  hostname: localhost.localdomain
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.11.3-200.fc40.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 1625894912
  memTotal: 2043133952
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.12.2-2.fc40.aarch64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.12.2
    package: netavark-1.12.2-1.fc40.aarch64
    path: /usr/libexec/podman/netavark
    version: netavark 1.12.2
  ociRuntime:
    name: crun
    package: crun-1.17-1.fc40.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.17
      commit: 000fa0d4eeed8938301f3bcf8206405315bc1017
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240906.g6b38f07-1.fc40.aarch64
    version: |
      pasta 0^20240906.g6b38f07-1.fc40.aarch64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-2.fc40.aarch64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 0
  swapTotal: 0
  uptime: 3h 5m 32.00s (Approximately 0.12 days)
  variant: v8
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 0
    stopped: 3
  graphDriverName: overlay
  graphOptions:
    overlay.imagestore: /usr/lib/containers/storage
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 106769133568
  graphRootUsed: 14310445056
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 19
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 5.2.5
  Built: 1729209600
  BuiltTime: Fri Oct 18 09:00:00 2024
  GitCommit: ""
  GoVersion: go1.22.7
  Os: linux
  OsArch: linux/arm64
  Version: 5.2.5

Luap99 · 2024-12-14T16:52:04Z

Version: 5.2.5

Your server version is is not updated automatically. You have to either remove and init the machine again if you have any data in there that you do not want to loose you can use
podman machine os apply --restart quay.io/podman/machine-os:5.3

gminova added the kind/bug Categorizes issue or PR as related to a bug. label Dec 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

podman + sam locally stuck on "Testing application behaviour against authorizers..." step #24839

podman + sam locally stuck on "Testing application behaviour against authorizers..." step #24839

gminova commented Dec 13, 2024 •

edited

Loading

baude commented Dec 13, 2024

gminova commented Dec 13, 2024 •

edited

Loading

gminova commented Dec 13, 2024 •

edited

Loading

mheon commented Dec 13, 2024

gminova commented Dec 13, 2024 •

edited

Loading

gminova commented Dec 13, 2024

mheon commented Dec 13, 2024

Luap99 commented Dec 13, 2024

gminova commented Dec 13, 2024 •

edited

Loading

Luap99 commented Dec 14, 2024

podman + sam locally stuck on "Testing application behaviour against authorizers..." step #24839

podman + sam locally stuck on "Testing application behaviour against authorizers..." step #24839

Comments

gminova commented Dec 13, 2024 • edited Loading

Issue Description

However, since sam works seamlessly with docker, I wonder if there is anything that could be changed on podman to avoid this issue. Thank you.

Steps to reproduce the issue

Describe the results you received

Describe the results you expected

Podman in a container

Privileged Or Rootless

Upstream Latest Release

baude commented Dec 13, 2024

gminova commented Dec 13, 2024 • edited Loading

gminova commented Dec 13, 2024 • edited Loading

mheon commented Dec 13, 2024

gminova commented Dec 13, 2024 • edited Loading

gminova commented Dec 13, 2024

mheon commented Dec 13, 2024

Luap99 commented Dec 13, 2024

gminova commented Dec 13, 2024 • edited Loading

Luap99 commented Dec 14, 2024

gminova commented Dec 13, 2024 •

edited

Loading

gminova commented Dec 13, 2024 •

edited

Loading

gminova commented Dec 13, 2024 •

edited

Loading

gminova commented Dec 13, 2024 •

edited

Loading

gminova commented Dec 13, 2024 •

edited

Loading