Experiment never finishes

[marupe-ambolt]

Describe the bug

I have set up a self-hosted ClearML instance, but every time I try to run an experiment, the worker running the experiment installs the necessary Ubuntu and Pip packages, and then just stalls without any further output and without ever finishing.

To reproduce

1. Follow the steps in the deployment guide https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_linux_mac.
2. Go to self-hosted ClearML webpage and clone e.g. the XGBoost simple example experiment.
3. Queue the cloned XGBoost experiment.
4. See that the experiment produces the following output in the console, and then stalls.

2024-09-04 10:23:24 task 21ee44a59ae74258bf1e8184fb7aacc8 pulled from 3a28160783b2462291244d51bd36faeb by worker clearml-services
2024-09-04 10:23:24 Running Task 21ee44a59ae74258bf1e8184fb7aacc8 inside default docker: ubuntu:18.04 arguments: []
2024-09-04 10:23:25 Executing: ['docker', 'run', '-t', '-l', 'clearml-worker-id=clearml-services:service:21ee44a59ae74258bf1e8184fb7aacc8', '-l', 'clearml-parent-worker-id=clearml-services', '-e', 'NVIDIA_VISIBLE_DEVICES=none', '-e', 'CLEARML_WORKER_ID=clearml-services:service:21ee44a59ae74258bf1e8184fb7aacc8', '-e', 'CLEARML_DOCKER_IMAGE=ubuntu:18.04', '-e', 'CLEARML_TASK_ID=21ee44a59ae74258bf1e8184fb7aacc8', '-e', 'CLEARML_AGENT_SERVICE_TASK=1', '-v', '/opt/clearml/agent/.clearml_agent.clearml-services%3Aservice%3A21ee44a59ae74258bf1e8184fb7aacc8.cfg:/tmp/clearml.conf', '-e', 'CLEARML_CONFIG_FILE=/tmp/clearml.conf', '-v', '/opt/clearml/agent/apt-cache:/var/cache/apt/archives', '-v', '/opt/clearml/agent/pip-cache:/root/.cache/pip', '-v', '/opt/clearml/agent/pip-download-cache:/root/.clearml/pip-download-cache', '-v', '/opt/clearml/agent/cache:/clearml_agent_cache', '-v', '/opt/clearml/agent/vcs-cache:/root/.clearml/vcs-cache', '-v', '/opt/clearml/agent/venvs-cache:/root/.clearml/venvs-cache', '--rm', 'ubuntu:18.04', 'bash', '-c', 'echo \'Binary::apt::APT::Keep-Downloaded-Packages "true";\' > /etc/apt/apt.conf.d/docker-clean ; chown -R root /root/.cache/pip ; export DEBIAN_FRONTEND=noninteractive ; export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL libsm6 libxext6 libxrender-dev libglib2.0-0" ; [ ! -z $(which git) ] || export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL git" ; declare LOCAL_PYTHON ; [ ! -z $LOCAL_PYTHON ] || for i in {15..5}; do which python3.$i && python3.$i -m pip --version && export LOCAL_PYTHON=$(which python3.$i) && break ; done ; [ ! -z $LOCAL_PYTHON ] || export CLEARML_APT_INSTALL="$CLEARML_APT_INSTALL python3-pip" ; [ -z "$CLEARML_APT_INSTALL" ] || (apt-get update -y ; apt-get install -y $CLEARML_APT_INSTALL) ; [ ! -z $LOCAL_PYTHON ] || export LOCAL_PYTHON=python3 ; $LOCAL_PYTHON -m pip install -U "pip<20.2 ; python_version < \'3.10\'" "pip<22.3 ; python_version >= \'3.10\' and python_version <= \'3.11\'" "pip>=23,<24.3 ; python_version >= \'3.12\'" ; $LOCAL_PYTHON -m pip install -U clearml-agent ; cp /tmp/clearml.conf ~/default_clearml.conf ; NVIDIA_VISIBLE_DEVICES=none $LOCAL_PYTHON -u -m clearml_agent execute --full-monitoring  --id 21ee44a59ae74258bf1e8184fb7aacc8']
2024-09-04 10:23:30 Unable to find image 'ubuntu:18.04' locally
18.04: Pulling from library/ubuntu
7c457f213c76: Pulling fs layer
7c457f213c76: Download complete
7c457f213c76: Pull complete
Digest: sha256:152dc042452c496007f07ca9127571cb9c29697f42acbfad72324b2bb2e43c98
Status: Downloaded newer image for ubuntu:18.04
2024-09-04 10:23:35 Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [102 kB]
Get:2 http://archive.ubuntu.com/ubuntu bionic InRelease [242 kB]
Get:3 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1637 kB]
Get:4 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [23.8 kB]
Get:5 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [1688 kB]
Get:6 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [3373 kB]
Get:7 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [102 kB]
Get:8 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [102 kB]
Get:9 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages [1344 kB]
Get:10 http://archive.ubuntu.com/ubuntu bionic/multiverse amd64 Packages [186 kB]
Get:11 http://archive.ubuntu.com/ubuntu bionic/restricted amd64 Packages [13.5 kB]
Get:12 http://archive.ubuntu.com/ubuntu bionic/universe amd64 Packages [11.3 MB]
Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [2411 kB]
Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [3786 kB]
Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [30.8 kB]
Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [1728 kB]
Get:17 http://archive.ubuntu.com/ubuntu bionic-backports/main amd64 Packages [64.0 kB]
Get:18 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [20.6 kB]
Fetched 28.2 MB in 3s (10.0 MB/s)
The following additional packages will be installed:
  binutils binutils-common binutils-x86-64-linux-gnu build-essential
  ca-certificates cpp cpp-7 dbus dh-python dirmngr dpkg-dev fakeroot file g++
  g++-7 gcc gcc-7 gcc-7-base gir1.2-glib-2.0 git-man gnupg gnupg-l10n
  gnupg-utils gpg gpg-agent gpg-wks-client gpg-wks-server gpgconf gpgsm
  krb5-locales less libalgorithm-diff-perl libalgorithm-diff-xs-perl
  libalgorithm-merge-perl libapparmor1 libasan4 libasn1-8-heimdal libassuan0
  libatomic1 libbinutils libbsd0 libc-dev-bin libc6-dev libcc1-0 libcilkrts5
  libcurl3-gnutls libdbus-1-3 libdpkg-perl libedit2 liberror-perl libexpat1
  libexpat1-dev libfakeroot libfile-fcntllock-perl libgcc-7-dev
  libgdbm-compat4 libgdbm5 libgirepository-1.0-1 libglib2.0-data libgomp1
  libgssapi-krb5-2 libgssapi3-heimdal libhcrypto4-heimdal libheimbase1-heimdal
  libheimntlm0-heimdal libhx509-5-heimdal libice6 libicu60 libisl19 libitm1
  libk5crypto3 libkeyutils1 libkrb5-26-heimdal libkrb5-3 libkrb5support0
  libksba8 libldap-2.4-2 libldap-common liblocale-gettext-perl liblsan0
  libmagic-mgc libmagic1 libmpc3 libmpdec2 libmpfr6 libmpx2 libnghttp2-14
  libnpth0 libperl5.26 libpsl5 libpthread-stubs0-dev libpython3-dev
  libpython3-stdlib libpython3.6 libpython3.6-dev libpython3.6-minimal
  libpython3.6-stdlib libquadmath0 libreadline7 libroken18-heimdal librtmp1
  libsasl2-2 libsasl2-modules libsasl2-modules-db libsqlite3-0 libssl1.0.0
  libssl1.1 libstdc++-7-dev libtsan0 libubsan0 libwind0-heimdal libx11-6
  libx11-data libx11-dev libx11-doc libxau-dev libxau6 libxcb1 libxcb1-dev
  libxdmcp-dev libxdmcp6 libxml2 libxmuu1 libxrender1 linux-libc-dev make
  manpages manpages-dev mime-support multiarch-support netbase openssh-client
  openssl patch perl perl-modules-5.26 pinentry-curses publicsuffix
  python-pip-whl python3 python3-asn1crypto python3-cffi-backend
  python3-crypto python3-cryptography python3-dbus python3-dev
  python3-distutils python3-gi python3-idna python3-keyring
  python3-keyrings.alt python3-lib2to3 python3-minimal python3-pkg-resources
  python3-secretstorage python3-setuptools python3-six python3-wheel
  python3-xdg python3.6 python3.6-dev python3.6-minimal readline-common
  shared-mime-info x11-common x11proto-core-dev x11proto-dev xauth
  xdg-user-dirs xorg-sgml-doctools xtrans-dev xz-utils
Suggested packages:
  binutils-doc cpp-doc gcc-7-locales default-dbus-session-bus
  | dbus-session-bus dbus-user-session libpam-systemd pinentry-gnome3 tor
  debian-keyring g++-multilib g++-7-multilib gcc-7-doc libstdc++6-7-dbg
  gcc-multilib autoconf automake libtool flex bison gdb gcc-doc gcc-7-multilib
  libgcc1-dbg libgomp1-dbg libitm1-dbg libatomic1-dbg libasan4-dbg
  liblsan0-dbg libtsan0-dbg libubsan0-dbg libcilkrts5-dbg libmpx2-dbg
  libquadmath0-dbg gettext-base git-daemon-run | git-daemon-sysvinit git-doc
  git-el git-email git-gui gitk gitweb git-cvs git-mediawiki git-svn
  parcimonie xloadimage scdaemon glibc-doc bzr gdbm-l10n krb5-doc krb5-user
  libsasl2-modules-gssapi-mit | libsasl2-modules-gssapi-heimdal
  libsasl2-modules-ldap libsasl2-modules-otp libsasl2-modules-sql
  libstdc++-7-doc libxcb-doc make-doc man-browser keychain libpam-ssh
  monkeysphere ssh-askpass ed diffutils-doc perl-doc libterm-readline-gnu-perl
  | libterm-readline-perl-perl pinentry-doc python3-doc python3-tk
  python3-venv python-crypto-doc python-cryptography-doc
  python3-cryptography-vectors python-dbus-doc python3-dbus-dbg gnome-keyring
  libkf5wallet-bin gir1.2-gnomekeyring-1.0 python-secretstorage-doc
  python-setuptools-doc python3.6-venv python3.6-doc binfmt-support
  readline-doc
The following NEW packages will be installed:
  binutils binutils-common binutils-x86-64-linux-gnu build-essential
  ca-certificates cpp cpp-7 dbus dh-python dirmngr dpkg-dev fakeroot file g++
  g++-7 gcc gcc-7 gcc-7-base gir1.2-glib-2.0 git git-man gnupg gnupg-l10n
  gnupg-utils gpg gpg-agent gpg-wks-client gpg-wks-server gpgconf gpgsm
  krb5-locales less libalgorithm-diff-perl libalgorithm-diff-xs-perl
  libalgorithm-merge-perl libapparmor1 libasan4 libasn1-8-heimdal libassuan0
  libatomic1 libbinutils libbsd0 libc-dev-bin libc6-dev libcc1-0 libcilkrts5
  libcurl3-gnutls libdbus-1-3 libdpkg-perl libedit2 liberror-perl libexpat1
  libexpat1-dev libfakeroot libfile-fcntllock-perl libgcc-7-dev
  libgdbm-compat4 libgdbm5 libgirepository-1.0-1 libglib2.0-0 libglib2.0-data
  libgomp1 libgssapi-krb5-2 libgssapi3-heimdal libhcrypto4-heimdal
  libheimbase1-heimdal libheimntlm0-heimdal libhx509-5-heimdal libice6
  libicu60 libisl19 libitm1 libk5crypto3 libkeyutils1 libkrb5-26-heimdal
  libkrb5-3 libkrb5support0 libksba8 libldap-2.4-2 libldap-common
  liblocale-gettext-perl liblsan0 libmagic-mgc libmagic1 libmpc3 libmpdec2
  libmpfr6 libmpx2 libnghttp2-14 libnpth0 libperl5.26 libpsl5
  libpthread-stubs0-dev libpython3-dev libpython3-stdlib libpython3.6
  libpython3.6-dev libpython3.6-minimal libpython3.6-stdlib libquadmath0
  libreadline7 libroken18-heimdal librtmp1 libsasl2-2 libsasl2-modules
  libsasl2-modules-db libsm6 libsqlite3-0 libssl1.0.0 libssl1.1
  libstdc++-7-dev libtsan0 libubsan0 libwind0-heimdal libx11-6 libx11-data
  libx11-dev libx11-doc libxau-dev libxau6 libxcb1 libxcb1-dev libxdmcp-dev
  libxdmcp6 libxext6 libxml2 libxmuu1 libxrender-dev libxrender1
  linux-libc-dev make manpages manpages-dev mime-support multiarch-support
  netbase openssh-client openssl patch perl perl-modules-5.26 pinentry-curses
  publicsuffix python-pip-whl python3 python3-asn1crypto python3-cffi-backend
  python3-crypto python3-cryptography python3-dbus python3-dev
  python3-distutils python3-gi python3-idna python3-keyring
  python3-keyrings.alt python3-lib2to3 python3-minimal python3-pip
  python3-pkg-resources python3-secretstorage python3-setuptools python3-six
  python3-wheel python3-xdg python3.6 python3.6-dev python3.6-minimal
  readline-common shared-mime-info x11-common x11proto-core-dev x11proto-dev
  xauth xdg-user-dirs xorg-sgml-doctools xtrans-dev xz-utils
0 upgraded, 178 newly installed, 0 to remove and 1 not upgraded.
Need to get 135 MB of archives.
After this operation, 461 MB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu bionic/main amd64 liblocale-gettext-perl amd64 1.07-3build2 [16.6 kB]
Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libssl1.1 amd64 1.1.1-1ubuntu2.1~18.04.23 [1303 kB]
0% [2 libssl1.1 2602 B/1303 kB 0%]
2024-09-04 10:23:40 Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libpython3.6-minimal amd64 3.6.9-1~18.04ubuntu1.12 [533 kB]
Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libexpat1 amd64 2.2.5-3ubuntu0.9 [82.8 kB]
Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3.6-minimal amd64 3.6.9-1~18.04ubuntu1.12 [1609 kB]

...

Setting up python3-setuptools (39.0.1-2ubuntu0.1) ...
Setting up python3-secretstorage (2.3.1-2) ...
Setting up dirmngr (2.2.4-1ubuntu1.6) ...
Setting up dh-python (3.20180325ubuntu2) ...
Setting up python3-keyring (10.6.0-1) ...
Setting up build-essential (12.4ubuntu1) ...
Setting up libcurl3-gnutls:amd64 (7.58.0-2ubuntu3.24) ...
Setting up python3-dev (3.6.7-1~18.04) ...
Setting up gpg-wks-client (2.2.4-1ubuntu1.6) ...
Setting up git (1:2.17.1-1ubuntu0.18) ...
Setting up gnupg (2.2.4-1ubuntu1.6) ...
Processing triggers for libc-bin (2.27-3ubuntu1.6) ...
Processing triggers for ca-certificates (20230311ubuntu0.18.04.1) ...
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
Ignoring pip: markers 'python_version >= "3.10" and python_version <= "3.11"' don't match your environment
Ignoring pip: markers 'python_version >= "3.12"' don't match your environment
Collecting pip<20.2
  Downloading https://files.pythonhosted.org/packages/43/84/23ed6a1796480a6f1a2d38f2802901d078266bda38388954d01d3f2e821d/pip-20.1.1-py2.py3-none-any.whl (1.5MB)
    100% |################################| 1.5MB 1.3MB/s 
?25hInstalling collected packages: pip
  Found existing installation: pip 9.0.1
    Not uninstalling pip at /usr/lib/python3/dist-packages, outside environment /usr
Successfully installed pip-20.1.1
2024-09-04 10:24:10 Collecting clearml-agent
  Downloading clearml_agent-1.9.1-py3-none-any.whl (437 kB)
     |################################| 437 kB 3.0 MB/s 
?25hCollecting urllib3<2,>=1.21.1
  Downloading urllib3-1.26.20-py2.py3-none-any.whl (144 kB)
     |################################| 144 kB 24.2 MB/s 
?25hCollecting virtualenv<21,>=16
  Downloading virtualenv-20.17.1-py3-none-any.whl (8.8 MB)
     |################################| 8.8 MB 19.1 MB/s 
?25hCollecting psutil<5.10.0,>=3.4.2
  Downloading psutil-5.9.8-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (288 kB)
     |################################| 288 kB 21.5 MB/s 
?25hCollecting pyparsing<3.2.0,>=2.0.3
  Downloading pyparsing-3.1.4-py3-none-any.whl (104 kB)
     |################################| 104 kB 36.3 MB/s 
?25hCollecting jsonschema<5.0.0,>=2.6.0
  Downloading jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)
     |################################| 56 kB 14.5 MB/s 
?25hCollecting six<1.17.0,>=1.13.0
  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting pyjwt<2.9.0,>=2.4.0
  Downloading PyJWT-2.4.0-py3-none-any.whl (18 kB)
Collecting requests<=2.31.0,>=2.20.0
  Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB)
     |################################| 63 kB 5.2 MB/s 
?25hCollecting attrs<24.0.0,>=18.0
  Downloading attrs-22.2.0-py3-none-any.whl (60 kB)
     |################################| 60 kB 16.6 MB/s 
?25hCollecting PyYAML<6.1,>=3.12
  Downloading PyYAML-6.0.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (677 kB)
     |################################| 677 kB 31.5 MB/s 
?25hCollecting furl<2.2.0,>=2.0.0
  Downloading furl-2.1.3-py2.py3-none-any.whl (20 kB)
Collecting pathlib2<2.4.0,>=2.3.0
  Downloading pathlib2-2.3.7.post1-py2.py3-none-any.whl (18 kB)
Collecting python-dateutil<2.9.0,>=2.4.2
  Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
     |################################| 247 kB 25.1 MB/s 
?25hCollecting importlib-metadata>=4.8.3; python_version < "3.8"
  Downloading importlib_metadata-4.8.3-py3-none-any.whl (17 kB)
Collecting filelock<4,>=3.4.1
  Downloading filelock-3.4.1-py3-none-any.whl (9.9 kB)
Collecting distlib<1,>=0.3.6
  Downloading distlib-0.3.8-py2.py3-none-any.whl (468 kB)
     |################################| 468 kB 36.6 MB/s 
?25hCollecting importlib-resources>=5.4; python_version < "3.7"
  Downloading importlib_resources-5.4.0-py3-none-any.whl (28 kB)
Collecting platformdirs<3,>=2.4
  Downloading platformdirs-2.4.0-py3-none-any.whl (14 kB)
Requirement already satisfied, skipping upgrade: setuptools in /usr/lib/python3/dist-packages (from jsonschema<5.0.0,>=2.6.0->clearml-agent) (39.0.1)
Collecting pyrsistent>=0.14.0
  Downloading pyrsistent-0.18.0-cp36-cp36m-manylinux1_x86_64.whl (117 kB)
     |################################| 117 kB 56.0 MB/s 
?25hRequirement already satisfied, skipping upgrade: idna<4,>=2.5; python_version >= "3" in /usr/lib/python3/dist-packages (from requests<=2.31.0,>=2.20.0->clearml-agent) (2.6)
Collecting charset-normalizer~=2.0.0; python_version >= "3"
  Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
Collecting certifi>=2017.4.17
  Downloading certifi-2024.8.30-py3-none-any.whl (167 kB)
     |################################| 167 kB 32.5 MB/s 
?25hCollecting orderedmultidict>=1.0.1
  Downloading orderedmultidict-1.0.1-py2.py3-none-any.whl (11 kB)
Collecting typing-extensions>=3.6.4; python_version < "3.8"
  Downloading typing_extensions-4.1.1-py3-none-any.whl (26 kB)
Collecting zipp>=0.5
  Downloading zipp-3.6.0-py3-none-any.whl (5.3 kB)
Installing collected packages: urllib3, typing-extensions, zipp, importlib-metadata, filelock, distlib, importlib-resources, platformdirs, virtualenv, psutil, pyparsing, six, attrs, pyrsistent, jsonschema, pyjwt, charset-normalizer, certifi, requests, PyYAML, orderedmultidict, furl, pathlib2, python-dateutil, clearml-agent
  Attempting uninstall: six
    Found existing installation: six 1.11.0
    Uninstalling six-1.11.0:
      Successfully uninstalled six-1.11.0
Successfully installed PyYAML-6.0.1 attrs-22.2.0 certifi-2024.8.30 charset-normalizer-2.0.12 clearml-agent-1.9.1 distlib-0.3.8 filelock-3.4.1 furl-2.1.3 importlib-metadata-4.8.3 importlib-resources-5.4.0 jsonschema-3.2.0 orderedmultidict-1.0.1 pathlib2-2.3.7.post1 platformdirs-2.4.0 psutil-5.9.8 pyjwt-2.4.0 pyparsing-3.1.4 pyrsistent-0.18.0 python-dateutil-2.8.2 requests-2.27.1 six-1.16.0 typing-extensions-4.1.1 urllib3-1.26.20 virtualenv-20.17.1 zipp-3.6.0
WARNING: You are using pip version 20.1.1; however, version 21.3.1 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.

Expected behaviour

The expected behaviour would be that the experiment finishes and gives the same output as the XGBoost simple example, which has the following console output:

2021-04-24 23:05:04 ClearML Task: created new task id=825e0c18fe0641d997dd70736402f7c0
2021-04-24 23:05:09 ClearML results page: https://app.community.clear.ml/projects/fe39a015c33548069314a7a4190c5418/experiments/825e0c18fe0641d997dd70736402f7c0/output/log
2021-04-24 23:05:11 ClearML Monitor: GPU monitoring failed getting GPU reading, switching off GPU monitoring
2021-04-24 23:05:36 [00:05:36] WARNING: /workspace/src/learner.cc:480: 
Parameters: { silent } might not be used.
  This may not be accurate due to some parameters are only used in language bindings but
  passed down to XGBoost core.  Or some parameters are not used but slip through this
  verification. Please open an issue if you find above cases.
2021-04-24 23:05:46 2021-04-25 00:05:46,989 - clearml.storage - INFO - Starting upload: /tmp/.clearml.upload_model_mi3qr561.tmp => https://files.community.clear.ml/examples/XGBoost simple example.825e0c18fe0641d997dd70736402f7c0/models/xgb.01.model
2021-04-24 23:05:48 2021-04-25 00:05:48,578 - clearml.Task - INFO - Completed model upload to https://files.community.clear.ml/examples/XGBoost%20simple%20example.825e0c18fe0641d997dd70736402f7c0/models/xgb.01.model
2021-04-24 23:05:56 2021-04-25 00:05:56,565 - clearml.storage - INFO - Starting upload: /tmp/.clearml.upload_model_n8g8ik7_.tmp => https://files.community.clear.ml/examples/XGBoost simple example.825e0c18fe0641d997dd70736402f7c0/models/xgb.02.model
Accuracy: 100.00%
2021-04-24 23:05:57 2021-04-25 00:05:57,817 - clearml.Task - INFO - Completed model upload to https://files.community.clear.ml/examples/XGBoost%20simple%20example.825e0c18fe0641d997dd70736402f7c0/models/xgb.02.model
2021-04-24 23:06:03 2021-04-25 00:06:03,896 - clearml.Task - INFO - Waiting to finish uploads
2021-04-24 23:06:06 2021-04-25 00:06:06,791 - clearml.Task - INFO - Finished uploading

Environment

• Server type: Self-hosted
• ClearML SDK Version: Not sure
• ClearML Server Version: 1.16.2-502
• Python Version: 3.6.13
• OS: Linux Ubuntu 22.04.4 LTS

Related Discussion

If this continues a slack thread, please provide a link to the original slack thread.

[shomratalon]

Hi @marupe-ambolt , from the logs it looks like you are running it from a services agent, any specific reason for it?