Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[2.1.3-0 ] error 47 from memcached_set #250

Closed
riadabdel opened this issue Apr 8, 2020 · 12 comments
Closed

[2.1.3-0 ] error 47 from memcached_set #250

riadabdel opened this issue Apr 8, 2020 · 12 comments

Comments

@riadabdel
Copy link

Hello,

I am running zulip on docker with version 2.1.3-0
I am getting spam by memecached errors:

Traceback (most recent call last):
  File "/home/zulip/deployments/2020-04-04-00-47-42/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 130, in get
    return super(PyLibMCCache, self).get(key, default, version)
  File "/home/zulip/deployments/2020-04-04-00-47-42/zulip-py3-venv/lib/python3.6/site-packages/django/core/cache/backends/memcached.py", line 79, in get
    val = self._cache.get(key)
pylibmc.ServerDown: error 47 from memcached_get(:1:f09f7cfd2fbcfc1ce4e37810fe48e): (0x23845d0) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY, memcached:11211,  host: memcached:11211 -> libmemcached/connect.cc:811

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zulip/deployments/2020-04-04-00-47-42/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 130, in get
    return super(PyLibMCCache, self).get(key, default, version)
  File "/home/zulip/deployments/2020-04-04-00-47-42/zulip-py3-venv/lib/python3.6/site-packages/django/core/cache/backends/memcached.py", line 79, in get
    val = self._cache.get(key)
pylibmc.ServerDown: error 47 from memcached_get(:1:RLF_TEST_KEY): (0x23845d0) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY,  host: memcached:11211 -> libmemcached/connect.cc:720
2020-04-08 14:50:23.664 ERR  [django.pylibmc] MemcachedError: error 47 from memcached_get(:1:f09f7cfd2fbcfc1ce4e37810fe48e): (0x23845d0) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY, memcached:11211,  host: memcached:11211 -> libmemcached/connect.cc:811
Traceback (most recent call last):
  File "/home/zulip/deployments/2020-04-04-00-47-42/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 130, in get
    return super(PyLibMCCache, self).get(key, default, version)
  File "/home/zulip/deployments/2020-04-04-00-47-42/zulip-py3-venv/lib/python3.6/site-packages/django/core/cache/backends/memcached.py", line 79, in get
    val = self._cache.get(key)
pylibmc.ServerDown: error 47 from memcached_get(:1:f09f7cfd2fbcfc1ce4e37810fe48e): (0x23845d0) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY, memcached:11211,  host: memcached:11211 -> libmemcached/connect.cc:811
2020-04-08 14:50:23.678 ERR  [django.pylibmc] MemcachedError: error 47 from memcached_set: (0x23845d0) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY,  host: memcached:11211 -> libmemcached/connect.cc:720
Traceback (most recent call last):
  File "/home/zulip/deployments/2020-04-04-00-47-42/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 140, in set
    **COMPRESS_KWARGS)
pylibmc.ServerDown: error 47 from memcached_set: (0x23845d0) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY,  host: memcached:11211 -> libmemcached/connect.cc:720

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zulip/deployments/2020-04-04-00-47-42/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 140, in set
    **COMPRESS_KWARGS)
pylibmc.ServerDown: error 47 from memcached_set: (0x23845d0) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY,  host: memcached:11211 -> libmemcached/connect.cc:720
2020-04-08 14:50:23.681 ERR  [django.pylibmc] MemcachedError: error 47 from memcached_get(:1:RLF_TEST_KEY): (0x23845d0) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY,  host: memcached:11211 -> libmemcached/connect.cc:720
Traceback (most recent call last):
  File "/home/zulip/deployments/2020-04-04-00-47-42/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 140, in set
    **COMPRESS_KWARGS)
pylibmc.ServerDown: error 47 from memcached_set: (0x23845d0) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY,  host: memcached:11211 -> libmemcached/connect.cc:720

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zulip/deployments/2020-04-04-00-47-42/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 130, in get
    return super(PyLibMCCache, self).get(key, default, version)
  File "/home/zulip/deployments/2020-04-04-00-47-42/zulip-py3-venv/lib/python3.6/site-packages/django/core/cache/backends/memcached.py", line 79, in get
    val = self._cache.get(key)
pylibmc.ServerDown: error 47 from memcached_get(:1:RLF_TEST_KEY): (0x23845d0) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY,  host: memcached:11211 -> libmemcached/connect.cc:720
2020-04-08 14:50:23.678 ERR  [django.pylibmc] MemcachedError: error 47 from memcached_set: (0x23845d0) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY,  host: memcached:11211 -> libmemcached/connect.cc:720
Traceback (most recent call last):
  File "/home/zulip/deployments/2020-04-04-00-47-42/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 140, in set
    **COMPRESS_KWARGS)
pylibmc.ServerDown: error 47 from memcached_set: (0x23845d0) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY,  host: memcached:11211 -> libmemcached/connect.cc:720
@timabbott
Copy link
Member

Are you using Docker Swarm?

If so, the issue is probably zulip/zulip#14456 (see #426 for debugging notes).

@riadabdel
Copy link
Author

No, i use only docker-compose

@maltokyo
Copy link
Contributor

I get this now too, after a restart. No updates on docker-compose file. Zulip still seems(?) to work fine. These come by email, periodically. Docker Zulip version 2.1.4

** EMAIL SUBJECT:**

[Django] 80753dd569c6: error 47 from memcached_set: (0x55c253aff6a0) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY, host: zulipstippy_memcached:11211 -> libmemcached/connect.cc:720

EMAIL BODY:

Logger django.pylibmc, from module django_pylibmc.memcached line 146:
Error generated by Anonymous user (not logged in) on 80753dd569c6 deployment

Traceback (most recent call last):
File "/srv/zulip-venv-cache/6176a52fab8737c09656fc8ad0352547e9801c07/zulip-py3-venv/lib/python3.6/site-packages/django/contrib/sessions/backends/base.py", line 202, in _get_session
return self._session_cache
AttributeError: 'SessionStore' object has no attribute '_session_cache'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/srv/zulip-venv-cache/6176a52fab8737c09656fc8ad0352547e9801c07/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 130, in get
return super(PyLibMCCache, self).get(key, default, version)
File "/srv/zulip-venv-cache/6176a52fab8737c09656fc8ad0352547e9801c07/zulip-py3-venv/lib/python3.6/site-packages/django/core/cache/backends/memcached.py", line 79, in get
val = self._cache.get(key)
pylibmc.Error: error 31 from memcached_get(:1:django.contrib.sessions.cache): (0x55c253aff6a0) A TIMEOUT OCCURRED, (number_of == 0), host: zulipstippy_memcached:11211 -> libmemcached/connect.cc:133

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/srv/zulip-venv-cache/6176a52fab8737c09656fc8ad0352547e9801c07/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 140, in set
**COMPRESS_KWARGS)
pylibmc.ServerDown: error 47 from memcached_set: (0x55c253aff6a0) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY, host: zulipstippy_memcached:11211 -> libmemcached/connect.cc:720

Deployed code:

  • ZULIP_VERSION: 2.1.4
  • version: docker

@timabbott
Copy link
Member

I think the memcached authentication setup we have breaks when the server's hostname changes, which probably happens often with Docker. zulip/zulip#14925 changes what library we're using to access memcached, which may help.

@timabbott
Copy link
Member

I believe Zulip server 2.1.7 fixes this issue; the problem was that the way we setup memcached authentication resulted in the default username used changing if the server was restarted with a different hostname, which effectively meant memcached would start throwing this exception every time access was attempted.

@riadabdel
Copy link
Author

Thank's for your help, i will try the new update.

@riadabdel
Copy link
Author

still getting the error with the new version 2.1.7-0


Traceback (most recent call last):
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 130, in get
    return super(PyLibMCCache, self).get(key, default, version)
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django/core/cache/backends/memcached.py", line 79, in get
    val = self._cache.get(key)
pylibmc.Error: error 41 from memcached_get(:1:RLF_TEST_KEY): (0x2e45430) AUTHENTICATION FAILURE,  host: memcached:11211 -> libmemcached/sasl.cc:292
2020-07-01 08:39:53.792 ERR  [django.pylibmc] MemcachedError: error 41 from memcached_get(:1:dbc3e0c23319b7d52e1d963bb050c): (0x2e45430) AUTHENTICATION FAILURE,  host: memcached:11211 -> libmemcached/sasl.cc:292
Traceback (most recent call last):
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 130, in get
    return super(PyLibMCCache, self).get(key, default, version)
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django/core/cache/backends/memcached.py", line 79, in get
    val = self._cache.get(key)
pylibmc.Error: error 41 from memcached_get(:1:dbc3e0c23319b7d52e1d963bb050c): (0x2e45430) AUTHENTICATION FAILURE,  host: memcached:11211 -> libmemcached/sasl.cc:292
2020-07-01 08:39:53.811 ERR  [django.pylibmc] MemcachedError: error 41 from memcached_set: (0x2e45430) AUTHENTICATION FAILURE,  host: memcached:11211 -> libmemcached/sasl.cc:292
Traceback (most recent call last):
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 140, in set
    **COMPRESS_KWARGS)
pylibmc.ServerDown: error 47 from memcached_set: (0x2e45430) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY,  host: memcached:11211 -> libmemcached/connect.cc:720

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 140, in set
    **COMPRESS_KWARGS)
pylibmc.Error: error 41 from memcached_set: (0x2e45430) AUTHENTICATION FAILURE,  host: memcached:11211 -> libmemcached/sasl.cc:292
2020-07-01 08:39:53.812 ERR  [django.pylibmc] MemcachedError: error 41 from memcached_get(:1:RLF_TEST_KEY): (0x2e45430) AUTHENTICATION FAILURE,  host: memcached:11211 -> libmemcached/sasl.cc:292
Traceback (most recent call last):
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 140, in set
    **COMPRESS_KWARGS)
pylibmc.ServerDown: error 47 from memcached_set: (0x2e45430) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY,  host: memcached:11211 -> libmemcached/connect.cc:720

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 130, in get
    return super(PyLibMCCache, self).get(key, default, version)
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django/core/cache/backends/memcached.py", line 79, in get
    val = self._cache.get(key)
pylibmc.Error: error 41 from memcached_get(:1:RLF_TEST_KEY): (0x2e45430) AUTHENTICATION FAILURE,  host: memcached:11211 -> libmemcached/sasl.cc:292
2020-07-01 08:39:53.811 ERR  [django.pylibmc] MemcachedError: error 47 from memcached_set: (0x2e45430) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY,  host: memcached:11211 -> libmemcached/connect.cc:720
Traceback (most recent call last):
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 140, in set
    **COMPRESS_KWARGS)
pylibmc.ServerDown: error 47 from memcached_set: (0x2e45430) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY,  host: memcached:11211 -> libmemcached/connect.cc:720
2020-07-01 08:39:56.808 ERR  [django.pylibmc] MemcachedError: error 47 from memcached_set: (0x4087f00) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY,  host: memcached:11211 -> libmemcached/connect.cc:720
Traceback (most recent call last):
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 130, in get
    return super(PyLibMCCache, self).get(key, default, version)
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django/core/cache/backends/memcached.py", line 79, in get
    val = self._cache.get(key)
pylibmc.Error: error 41 from memcached_get(:1:dbc3e0c23319b7d52e1d963bb050c): (0x4087f00) AUTHENTICATION FAILURE,  host: memcached:11211 -> libmemcached/sasl.cc:292

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 140, in set
    **COMPRESS_KWARGS)
pylibmc.ServerDown: error 47 from memcached_set: (0x4087f00) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY,  host: memcached:11211 -> libmemcached/connect.cc:720
2020-07-01 08:39:56.820 ERR  [django.pylibmc] MemcachedError: error 41 from memcached_get(:1:RLF_TEST_KEY): (0x4087f00) AUTHENTICATION FAILURE,  host: memcached:11211 -> libmemcached/sasl.cc:292
Traceback (most recent call last):
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 130, in get
    return super(PyLibMCCache, self).get(key, default, version)
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django/core/cache/backends/memcached.py", line 79, in get
    val = self._cache.get(key)
pylibmc.Error: error 41 from memcached_get(:1:dbc3e0c23319b7d52e1d963bb050c): (0x4087f00) AUTHENTICATION FAILURE,  host: memcached:11211 -> libmemcached/sasl.cc:292

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 130, in get
    return super(PyLibMCCache, self).get(key, default, version)
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django/core/cache/backends/memcached.py", line 79, in get
    val = self._cache.get(key)
pylibmc.Error: error 41 from memcached_get(:1:RLF_TEST_KEY): (0x4087f00) AUTHENTICATION FAILURE,  host: memcached:11211 -> libmemcached/sasl.cc:292
2020-07-01 08:39:56.808 ERR  [django.pylibmc] MemcachedError: error 41 from memcached_get(:1:dbc3e0c23319b7d52e1d963bb050c): (0x4087f00) AUTHENTICATION FAILURE,  host: memcached:11211 -> libmemcached/sasl.cc:292
Traceback (most recent call last):
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 130, in get
    return super(PyLibMCCache, self).get(key, default, version)
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django/core/cache/backends/memcached.py", line 79, in get
    val = self._cache.get(key)
pylibmc.Error: error 41 from memcached_get(:1:dbc3e0c23319b7d52e1d963bb050c): (0x4087f00) AUTHENTICATION FAILURE,  host: memcached:11211 -> libmemcached/sasl.cc:292
2020-07-01 08:39:56.826 ERR  [django.pylibmc] MemcachedError: error 41 from memcached_set: (0x4087f00) AUTHENTICATION FAILURE,  host: memcached:11211 -> libmemcached/sasl.cc:292
Traceback (most recent call last):
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 140, in set
    **COMPRESS_KWARGS)
pylibmc.ServerDown: error 47 from memcached_set: (0x4087f00) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY,  host: memcached:11211 -> libmemcached/connect.cc:720

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 140, in set
    **COMPRESS_KWARGS)
pylibmc.Error: error 41 from memcached_set: (0x4087f00) AUTHENTICATION FAILURE,  host: memcached:11211 -> libmemcached/sasl.cc:292
2020-07-01 08:39:56.827 ERR  [django.pylibmc] MemcachedError: error 41 from memcached_get(:1:RLF_TEST_KEY): (0x4087f00) AUTHENTICATION FAILURE,  host: memcached:11211 -> libmemcached/sasl.cc:292
Traceback (most recent call last):
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 140, in set
    **COMPRESS_KWARGS)
pylibmc.ServerDown: error 47 from memcached_set: (0x4087f00) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY,  host: memcached:11211 -> libmemcached/connect.cc:720

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 130, in get
    return super(PyLibMCCache, self).get(key, default, version)
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django/core/cache/backends/memcached.py", line 79, in get
    val = self._cache.get(key)
pylibmc.Error: error 41 from memcached_get(:1:RLF_TEST_KEY): (0x4087f00) AUTHENTICATION FAILURE,  host: memcached:11211 -> libmemcached/sasl.cc:292
2020-07-01 08:39:56.826 ERR  [django.pylibmc] MemcachedError: error 47 from memcached_set: (0x4087f00) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY,  host: memcached:11211 -> libmemcached/connect.cc:720
Traceback (most recent call last):
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 140, in set
    **COMPRESS_KWARGS)
pylibmc.ServerDown: error 47 from memcached_set: (0x4087f00) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY,  host: memcached:11211 -> libmemcached/connect.cc:720
2020-07-01 08:39:58.975 ERR  [django.pylibmc] MemcachedError: error 47 from memcached_set: (0x2219f00) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY,  host: memcached:11211 -> libmemcached/connect.cc:720
Traceback (most recent call last):
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 130, in get
    return super(PyLibMCCache, self).get(key, default, version)
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django/core/cache/backends/memcached.py", line 79, in get
    val = self._cache.get(key)
pylibmc.Error: error 41 from memcached_get(:1:dbc3e0c23319b7d52e1d963bb050c): (0x2219f00) AUTHENTICATION FAILURE,  host: memcached:11211 -> libmemcached/sasl.cc:292

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 140, in set
    **COMPRESS_KWARGS)
pylibmc.ServerDown: error 47 from memcached_set: (0x2219f00) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY,  host: memcached:11211 -> libmemcached/connect.cc:720
2020-07-01 08:39:58.991 ERR  [django.pylibmc] MemcachedError: error 41 from memcached_get(:1:RLF_TEST_KEY): (0x2219f00) AUTHENTICATION FAILURE,  host: memcached:11211 -> libmemcached/sasl.cc:292
Traceback (most recent call last):
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 130, in get
    return super(PyLibMCCache, self).get(key, default, version)
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django/core/cache/backends/memcached.py", line 79, in get
    val = self._cache.get(key)
pylibmc.Error: error 41 from memcached_get(:1:dbc3e0c23319b7d52e1d963bb050c): (0x2219f00) AUTHENTICATION FAILURE,  host: memcached:11211 -> libmemcached/sasl.cc:292

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 130, in get
    return super(PyLibMCCache, self).get(key, default, version)
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django/core/cache/backends/memcached.py", line 79, in get
    val = self._cache.get(key)
pylibmc.Error: error 41 from memcached_get(:1:RLF_TEST_KEY): (0x2219f00) AUTHENTICATION FAILURE,  host: memcached:11211 -> libmemcached/sasl.cc:292
2020-07-01 08:39:58.975 ERR  [django.pylibmc] MemcachedError: error 41 from memcached_get(:1:dbc3e0c23319b7d52e1d963bb050c): (0x2219f00) AUTHENTICATION FAILURE,  host: memcached:11211 -> libmemcached/sasl.cc:292
Traceback (most recent call last):
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 130, in get
    return super(PyLibMCCache, self).get(key, default, version)
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django/core/cache/backends/memcached.py", line 79, in get
    val = self._cache.get(key)
pylibmc.Error: error 41 from memcached_get(:1:dbc3e0c23319b7d52e1d963bb050c): (0x2219f00) AUTHENTICATION FAILURE,  host: memcached:11211 -> libmemcached/sasl.cc:292
2020-07-01 08:39:58.997 ERR  [django.pylibmc] MemcachedError: error 41 from memcached_set: (0x2219f00) AUTHENTICATION FAILURE,  host: memcached:11211 -> libmemcached/sasl.cc:292
Traceback (most recent call last):
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 140, in set
    **COMPRESS_KWARGS)
pylibmc.ServerDown: error 47 from memcached_set: (0x2219f00) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY,  host: memcached:11211 -> libmemcached/connect.cc:720

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 140, in set
    **COMPRESS_KWARGS)
pylibmc.Error: error 41 from memcached_set: (0x2219f00) AUTHENTICATION FAILURE,  host: memcached:11211 -> libmemcached/sasl.cc:292
2020-07-01 08:39:58.997 ERR  [django.pylibmc] MemcachedError: error 41 from memcached_get(:1:RLF_TEST_KEY): (0x2219f00) AUTHENTICATION FAILURE,  host: memcached:11211 -> libmemcached/sasl.cc:292
Traceback (most recent call last):
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 140, in set
    **COMPRESS_KWARGS)
pylibmc.ServerDown: error 47 from memcached_set: (0x2219f00) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY,  host: memcached:11211 -> libmemcached/connect.cc:720

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 130, in get
    return super(PyLibMCCache, self).get(key, default, version)
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django/core/cache/backends/memcached.py", line 79, in get
    val = self._cache.get(key)
pylibmc.Error: error 41 from memcached_get(:1:RLF_TEST_KEY): (0x2219f00) AUTHENTICATION FAILURE,  host: memcached:11211 -> libmemcached/sasl.cc:292
2020-07-01 08:39:58.996 ERR  [django.pylibmc] MemcachedError: error 47 from memcached_set: (0x2219f00) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY,  host: memcached:11211 -> libmemcached/connect.cc:720
Traceback (most recent call last):
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 140, in set
    **COMPRESS_KWARGS)
pylibmc.ServerDown: error 47 from memcached_set: (0x2219f00) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY,  host: memcached:11211 -> libmemcached/connect.cc:720
2020-07-01 08:40:02.278 ERR  [django.pylibmc] MemcachedError: error 47 from memcached_set: (0x314ef00) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY,  host: memcached:11211 -> libmemcached/connect.cc:720
Traceback (most recent call last):
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 130, in get
    return super(PyLibMCCache, self).get(key, default, version)
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django/core/cache/backends/memcached.py", line 79, in get
    val = self._cache.get(key)
pylibmc.Error: error 41 from memcached_get(:1:dbc3e0c23319b7d52e1d963bb050c): (0x314ef00) AUTHENTICATION FAILURE,  host: memcached:11211 -> libmemcached/sasl.cc:292

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 140, in set
    **COMPRESS_KWARGS)
pylibmc.ServerDown: error 47 from memcached_set: (0x314ef00) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY,  host: memcached:11211 -> libmemcached/connect.cc:720
2020-07-01 08:40:02.291 ERR  [django.pylibmc] MemcachedError: error 41 from memcached_get(:1:RLF_TEST_KEY): (0x314ef00) AUTHENTICATION FAILURE,  host: memcached:11211 -> libmemcached/sasl.cc:292
Traceback (most recent call last):
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 130, in get
    return super(PyLibMCCache, self).get(key, default, version)
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django/core/cache/backends/memcached.py", line 79, in get
    val = self._cache.get(key)
pylibmc.Error: error 41 from memcached_get(:1:dbc3e0c23319b7d52e1d963bb050c): (0x314ef00) AUTHENTICATION FAILURE,  host: memcached:11211 -> libmemcached/sasl.cc:292

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 130, in get
    return super(PyLibMCCache, self).get(key, default, version)
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django/core/cache/backends/memcached.py", line 79, in get
    val = self._cache.get(key)
pylibmc.Error: error 41 from memcached_get(:1:RLF_TEST_KEY): (0x314ef00) AUTHENTICATION FAILURE,  host: memcached:11211 -> libmemcached/sasl.cc:292
2020-07-01 08:40:02.278 ERR  [django.pylibmc] MemcachedError: error 41 from memcached_get(:1:dbc3e0c23319b7d52e1d963bb050c): (0x314ef00) AUTHENTICATION FAILURE,  host: memcached:11211 -> libmemcached/sasl.cc:292
Traceback (most recent call last):
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 130, in get
    return super(PyLibMCCache, self).get(key, default, version)
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django/core/cache/backends/memcached.py", line 79, in get
    val = self._cache.get(key)
pylibmc.Error: error 41 from memcached_get(:1:dbc3e0c23319b7d52e1d963bb050c): (0x314ef00) AUTHENTICATION FAILURE,  host: memcached:11211 -> libmemcached/sasl.cc:292
2020-07-01 08:40:02.297 ERR  [django.pylibmc] MemcachedError: error 41 from memcached_set: (0x314ef00) AUTHENTICATION FAILURE,  host: memcached:11211 -> libmemcached/sasl.cc:292
Traceback (most recent call last):
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 140, in set
    **COMPRESS_KWARGS)
pylibmc.ServerDown: error 47 from memcached_set: (0x314ef00) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY,  host: memcached:11211 -> libmemcached/connect.cc:720

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 140, in set
    **COMPRESS_KWARGS)
pylibmc.Error: error 41 from memcached_set: (0x314ef00) AUTHENTICATION FAILURE,  host: memcached:11211 -> libmemcached/sasl.cc:292
2020-07-01 08:40:02.297 ERR  [django.pylibmc] MemcachedError: error 41 from memcached_get(:1:RLF_TEST_KEY): (0x314ef00) AUTHENTICATION FAILURE,  host: memcached:11211 -> libmemcached/sasl.cc:292
Traceback (most recent call last):
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 140, in set
    **COMPRESS_KWARGS)
pylibmc.ServerDown: error 47 from memcached_set: (0x314ef00) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY,  host: memcached:11211 -> libmemcached/connect.cc:720

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 130, in get
    return super(PyLibMCCache, self).get(key, default, version)
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django/core/cache/backends/memcached.py", line 79, in get
    val = self._cache.get(key)
pylibmc.Error: error 41 from memcached_get(:1:RLF_TEST_KEY): (0x314ef00) AUTHENTICATION FAILURE,  host: memcached:11211 -> libmemcached/sasl.cc:292
2020-07-01 08:40:02.296 ERR  [django.pylibmc] MemcachedError: error 47 from memcached_set: (0x314ef00) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY,  host: memcached:11211 -> libmemcached/connect.cc:720
Traceback (most recent call last):
  File "/home/zulip/deployments/2020-06-26-05-15-05/zulip-py3-venv/lib/python3.6/site-packages/django_pylibmc/memcached.py", line 140, in set
    **COMPRESS_KWARGS)
pylibmc.ServerDown: error 47 from memcached_set: (0x314ef00) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY,  host: memcached:11211 -> libmemcached/connect.cc:720

My /var/log/zulip/error.log log file size...:
Capture d’écran de 2020-07-01 09-59-59

@timabbott
Copy link
Member

timabbott commented Jul 1, 2020 •
edited
Loading

Did you apply the changes to docker-compose.yml in 0810e2b when updating? I suppose I should have mentioned one needs to do that.

(And yes, memcached authentication being broken is going to do a ton of logging because every single request will throw at least one exception trying to access the cache. I would have expected our logrotate configuration to handle that on a normal Zulip server, but I suppose Docker ignores that configuration).

@andersk FYI.

@riadabdel
Copy link
Author

No i didnt, but i added manually in memcached container, its ok like that ?
i put a new dev zulip stack, and i got the error with the latest version, so i added MEMCACHED_USERNAME = 'zulip' on "/etc/zulip/settings.py", its work!! no more errors. And when i add MEMCACHED_USERNAME = 'zulip@localhost', its not working.
So i did the same thing on my prod, this didnt solved the issue..

@riadabdel
Copy link
Author

Another thing, the server.log have the same size to :
Capture d’écran de 2020-07-02 08-35-10

@timabbott
Copy link
Member

Zulip's default configuration will use memcached SASL2 authentication when connecting to memcached. Can you explain what you mean by "I added manually in memcached container"?

@timabbott timabbott mentioned this issue Jul 7, 2020
Open
@riadabdel
Copy link
Author

Hello, after 3 days of searching, i find the solution :D
So what i did:
In the file:
/home/zulip/deployments/current/zproject/default_settings.py
I replaced this line
MEMCACHED_USERNAME = None if get_secret("memcached_password") is None else "zulip"
By this:
MEMCACHED_USERNAME = 'zulip@localhost'

I have a question, where(how) the server decide to take settings from /home/zulip/deployments/current/zproject/default_settings.py instead of /etc/zulip/settings.py

thx for the support, i appreciate.
sorry for my english

@H8to H8to mentioned this issue Jul 18, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@maltokyo @timabbott @riadabdel