python - Django Celery ConnectionError: Too many heartbeats missed -

python - Django Celery ConnectionError: Too many heartbeats missed -

question

how can solve connectionerror: many heartbeats missed celery?

example error [2013-02-11 15:15:38,513: error/mainprocess] error in timer: connectionerror('too many heartbeats missed', none, none, none, '') traceback (most recent phone call last): file "/app/.heroku/python/lib/python2.7/site-packages/celery/utils/timer2.py", line 97, in apply_entry entry() file "/app/.heroku/python/lib/python2.7/site-packages/celery/utils/timer2.py", line 51, in __call__ homecoming self.fun(*self.args, **self.kwargs) file "/app/.heroku/python/lib/python2.7/site-packages/celery/utils/timer2.py", line 153, in _reschedules homecoming fun(*args, **kwargs) file "/app/.heroku/python/lib/python2.7/site-packages/kombu/connection.py", line 265, in heartbeat_check homecoming self.transport.heartbeat_check(self.connection, rate=rate) file "/app/.heroku/python/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 134, in heartbeat_check homecoming connection.heartbeat_tick(rate=rate) file "/app/.heroku/python/lib/python2.7/site-packages/amqp/connection.py", line 837, in heartbeat_tick raise connectionerror('too many heartbeats missed') connectionerror: many heartbeats missed app overview django app using celery periodic background tasks hosted on heroku single task scheduled run every 15 minutes via settings / celerybeat messaging handled via cloudamqp add-on processes run web: newrelic-admin run-program gunicorn --workers=2 --worker-class=gevent someapp.wsgi:application scheduler: newrelic-admin run-program python manage.py celery worker -b -e --maxtasksperchild=1000 --loglevel=warning package versions

just think relevant:

django==1.4.3 amqp==1.0.8 billiard==2.7.3.20 celery==3.0.14 gevent==0.13.8 greenlet==0.4.0 kombu==2.5.6 raven==3.1.10 what i've tried far correlating error activities (doesn't seem correlate user's visiting app, background tasks beingness called, app idling) googling / searching until fingers numb upgrading packages latest versions various levels of logging exception capturing sentry (doesn't appear in sentry) cannot reproduce error locally under development environment, in production on heroku possible relevant info i'm not sure when error first appeared (~ 1 month ago?) may related in way next changes (don't recall error before this, not 100% sure though) celery==3.0.13 celery==3.0.14 amqplib -> amqp kombu==2.4.8 kombu==2.5.4 error appears in logs (doesn't picked new relic or getsentry.com)

how happen?

it may heartbeat monitoring not working in case. heartbeat back upwards introduced recently, there may bugs. cannot reproduce here, need more info understand going on.

you can disable heartbeats setting broker_heartbeat=0. if bug worker should run fine, not able observe broken connection. beingness unable observe connection loss problem in environments (usually caused specific router/firewall configurations)

python django heroku celery django-celery