Running tasks in the background using Celery¶
Background tasks are run using Celery, which delegates to Redis.
The project template’s docker-compose.yml
file configures redis
,
celery
, celerybeat
and celeryflower
machines.
Running background tasks¶
See the Celery, documentation for writing and running background tasks that take place in response to an event.
In development¶
The Celery-related services are started when you run docker-compose run
.
If you’re not running Docker (and have Redis running on your machine) then, in 3 separate tabs, run:
./go.sh celery.sh # celery task runner
./go.sh celerybeat.sh # celery task scheduler
./go.sh celeryflower.sh # celery inspection service
On a server¶
Docker uses the included docker-compose.yml
file to deploy your application
with all necessary services, including the Celery-related ones.
For more information, see Deploying your site.
Scheduling tasks¶
Tasks that need to run regularly are scheduled in crontab
-like fashion
using CeleryBeat.
You can define your tasks, wherever you like (we suggest in a file called tasks.py). For example:
from celery.task import Task
from django.core.management import call_command
from icekit.tasks import one_instance
class UpdateSearchIndexTask(Task):
@one_instance(key='UpdateSearchIndexTask')
def run(self, **kwargs):
call_command('update_index', remove=True)
The one_instance
decorator
The icekit.tasks.one_instance
decorator uses a Redis token to ensure that
only one instance of the function will run at a time. This is useful in
case tasks take longer to run than the interval between calls, but means
that if tasks crash that the lock won’t be released until
settings.DEFAULT_ONE_INSTANCE_TIMEOUT
seconds (default 7200) have
elapsed.
Then you need to specify the task in a schedule, which is defined in
settings.CELERYBEAT_SCHEDULE
, in this format:
from celery.schedules import crontab
CELERYBEAT_SCHEDULE = {
'UpdateSearchIndexTask': {
'task': 'icekit.tasks.UpdateSearchIndexTask',
'schedule': crontab(minute='*/15'), # Every 15 minutes.
},
}
Add your tasks to the dictionary to include them in the scheduler.
Troubleshooting¶
You can use Celery Flower to monitor running tasks. Visit http://localhost:5555 (or http://{{yourproject}}.lvh.me:5555 )to see the interface.
Note
Restarting Flower (e.g. on deployment) will clear its records of tasks, so use with caution in production environments.
If Celery tasks fail, they will show up as failures. However, if celery tasks
‘succeed’ but with a very quick runtime (0.02 secs), the task
is possibly prevented from running by a one_instance
lock.
Logging¶
Using Docker Cloud, you can also check the log (stdout) of the Celery Docker container for actual tasks being executed. Check the logs (stdout) of the Celery Beat container to ensure the scheduled tasks process is running.
See the Celery Logging docs for how to log, e.g.:
from celery.utils.log import get_task_logger
logger = get_task_logger(__name__)
@app.task
def add(x, y):
logger.info('Adding {0} + {1}'.format(x, y))
return x + y
Freeing task locks¶
Locks are stored in Redis as keys with a TTL. You can see the locks in a Python shell with:
>>> import redis
>>> REDIS_CLIENT = redis.Redis()
>>> [x for x in REDIS_CLIENT.keys() if 'Task' in x]
['FetchCollectionDataTask', 'FindAndFetchNetxImagesTask', 'DeleteRemovedNetxImagesTask']
>>> REDIS_CLIENT.pttl('FindAndFetchNetxImagesTask') # ms to live
7693886L
(note that 'Task' in x
part assumes that your CELERYBEAT_SCHEDULE
entries have Task
in the key name. You can delete a key, thus freeing up
the lock, with:
>>> REDIS_CLIENT.delete('FindAndFetchNetxImagesTask')
(a response of 1 means the lock was deleted, a response of 0 means it was not found.)