About Tasks

Tasks are the basic unit to be scheduled.


  • A task is differentiated by its taskid. (Default: md5(url), can be changed by overriding the def get_taskid(self, task) method)
  • Tasks are isolated between different projects.
  • A Task has 4 status:
    • active
    • failed
    • success
    • bad - not used
  • Only tasks in active status will be scheduled.
  • Tasks are served in order of priority.


new task

When a new task (never seen before) comes in:

  • If exetime is set but not arrived, it will be put into a time-based queue to wait.
  • Otherwise it will be accepted.

When the task is already in the queue:

  • Ignored unless force_update

When a completed task comes out:

  • If age is set, last_crawl_time + age < now it will be accepted. Otherwise discarded.
  • If itag is set and not equal to it's previous value, it will be accepted. Otherwise discarded.

task retry

When a fetch error or script error happens, the task will retry 3 times by default.

The first retry will execute every time after 30 seconds, 1 hour, 6 hours, 12 hours and any more retries will postpone 24 hours.

If age is specified, the retry delay will not larger then age.

You can config the retry delay by adding a variable named retry_delay to handler. retry_delay is a dict to specify retry intervals. The items in the dict are {retried: seconds}, and a special key: '' (empty string) is used to specify the default retry delay if not specified.

e.g. the default retry_delay declares like:

class MyHandler(BaseHandler):
    retry_delay = {
        0: 30,
        1: 1*60*60,
        2: 6*60*60,
        3: 12*60*60,
        '': 24*60*60