How Can I Make Scrapy To Process The Url Sequentially

I have this code def parse(self, response): hxs = HtmlXPathSelector(response) sites = hxs.select('//div[@class='headline_area']') items = [] for site in sites[:5]

Solution 1:

Does the link has some regularity on different page? For example, if the 5th page's link is www.xxxx.net/nForum/#!article/Bet/447540?p=5. You can scrappy link with p=5 directly.

Solution 2:

You can use the inline_requests decorator.