Scrapy Logging Levels

When executing a Scrapy spider using the command scrapy crawl spider_name, the terminal displays logging information generated by the framework.

Scrapy provides different logging levels:

• ERROR: Critical errors that affect the crawling process
• WARNING: Potential issues that don't stop execution
• INFO: General information about the crawling process
• DEBUG: Detailed debugging information

To specify which logging levels to output, add the following to your settings.py file:

LOG_LEVEL = 'desired_log_level'

To save logs to a file instead of displaying them in the terminal, use:

LOG_FILE = 'scrapy_logs.txt'

Passing Parameters Between Requests

In some scenarios, the data you need to scrape exists across multiple pages. For example, when scraping an e-commerce website, product names and prices might be on the main page, while detailed specifications are on individual product pages. In such cases, request parameter passing becomes essential.

Scrapy allows you to pass data between requests using the meta parameter.

Case Study: Scraping Book Information

Let's create a spider that scrapes book titles and ratings from a main page and publication dates, authors, and page counts from individual book detail pages.

Spider File:

# -*- coding: utf-8 -*-
import scrapy
from bookstore.items import BookItem

class BookSpider(scrapy.Spider):
    name = 'book'
    allowed_domains = ['www.example-books.com']
    start_urls = ['http://www.example-books.com/']

    def parse(self, response):
        book_containers = response.xpath('//div[@class="book-item"]')

        for container in book_containers:
            item = BookItem()
            item['title'] = container.xpath('.//h2/a/text()').extract_first()
            item['rating'] = container.xpath('.//span[@class="rating"]/text()').extract_first()
            item['category'] = container.xpath('.//div[@class="category"]/text()').extract_first()
            item['detail_url'] = container.xpath('./a/@href').extract_first()
            
            # Request detail page, passing item data via meta
            yield scrapy.Request(
                url=item['detail_url'], 
                callback=self.parse_detail, 
                meta={'book_item': item}
            )

    def parse_detail(self, response):
        # Retrieve item from meta
        item = response.meta['book_item']
        item['author'] = response.xpath('//div[@class="author"]/text()').extract_first()
        item['publish_date'] = response.xpath('//div[@class="publish-date"]/text()').extract_first()
        item['page_count'] = response.xpath('//div[@class="page-count"]/text()').extract_first()
        
        # Submit item to pipeline
        yield item

Items File:

# -*- coding: utf-8 -*-
import scrapy

class BookItem(scrapy.Item):
    title = scrapy.Field()
    rating = scrapy.Field()
    category = scrapy.Field()
    author = scrapy.Field()
    publish_date = scrapy.Field()
    page_count = scrapy.Field()
    detail_url = scrapy.Field()

Pipeline File:

# -*- coding: utf-8 -*-
import 

class BookPipeline(object):
    def __init__(self):
        self.file = open('books.', 'w')
    
    def process_item(self, item, spider):
        book_data = dict(item)
        .dump(book_data, self.file, ensure_ascii=False)
        return item
    
    def close_spider(self, spider):
        self.file.close()

Improving Scrapy Crawling Efficiency

To optimize your Scrapy spiders for better performance, consider the following configuration adjustments:

• Increase Concurrency: By default, Scrapy uses 32 concurrent threads. You can increase this value in settings.py:

  CONCURRENT_REQUESTS = 100

• Lower Log Level: Reduce CPU usage by setting less verbose logging:

  LOG_LEVEL = 'INFO'

• Disable Cookies: If cookies aren't necessary for your scraping task, disable them to reduce CPU overhead:

  COOKIES_ENABLED = False

• Disable Retry Mechanism: Preventing automatic retries for failed requests can speed up crawling:

  RETRY_ENABLED = False

• Reduce Download Timeout: Setting shorter timeouts helps quickly abandon unresponsive URLs:

  DOWNLOAD_TIMEOUT = 10

Case Study: Scraping Recipe Information

Here's an example spider that scrapes recipe names and images from a recipe website:

Spider File:

# -*- coding: utf-8 -*-
import scrapy
from recipeScraper.items import RecipeItem

class RecipeSpider(scrapy.Spider):
    name = 'recipe'
    allowed_domains = ['www.recipesite.com']
    start_urls = ['http://www.recipesite.com/recipes/']
    
    page_number = 1
    base_url = 'http://www.recipesite.com/recipes/page/%d'

    def parse(self, response):
        recipe_items = response.xpath('//div[@class="recipe-card"]')
        
        for item in recipe_items:
            name = item.xpath('./a/h3/text()').extract_first()
            image_url = item.xpath('./a/img/@src').extract_first()
            
            recipe_item = RecipeItem()
            recipe_item['name'] = name
            recipe_item['image_url'] = 'http://www.recipesite.com' + image_url
            
            yield recipe_item
        
        if self.page_number < 10:
            self.page_number += 1
            next_page_url = self.base_url % self.page_number
            yield scrapy.Request(url=next_page_url, callback=self.parse)

Items File:

# -*- coding: utf-8 -*-
import scrapy

class RecipeItem(scrapy.Item):
    name = scrapy.Field()
    image_url = scrapy.Field()

Pipeline File:

# -*- coding: utf-8 -*-
import 
import os
import urllib.request

class RecipePipeline(object):
    def __init__(self):
        self.output_file = None
    
    def open_spider(self, spider):
        self.output_file = open('./recipes.', 'w')
    
    def download_image(self, item):
        url = item['image_url']
        filename = item['name'].replace(' ', '_') + '.jpg'
        
        if not os.path.exists('./recipe_images'):
            os.makedirs('./recipe_images')
        
        filepath = os.path.join('./recipe_images', filename)
        urllib.request.urlretrieve(url, filepath)
        print(f"Downloaded: {filename}")
    
    def process_item(self, item, spider):
        recipe_data = dict(item)
        .dump(recipe_data, self.output_file, ensure_ascii=False)
        self.output_file.write('\n')
        
        # Download recipe images
        self.download_image(item)
        return item
    
    def close_spider(self, spider):
        self.output_file.close()

Configuration File (settings.py):

# User agent header
USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'

# Respect robots.txt
ROBOTSTXT_OBEY = False

# Increase concurrent requests
CONCURRENT_REQUESTS = 100

# Disable cookies for efficiency
COOKIES_ENABLED = False

# Set log level to reduce verbosity
LOG_LEVEL = 'ERROR'

# Disable retry mechanism
RETRY_ENABLED = False

# Set download timeout
DOWNLOAD_TIMEOUT = 3

# Add delay between requests
DOWNLOAD_DELAY = 3