Taobao employs sophisticated anti-bot mechanisms that present significant challenges even for browser automation tools. While functional, the approach outlined here is more educational than practical for large-scale extraction.

Prerequisites

• Google Chrome browser
• ChromeDriver matching your Chrome version
• Python 3.x
• Selenium library (pip install selenium)
• lxml library (pip install lxml)

Core Implementation

The scraper targets product listings by keyword, sorting results by sales volume and extracting key metrics:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.common.exceptions import NoSuchElementException
from lxml import etree
import time
import random
import os

class TaobaoDataExtractor:
    def __init__(self, output_file='taobao_products.tsv'):
        self.output_path = output_file
        self.session_active = True
        
    def configure_browser(self, debug_mode=False):
        chrome_options = Options()
        chrome_options.add_argument('--disable-blink-features=AutomationControlled')
        chrome_options.add_argument('--no-sandbox')
        chrome_options.add_argument('--disable-dev-shm-usage')
        
        if debug_mode:
            chrome_options.add_experimental_option('debuggerAddress', '127.0.0.1:9222')
        
        return webdriver.Chrome(options=chrome_options)
    
    def parse_product_grid(self, page_content):
        html_parser = etree.HTML(page_content)
        product_entries = []
        
        # Use more robust selectors based on common Taobao page structure
        product_cards = html_parser.xpath('//div[contains(@class, "item")]')
        
        for card in product_cards[:50]:  # Limit to first 50 items
            try:
                name = card.xpath('.//div[contains(@class, "title")]//text()')
                price_whole = card.xpath('.//span[contains(@class, "price-int")]/text()')
                price_decimal = card.xpath('.//span[contains(@class, "price-float")]/text()')
                sales = card.xpath('.//div[contains(@class, "deal-cnt")]/text()')
                shop = card.xpath('.//a[contains(@class, "shopname")]//text()')
                
                if name and price_whole and price_decimal:
                    full_price = f"{price_whole[0]}.{price_decimal[0]}"
                    sales_volume = sales[0].replace('人付款', '') if sales else '0'
                    store_name = shop[0].strip() if shop else 'Unknown'
                    
                    product_entries.append({
                        'name': ''.join(name).strip(),
                        'price': full_price,
                        'sales': sales_volume,
                        'shop': store_name
                    })
            except Exception:
                continue
                
        return product_entries
    
    def execute_search(self, browser, search_term, max_pages=3):
        base_url = f"https://s.taobao.com/search?q={search_term}"
        browser.get(base_url)
        time.sleep(random.uniform(0.5, 1.0))
        
        # Sort by sales volume using data attribute
        try:
            sales_sort = browser.find_element(By.CSS_SELECTOR, 'li[data-sort="sale-desc"]')
            browser.execute_script("arguments[0].click();", sales_sort)
            time.sleep(random.uniform(0.5, 1.0))
        except NoSuchElementException:
            pass
        
        all_results = []
        
        for page_num in range(max_pages):
            # Scroll to ensure all items are loaded
            browser.find_element(By.TAG_NAME, 'body').send_keys(Keys.END)
            time.sleep(random.uniform(0.5, 1.0))
            
            # Scroll back to top for stability
            browser.find_element(By.TAG_NAME, 'body').send_keys(Keys.HOME)
            time.sleep(0.2)
            
            page_data = self.parse_product_grid(browser.page_source)
            all_results.extend(page_data)
            
            if page_num < max_pages - 1:
                try:
                    next_button = browser.find_element(By.CSS_SELECTOR, 'button.next-btn')
                    browser.execute_script("window.scrollBy(0, -400)")
                    next_button.click()
                    time.sleep(random.uniform(1.0, 1.5))
                except NoSuchElementException:
                    break
        
        return all_results

# Main execution block
if __name__ == '__main__':
    search_keywords = ['laptop computer', 'essential oil']
    extractor = TaobaoDataExtractor()
    
    # Check for existing progress
    start_index = 0
    if os.path.exists(extractor.output_path):
        with open(extractor.output_path, 'r', encoding='utf-8') as f:
            lines = f.readlines()
            if len(lines) > 1:
                last_line = lines[-1].strip().split('\t')
                if last_line and last_line[0] in search_keywords:
                    start_index = search_keywords.index(last_line[0])
                    print(f"Resuming from keyword: '{last_line[0]}' at position {start_index}")
    
    keywords_to_process = search_keywords[start_index:]
    
    # Initialize browser instance
    browser = extractor.configure_browser(debug_mode=False)
    
    file_mode = 'a' if start_index > 0 else 'w'
    with open(extractor.output_path, file_mode, encoding='utf-8') as output_file:
        if file_mode == 'w':
            output_file.write('\t'.join(['Search Term', 'Product Name', 'Price', 'Monthly Sales', 'Store Name']) + '\n')
        
        for keyword in keywords_to_process:
            print(f"Scraping results for: {keyword}")
            try:
                products = extractor.execute_search(browser, keyword)
                for product in products:
                    output_file.write('\t'.join([
                        keyword,
                        product['name'],
                        product['price'],
                        product['sales'],
                        product['shop']
                    ]) + '\n')
                output_file.flush()  # Ensure data is written immediately
            except Exception as error:
                print(f"Fatal error occurred: {error}")
                user_input = input("Exit scraping? (Enter '1' to quit, any other key to continue after manual intervention)\n")
                if user_input.strip() == '1':
                    print("Terminating scraping session...")
                    break
        
    browser.quit()

Verification Mechanism Workarounds

After approximately 10-15 requests, Taobao's verification system activates. The process requires manual intervention:

1. Initial Detection: Taobao identifies automated Chrome instances. The verification loop persists with "Click to retry" errors.

2. Manual Browser Launch: Open Chrome via command line with remote debugging:

   cd "C:\Program Files\Google\Chrome\Application"
   chrome.exe --remote-debugging-port=9222
   

3. Selenium Connection: Modify the browser initialization:

   chrome_options = Options()
   chrome_options.add_experimental_option("debuggerAddress", "127.0.0.1:9222")
   browser = webdriver.Chrome(options=chrome_options)
   

This approach reduces detection rates, though verifications still occur periodically.

Verification Types

Slider Captcha: Move the slider slowly toward the right edge without reaching the end completely. This gradual movement increases success probability.

Image Recognition: A more advanced challenge requiring users to align an object within a frame. While straightforward for humans, it interrutps automated workflows.

Rate Limiting Considerations

Verifications intensify with usage, sometimes triggering after every 1-3 keywords. The success rate diminishes significantly under frequent requests. Implementing delays of several minutes between search terms improves long-term viability.

When verification appears, complete it manually in the CMD window and press Enter to resume. Terminate the session by entering 1 followed by Enter.