Project Overview

With the rapid growth of online shopping, understanding market dynamics through data analysis has become increasingly valuable. This project demonstrates how to extract product information from a major e-commerce platform and perform various analytical tasks to uncover meaningful patterns.

Analysis Strategy

The analysis focuses on the following aspects:

• Extracting product data using Python web scraping techniques targeting the snacks category
• Investigating the correlation between pricing and sales volume
• Examining how pricing affects total revenue
• Evaluating shipping cost impact on purchase behavior
• Measuring shipping costs against gross merchandise volume
• Visualizing merchant distribution across different regions
• Creating 3D plots to reveal relationships between price, shipping, and sales
• Generating geographic heatmaps using mapping APIs

Data Collection Phase

The scraping module collects product data from search results, exporting the dataset to a Excel file for subsequent processing.

Data Points Collected

The following field are captured during the scraping process:

• Merchant name (nick)
• Merchant identifier (user_id)
• Product location (item_loc)
• Product price (view_price)
• Shipping fee (view_fee)
• Total sales count (view_sales)

Implementation Code

import requests
import re
import xlsxwriter

def generate_search_urls(num_pages):
    base_url = 'https://s.taobao.com/search?q=%E9%9B%B6%E9%A3%9F'
    url_list = []
    for page in range(num_pages):
        offset = str(20 * page)
        full_url = base_url + offset
        url_list.append(full_url)
    return url_list

def fetch_page_content(target_url):
    try:
        request_headers = {
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) '
                          'AppleWebKit/537.36 (KHTML, like Gecko) '
                          'Chrome/63.0.3239.84 Safari/537.36',
            'Cookie': 'thw=cn; t=2849901e12f931031b18c00202a1bb10; '
                      'cna=D858FVmxHTsCAd75gxEtAE8S; '
                      'tracknick=%5Cu5C0F%5Cu956D%5Cu5C0F%5Cu956D%5Cu5C0F%5Cu956D%5Cu554A'
        }
        response = requests.get(target_url, headers=request_headers, timeout=30)
        response.raise_for_status()
        response.encoding = response.apparent_encoding
        return response.text
    except:
        return 'Network error'

def parse_product_details(html_content):
    merchant_names = re.findall('"nick":"(.*?)"', html_content)
    merchant_ids = re.findall('"user_id":"(.*?)"', html_content)
    locations = re.findall('"item_loc":(.*?),', html_content)
    prices = re.findall('"view_price":(.*?),', html_content)
    shipping_fees = re.findall('"view_fee":(.*?),', html_content)
    sales_counts = re.findall('"view_sales":(.*?),', html_content)
    return merchant_names, merchant_ids, locations, prices, shipping_fees, sales_counts

def execute_scrape(num_pages):
    page_count = 0
    target_urls = generate_search_urls(num_pages)
    workbook = xlsxwriter.Workbook('c:\\python1\\ecommerce_data.xlsx')
    data_sheet = workbook.add_worksheet()
    headers = ['Merchant', 'Merchant ID', 'Location', 'Price', 'Shipping', 'Sales']
    data_sheet.write_row('A1', headers)
    data_sheet.set_column('A:D', 25)
    
    for url in target_urls:
        page_html = fetch_page_content(url)
        names, ids, locs, prcs, ships, sls = parse_product_details(page_html)
        data_sheet.write_column(1 + 20 * page_count, 0, names)
        data_sheet.write_column(1 + 20 * page_count, 1, ids)
        data_sheet.write_column(1 + 20 * page_count, 2, locs)
        data_sheet.write_column(1 + 20 * page_count, 3, prcs)
        data_sheet.write_column(1 + 20 * page_count, 4, ships)
        data_sheet.write_column(1 + 20 * page_count, 5, sls)
        page_count += 1
        if not names:
            print(f'Page {page_count} extraction failed')
        else:
            print(f'Page {page_count} extraction successful')
    workbook.close()

if __name__ == '__main__':
    pages_to_scrape = input('Enter number of pages to scrape (recommended: less than 10): ')
    execute_scrape(int(pages_to_scrape))

Data Preparation

Before analysis, preprocess the data:

1. Rename column headers for easier coding: NAME, ID, LOC, PRICE, FEE, SALES
2. Open the CSV file in a text editor and remove quotation marks to convert values from object to float types

Analysis Implementation

Price and Sales Correlation

import seaborn as sns
import pandas as pd
import matplotlib.pyplot as plt

product_df = pd.read_csv('c:/python1/ecommerce_data.csv')
sns.set(style='darkgrid')
sns.jointplot(x='PRICE', y='SALES', data=product_df, kind='scatter', color='purple')
plt.show()

Observations:

• Products priced between 0 and 60 yuan show the highest sales volume
• Sales decline as price increases beyond the optimal range
• Peak sales occur around the 60 yuan price point
• The 10-40 yuan range demonstrates consistently strong sales performance

Revenue Impact Analysis

product_df['TOTAL_REVENUE'] = product_df['PRICE'] * product_df['SALES']
sns.regplot(x='PRICE', y='TOTAL_REVENUE', data=product_df, color='purple')
plt.show()

Key Findings:

• Maximum revenue generation occurs at the 60 yuan price point
• Secondary revenue peaks exist in the 20-40 yuan bracket
• Strategic pricing at 60 yuan can optimize total sales revenue

Shipping Fee and Sales Relationship

sns.set(style='darkgrid')
sns.jointplot(x='FEE', y='SALES', data=product_df, kind='scatter', color='purple')
plt.show()

Insights:

• Free shipping (0 yuan) correlates with highest purchase frequency
• Sales decrease proportionally with increasing shipping costs up to 50 yuan
• An anomaly at the 50 yuan shipping mark shows unusually high sales, suggesting premium imported goods

Shipping Cost and Revenue Correlation

product_df['TOTAL_REVENUE'] = product_df['PRICE'] * product_df['SALES']
sns.regplot(x='FEE', y='TOTAL_REVENUE', data=product_df, color='purple')
plt.show()

Conclusions:

• Zero shipping cost products generate both maximum sales and revenue
• Revenue and shipping fees show an inverse relationship

Regional Distribution Analysis

plt.rcParams['font.sans-serif'] = ['SimHei']
plt.rcParams['axes.unicode_minus'] = False
plt.figure(figsize=(8, 4))
product_df['region'] = product_df['LOC']
product_df['region'].value_counts().plot(kind='bar', color='purple')
plt.xticks(rotation=90)
plt.xlabel('Region')
plt.ylabel('Merchant Count')
plt.title('Merchant Distribution by Region')
plt.show()

Distribution Pattern:

• Shanghai dominates merchant concentration
• Coastal cities including Nanjing, Hangzhou, and Shenzhen follow
• Merchant density correlates strongly with regional economic development

Three-Dimensional Analysis

import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
import numpy as np

price_range, shipping_range = np.mgrid[-2:2:20j, -2:2:20j]
volume_values = price_range * np.exp(-price_range**2 - shipping_range**2)
figure = plt.figure()
axis = figure.add_subplot(111, projection='3d')
axis.plot_surface(price_range, shipping_range, volume_values, 
                  rstride=2, cstride=1, cmap='Blues_r')
axis.set_xlabel('PRICE')
axis.set_ylabel('SHIPPING')
axis.set_zlabel('SALES')
plt.show()

Visualization Results:

• Price and sales volume demonstrate an inverse relationship
• Shipping costs and sales also show inverse correlation
• No direct correlation exists between price and shipping cost

Geographic Heatmap Generation

To create an interactive heatmap showing merchant distribution across China, obtain coordinates through geocoding:

1. Register at the Baidu Maps Open Platform to receive an API key (AK)
2. Reference the JavaScript API overlay examples for the base HTML structure
3. Use Python to retrieve latitude and longitude coordinates for each location

import json
from urllib.request import urlopen, quote
import csv

def geocode_location(address):
    api_endpoint = 'http://api.map.baidu.com/geocoder/v2/'
    output_format = 'json'
    api_key = 'YOUR_BAIDU_API_KEY'
    encoded_address = quote(address)
    request_uri = f'{api_endpoint}?address={encoded_address}&output={output_format}&ak={api_key}'
    response = urlopen(request_uri)
    json_response = response.read().decode()
    parsed_data = json.loads(json_response)
    return parsed_data

output_file = open(r'c:/python1/coordinates.json', 'w')
with open(r'c:/python1/ecommerce_data.csv', 'r') as csv_file:
    csv_reader = csv.reader(csv_file)
    for row in csv_reader:
        if csv_reader.line_num == 1:
            continue
        location_name = row[0].strip()
        count_value = row[1].strip()
        geo_result = geocode_location(location_name)
        longitude = geo_result['result']['location']['lng']
        latitude = geo_result['result']['location']['lat']
        coordinate_entry = f'{{"lat":{latitude},"lng":{longitude},"count":{count_value}}}'
        output_file.write(coordinate_entry)
output_file.close()

Replace the coordinates in the HTML template with the generated data to produce the heatmap visualization.