Automating data extraction from recruitment platforms using point-and-click tools often introduces maintenance overhead and performance bottlenecks. A more efficient strategy involves intercepting and replicating the underlying network traffic. By inspecting the browser developer tools during a search operation, it becomes clear that pagination does not requirre full page reloads. Instead, the frontend triggers asynchronous requests to a dedicated API endpoint that returns structured job data.
The target endpoint accepts standard query parameters to filter results. The keyword identifier and page index are passed directly in the URL. The server responds with a JSON payload containing the complete dataset for the requested page. While the original implementation uses POST, network inspection confirms that a standard GET request yields identical results, simplifying the client-side implementation.
Architecture and Dependencies
The solution is built on the .NET Framework and leverages several established libraries. Network communication is handled via HttpWebRequest, HTML parsing relies on AngleSharp, and data persistence utilizes XDocument for XML serialization. Execution tracking and error reporting are managed through Log4Net. The core functionality is encapsulated within a dedicated service class, designed to process a batch of related keywords and consolidate the output into a single XML document.
Service Initialization
The scraper class accepts configuration parameters including the local SSL certificate path, output directory, target keywords, and a logging instance. These dependencies are injected via the constructor to promote loose coupling and testability.
public class LagouJobExtractor
{
private readonly string _certLocation;
private readonly string _outputDirectory;
private readonly string[] _searchTerms;
private readonly ILog _logger;
public LagouJobExtractor(string certLocation, string outputDirectory, string[] searchTerms, ILog logger)
{
_certLocation = certLocation;
_outputDirectory = outputDirectory;
_searchTerms = searchTerms;
_logger = logger;
}
}
Orchestration Logic
The primary execution method iterates through each keyword and paginates through the results. It constructs the request URL, bypases SSL certificate validation to prevent handshake failures, retrieves the JSON response, and processes the payload. Duplicate entries are tracked using a hash set of position identifiers to ensure data uniqueness across keyword variations.
private XDocument _document;
private XElement _rootNode;
private HashSet<int> _processedIds = new HashSet<int>();
private int _newRecords = 0;
private int _duplicateRecords = 0;
public int[] ExecuteExtraction()
{
_document = new XDocument();
_rootNode = new XElement("JobListings");
foreach (var term in _searchTerms)
{
for (int page = 1; page <= 30; page++)
{
string endpoint = $"https://www.lagou.com/jobs/positionAjax.json?px=new&needAddtionalResult=false&first=false&kd={term}&pn={page}";
ServicePointManager.ServerCertificateValidationCallback = (s, c, ch, e) => true;
string rawJson = RetrieveResource(endpoint);
var match = Regex.Match(rawJson, @"\[[\S\s]+\]");
if (!match.Success) break;
string formattedJson = $"{{\"result\":{match.Value}}}";
var serializer = new JavaScriptSerializer();
try
{
var payload = (Dictionary<string, object>)serializer.DeserializeObject(formattedJson);
var items = (object[])payload["result"];
foreach (var item in items)
{
var rawData = (Dictionary<string, object>)item;
var cleanData = new Dictionary<string, string>();
foreach (var pair in rawData)
{
string value = pair.Value?.ToString() ?? "";
if (value == "System.Object[]")
value = string.Join(" ", (object[])pair.Value);
cleanData[pair.Key] = PurgeControlCharacters(value);
}
if (!MapRecordToDocument(cleanData)) return new int[] { 0, 0 };
}
}
catch (Exception ex)
{
_logger.Error($"JSON parsing failed at {endpoint}: {ex.Message}");
return new int[] { 0, 0 };
}
}
}
try
{
if (_newRecords > 0)
{
_document.Add(_rootNode);
_document.Save(_outputDirectory);
_logger.Info($"Extraction complete. Added {_newRecords} records to {_outputDirectory}");
}
return new int[] { _newRecords, _duplicateRecords };
}
catch (Exception ex)
{
_logger.Error($"XML serialization failed at {_outputDirectory}: {ex.Message}");
return new int[] { 0, 0 };
}
}
Data Extraction and Mapping
Once the JSON payload is deserialized into a dictionary structure, each job record is validated against the duplicate registry. Valid records trigger a secondary request to fetch the full job description HTML. The DOM is parsed to extract additional fields such as company details, location, and role requirements. All text values undergo sanitization before being appended to the XML tree.
private bool MapRecordToDocument(Dictionary<string, string> record)
{
int jobId = Convert.ToInt32(record["positionId"]);
if (_processedIds.Contains(jobId))
{
_duplicateRecords++;
return true;
}
_processedIds.Add(jobId);
var jobNode = new XElement("Job", new XAttribute("id", jobId));
string detailUrl = $"https://www.lagou.com/jobs/{jobId}.html";
try
{
jobNode.SetElementValue("Title", record["positionName"]);
jobNode.SetElementValue("Salary", record["salary"]);
jobNode.SetElementValue("Experience", record["workYear"]);
jobNode.SetElementValue("Education", record["education"]);
jobNode.SetElementValue("Location", record["city"]);
jobNode.SetElementValue("Type", record["jobNature"]);
jobNode.SetElementValue("PostDate", Regex.Match(record["createTime"], @"\d{4}-\d{1,2}-\d{1,2}").Value);
jobNode.SetElementValue("DetailLink", detailUrl);
jobNode.SetElementValue("Benefits", record["positionAdvantage"]);
string detailHtml = RetrieveResource(detailUrl);
var parsedDoc = new HtmlParser().Parse(detailHtml);
jobNode.SetElementValue("Department", parsedDoc.QuerySelector("div.company").TextContent.Replace(record["companyShortName"], "").Replace("招聘", ""));
jobNode.SetElementValue("WorkAddress", parsedDoc.QuerySelector("div.work_addr").TextContent.Replace("\n", "").Replace(" ", "").Replace("查看地图", ""));
string description = parsedDoc.QuerySelector("dd.job_bt>div").TextContent;
description = string.Join("\n", description.Split(new[] { "\n ", " \n", "\n" }, StringSplitOptions.RemoveEmptyEntries));
jobNode.SetElementValue("Description", description);
jobNode.SetElementValue("CompanyWebsite", parsedDoc.QuerySelector("ul.c_feature a[rel=nofollow]").TextContent);
jobNode.SetElementValue("CompanyShort", record["companyShortName"]);
jobNode.SetElementValue("CompanyFull", record["companyFullName"]);
jobNode.SetElementValue("CompanySize", record["companySize"]);
jobNode.SetElementValue("FundingStage", record["financeStage"]);
jobNode.SetElementValue("Industry", record["industryField"]);
jobNode.SetElementValue("CompanyLink", $"https://www.lagou.com/gongsi/{record["companyId"]}.html");
_rootNode.Add(jobNode);
_newRecords++;
return true;
}
catch (Exception ex)
{
_logger.Error($"Failed mapping job ID {jobId}: {ex.Message}");
return false;
}
}
Network Request Handler
To mitigate rate limiting and IP bans, a deliberate delay is introduced between requests. The HTTP client is configured with browser-like headers, and auto-redirect is disabled to maintain control over the request lifecycle. The resposne stream is read into a UTF-8 string before being passed to the sanitizer.
private string RetrieveResource(string url)
{
Thread.Sleep(1500);
try
{
var request = (HttpWebRequest)WebRequest.Create(url);
request.ClientCertificates.Add(new X509Certificate(_certLocation));
request.Method = "GET";
request.KeepAlive = true;
request.Accept = "text/html, application/xhtml+xml, */*";
request.ContentType = "text/html";
request.UserAgent = "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)";
request.AllowAutoRedirect = false;
using var response = (HttpWebResponse)request.GetResponse();
using var memoryStream = new MemoryStream();
response.GetResponseStream().CopyTo(memoryStream);
byte[] data = memoryStream.ToArray();
string content = Encoding.UTF8.GetString(data);
return PurgeControlCharacters(content);
}
catch (Exception ex)
{
_logger.Error($"Request failed for {url}: {ex.Message}");
return string.Empty;
}
}
String Sanitization
Web content often contains non-printable control characters that can break XML serialization. A dedicated filtering method iterates through the character array, replacing problematic ASCII codes with safe placeholders while preserving standard whitespace and printable characters. This step is applied twice: once during network retrieval and again during dictionary mapping, to ensure complete compatibility with the XML writer.
private string PurgeControlCharacters(string input)
{
var builder = new StringBuilder(input.Length);
foreach (char c in input)
{
int code = (int)c;
if ((code >= 0 && code <= 8) || (code >= 11 && code <= 12) || (code >= 14 && code <= 31))
builder.Append(' ');
else
builder.Append(c);
}
return builder.ToString();
}
Execution Configuration
The console application initializes the logging framework, instantiates the scraper service with the desired keywords, and measures the total execution time. The output includes a summary of successfully processed records and any duplicates encountered during the run.
var timer = new Stopwatch();
timer.Start();
var certPath = @"C:\certs\lagou.cer";
var outputPath = @"C:\data\lagou_output.xml";
log4net.Config.XmlConfigurator.Configure();
var log = LogManager.GetLogger(typeof(Program));
var keywords = new[] { ".Net", "C#" };
var extractor = new LagouJobExtractor(certPath, outputPath, keywords, log);
var result = extractor.ExecuteExtraction();
timer.Stop();
if (result[0] + result[1] > 0)
{
Console.WriteLine($"Completed in {timer.Elapsed}. New: {result[0]}, Duplicates: {result[1]}");
}
else
{
Console.WriteLine("Extraction failed. Check logs for details.");
}