Overview

This guide demonstrates how to build a Windows desktop application using C# to extract novel content from websites. Many web novels lack official download options, but since they can be viewed in a browser, the content can typically be retrieved by parsing the underlying HTML structure.

Prerequisites

• Visual Studio 2022
• .NET Framework 2.0 for broad compatibility
• A target novel website for testing

Implementation Steps

1. Analyzing Website HTML Structure

Open the target novel page in Chrome and press F12 to access developer tools. Alternatively, save the webpage as HTML using "Save As" from the context menu, then open the file in a text editor to examine the structure. Identify the HTML elements containing the chapter content you want to extract.

2. Creating the Project

Create a new Windows Forms Application in Visual Studio. If you prefer a console-based approach instead of a GUI, select the Console Application template during project creation.

3. Installing Required Packages

Navigate to Tools → NuGet Package Manager → Manage NuGet Packages for Solution. In the Browse tab, search for "HtmlAgilityPack". Select the package and click Install. Once installed, the reference appears in the Solution Explorer's References folder.

4. Designing the Form

Change the form's Text property to set a descriptive window title. Add the following controls:

• TextBox: For entering the target URL
• Button: To trigger the scraping operation
• ListBox: For displaying status messages and progress

Double-click the button to generate the click event handler.

5. Writing the Code

Initializing Form Constructor

Add the following line to the form constructor:

CheckForIllegalCrossThreadCalls = false;

This allows cross-thread access to UI controls from background operations.

Status Message Helper

Add a method to the Form class for displaying status updates:

private void UpdateStatusDisplay(string message)
{
    listbStatus.Items.Add($"{DateTime.Now:HH:mm:ss}: {message}");
    listbStatus.SelectedIndex = listbStatus.Items.Count - 1;
    
    while (listbStatus.Items.Count > 50)
    {
        listbStatus.Items.RemoveAt(0);
    }
}

This function appends timestamped messages to the ListBox, automatically scrolls to the newest entry, and maintains a rolling buffer of 50 entries by removing the oldest item.

HTML Retrieval Function

Add this function to the Form class:

private string FetchPageContent(string targetUrl)
{
    string pageContent = string.Empty;
    
    try
    {
        HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(targetUrl);
        webRequest.Method = "GET";
        webRequest.Timeout = 5000;
        
        HttpWebResponse webResponse = (HttpWebResponse)webRequest.GetResponse();
        StreamReader contentReader = new StreamReader(
            webResponse.GetResponseStream(), 
            Encoding.UTF8
        );
        pageContent = contentReader.ReadToEnd();
        contentReader.Close();
        webResponse.Close();
    }
    catch (Exception ex)
    {
        UpdateStatusDisplay($"Error retrieving page: {ex.Message}");
    }
    
    return pageContent;
}

Content Parsing Function

Add a method to extract novel content from the HTML:

private string ExtractNovelContent(string htmlContent)
{
    HtmlDocument document = new HtmlDocument();
    document.LoadHtml(htmlContent);
    
    HtmlNodeCollection contentNodes = document.DocumentNode
        .SelectNodes("//div[@class='novel-content']");
    
    if (contentNodes == null)
    {
        return string.Empty;
    }
    
    StringBuilder contentBuilder = new StringBuilder();
    foreach (HtmlNode node in contentNodes)
    {
        contentBuilder.AppendLine(node.InnerText);
    }
    
    return contentBuilder.ToString();
}

Adjust the XPath selector to match the actual HTML structure of your target website.

Button Click Handler

Implement the button's click event:

private void btnStartScraping_Click(object sender, EventArgs e)
{
    string inputUrl = txtUrl.Text.Trim();
    
    if (string.IsNullOrEmpty(inputUrl))
    {
        UpdateStatusDisplay("Please enter a valid URL");
        return;
    }
    
    UpdateStatusDisplay("Initiating page fetch...");
    string htmlData = FetchPageContent(inputUrl);
    
    if (string.IsNullOrEmpty(htmlData))
    {
        UpdateStatusDisplay("Failed to retrieve page content");
        return;
    }
    
    UpdateStatusDisplay("Parsing novel content...");
    string novelText = ExtractNovelContent(htmlData);
    
    if (!string.IsNullOrEmpty(novelText))
    {
        System.IO.File.WriteAllText("novel_output.txt", novelText);
        UpdateStatusDisplay("Content saved to novel_output.txt");
    }
    else
    {
        UpdateStatusDisplay("No content extracted - verify XPath selector");
    }
}

6. Testing the Application

Run the application, paste a novel chapter URL into the text field, and click the button. Monitor the ListBox for status updates. Successfully extracted content saves to novel_output.txt in the application's working directory.

7. Handling Pagination

To scrape multiple chapters, implement navigation by locating "Next Chapter" links in the HTML and iterating through them:

private List<string> GatherChapterUrls(string startUrl)
{
    List<string> chapterUrls = new List<string>();
    string currentUrl = startUrl;
    
    while (!string.IsNullOrEmpty(currentUrl))
    {
        chapterUrls.Add(currentUrl);
        string html = FetchPageContent(currentUrl);
        
        HtmlDocument doc = new HtmlDocument();
        doc.LoadHtml(html);
        
        HtmlNode nextLink = doc.DocumentNode
            .SelectSingleNode("//a[contains(text(),'Next')]");
        
        currentUrl = nextLink?.GetAttributeValue("href", string.Empty);
    }
    
    return chapterUrls;
}

This approach collects sequential chapter URLs until no "Next" link is found, enabling batch extraction of entire novels.