When processing bulk data imports, it's common to filter out duplicates before insertion. Without LINQ, developers often resort to manually iterating through collections and using auxiliary lists to track uniqueness. LINQ’s Distinct() method simplifies this process—but its behavior depends heavily on how object equality is defined.

Consider a simple Person class:

public class Person
{
    public string Name { get; set; }
    public int Age { get; set; }

    public Person(string name, int age)
    {
        Name = name;
        Age = age;
    }
}

We create a list containing multiple Person instances, including duplicates by value (same name and age) but different references:

var personList = new List<Person>()
{
    new Person("ZhangSan", 26),
    new Person("XiaoMing", 25),
    new Person("CuiYanWei", 25),
    new Person("XiaoMing", 26),
    new Person("XiaoMing", 25), // Duplicate by value
    new Person("LaoWang", 26),
    new Person("XiaoMing", 26), // Duplicate by value
    new Person("ZhangSan", 26)  // Same reference as first
};

Now, applying Distinct() without a custom comparer:

var defaultDistinct = personList.Distinct().ToList();
foreach (var p in defaultDistinct)
    Console.WriteLine($"Name: {p.Name}, Age: {p.Age}");

This removes only the exact same reference (ZhangSan, 26 appears twice in the list, but both are the same object instance). The other duplicates—like two separate XiaoMing, 26 objects—are retained because Distinct() by default uses reference equality, not value equality.

To compare by property values, implement IEqualityComparer<T>:

public class PersonValueComparer : IEqualityComparer<Person>
{
    public bool Equals(Person x, Person y)
    {
        if (x == null || y == null)
            return false;
        return x.Name == y.Name && x.Age == y.Age;
    }

    public int GetHashCode(Person obj)
    {
        if (obj == null)
            return 0;
        return (obj.Name?.GetHashCode() ?? 0) ^ (obj.Age.GetHashCode() << 2);
    }
}

Use it like this:

var customDistinct = personList.Distinct(new PersonValueComparer()).ToList();
foreach (var p in customDistinct)
    Console.WriteLine($"Name: {p.Name}, Age: {p.Age}");

Now, all duplicates by name and age are removed. But why does this work? The key lies in how Distinct() internally uses hash codes.

When comparing objects, Distinct() first computes the hash code for each item. If two objects have different hash codes, they’re immediately considered unequal—no further comparision needed. If hash codes match, Equals() is invoked to confirm true equality.

That’s why overriding GetHashCode() is critical: if two logically equal objects return different hash codes, Equals() will never be called, and duplicates will persist. Conversely, if hash codes are poorly distributed (e.g., all return the same value), performance degrades due to excessive Equals() calls.

In the corrected PersonValueComparer, we combine the hash codes of Name and Age using XOR and bit shifting to ensure similar values produce distinct hashes while maintaining performence. This guarantees that objects with identical Name and Age will consistently produce the same hash, allowing Distinct() to correctly identify and eliminate duplicates.

Always ensure that:

• If Equals(x, y) returns true, then x.GetHashCode() must equal y.GetHashCode().
• Hash codes should be stable during the object’s lifetime.
• Hash code computation should be fast and minimize collisions.

Failure to adhere to these rules results in unpredictable behavior—no matter how correct your Equals() logic may be.