How Everything Works

Everything achieves remarkable search performance by leveraging NTFS filesystem features rather than traditional directory traversal. When first launched, the appplication scans the entire disk and builds an index database. The speed advantage comes from two core NTFS mechanisms: the Master File Table (MFT) and USN journal.

Master File Table (MFT)

NTFS maintains a special table called the Master File Table that stores metadata for all files and directories. Everything traverses this table directly to capture file names and paths with out recursively walking through the filesystem. This approach is significantly faster than conventional file system enumeration.

USN Journal

The Update Sequence Number (USN) journal acts as a change log for the NTFS filesystem. Every modification to the filesystem is recorded in this journal. Everything monitors this log to keep its index synchronized with actual file system changes.

Search Mechanism

After the index is built, search operations perform simple string matching against the cached data. This pre-computed index eliminates the need for real-time filesystem access during searches.

Building a Custom File Search Implementation

Step 1: Establishing the Database Index

The first task involves creating a comprehensive index of file paths. SQLite serves as the storage backend. The core indexing mechanism uses USN journal APIs to enumerate files and reconstruct their full paths.

Obtaining a Volume Handle

Use CreateFile to open the target drive and obtain a handle:

bool getVolumeHandle()
{
    CString devicePath(_T("\\\\.\\C:"));
    devicePath.SetAt(4, driveLetter);
    
    volumeHandle = CreateFile(
        devicePath,
        GENERIC_READ | GENERIC_WRITE,
        FILE_SHARE_READ | FILE_SHARE_WRITE,
        nullptr,
        OPEN_EXISTING,
        FILE_ATTRIBUTE_READONLY,
        nullptr);

    return volumeHandle != INVALID_HANDLE_VALUE;
}

Creating the USN Journal

Send the FSCTL_CREATE_USN_JOURNAL command via DeviceIoControl:

bool createJournal()
{
    CREATE_USN_JOURNAL_DATA createData = {};
    createData.MaximumSize = 0;
    createData.AllocationDelta = 0;
    
    DWORD bytesReturned;
    return DeviceIoControl(
        volumeHandle,
        FSCTL_CREATE_USN_JOURNAL,
        &createData,
        sizeof(createData),
        nullptr,
        0,
        &bytesReturned,
        nullptr) != 0;
}

Querying Journal Information

Retrieve current journal state with FSCTL_QUERY_USN_JOURNAL:

bool queryJournalInfo(USN_JOURNAL_DATA& journalData)
{
    DWORD bytesReturned;
    return DeviceIoControl(
        volumeHandle,
        FSCTL_QUERY_USN_JOURNAL,
        nullptr,
        0,
        &journalData,
        sizeof(journalData),
        &bytesReturned,
        nullptr) != 0;
}

Enumerating USN Records

Iterate through all entries using FSCTL_ENUM_USN_DATA:

bool enumerateRecords()
{
    MFT_ENUM_DATA enumData = {};
    enumData.StartFileReferenceNumber = 0;
    enumData.LowUsn = journalInfo.FirstUsn;
    enumData.HighUsn = journalInfo.NextUsn;
    
    constexpr size_t BUFFER_SIZE = 0x3900;
    char buffer[BUFFER_SIZE];
    
    while (DeviceIoControl(
        volumeHandle,
        FSCTL_ENUM_USN_DATA,
        &enumData,
        sizeof(enumData),
        buffer,
        BUFFER_SIZE,
        &bytesReturned,
        nullptr))
    {
        DWORD recordDataSize = bytesReturned - sizeof(USN);
        auto recordPtr = reinterpret_cast<PUSN_RECORD>(
            buffer + sizeof(USN));
        
        while (recordDataSize > 0)
        {
            FileEntry entry;
            entry.name = CString(
                recordPtr->FileName,
                recordPtr->FileNameLength / 2);
            entry.parentRef = recordPtr->ParentFileReferenceNumber;
            
            fileIndex[recordPtr->FileReferenceNumber] = entry;
            
            recordDataSize -= recordPtr->RecordLength;
            recordPtr = reinterpret_cast<PUSN_RECORD>(
                reinterpret_cast<char*>(recordPtr) + 
                recordPtr->RecordLength);
        }
        
        enumData.StartFileReferenceNumber = 
            *reinterpret_cast<USN*>(buffer);
    }
    return true;
}

The USN record structure contains essential fields:

struct USN_RECORD_V2 {
    DWORD RecordLength;
    WORD MajorVersion;
    WORD MinorVersion;
    DWORDLONG FileReferenceNumber;
    DWORDLONG ParentFileReferenceNumber;
    USN Usn;
    LARGE_INTEGER TimeStamp;
    DWORD Reason;
    DWORD SourceInfo;
    DWORD SecurityId;
    DWORD FileAttributes;
    WORD FileNameLength;
    WORD FileNameOffset;
    WCHAR FileName[1];
};

The critical fields are FileReferenceNumber, ParentFileReferenceNumber, and FileName. The USN journal maintains a tree structure where path reconstruction happens by following parent references upward until reaching the root.

Path Reconstruction

Traverse the parent chain to build complete file paths:

void buildFullPath(DWORDLONG fileRef, CString& resultPath)
{
    auto endIterator = fileIndex.end();
    while (true)
    {
        auto entry = fileIndex.find(fileRef);
        if (entry == endIterator)
        {
            resultPath = L":" + resultPath;
            return;
        }
        resultPath = _T("\\") + entry->second.name + resultPath;
        fileRef = entry->second.parentRef;
    }
}

Cleaning Up

Remove the temporary journal after indexing:

bool removeJournal()
{
    DELETE_USN_JOURNAL_DATA deleteData = {};
    deleteData.UsnJournalID = journalInfo.UsnJournalID;
    deleteData.DeleteFlags = USN_DELETE_FLAG_DELETE;
    
    DWORD bytesReturned;
    bool success = DeviceIoControl(
        volumeHandle,
        FSCTL_DELETE_USN_JOURNAL,
        &deleteData,
        sizeof(deleteData),
        nullptr,
        0,
        &bytesReturned,
        nullptr) != 0;
    
    CloseHandle(volumeHandle);
    return success;
}

Step 2: Monitoring File System Changes

Track modifications using ReadDirectoryChangesW. The API reports four action types: file creation, modification, deletion, and rename operations.

void watchDirectory(const char* directoryPath)
{
    char notifyBuffer[1024];
    char fileNameBuffer[1000];
    char renamedBuffer[1000];
    
    WCHAR widePath[1000];
    MultiByteToWideChar(CP_ACP, 0, directoryPath, 
        strlen(directoryPath), widePath, 1000);
    
    HANDLE dirHandle = CreateFile(
        widePath,
        GENERIC_READ | GENERIC_WRITE | FILE_LIST_DIRECTORY,
        FILE_SHARE_READ | FILE_SHARE_WRITE,
        nullptr,
        OPEN_EXISTING,
        FILE_FLAG_BACKUP_SEMANTICS,
        nullptr);
    
    if (dirHandle == INVALID_HANDLE_VALUE)
    {
        return;
    }
    
    auto notifyInfo = reinterpret_cast<FILE_NOTIFY_INFORMATION*>(
        notifyBuffer);
    
    while (isMonitoring)
    {
        DWORD bytesReturned;
        if (ReadDirectoryChangesW(
            dirHandle,
            notifyBuffer,
            1024,
            TRUE,
            FILE_NOTIFY_CHANGE_FILE_NAME | 
            FILE_NOTIFY_CHANGE_DIR_NAME | 
            FILE_NOTIFY_CHANGE_SIZE,
            &bytesReturned,
            nullptr,
            nullptr))
        {
            if (notifyInfo->FileNameLength > 0)
            {
                WideCharToMultiByte(CP_ACP, 0, 
                    notifyInfo->FileName,
                    notifyInfo->FileNameLength / 2,
                    fileNameBuffer, 250, nullptr, nullptr);
            }
            
            if (notifyInfo->NextEntryOffset != 0)
            {
                auto nextEntry = reinterpret_cast<PFILE_NOTIFY_INFORMATION>(
                    reinterpret_cast<char*>(notifyInfo) + 
                    notifyInfo->NextEntryOffset);
                WideCharToMultiByte(CP_ACP, 0,
                    nextEntry->FileName,
                    nextEntry->FileNameLength / 2,
                    renamedBuffer, 250, nullptr, nullptr);
            }
            
            string fullPath = directoryPath;
            fullPath += fileNameBuffer;
            
            switch (notifyInfo->Action)
            {
            case FILE_ACTION_ADDED:
                if (!containsRecycleBin(fileNameBuffer))
                {
                    insertIndexEntry(fullPath);
                }
                break;
                
            case FILE_ACTION_MODIFIED:
                if (!containsRecycleBin(fileNameBuffer) &&
                    !isTemporaryFile(fileNameBuffer))
                {
                    insertIndexEntry(fullPath);
                }
                break;
                
            case FILE_ACTION_REMOVED:
                if (!containsRecycleBin(fileNameBuffer))
                {
                    removeIndexEntry(fullPath);
                }
                break;
                
            case FILE_ACTION_RENAMED_OLD_NAME:
                if (!containsRecycleBin(fileNameBuffer))
                {
                    removeIndexEntry(fullPath);
                    fullPath = directoryPath;
                    fullPath += renamedBuffer;
                    insertIndexEntry(fullPath);
                }
                break;
            }
        }
    }
    CloseHandle(dirHandle);
}

Step 3: Implementing the User Interface

The search interface requires a text input field with real-time response. Listen for document changes and trigger asynchronous search operations:

private void setupSearchListener() {
    searchField.getDocument().addDocumentListener(new DocumentListener() {
        @Override
        public void insertUpdate(DocumentEvent e) {
            resetDisplay();
            if (shouldExecuteSearch()) {
                startSearchTimer();
                submitSearchSignal();
            }
        }
        
        @Override
        public void removeUpdate(DocumentEvent e) {
            resetDisplay();
            if (searchField.getText().isEmpty()) {
                clearResults();
                cancelSearchSignal();
            } else if (shouldExecuteSearch()) {
                startSearchTimer();
                submitSearchSignal();
            }
        }
        
        @Override
        public void changedUpdate(DocumentEvent e) {
            cancelSearchSignal();
        }
    });
}

Dynamic Query Generation

Generate SQL queries based on search parameters and priority mapping:

private Map<Map<String, String>, Set<String>> constructQueries() {
    Map<Map<String, String>, Set<String>> queryMap = new LinkedHashMap<>();
    
    if (priorityList.isEmpty()) {
        return queryMap;
    }
    
    String tableName = determineTargetTable();
    
    for (PriorityLevel level : priorityList) {
        Map<String, String> querySet = new LinkedHashMap<>();
        String sql = String.format("SELECT %%s FROM %s WHERE PRIORITY=%d",
            tableName, level.getValue());
        querySet.put(sql, tableName);
        
        for (String altTable : alternativeTables) {
            if (!altTable.equals(tableName)) {
                String altSql = String.format("SELECT %%s FROM %s WHERE PRIORITY=%d",
                    altTable, level.getValue());
                querySet.put(altSql, altTable);
            }
        }
        
        Set<String> resultContainer = new ConcurrentSkipListSet<>();
        queryMap.put(querySet, resultContainer);
    }
    
    return queryMap;
}

Pattern Matching

Filter results based on user-specified criteria. The matching logic supports case-insensitive search, path-based filtering, and optional pinyin matching for Chinese characters:

private boolean failsToMatch(String filePath, boolean ignoreCase, 
    String[] searchTerms) {
    for (String term : searchTerms) {
        if (term == null || term.isEmpty()) continue;
        
        boolean isPathSearch = term.charAt(0) == '/' || 
            term.charAt(0) == File.separatorChar;
        
        String targetString;
        if (isPathSearch) {
            targetString = extractParentPath(filePath);
            term = stripPathSeparators(term);
        } else {
            targetString = extractFileName(filePath);
        }
        
        if (ignoreCase) {
            targetString = targetString.toLowerCase();
            term = term.toLowerCase();
        }
        
        if (targetString.indexOf(term) == -1) {
            if (isPathSearch) {
                return true;
            } else {
                if (containsChinese(targetString)) {
                    if (!convertToPinyin(targetString).contains(term)) {
                        return true;
                    }
                } else {
                    return true;
                }
            }
        }
    }
    return false;
}

The string matching avoids complex algorithms like KMP because file paths lack repetitive patterns, and typical path lengths don't benefit from preprocessing overhead.

Result Display

Render search results in a scrollable list, updating dynamically as matches are found.