Understanding Character Sets and Encoding Standards

Text processing in modern computing relies on two distinct layers: an abstract character repertoire and a concrete binary representation. Unicode acts as the universal character map, assigning a unique numeric code point to every glyph across all human writing systems. While Unicode defines what characters exist, encoding specifications determine how those code points translate into bytes.

UTF-8 is a variable-width encoding that allocates one to four bytes per code point. It maintains strict backward compatibility with ASCII, meaning standard Latin letters, digits, and common punctuation occupy a single byte. Less frequent scripts, mathematical symbols, and emojis expand to two, three, or four bytes as required. UTF-16, conversely, operates on 16-bit code units. The first 65,536 characters (the Basic Multilingual Plane) are represented directly. Characters outside this range utilize surrogate pairs, consuming 32 bits total. UTF-8 dominates web infrastructure, APIs, and file storage due to its compact representation of ASCII text and robust cross-platform compatibility.

Unicode-to-Base64 Conversion in JavaScript

Native JavaScript provides btoa() and atob() for Base64 transformation, but these functions exclusively handle 8-bit Latin-1 strings. Supplying raw multibyte characters (such as CJK ideographs, emojis, or accented European letters) triggers an InvalidCharacterError. To safely encode arbitrary Unicode text, the string must first be serialized into percent-escaped UTF-8 octets before Base64 encoding.

The following utility implements a reliable pipeline for bidirectional conversion:

function encodeToBase64(rawInput) {
  const utf8Sequence = encodeURIComponent(rawInput);
  return window.btoa(utf8Sequence);
}

function decodeFromBase64(encodedData) {
  const percentEscaped = window.atob(encodedData);
  return decodeURIComponent(percentEscaped);
}

// Execution flow
const sourceText = "数据编码示例";
const encodedPayload = encodeToBase64(sourceText);
const recoveredText = decodeFromBase64(encodedPayload);

console.log(encodedPayload);
console.log(recoveredText);

This approach guarantees that all code points are transformed into a safe ASCII-compatible format via URI encoding. The Base64 engine then processes the escaped octets without data loss. Decoding reverses the pipeline: Base64 restoration yields the percent-escaped sequence, wich decodeURIComponent maps back to the original multibyte characters.

Byte Serialization and Deserialization in C#

The .NET runtime represents strings internally as UTF-16, but external data exchange typically requires explicit encoding to UTF-8 or other byte-oriented formats. The System.Text.Encoding namespace provides static methods to bridge managed strings and raw byte arrays.

The example below demonstrates direct UTF-8 serialization, hex visualization, and reconstruction:

using System;
using System.Text;

namespace EncodingUtilities
{
    class ByteConverter
    {
        public static void Run() 
        {
            string inputMessage = "你好世界";
            
            // Serialize string to UTF-8 byte array
            byte[] rawBytes = Encoding.UTF8.GetBytes(inputMessage);
            
            // Format bytes as hexadecimal for inspection
            string hexRepresentation = BitConverter.ToString(rawBytes).Replace("-", " ");
            Console.WriteLine($"Encoded Bytes: {hexRepresentation}");
            
            // Deserialize bytes back to managed string
            string outputMessage = Encoding.UTF8.GetString(rawBytes);
            Console.WriteLine($"Decoded Text: {outputMessage}");
        }
        
        static void Main() => Run();
    }
}

Encoding.UTF8.GetBytes applies the UTF-8 algorithm to map each character's code units into a contiguous memory buffer. Encoding.UTF8.GetString reverses the operation by interpreting the octet stream according to UTF-8 continuation bit rules. In production environments, these operations are typically integrated with Stream readers and writers, accompanied by DecoderFallback configurations to gracefully handle malformed sequences during network transmission or file I/O.