Detect whether hex-encoded bytes are ASCII, UTF-8, or UTF-16 with a BOM, by decode-validity and BOM presence (not statistical language detection).