fix reality handshake

[SKTT1Ryze]

VLESS Reality Protocol: Fix AES-256-GCM and Implement RealityServerCertVerifier

Summary

This PR fixes a critical authentication bug in the Reality protocol implementation
and completes the handshake by adding a custom certificate verifier that understands
Reality's non-standard server certificate format.

---

Background

VLESS Reality is a TLS extension used by proxy
servers to authenticate clients without being distinguishable from normal TLS traffic.
It works as follows:

1. The client generates an ephemeral X25519 keypair.
2. The client performs ECDH with the server's static public key to derive an
   auth_shared_secret.
3. auth_key = HKDF-SHA256(auth_shared_secret, salt=random[:20], info="REALITY") —
   32 bytes.
4. The client encrypts a 16-byte plaintext [version(3) | reserved(1) | timestamp(4) | short_id(8)] with AES-256-GCM (key=auth_key, nonce=random[20:32],
   aad=ClientHello bytes) and places the 32-byte result in the TLS session_id
   field.
5. The same ephemeral X25519 keypair is used for the normal TLS 1.3 key exchange
   (ECDH with the server's ephemeral key from ServerHello).
6. The server verifies the session_id; on failure it transparently proxies the
   connection to the SNI destination (fallback).
7. On success the server presents a minimal Ed25519 X.509v1 certificate whose last
   64 bytes are overwritten with HMAC-SHA512(auth_key, ed25519_pubkey). Standard
   CA-chain verifiers reject this cert as BadEncoding.

---

Bug Fix: AES-128-GCM → AES-256-GCM

The previous implementation derived only 16 bytes from HKDF and used AES-128-GCM,
while the Xray-core reference implementation
derives 32 bytes and uses AES-256-GCM. This caused every Reality handshake to
fail authentication (server received an invalid session_id and fell back to the SNI
destination).

- let mut auth_key = [0u8; 16];
- auth_key_expander.expand_slice(&[b"REALITY"], &mut auth_key)?;
- let result = aes_128_gcm_encrypt(&auth_key, nonce, hello_bytes, &plaintext)?;

+ let mut auth_key = [0u8; 32];
+ auth_key_expander.expand_slice(&[b"REALITY"], &mut auth_key)?;
+ let result = aes_256_gcm_encrypt(&auth_key, nonce, hello_bytes, &plaintext)?;

---
New: RealityServerCertVerifier

After fixing the auth bug, the TLS handshake fails at certificate verification
because webpki requires X.509v3 and rejects the server's minimal v1 cert with
InvalidCertificate(BadEncoding).

RealityServerCertVerifier wraps any existing ServerCertVerifier and handles
both cases transparently:

verify_server_cert

1. Read the auth_key that was stored in a shared slot during compute_session_id.
2. Extract the Ed25519 public key from the cert DER by scanning for the OID bytes
06 03 2b 65 70 (1.3.101.112) followed by the BIT STRING header 03 21 00.
3. Compute HMAC-SHA512(auth_key, ed25519_pubkey) and compare (constant-time) with
the last 64 bytes of the cert DER.
4. If the HMAC matches → accept (it is a valid Reality cert for this session).
5. Otherwise → delegate to the inner verifier (handles normal TLS destinations that
Reality falls back to).

verify_tls13_signature

1. Try the inner verifier first (normal TLS destinations work without changes).
2. If the inner verifier fails, extract the Ed25519 public key from the cert DER and
verify the CertificateVerify signature directly.

supported_verify_schemes

Returns the inner verifier's schemes plus ED25519, ensuring the server's Ed25519
CertificateVerify is not filtered out during negotiation.

---
Auth-key Slot

RealityConfig gains a Arc<Mutex<Option<[u8; 32]>>> field (auth_key_slot,
std-only). compute_session_id writes the derived auth_key into this slot.
with_reality() in the config builder clones the slot reference and passes it to
RealityServerCertVerifier, so the verifier sees the key that was actually used for
the current handshake without any additional API surface.

---
Files Changed

┌─────────────────────────────────────────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│                  File                   │                                                     Change                                                      │
├─────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ rustls/src/client/reality.rs            │ Fix AES-128→256-GCM; add auth_key_slot to RealityConfig; add RealityServerCertVerifier and HMAC/Ed25519 helpers │
├─────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ rustls/src/client/builder.rs            │ with_reality() wraps the existing verifier with RealityServerCertVerifier                                       │
├─────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ examples/src/bin/reality-vless-probe.rs │ Install default crypto provider; fix cfg guards                                                                 │
└─────────────────────────────────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

---
Testing

- All 11 existing unit tests in client::reality continue to pass.
- Manual end-to-end test against a live VLESS Reality server (tw05.ctg.wtf:443):
  - Before fix: TLS handshake completed but server fell back to www.microsoft.com
(authentication failed).
  - After AES fix: Server recognised the client as a Reality session but rejected
the cert with InvalidCertificate(BadEncoding).
  - After verifier: Handshake completes fully; VLESS frame is delivered over the
authenticated TLS channel.