Generate TLS alerts on certificate validation failure on macOS

[liveans]

Note

This pull request was prepared with AI assistance (GitHub Copilot CLI). The code, build, and test validation were performed by the assistant under my supervision.

Fixes #127053.

Summary

On macOS SecureTransport, when SslStream rejects the peer's certificate (chain build failure, name mismatch, missing cert, etc.) the handshake was aborted without putting a TLS alert on the wire — the peer just saw a
TCP RST / EOF. Windows / Linux / Android all send a fatal alert (unknown_ca, bad_certificate, …) per RFC 5246 §7.2 / RFC 8446 §6.2 before aborting. This PR brings macOS to parity.

Previous behavior (macOS, SecureTransport)

Step	Today
errSSLServerAuthCompleted / errSSLClientAuthCompleted from SSLHandshake	handled internally — no opportunity to inject an alert
VerifyRemoteCertificate fails in CompleteHandshake	exception is thrown, but no alert frame is emitted — peer sees a closed socket

New behavior

1. The Apple PAL now surfaces SecurityStatusPalErrorCode.CertValidationNeeded when SecureTransport pauses at the auth-completed state, before CompleteHandshake runs.
2. SslStream invokes its existing cert validation pipeline at that point and, on failure, builds a fatal alert record via TlsFrameHelper.CreateAlertFrame and writes it to the wire before tearing down the handshake.
3. The alert description follows the existing Windows/Linux mapping (GetAlertMessageFromChain for chain errors, BadCertificate for name mismatch, CertificateUnknown for missing cert).
4. The fallback is gated to SecureTransport (SafeDeleteSslContext) only — SafeDeleteNwContext (Network.framework, opt-in via System.Net.Security.UseNetworkFramework) keeps its async-context behavior untouched.

Design notes

Why an explicit plaintext alert frame?

AppleCryptoNative_SslSetError (new native shim around SSLSetError) sets SecureTransport's internal error state so subsequent SSLHandshake / SSLClose calls return the desired OSStatus — but empirical A/B testing
showed it does not itself write an alert record to the BIO. The PR therefore builds and writes the alert frame directly using TlsFrameHelper.CreateAlertFrame, which is the same code path Windows uses for
protocol_version alert injection. SSLSetError is still called so SecureTransport's internal state lines up with what we sent on the wire.

This is a SecureTransport-only behavior; Network.framework manages its own alert handling and is excluded via SslStreamPal.IsAsyncSecurityContext.

CanGenerateCustomAlertsForContext

CanGenerateCustomAlerts was a per-PAL const bool. It becomes a per-context method CanGenerateCustomAlertsForContext(SafeDeleteContext?) so the OSX PAL can answer true only for SafeDeleteSslContext. Windows /
Unix / Android keep the constant value as the implementation.

Cross-platform exception parity

VerifyRemoteCertificateAndGenerateNextToken initially surfaced the cert exception via SecurityStatusPalErrorCode.InternalError, which caused ForceAuthenticationAsync to wrap it inside
AuthenticationException(SR.net_auth_SSPI, …). On Windows / Linux / Android the cert exception is thrown directly by SendAuthResetSignal via ExceptionDispatchInfo.Throw. A new
SecurityStatusPalErrorCode.CertValidationFailed code is introduced; when the IO loop sees it, the inner cert exception is rethrown directly so the user-visible message matches the other platforms exactly. The new test
cases pin this behavior via Assert.Null(ex.InnerException) and message-content assertions.

Tests

• New OSX-only functional tests in SslStreamAlertsTest:
  • SslStream_NoCallback_UntrustedCert_SendsUnknownCAAlert_OSX — client side, server cert untrusted, asserts unknown_ca(48) reaches the server's recorded read stream.
  • SslStream_NoCallback_UntrustedClientCert_ServerSendsUnknownCAAlert_OSX — server side, mutual-auth, asserts unknown_ca(48) reaches the client's recorded read stream.
  • Both tests verify the actual on-the-wire bytes (via a RecordingReadStream that wraps the inner stream and matches alert framing), not just exception types.
• New unit test CreateAlertFrame_NonProtocolAlert_UsesRequestedVersion in TlsAlertsMatchWindowsInterop pins the TLS 1.0 / 1.1 / 1.2 record framing of CreateAlertFrame, including the (int)version >= Tls fix that
  previously excluded TLS 1.0.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/ncl, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

This PR updates the macOS (SecureTransport) SslStream handshake flow to explicitly emit a fatal TLS alert record when certificate validation fails, aligning behavior more closely with other platforms. It introduces new handshake/status plumbing for “cert validation needed” and adds Apple-native interop to set SecureTransport’s internal error state.

Changes:

• Add Apple PAL support to surface a “certificate validation needed” handshake state and to set a specific TLS alert via a new AppleCryptoNative_SslSetError export.
• Update SslStream handshake/token generation to run cert validation at the new pause point on Apple and (on failure) send an alert frame + preserve exception parity via a dedicated CertValidationFailed status.
• Add/extend unit + functional tests (including on-the-wire alert bytes capture) and adjust TlsFrameHelper.CreateAlertFrame to handle TLS 1.0 framing.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/native/libs/System.Security.Cryptography.Native.Apple/pal_ssl.h	Adds PAL TLS alert message enum and declares AppleCryptoNative_SslSetError.
src/native/libs/System.Security.Cryptography.Native.Apple/pal_ssl.c	Implements alert→OSStatus mapping and AppleCryptoNative_SslSetError.
src/native/libs/System.Security.Cryptography.Native.Apple/entrypoints.c	Exposes AppleCryptoNative_SslSetError for managed P/Invoke resolution.
src/libraries/Common/src/Interop/OSX/System.Security.Cryptography.Native.Apple/Interop.Ssl.cs	Adds managed SslSetError P/Invoke + wrapper.
src/libraries/System.Net.Security/src/System/Net/SecurityStatusPal.cs	Introduces CertValidationNeeded and CertValidationFailed internal status codes.
src/libraries/System.Net.Security/src/System/Net/Security/TlsFrameHelper.cs	Fixes TLS 1.0 inclusion in CreateAlertFrame version handling.
src/libraries/System.Net.Security/src/System/Net/Security/SslStreamPal.Windows.cs	Adds CanGenerateCustomAlertsForContext shim (constant-backed).
src/libraries/System.Net.Security/src/System/Net/Security/SslStreamPal.Unix.cs	Adds CanGenerateCustomAlertsForContext shim (constant-backed).
src/libraries/System.Net.Security/src/System/Net/Security/SslStreamPal.Android.cs	Adds CanGenerateCustomAlertsForContext shim (constant-backed).
src/libraries/System.Net.Security/src/System/Net/Security/SslStreamPal.OSX.cs	Enables custom alerts for SecureTransport contexts and returns CertValidationNeeded at auth-complete pause; applies alerts via SslSetError.
src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Protocol.cs	Runs cert validation when Apple PAL signals CertValidationNeeded; generates on-wire alert payload for SecureTransport contexts; switches alert gating to per-context.
src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs	Rethrows cert validation exception directly for CertValidationFailed to match other platforms’ exception shape.
src/libraries/System.Net.Security/tests/UnitTests/TlsAlertsMatchWindowsInterop.cs	Adds coverage to pin CreateAlertFrame record version bytes for TLS 1.0/1.1/1.2.
src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamAlertsTest.cs	Adds macOS-only functional tests that assert the actual alert bytes appear on the wire using a recording stream wrapper.

[rzikm]

LGTM, modulo comments