tls: add optional server certificate verification

[KernelClint]

TLSClientAutomaton completes a handshake and sends application data without checking the server's
certificate against a trust store or the hostname it asked for. It is a test client, and it does
not claim otherwise — but there is no way to ask it to check, which makes it awkward to use against
a real service where that is the point.

The relevant states pass the certificate through without a policy:
scapy/layers/tls/automaton_cli.py:118-142
takes the server name but has no trust store, and :396-405 advances past a TLS 1.2 Certificate
message without authenticating it.

The change adds verification and makes it selectable. When enabled, an explicit CA bundle is used if
one is supplied and the system trust store otherwise, the requested hostname is checked, and the
connection closes on failure. verify=False keeps the current behaviour for the cases where
talking to an untrusted server is the whole point of the exercise.

The added regressions cover a self-signed server, a valid certificate for the wrong hostname, and a
correct server, asserting the first two close the connection and the third completes. Without the
source change they fail.

On cost. Performance was measured on one computer, before and after the fix, but there is not
much of a "before" to measure: the client performs no verification today, so its valid path does
nothing. Verifying took 2.9 µs per handshake, against 819 ns of test overhead — roughly 2.1 µs
added, once per connection, for a full chain and hostname check. A percentage against a path that
does no work would not mean anything, so none is quoted.

[codecov]

Codecov Report

❌ Patch coverage is 79.72028% with 58 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.84%. Comparing base (e4742a5) to head (e54bcc2).
⚠️ Report is 10 commits behind head on master.

Files with missing lines	Patch %	Lines
scapy/layers/tls/cert.py	70.54%	38 Missing ⚠️
scapy/layers/tls/automaton_srv.py	72.41%	8 Missing ⚠️
scapy/layers/tls/automaton_cli.py	86.48%	5 Missing ⚠️
scapy/automaton.py	77.77%	4 Missing ⚠️
scapy/layers/tls/crypto/pkcs1.py	40.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5144      +/-   ##
==========================================
- Coverage   80.92%   80.84%   -0.09%     
==========================================
  Files         393      393              
  Lines       98107    98316     +209     
==========================================
+ Hits        79392    79480      +88     
- Misses      18715    18836     +121     

Files with missing lines	Coverage Δ
scapy/asn1/mib.py	91.87% <ø> (ø)
scapy/layers/tls/automaton.py	86.33% <100.00%> (-2.80%)	⬇️
scapy/layers/tls/handshake.py	88.30% <100.00%> (-0.46%)	⬇️
scapy/layers/tls/handshake_sslv2.py	93.58% <100.00%> (ø)
scapy/layers/tls/session.py	86.05% <100.00%> (+0.01%)	⬆️
scapy/layers/x509.py	98.03% <100.00%> (+0.05%)	⬆️
scapy/tools/UTscapy.py	64.99% <100.00%> (-0.39%)	⬇️
scapy/layers/tls/crypto/pkcs1.py	74.59% <40.00%> (-1.69%)	⬇️
scapy/automaton.py	80.54% <77.77%> (-1.22%)	⬇️
scapy/layers/tls/automaton_cli.py	86.19% <86.48%> (-0.80%)	⬇️
... and 2 more

... and 31 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[gpotter2]

It also seems the test against google.com fails to verify the certificates :(

[gpotter2]

Sorry, I still have comments :( Thanks again for the work and sorry about that

[KernelClint]

I had a look at the failing test. Verification itself isn't the problem (the client rejects the server as expected) but the test hangs afterwards while stopping the server.

When the client fails, test_tls_client calls atmtsrv.stop() while the server thread is still inside atmtsrv.run(), waiting on cmdout. stop() then empties cmdout in _flush_inout(). ObjectPipe.clear() sees a message in the queue and does a blocking os.read(). If the server thread has already read that byte, it waits forever. That's why your log shows "Flushing..." and never "self.started.done()". It's a race, so which jobs fail changes from run to run. Locally I saw the test hang 2 out of every 3 tries...

Letting the server thread read END before calling stop() fixes it (no more hangs locally). That needs the new TLSServerAutomaton.stop()/forcestop() to accept wait again:

--- a/scapy/layers/tls/automaton_srv.py
+++ b/scapy/layers/tls/automaton_srv.py
-    def stop(self):
+    def stop(self, wait=True):
 ...
-        return super(TLSServerAutomaton, self).stop()
+        return super(TLSServerAutomaton, self).stop(wait=wait)
 
-    def forcestop(self):
+    def forcestop(self, wait=True):
 ...
-        return super(TLSServerAutomaton, self).forcestop()
+        return super(TLSServerAutomaton, self).forcestop(wait=wait)
--- a/test/scapy/layers/tls/tlsclientserver.uts
+++ b/test/scapy/layers/tls/tlsclientserver.uts
         print("Client crashed:", final_reason.atmt_origfunc)
+        # The server is still running, and th_ is waiting in atmtsrv.run()
+        # for its END message. Let th_ read it: stop() empties the same pipe,
+        # and hangs if th_ takes the message first.
+        atmtsrv.stop(wait=False)
+        th_.join(timeout=5)
     else:

Two other things I found while testing:

• _verify_server_cert() starts with raise self.INVALID_SERVER_CERTIFICATE(), so with verification on, every server is rejected and the CertTree check below never runs. There's also a print(verify_server) in parse_args. I think both are left over from debugging.
• The "TLS 1.3 client rejects an invalid server CertificateVerify" test was removed. It would have caught the first point: with that raise in place, it rejects the correctly signed server too. With the raise removed, it passes. Was removing it intentional? Same question for "Reject a CertificateVerify made with a key unrelated to the client certificate", which still passes.

With these changes, tlsclientserver.uts and cert.uts pass locally (macOS, Python 3.13). I haven't tried Windows.

[gpotter2]

Thanks a lot for lookint into this @KernelClint !

I removed the tests because they were too different from the other tests in the file (and one of them was misplaced in the wrong section). They are useful, of course, but I didn't think I'd have time fixing it before 2.8.0 (I'd wanted to merge this PR quickly)... it's a bit lazy i know