Conversation
PCAP-NG reader: - Stream blocks one at a time instead of reading the whole file into memory, and parse them directly, removing the pcapngdecoder dependency - Track interfaces per section: timestamps honor if_tsresol and if_tsoffset, and each packet is decoded for its interface's link type - Decode Ethernet (with 802.1Q/QinQ tags), raw IP, BSD loopback, PPP, Linux SLL/SLL2, and 802.11 / radiotap data frames - Add interface_id, interface_name, link_type, captured_length, comment, direction, reception_type, fcs_length, drop_count and packet_hash columns - Skip unknown block types instead of stopping, and tolerate a truncated final block - Decode each packet once per row instead of once per column - Stat queries: return null for missing options, add comment, make if_speed and if_tsoffset BIGINT, and use the interface's resolution for statistics timestamps - Support sessionizeTCPStreams Packet decoding (both readers): - Fix the TCP header length and payload offsets, bound payloads by the IP length and captured length, and walk IPv6 extension headers correctly; IPv6 TCP sequence numbers, flags and ports were misread - Detect ARP by EtherType and label ICMPv6 as ICMP; IPv6 packets with a hop-by-hop header were labeled ARP Format plugin: - Detect PCAP vs PCAP-NG per file; the format was a static field and was detected from the first file in the scan TCP sessions (both readers): - Share one TcpSessionizer; only TCP packets are tracked - Return sessions still open at end of file, with a new session_closed column, and ignore packets trailing a closed session - connection_time is null when the handshake was not captured Tests: replace the corrupted todo/ fixtures with valid captures, add fixtures for metadata, link types and sessionization, and correct expectations that encoded the old decoding bugs.
cgivre
added
enhancement
OutputBatchBuilder rebuilt any nested column whose metadata isMap() as a new map vector, including map arrays. A map array inside a map lost its offsets, so the batch failed validation (row count n, value count 0).
ScanProjectionParser took a column's array dimensions from its depth in the whole path, so m.a[0] became a two-dimensional array, its index was dropped, and projecting an element of an array inside a map failed as incompatible.
cgivre
marked this pull request as ready for review
October 2, 2026 19:15
A leftover probe test and the three ServiceLoader registration files had no license header, which failed the Apache RAT check.
Interim 1xx responses such as 100 Continue were paired as answers, shifting every later request-response pair on the connection by one.
- Cap PCAP-NG block lengths at 16 MiB and the PCAP snapshot length at libpcap's 262144 bytes, so a damaged or crafted header becomes an error row instead of a multi-gigabyte allocation - In session mode, where packets are not rows, report each distinct packet or interface error once as an error row instead of dropping it
timestamp is a reserved word, so every query had to quote it, and an unquoted reference such as MIN(timestamp) was a parse error. packet_timestamp is also the name classic PCAP files already use.
- Keep the IP addresses of a packet whose IPv4 header cannot be parsed, with ports left empty, instead of discarding everything after the link layer - DNS records past the 64-item cap are skipped, so later sections are kept - An invalid HTTP Content-Length is reported once - Compare HTTP chunk sizes without risk of overflow - Give each reader its own copy of the parsed_data schema
getLinkPayload returns the bytes after the link-layer header (the ARP message of an ARP frame); getIpPayload returns the bytes after the IP header (the ICMP message of an ICMP packet). ARP frames read from PCAP-NG now record where their link payload starts on every link type.
- PCAP: refill the buffer by appending after the unread bytes until it is full. Topping up at the read position overwrote unread data whenever a read came back short, so gzip-compressed captures could not be read at all - PCAP: read the file header fully instead of with a single read - Each PCAP packet copies its record; packets kept for sessionization pointed into the reader's buffer and changed when it was reused, corrupting session data in captures larger than the buffer - A read that fails partway through a file (for example a corrupted gzip stream) now returns the packets read before it and an error row, in both readers, instead of ending silently (PCAP) or failing the query (PCAP-NG)
Decode the TLS ClientHello and ServerHello that start a TCP segment on the TLS ports: versions, session ID, SNI, ALPN, cipher suites, extensions, groups, point formats, signature algorithms, and the JA3 and JA3S fingerprints.
Decode STUN messages (RFC 5389) on UDP 3478 and 19302: class, method, transaction ID, mapped and XOR-mapped addresses, SOFTWARE, REALM, NONCE, ERROR-CODE, USERNAME and the attribute list. Add query tests for the TLS and STUN decoders.
# Conflicts: # contrib/format-pcapng/src/main/resources/META-INF/services/org.apache.drill.exec.store.pcap.protocol.PacketProtocolDecoder
# Conflicts: # contrib/format-pcapng/src/main/resources/META-INF/services/org.apache.drill.exec.store.pcap.protocol.PacketProtocolDecoder
# Conflicts: # contrib/format-pcapng/src/main/resources/META-INF/services/org.apache.drill.exec.store.pcap.protocol.PacketProtocolDecoder
# Conflicts: # contrib/format-pcapng/src/main/resources/META-INF/services/org.apache.drill.exec.store.pcap.protocol.PacketProtocolDecoder
Harvesting a batch fills in values for columns that were not written, and that fill may need to grow a vector. When the batch was already over its memory budget, the growth was refused and treated as overflow, which is not possible during harvest, so the query failed with Unexpected state: FULL_BATCH. Readers with many sparse columns, such as nested maps that most rows leave empty, hit this on ordinary files.
Decodes the cleartext TLS handshake of TCP sessions on TLS ports: SNI, ALPN, offered and selected versions, cipher suite, session resumption and, for TLS 1.2 and earlier, the server certificate chain (parsed with the JDK X.509 certificate factory). Records and handshake messages are reassembled across segments and records, bounded at 64 KB per direction. In TLS 1.3 the certificate is encrypted and only that fact is recorded.
# Conflicts: # contrib/format-pcapng/src/test/resources/fixtures/tls_fixtures.py
FTP control channel (port 21): banner, login, SYST, directories, transfers with their data addresses, commands and replies; stops at an accepted AUTH TLS. SSH (ports 22 and 2222): identification strings and the first KEXINIT of each side with HASSH client and server fingerprints. DNS over TCP (port 53): every length-prefixed message of a session, including zone transfers, reusing DnsParser.
# Conflicts: # contrib/format-pcapng/src/main/resources/META-INF/services/org.apache.drill.exec.store.pcap.protocol.SessionProtocolDecoder
Session decoders for the mail protocols, sharing header parsing (folded lines, RFC 2047 UTF-8 encoded-words), SASL credential handling and bounded lists in protocol/mail. Each records the greeting, commands and replies with passwords masked unless exposeCredentials is set, STARTTLS/STLS switches, and the headers and sizes of transferred or retrieved messages.
# Conflicts: # contrib/format-pcapng/src/main/resources/META-INF/services/org.apache.drill.exec.store.pcap.protocol.SessionProtocolDecoder
Add an identity-only NTLMSSP parser (protocol.ntlm) and an SMB2/SMB3 session decoder (protocol.smb, TCP 445 and 139). The decoder reports the negotiated and offered dialects, signing and encryption state, the server and client GUIDs, and the user, domain and workstation from the NTLMSSP blob inside the SPNEGO token of SESSION_SETUP. Pure SMB1 sessions report only dialect SMB1; an SMB3 transform header marks the stream encrypted and opaque; Kerberos is reported as auth_type kerberos with no identity fields. The NTLM helper exposes identity metadata only: server challenges, LM/NT responses and session keys are never read out, regardless of exposeCredentials.
# Conflicts: # contrib/format-pcapng/src/main/resources/META-INF/services/org.apache.drill.exec.store.pcap.protocol.SessionProtocolDecoder
Decodes the cleartext metadata of Kerberos v5 (RFC 4120) AS-REQ, AS-REP, TGS-REQ, TGS-REP and KRB-ERROR messages over UDP and single-segment TCP port 88: message type, realm, client and server principal names, offered encryption types, the ticket enc-part etype (the Kerberoasting signal), the error code and name, pre-auth presence and the requested till time. Encrypted parts are never read. Adds a shared, bounded ASN.1 BER reader (protocol.asn1.Asn1Reader), modelled on the SNMP decoder's package-private BerReader, with value accessors for integers, strings, OIDs and GeneralizedTime.
Reassembles LDAP (RFC 4511) messages across TCP segments on port 389 and reads the bind version, DN and authentication type, whether a password was sent (and the cleartext password only with exposeCredentials), the bind result code and name, search base DNs, scopes and RFC 4515 filter strings, and the DNs of returned entries. Lists are capped at 64 and sessions at 1000 messages. Reuses the shared protocol.asn1.Asn1Reader.
# Conflicts: # contrib/format-pcapng/src/main/resources/META-INF/services/org.apache.drill.exec.store.pcap.protocol.PacketProtocolDecoder # contrib/format-pcapng/src/main/resources/META-INF/services/org.apache.drill.exec.store.pcap.protocol.SessionProtocolDecoder
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
|
||
| private static byte[] aesEcb(byte[] key, byte[] data) { | ||
| try { | ||
| Cipher cipher = Cipher.getInstance("AES/ECB/NoPadding"); |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DRILL-8556: Various PCAP and PCAP-NG Improvements
Description
The PCAP-NG reader read the whole file into memory (twice: the raw bytes plus the third-party library's list of every parsed block), so large captures needed several times their size in heap. This PR streams PCAP-NG files block by block, removes the
pcapngdecoderdependency, and fixes a set of decoding bugs in both the PCAP and PCAP-NG readers.PCAP-NG reader
if_tsresolandif_tsoffset(nanosecond captures were off by 1000x), and packets are decoded for their interface's link type.sessionizeTCPStreams.Packet decoding (both readers)
ARP_PROTOCOLwas defined as IP protocol 0 (IPv6 hop-by-hop), so IPv6 multicast packets were labeled ARP and real ARP frames were not. ARP is now detected by EtherType; ICMPv6 is labeled ICMP.Format plugin
staticfield shared by all readers and was detected from the first file in the scan, so directories mixing.pcapand.pcapngsent files to the wrong reader.TCP sessions (both readers)
TcpSessionizer; only TCP packets are tracked (all non-TCP packets previously accumulated in a "session 0" that was never written).session_closedcolumn instead of dropped. Packets trailing a closed session no longer start a phantom session.connection_timeis null when the handshake was not captured.Protocol decoders
ServiceLoader:PacketProtocolDecoderfor single packets andSessionProtocolDecoderfor reassembled TCP sessions. A decoder only handles traffic that parses as its protocol. Design and decoder-writing guide:docs/dev/PcapProtocolDecoders.md.parsed_protocol,parsed_data(one map per decoder) anddecode_error. Decoding runs only when these columns are queried.exposeCredentials, since those are offline-crackable material rather than a password; Kerberos records only the encryption types, never the encrypted parts. Only JA4 is implemented, not the JA4S/JA4H/JA4SSH variants, whose license is not compatible with Apache.TestDecoderFieldNameschecks this for every registered decoder.decode_error, and damaged or non-capture files return a row with onlydecode_errorset instead of failing the query.exposeCredentials(default false): cleartext passwords found by decoders are only returned when it is set; usernames and apassword_presentflag are always returned.Drill scan framework fixes (exec/java-exec), found while implementing the decoders:
OutputBatchBuilderrebuilt map arrays nested in maps as plain maps, dropping their offsets, so such batches failed validation.ScanProjectionParsercounted array dimensions from the root of the path, so projecting an element of an array inside a map (m.a[0]) failed as incompatible.ResultSetLoaderImplcould fail a query while harvesting a full batch: filling the many unwritten decoder columns asked for more memory than the batch budget allowed. Those fills are now allowed while harvesting.Read reliability (classic PCAP)
Documentation
Breaking change: the PCAP-NG
timestampcolumn is renamedpacket_timestamp, the name PCAP files already use.timestampis a reserved word, so it had to be quoted in every query and an unquotedMIN(timestamp)was a parse error.New PCAP-NG packet columns:
captured_length,interface_id,interface_name,link_type,comment,direction,reception_type,fcs_length,drop_count,packet_hash. New session column:session_closed. In stat mode (stat: true), options a block does not have are now null (previously""or-1), acommentcolumn is added, andif_speed/if_tsoffsetare BIGINT. The plugin README documents the columns and supported link types.User-visible value changes from the bug fixes: classic PCAP
datapayloads, IPv6 TCP fields,typefor ARP/ICMPv6, andis_corrupt(testv1.pcapreported 16 corrupt packets that were snaplen-truncated, not corrupt).Testing
Replaced the corrupted
todo/fixtures (they had been run through a UTF-8 conversion) with valid Wireshark captures, and added generated fixtures covering link types, timestamp resolutions, packet options, multiple sections, encapsulations and sessionization. Fixtures were cross-checked with scapy.Every classic PCAP payload in the test fixtures was compared against scapy (6,736 exact matches; 17 encrypted payloads differ only in how invalid UTF-8 renders). Test expectations that encoded the old bugs were corrected after verifying the new values against scapy.
Protocol decoders: unit tests for the registry, TCP stream reassembly, DNS and HTTP parsers (including compression-pointer loops, record and header caps, keep-alive pairing with chunked bodies), and query tests on generated fixtures for decoder columns and every error-handling case. Expected values were cross-checked with scapy.
Every decoder has unit tests and a query test on a generated fixture holding valid traffic, other traffic on the same port (left undecoded) and a malformed message (reported in
decode_error). Fixtures are cross-checked with scapy where scapy supports the protocol (DNS, DHCP, NTP, SNMP, TLS, SSH KEXINIT and others); HASSH and JA3 values are computed independently.The format-pcapng module: 332 tests pass, plus checkstyle and the license check.
Scan framework fixes: regression tests
TestOutputBatchBuilder.testMapArrayInMap,TestProjectionParser.testArrayInMapandTestResultSetLoaderLimits.testHarvestFillsUnwrittenColumnsOverBudget; 501 scan/result-set-loader/projection tests and 246 JSON reader tests pass.