aboutsummaryrefslogtreecommitdiff
path: root/src/lib/protocols/quic.c
Commit message (Collapse)AuthorAge
* QUIC: remove extraction of user-agent (#2650)Ivan Nardi2025-01-07
| | | | | In very old (G)QUIC versions by Google, the user agent was available on plain text. That is not true anymore, since about end of 2021. See: https://github.com/google/quiche/commit/f282c934f4731a9f4be93409c9f3e8687f0566a7
* Add the ability to enable/disable every specific flow risks (#2653)Ivan Nardi2025-01-06
|
* QUIC: extract "max idle timeout" parameter (#2649)Ivan Nardi2025-01-06
| | | | | Even if it is only the proposed value by the client (and not the negotiated one), it might be use as hint for timeout by the (external) flows manager
* QUIC: add a basic heuristic to detect mid-flowsNardi Ivan2024-09-10
|
* TLS: better state about handshake (#2534)Ivan Nardi2024-09-03
| | | | Keep track if we received CH or/and SH messsages: usefull with unidirectional flows
* Introduced ndpi_master_app_protocol typedefLuca Deri2024-08-24
|
* Domain Classification Improvements (#2396)Luca Deri2024-04-18
| | | | | | | | | | | | | | | | | | | * Added size_t ndpi_compress_str(const char * in, size_t len, char * out, size_t bufsize); size_t ndpi_decompress_str(const char * in, size_t len, char * out, size_t bufsize); used to compress short strings such as domain names. This code is based on https://github.com/Ed-von-Schleck/shoco * Major code rewrite for ndpi_hash and ndpi_domain_classify * Improvements to make sure custom categories are loaded and enabled * Fixed string encoding * Extended SalesForce/Cloudflare domains list
* Disable `-Wno-unused-parameter -Wno-unused-function`. (#2358)Toni2024-04-03
| | | | | * unused parameters and functions pollute the code and decrease readability Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Improve normalization of `flow->host_server_name` (#2310)Ivan Nardi2024-02-09
| | | | | | | | | | | | | Follow-up of 4543385d107fcc5a7e8632e35d9a60bcc40cb4f4 Remove trailing spaces for any HTTP header (we already remove leading spaces) We want: * a "normalized" string in `flow->host_server_name`, but * to parse the original string for flow risk checking `ndpi_hostname_sni_set()` is a private function, so there is no need to export its flags.
* Normalization of host_server_name (#2299)Vitaly Lavrov2024-02-05
| | | | | | | | | * Normalization of host_server_name The ndpi_hostname_sni_set() function replaces all non-printable characters with the "?" character and removing whitespace characters at the end of the line. * Added conditional hostname normalization.
* QUIC: fix decryption with CH fragments with different Destination CID (#2278)Ivan Nardi2024-01-24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | QUIC decryption fails when the Client Hello is split into multiple UDP packets and these packets have different Destination Connection IDs (because the server told the client to switch to a different CID; see RFC 9000 7.2) ``` The Destination Connection ID field from the first Initial packet sent by a client is used to determine packet protection keys for Initial packets. [..] Upon first receiving an Initial or Retry packet from the server, the client uses the Source Connection ID supplied by the server as the Destination Connection ID for subsequent packets ``` From a logical point of view, the ciphers used for decryption should be initialized only once, with the first Initial pkt sent by the client and kept for later usage with the following packets (if any). However it seems that we can safely initialize them at each packet, if we keep using the DCID of the **first** packet sent by the client. Keep initializing the ciphers at each packet greatly simplifie this patch. This issue has been undetected for so long because: * in the vast majority of the cases we only decrypt one packet per flow; * the available traces with the Client Hello split into multiple packets (i.e. cases where we need to decrypt at least two packets per flow) were created in a simple test environment to simulate Post-Quantum handshake, and in that scenario the client sent all the packets (with the same DCID) before any reply from the server. However, in the last months all major browsers started supporting PQ key, so it is now common to have split CH in real traffic. Please note that in the attached example, the CH is split into 2 (in-order) fragments (in different UDP packets) and the second one in turn is divided into 9 (out-of-order) CRYPTO frames; the reassembler code works out-of-the-box even in this (new) scenario.
* QUIC: add heuristic to detect unidirectional *G*QUIC flows (#2207)Ivan Nardi2023-12-13
| | | Fix extraction of `flow->protos.tls_quic.quic_version` metadata.
* Have a clear distinction between public and private/internal API (#2137)Ivan Nardi2023-11-09
| | | | | | 1) Public API/headers in `src/include/` [as it has always been] 2) Private API/headers in `src/lib/` Try to keep the "ndpi_" prefix only for the public functions
* QUIC: export QUIC version as metadataNardi Ivan2023-10-11
|
* QUIC: fix dissection of packets forcing VNNardi Ivan2023-06-08
|
* QUIC: fix a memory access errorNardi Ivan2023-06-08
| | | | Found while fuzzing
* QUIC: add support for QUIC version 2Nardi Ivan2023-06-08
| | | | | | See: https://www.rfc-editor.org/rfc/rfc9369.txt Old v2-01 version has been removed, since it has never been really used.
* fuzz: some improvements and add two new fuzzers (#1881)Ivan Nardi2023-02-09
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Remove `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` define from `fuzz/Makefile.am`; it is already included by the main configure script (when fuzzing). Add a knob to force disabling of AESNI optimizations: this way we can fuzz also no-aesni crypto code. Move CRC32 algorithm into the library. Add some fake traces to extend fuzzing coverage. Note that these traces are hand-made (via scapy/curl) and must not be used as "proof" that the dissectors are really able to identify this kind of traffic. Some small updates to some dissectors: CSGO: remove a wrong rule (never triggered, BTW). Any UDP packet starting with "VS01" will be classified as STEAM (see steam.c around line 111). Googling it, it seems right so. XBOX: XBOX only analyses UDP flows while HTTP only TCP ones; therefore that condition is false. RTP, STUN: removed useless "break"s Zattoo: `flow->zattoo_stage` is never set to any values greater or equal to 5, so these checks are never true. PPStream: `flow->l4.udp.ppstream_stage` is never read. Delete it. TeamSpeak: we check for `flow->packet_counter == 3` just above, so the following check `flow->packet_counter >= 3` is always false.
* Some small changes (#1869)Ivan Nardi2023-01-25
| | | | | | | | All dissector callbacks should not be exported by the library; make static some other local functions. The callback logic in `ndpiReader` has never been used. With internal libgcrypt, `gcry_control()` should always return no errors. We can check `categories` length at compilation time.
* Improve support for Snapchat voip calls (#1858)Ivan Nardi2023-01-17
| | | | Latest Snapchat versions use QUICv1 for their audio/video real time sessions. See c50a8d480
* Add protocol disabling feature (#1808)Ivan Nardi2022-12-18
| | | | | | | | | | | | | | | | | | | | | | The application may enable only some protocols. Disabling a protocol means: *) don't register/use the protocol dissector code (if any) *) disable classification by-port for such a protocol *) disable string matchings for domains/certificates involving this protocol *) disable subprotocol registration (if any) This feature can be tested with `ndpiReader -B list_of_protocols_to_disable`. Custom protocols are always enabled. Technically speaking, this commit doesn't introduce any API/ABI incompatibility. However, calling `ndpi_set_protocol_detection_bitmask2()` is now mandatory, just after having called `ndpi_init_detection_module()`. Most of the diffs (and all the diffs in `/src/lib/protocols/`) are due to the removing of some function parameters. Fix the low level macro `NDPI_LOG`. This issue hasn't been detected sooner simply because almost all the code uses only the helpers `NDPI_LOG_*`
* fuzz: add a new fuzzer testing memory allocation failures (#1818)Ivan Nardi2022-12-06
| | | | | | | | | Try to fuzz error paths triggered by allocation errors. Fix some errors already found by this new fuzzer. Basic idea taken from: https://github.com/harfbuzz/harfbuzz/pull/2566/files `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` is a standard define used to (not)compile specific code in fuzzing builds. See: https://llvm.org/docs/LibFuzzer.html
* Add a new fuzzer for QUIC (#1800)Ivan Nardi2022-12-01
| | | | | | | | | | | | QUIC packets are encrypted/obfuscated; that means that we need to decrypt them before parsing the real (TLS) message. Fuzzing is not effective here, since a random buffer is hardly a valid encrypted QUIC packet. Add a new fuzzer, testing *decrypted* QUIC packets. Add a basic corpus. Fix a few bugs already found by this fuzzer.
* TLS: improve handling of ALPN(s) (#1784)Ivan Nardi2022-10-25
| | | | | | | | Tell "Advertised" ALPN list from "Negotiated" ALPN; the former is extracted from the CH, the latter from the SH. Add some entries to the known ALPN list. Fix printing of "TLS Supported Versions" field.
* LINE_CALL: add detection of LINE voip calls (#1761)Ivan Nardi2022-10-06
| | | | | | | | These flows are classifed as `LINE_CALL`; another option was `RTP/LINE_CALL`. No sure about the best solution... Extend LINE domains list. Remove RTP dead code.
* Fix warnings on OpenWrt CCs, caused by the memcpy builtin. (#1741)Toni2022-09-18
| | | | | | | | | | | | | | | | | | | In file included from ../include/ndpi_includes.h:31, from ../include/ndpi_main.h:27, from ../include/ndpi_api.h:28, from protocols/quic.c:27: In function 'memcpy', inlined from 'tls13_hkdf_expand_label_context' at protocols/quic.c:473:5, inlined from 'tls13_hkdf_expand_label' at protocols/quic.c:498:10, inlined from 'quic_hkdf_expand_label.constprop' at protocols/quic.c:512:6: /home/build/openwrt/staging_dir/toolchain-mips_24kc_gcc-11.3.0_musl/include/fortify/string.h:53:16: error: argument 2 null where non-null expected [-Werror=nonnull] 53 | return __builtin_memcpy(__od, __os, __n); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ protocols/quic.c: In function 'quic_hkdf_expand_label.constprop': /home/build/openwrt/staging_dir/toolchain-mips_24kc_gcc-11.3.0_musl/include/fortify/string.h:53:16: note: in a call to built-in function '__builtin_memcpy' Signed-off-by: Toni Uhlig <matzeton@googlemail.com> Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Avoid useless host automa lookup (#1724)Ivan Nardi2022-09-05
| | | | | | | | | | | | | | | | | | | The host automa is used for two tasks: * protocol sub-classification (obviously); * DGA evaluation: the idea is that if a domain is present in this automa, it can't be a DGA, regardless of its format/name. In most dissectors both checks are executed, i.e. the code is something like: ``` ndpi_match_host_subprotocol(..., flow->host_server_name, ...); ndpi_check_dga_name(..., flow->host_server_name,...); ``` In that common case, we can perform only one automa lookup: if we check the sub-classification before the DGA, we can avoid the second lookup in the DGA function itself.
* QUIC: fix heap-buffer-overflowNardi Ivan2022-08-25
| | | | | | | | | | | | | | | | | | | | | | | | | | ``` ==12318==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x604000000032 at pc 0x55a59ec97959 bp 0x7fffee67fdd0 sp 0x7fffee67fdc8 READ of size 1 at 0x604000000032 thread T0 #0 0x55a59ec97958 in may_be_0rtt /home/ivan/svnrepos/nDPI/src/lib/protocols/quic.c:1483:24 #1 0x55a59ec9515f in ndpi_search_quic /home/ivan/svnrepos/nDPI/src/lib/protocols/quic.c:1708:13 #2 0x55a59ec32e95 in check_ndpi_detection_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5428:6 #3 0x55a59ec33c5b in check_ndpi_udp_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5464:10 #4 0x55a59ec335fc in ndpi_check_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5497:12 #5 0x55a59ec44615 in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:6322:15 #6 0x55a59eb8884e in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:29:5 #7 0x55a59eb889c7 in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:101:17 #8 0x7fb5b3ba2082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16 #9 0x55a59eac742d in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet_with_main+0x47d42d) (BuildId: 712c87b21cf5c05f64174745909c693d3ba0b62e) 0x604000000032 is located 0 bytes to the right of 34-byte region [0x604000000010,0x604000000032) allocated by thread T0 here: #0 0x55a59eb4bfee in malloc (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet_with_main+0x501fee) (BuildId: 712c87b21cf5c05f64174745909c693d3ba0b62e) #1 0x55a59eb8899c in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:87:17 ``` Found by CI tests. See: https://github.com/ntop/nDPI/runs/7996151458?check_suite_focus=true
* QUIC: add support for 0-RTT packets received before the InitialNardi Ivan2022-08-24
| | | | | | | | | | | | | | | | | | | | | RFC9001 4.6.1: "A client that wishes to send 0-RTT packets uses the early_data extension in the ClientHello message of a subsequent handshake; see Section 4.2.10 of [TLS13]. It then sends application data in 0-RTT packets." That means the client sends before the CH (in the Initial) and then the 0-RTT (in the same UDP datagram or not)". However, because of packet loss or out-of-order delivery, it might happens that a 0-RTT packet is received before the Initial (the original one or a retransmission). For example, Google and Facebook servers save 0-RTT packets for a small amount of time in hopes of receiving the corresponding Initial. Update the QUIC dissector to detect 0-RTT packets and keep looking for the Initial. Issue found by @utoni in #1706; the trace example has been taken from that PR.
* Further simplification of `ndpi_process_extra_packet()` (#1698)Ivan Nardi2022-08-05
| | | | | | | | See 95e16872. After c0732eda, we can safely remove the protocol list from `ndpi_process_extra_packet()`. The field `flow->check_extra_packets` is redundant; remove it.
* Skype_Teams, Mining, SnapchatCall: fix flow category (#1624)Ivan Nardi2022-07-03
|
* Updated tests resultsLuca Deri2022-05-30
| | | | Code cleanup
* Added RiskInfo stringLuca Deri2022-05-30
|
* Fix dissection of IPv4 header (#1561)Ivan Nardi2022-05-29
| | | | | See: https://github.com/ntop/nDPI/runs/6643914510?check_suite_focus=true Convert al the `MIN(a,b)` calls to `ndpi_min(a,b)`
* Compilation fixesLuca Deri2022-05-27
|
* Added invalid SNI check in QUICLuca Deri2022-05-24
|
* QUIC: handle retransmissions and overlapping fragments in reassembler ↵Vinicius Silva Nogueira2022-04-07
| | | | | | | | | | | | | | | | | | | | | (#1195) (#1498) * QUIC: handle retransmissions and overlapping fragments in reassembler * Trigger CI * minor fix: parentheses * Changing ndpi_malloc to ndpi_calloc * fix memory leak * quic_reasm_buf calloc to malloc * change order of is_ch_complete && is_reasm_buf_complete call * is_reasm_buf_complete: added handling for case where frame size is not multiple of 8 * add extra check
* QUIC: add support for version 2 draft 01 (#1493)Ivan Nardi2022-03-25
| | | | | | Support for v2-00 has been removed (it has never been used in real networks and it is incompatible with v2-01). Chrome already supports v2-01 in latest versions in Chrome Beta channel.
* Trying to improve QUIC reassembler (#1195) (#1489)Vinicius Silva Nogueira2022-03-24
| | | | | | | | | | | * handling QUIC out-of-order fragments * minor fix * updated quic_frags_ch_out_of_order_same_packet_craziness.pcapng.out * quic test: buf_len + last_pos * QUIC: comment update in __reassemble function and minor change is_ch_complete function
* QUIC: convert logs to standard mechanism (#1485)Ivan Nardi2022-03-15
|
* QUIC: fix dissection of draft-34 (#1484)dev-1Ivan Nardi2022-03-09
| | | | QUIC-34 is probably not used in production, but fixing it is trivial and it doesn't add any noise to the already complex QUIC code.
* configure: fix usage of libgpg-error with `--with-local-libgcrypt` (#1472)Ivan Nardi2022-03-05
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Right now, using external libgcrypt, nDPI is not linked to libgpg-error because configure script never checks for it. ``` ivan@ivan-Latitude-E6540:~/svnrepos/nDPI(dev)$ CC=gcc-11 CXX=g++-11 CFLAGS="-O3 -g -Werror" ./autogen.sh --enable-debug-messages --with-pcre --with-local-libgcrypt && make -s -j [...] checking for numa_available in -lnuma... yes checking for pcap_open_live in -lpcap... yes checking for pthread_setaffinity_np in -lpthread... yes checking for gcry_cipher_checktag in -lgcrypt... yes <------- missing check for libgpg-error checking for pcre_compile in -lpcre... yes checking that generated files are newer than configure... done [...] ivan@ivan-Latitude-E6540:~/svnrepos/nDPI(dev)$ grep HAVE_LIBGPG_ERROR src/include/ndpi_config.h /* #undef HAVE_LIBGPG_ERROR */ ``` Make both libgcrypt and libgpg-error mandatory if `--with-local-libgcrypt` is used. Technically speaking, libgpg-error might be optional, because it is used only for debug messages. However having both libraries mandatory slightly simplified the logic. In most environments, libgpg-error is a dependency of libgcrypt anyway, so having both libraries should be the standard case.
* Drop support for non-gcrypt builds. (#1469)Toni2022-03-02
| | | | | | | | * As there is now a builtin, lightweight libgcrypt there is no need to disable tls-clho decryption. * It is still possible to use a host libgcrypt with `--with-local-libgcrypt'. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Added lightweight implementation of libgcrypt. (#1444)Vitaly Lavrov2022-02-20
| | | | | | | | | | | | | | | | | | | | | Implementation borrowed from the https://github.com/ARMmbed/mbedtls.git project (v3.1.0) Speed testing (Xeon(R) CPU E3-1230 V2 @ 3.30GHz): gcrypt-gnu Test md 2897 ms enc 2777 ms dec 942 ms gcrypt-int Test md 3668 ms enc 1312 ms dec 2836 ms gcrypt-int-noaesni Test md 3652 ms enc 1916 ms dec 4458 ms gcrypt-gnu-nonopt Test md 3763 ms enc 4978 ms dec 3999 ms gcrypt-gnu-nonopt - libgcrypt compiled without hardware acceleration --disable-padlock-support --disable-aesni-support \ --disable-shaext-support --disable-pclmul-support \ --disable-sse41-support --disable-drng-support \ --disable-avx-support --disable-avx2-support \ --disable-neon-support --disable-arm-crypto-support \ --disable-ppc-crypto-support --disable-amd64-as-feature-detection
* QUIC: fix an integer overflow (#1396)Ivan Nardi2022-01-11
| | | | | | | | | | | | | | | | | Reported by oss-fuzz: ``` ==685288==ERROR: AddressSanitizer: SEGV on unknown address 0x61a100000687 (pc 0x0000005aba64 bp 0x7ffe3f29f510 sp 0x7ffe3f29f400 T0) ==685288==The signal is caused by a READ memory access. SCARINESS: 20 (wild-addr-read) #0 0x5aba64 in quic_len ndpi/src/lib/protocols/quic.c:203:12 #1 0x5aba64 in decrypt_initial_packet ndpi/src/lib/protocols/quic.c:993:16 #2 0x5aba64 in get_clear_payload ndpi/src/lib/protocols/quic.c:1302:21 #3 0x5aba64 in ndpi_search_quic ndpi/src/lib/protocols/quic.c:1658:19 #4 0x579f00 in check_ndpi_detection_func ndpi/src/lib/ndpi_main.c:4683:6 #5 0x57abe6 in ndpi_check_flow_func ndpi/src/lib/ndpi_main.c:0 #6 0x583b2c in ndpi_detection_process_packet ndpi/src/lib/ndpi_main.c:5545:15 #7 0x55e75e in LLVMFuzzerTestOneInput ndpi/fuzz/fuzz_process_packet.c:30:3 [...] ```
* Add a "confidence" field about the reliability of the classification. (#1395)Ivan Nardi2022-01-11
| | | | | | | | | | | | | As a general rule, the higher the confidence value, the higher the "reliability/precision" of the classification. In other words, this new field provides an hint about "how" the flow classification has been obtained. For example, the application may want to ignore classification "by-port" (they are not real DPI classifications, after all) or give a second glance at flows classified via LRU caches (because of false positives). Setting only one value for the confidence field is a bit tricky: more work is probably needed in the next future to tweak/fix/improve the logic.
* Update copyrightAlfredo Cardigliano2022-01-03
|
* QUIC: fix old versions of GQUIC on big-endian machines (#1387)Ivan Nardi2021-12-17
| | | Credits to @viniciussn (see #1312)
* Improve IPv6 support, enabling IPv6 traffic on (almost) all dissectors. (#1380)Ivan Nardi2021-12-04
| | | | | | | | | | | There are no valid reasons for a (generic) protocol to ignore IPv6 traffic. Note that: * I have not found the specifications of "CheckPoint High Availability Protocol", so I don't know how/if it supports IPv6 * all LRU caches are still IPv4 only Even if src_id/dst_id stuff is probably useless (see #1279), the right way to update the protocol classification is via `ndpi_set_detected_protocol()`
* QUIC: add support for QUICv2 (draft 00) (#1379)Ivan Nardi2021-12-04
| | | | It is already time to start looking at the new QUIC version. See: https://datatracker.ietf.org/doc/html/draft-ietf-quic-v2-00
Powered by cgit, Themed by Ted Yin.