aboutsummaryrefslogtreecommitdiff
path: root/src/lib/protocols/http.c
Commit message (Collapse)AuthorAge
...
* HTTP: correctly set the classification protocolsNardi Ivan2022-09-06
| | | | | | | | | | | | Classification should always be set via `ndpi_set_detected_protocol()` to be sure to set a correct `confidence` value, too. Having a "known" protocol stack with `NDPI_CONFIDENCE_UNKNOWN` as confidence, is not valid. This code in HTTP dissector likely needs some more thoughts (the classification itself of the attached example doesn't make a lot of sense), but the goal of this commit is only to always have a valid `confidence` value.
* Avoid useless host automa lookup (#1724)Ivan Nardi2022-09-05
| | | | | | | | | | | | | | | | | | | The host automa is used for two tasks: * protocol sub-classification (obviously); * DGA evaluation: the idea is that if a domain is present in this automa, it can't be a DGA, regardless of its format/name. In most dissectors both checks are executed, i.e. the code is something like: ``` ndpi_match_host_subprotocol(..., flow->host_server_name, ...); ndpi_check_dga_name(..., flow->host_server_name,...); ``` In that common case, we can perform only one automa lookup: if we check the sub-classification before the DGA, we can avoid the second lookup in the DGA function itself.
* HTTP, SoftEther, Florensia: fix some memory corruptionsNardi Ivan2022-08-22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | ``` ==12308==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60600068f4b4 at pc 0x5603d2b56098 bp 0x7ffc967bd3b0 sp 0x7ffc967bd3a8 READ of size 2 at 0x60600068f4b4 thread T0 #0 0x5603d2b56097 in ndpi_search_florensia /home/ivan/svnrepos/nDPI/src/lib/protocols/florensia.c:93:40 #1 0x5603d2a2e835 in check_ndpi_detection_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5414:6 #2 0x5603d2a2f247 in check_ndpi_tcp_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5462:12 #3 0x5603d2a2eee7 in ndpi_check_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5481:12 #4 0x5603d2a3ffb5 in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:6308:15 #5 0x5603d2955f4c in packet_processing /home/ivan/svnrepos/nDPI/example/reader_util.c:1612:31 #6 0x5603d294ee64 in ndpi_workflow_process_packet /home/ivan/svnrepos/nDPI/example/reader_util.c:2182:10 #7 0x5603d290aa50 in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:107:7 #8 0x5603d290ab8b in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:179:17 #9 0x7f972cdc9082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16 #10 0x5603d28494fd in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader_with_main+0x4be4fd) (BuildId: e81373a75c70e248ba9ae985fcb4134e7113121a) ``` ``` ==14164==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60d0000000c1 at pc 0x561608bada2c bp 0x7ffc40d38ed0 sp 0x7ffc40d38690 READ of size 10 at 0x60d0000000c1 thread T0 #0 0x561608bada2b in __interceptor_strncpy (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet_with_main+0x4e9a2b) (BuildId: 2e4fd67f1aeb93c491a04ae424331669c993ee9b) #1 0x561608e66f2a in dissect_softether_host_fqdn /home/ivan/svnrepos/nDPI/src/lib/protocols/softether.c:226:2 #2 0x561608e64e88 in ndpi_search_softether /home/ivan/svnrepos/nDPI/src/lib/protocols/softether.c:316:8 #3 0x561608ca7475 in check_ndpi_detection_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5414:6 #4 0x561608ca823b in check_ndpi_udp_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5450:10 #5 0x561608ca7bdc in ndpi_check_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5483:12 #6 0x561608cb8bf5 in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:6308:15 #7 0x561608bff84e in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:29:5 #8 0x561608bff9c7 in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:101:17 #9 0x7fc50662d082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16 #10 0x561608b3e42d in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet_with_main+0x47a42d) (BuildId: 2e4fd67f1aeb93c491a04ae424331669c993ee9b) ``` Found by oss-fuzzer. See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=50105 See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=50110 See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=50268
* HTTP: fix classification in some corner cases (#1704)Ivan Nardi2022-08-09
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Initializing `low->detected_protocol_stack[1]` but not `low->detected_protocol_stack[0]` lead to *very* strange errors. Oss-fuzzer, exploiting this bug, has been able to crash the application or to leak some memory ``` ==19775==ERROR: AddressSanitizer: SEGV on unknown address (pc 0x5624c0dd1802 bp 0x5624c197e848 sp 0x7fff4b4db690 T0) ==19775==The signal is caused by a READ memory access. ==19775==Hint: this fault was caused by a dereference of a high value address (see register values below). Disassemble the provided pc to learn which register was used. #0 0x5624c0dd1802 in __asan::Allocator::Deallocate(void*, unsigned long, unsigned long, __sanitizer::BufferedStackTrace*, __asan::AllocType) (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader_with_main+0x522802) (BuildId: 9a1ed9efcaf47e345767c86520372d28e31ca1aa) #1 0x5624c0e53ea6 in __interceptor_free (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader_with_main+0x5a4ea6) (BuildId: 9a1ed9efcaf47e345767c86520372d28e31ca1aa) #2 0x5624c0ebb474 in free_wrapper /home/ivan/svnrepos/nDPI/example/reader_util.c:330:3 #3 0x5624c0f75561 in ndpi_free /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:240:7 #4 0x5624c0faa921 in ndpi_free_flow_data /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:4915:2 #5 0x5624c0f755ef in ndpi_free_flow /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:8076:5 #6 0x5624c0f755c4 in ndpi_flow_free /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:253:5 #7 0x5624c0eba981 in ndpi_free_flow_info_half /home/ivan/svnrepos/nDPI/example/reader_util.c:305:25 #8 0x5624c0ecb4bc in process_ndpi_collected_info /home/ivan/svnrepos/nDPI/example/reader_util.c:1317:5 #9 0x5624c0edca36 in packet_processing /home/ivan/svnrepos/nDPI/example/reader_util.c:1639:2 #10 0x5624c0ed5172 in ndpi_workflow_process_packet /home/ivan/svnrepos/nDPI/example/reader_util.c:2182:10 #11 0x5624c0e90a80 in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:107:7 #12 0x5624c0e90bbb in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:179:17 #13 0x7f580731f082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16 #14 0x5624c0dcf52d in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader_with_main+0x52052d) (BuildId: 9a1ed9efcaf47e345767c86520372d28e31ca1aa) ``` ``` ==17492==ERROR: LeakSanitizer: detected memory leaks Direct leak of 5 byte(s) in 1 object(s) allocated from: #0 0x55b799b0f01e in malloc (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet_with_main+0x56101e) (BuildId: d2adbfb29a6eda6dc59fdfb8930d7e6496ac7b8b) #1 0x55b799bb3bf4 in ndpi_malloc /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:212:46 #2 0x55b799bb3f28 in ndpi_strdup /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:279:13 #3 0x55b799c8b9bf in processCertificateElements /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:683:39 #4 0x55b799c81e89 in processCertificate /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:835:7 #5 0x55b799c8d500 in processTLSBlock /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:889:13 #6 0x55b799ca80f1 in ndpi_search_tls_tcp /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:1034:2 #7 0x55b799ca3be2 in ndpi_search_tls_wrapper /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:2475:5 #8 0x55b799bf9345 in check_ndpi_detection_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5414:6 #9 0x55b799bf9d57 in check_ndpi_tcp_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5462:12 #10 0x55b799bf99f7 in ndpi_check_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5481:12 #11 0x55b799c0b838 in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:6308:15 #12 0x55b799b4b87e in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:29:5 ``` ``` ==17511==ERROR: LeakSanitizer: detected memory leaks Direct leak of 1 byte(s) in 1 object(s) allocated from: #0 0x55aed645e01e in malloc (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet_with_main+0x56101e) (BuildId: d2adbfb29a6eda6dc59fdfb8930d7e6496ac7b8b) #1 0x55aed6502bf4 in ndpi_malloc /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:212:46 #2 0x55aed6502f28 in ndpi_strdup /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:279:13 #3 0x55aed65ea0c6 in processClientServerHello /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:2153:34 #4 0x55aed65dbe77 in processTLSBlock /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:867:5 #5 0x55aed65f70f1 in ndpi_search_tls_tcp /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:1034:2 #6 0x55aed65f2be2 in ndpi_search_tls_wrapper /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:2475:5 #7 0x55aed6548345 in check_ndpi_detection_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5414:6 #8 0x55aed6548d57 in check_ndpi_tcp_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5462:12 #9 0x55aed65489f7 in ndpi_check_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5481:12 #10 0x55aed655a838 in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:6308:15 #11 0x55aed649a87e in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:29:5 ``` See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=49844 See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=49842 See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=49906 See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=49915
* HTTP, Bittorrent, Kerberos: fix stopping of extra dissection (#1700)Ivan Nardi2022-08-06
| | | | | | | The return value of the extra-dissection callback indicates if the extra dissection needs to be called again. In the HTTP cose, this setting to NULL of the callabck is wrong since we stop extra dissection only if we have a hostname *and* a return code.
* Further simplification of `ndpi_process_extra_packet()` (#1698)Ivan Nardi2022-08-05
| | | | | | | | See 95e16872. After c0732eda, we can safely remove the protocol list from `ndpi_process_extra_packet()`. The field `flow->check_extra_packets` is redundant; remove it.
* HTTP: improve sub-classification (#1696)Ivan Nardi2022-08-05
| | | | | | | | | | Content-matched sub-protocols (DASH, IPP, MPEGDASH...) shouldn't ovewrite the previous master protocol (if any; usually HTTP). Furthermore. the HTTP dissector shouldn't update the classification (in the extra-dissection code path) if a content-matched sub-protocols has already been found. This commit should address the first part of the changes described in #1687.
* HTTP: fix classification (#1692)Ivan Nardi2022-07-30
| | | | | | | | | | If we have a valid HTTP sessions, we should ignore `flow->guessed_protocol_id` field (i.e. classification "by-port") altogheter. The attached trace was classified as "SIP/HTTP" only because the *client* port was 5060... As a general rule, having a classification such as "XXXX/HTTP" is *extremely* suspicious.
* Update the protocol bitmask for some protocols (#1675)Ivan Nardi2022-07-27
| | | | | | | Tcp retransmissions should be ignored. Remove some unused protocol bitmasks. Update script to download Whatsapp IP list.
* Improve handling of HTTP-Proxy and HTTP-Connect (#1673)Ivan Nardi2022-07-25
| | | | | | | | | | | | Treat HTTP-Proxy and HTTP-Connect flows like the HTTP ones: print/serialize all the attributes and allow parsing of replies. The line about "1kxun" has been removed to avoid regressions in 1KXUN classification in `tests/pcap/1kxun.pcap`. I haven't fully understod what was happening but the comment at the beginning of `static ndpi_category_match category_match[]` says that we can't have overlaps between `host_match` and `category_match` lists and that is no longer true since 938e89ca. Bottom line: removing this line seems the right thing to do, anyway.
* HTTP: improve detection of WindowsUpdate (#1658)Ivan Nardi2022-07-10
| | | | WindowsUpdate is also transported over HTTP, using a numeric IP as hostname (some kinds of CDN?)
* Prohibit MPEG-DASH to set HTTP as application protocol. (#1560)Toni2022-05-30
| | | Signed-off-by: lns <matzeton@googlemail.com>
* HTTP: fix heap-buffer-overflow error (#1564)Ivan Nardi2022-05-30
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | ``` ==222479==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60b000014739 at pc 0x55af06f2364f bp 0x7ffd7b6f4bf0 sp 0x7ffd7b6f4378 READ of size 12 at 0x60b000014739 thread T0 #0 0x55af06f2364e in printf_common(void*, char const*, __va_list_tag*) asan_interceptors.cpp.o #1 0x55af06f24f70 in __interceptor_snprintf (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader_with_main+0x613f70) (BuildId: f6545ec2bd7663bc3f16aeeb87bddc64d173a2a8) #2 0x55af0720927f in ndpi_check_http_header /home/ivan/svnrepos/nDPI/src/lib/protocols/http.c:982:2 #3 0x55af071f4797 in ndpi_check_http_tcp /home/ivan/svnrepos/nDPI/src/lib/protocols/http.c:1219:5 #4 0x55af071f05c5 in ndpi_search_http_tcp /home/ivan/svnrepos/nDPI/src/lib/protocols/http.c:1402:3 #5 0x55af07080d1e in check_ndpi_detection_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5150:6 #6 0x55af07081734 in check_ndpi_tcp_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5198:12 #7 0x55af070813d7 in ndpi_check_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5217:12 #8 0x55af070939f7 in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:6076:15 #9 0x55af06fc7e1f in packet_processing /home/ivan/svnrepos/nDPI/example/reader_util.c:1541 #10 0x55af06fc7e1f in ndpi_workflow_process_packet /home/ivan/svnrepos/nDPI/example/reader_util.c:2110 #11 0x55af06fc2859 in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:109:7 #12 0x55af06fc2feb in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:181:17 #13 0x7efe5eaac082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16 #14 0x55af06f0055d in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader_with_main+0x5ef55d) (BuildId: f6545ec2bd7663bc3f16aeeb87bddc64d173a2a8) 0x60b000014739 is located 0 bytes to the right of 105-byte region [0x60b0000146d0,0x60b000014739) allocated by thread T0 here: #0 0x55af06f84bae in malloc (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader_with_main+0x673bae) (BuildId: f6545ec2bd7663bc3f16aeeb87bddc64d173a2a8) #1 0x55af06fc2673 in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:103:31 #2 0x55af06fc2feb in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:181:17 #3 0x7efe5eaac082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16 ``` Fiund by oss-fuzzer. See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=47724
* Updated tests resultsLuca Deri2022-05-30
| | | | Code cleanup
* Added RiskInfo stringLuca Deri2022-05-30
|
* Added detection for WordPress exploitsLuca Deri2022-05-29
| | | | Fixed ndpi_iph_is_valid_and_not_fragmented() that was bugged with non UDP traffic
* Added MPEG-DASH dissector. Fixes #1223. (#1555)Toni2022-05-29
| | | | | | * Improved HTTP POST detection * Refactored subprotocol detection Signed-off-by: lns <matzeton@googlemail.com>
* Fixed HTTP lower/upper protocol mess for Aimini/IPP. (#1557)Toni2022-05-28
| | | | | * If HTTP was already detected, there is no need to overwrite the detection again Signed-off-by: lns <matzeton@googlemail.com>
* Improved detection of invalid SNI and hostnames in TLS, HTTPLuca Deri2022-05-24
|
* Moved RTSP http patterns to the protocol source file.lns2022-05-19
| | | | Signed-off-by: lns <matzeton@googlemail.com>
* Improved Xiaomi HTTP detection. (#1546)Toni2022-05-10
| | | | | * Merged Xiaomi pcap files Signed-off-by: lns <matzeton@googlemail.com>
* Improved suspicious http user agent detection. (#1537)Toni2022-05-02
| | | Signed-off-by: lns <matzeton@googlemail.com>
* Added generic user agent setter. (#1530)Toni2022-04-25
| | | | | * ndpiReader: Print user agent if one was set and not just for certain protocols. Signed-off-by: lns <matzeton@googlemail.com>
* Fix compilation and sync unit tests results (#1445)Ivan Nardi2022-02-19
| | | | 'strcasestr' is not defined in all enviroments: quicker fix is to use 'ndpi_strncasestr' instead.
* Added newflow risk NDPI_HTTP_CRAWLER_BOTLuca Deri2022-02-17
|
* Added NDPI_ERROR_CODE_DETECTED riskLuca Deri2022-02-03
|
* Renamed DCERPC to more generic RPC protocol so we can use also for other ↵Luca Deri2022-02-03
| | | | | | | types of RPCs (not limited to DCE) Extended HTTP plugin to support RPC Improved HTTP crear text detection to limit it to Basic and Digest
* Add a "confidence" field about the reliability of the classification. (#1395)Ivan Nardi2022-01-11
| | | | | | | | | | | | | As a general rule, the higher the confidence value, the higher the "reliability/precision" of the classification. In other words, this new field provides an hint about "how" the flow classification has been obtained. For example, the application may want to ignore classification "by-port" (they are not real DPI classifications, after all) or give a second glance at flows classified via LRU caches (because of false positives). Setting only one value for the confidence field is a bit tricky: more work is probably needed in the next future to tweak/fix/improve the logic.
* Improved user agent analysisLuca Deri2022-01-09
|
* Invalid checkLuca Deri2022-01-06
|
* Update copyrightAlfredo Cardigliano2022-01-03
|
* Added support for Log4J/Log4Shell detection in nDPI via a new flow risk ↵Luca Deri2021-12-23
| | | | named NDPI_POSSIBLE_EXPLOIT
* A final(?) effort to reduce memory usage per flow (#1389)Ivan Nardi2021-12-22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Remove some unused fields and re-organize other ones. In particular: * Update the parameters of `ndpi_ssl_version2str()` function * Zattoo, Thunder: these timestamps aren't really used. * Ftp/mail: these protocols are dissected only over TCP. * Attention must be paid to TLS.Bittorrent flows to avoid invalid read/write to `flow->protos.bittorrent.hash` field. This is the last(?) commit of a long series (see 22241a1d, 227e586e, 730c2360, a8ffcd8b) aiming to reduce library memory consumption. Before, at nDPI 4.0 (more precisly, at a6b10cf7, because memory stats were wrong until that commit): ``` nDPI Memory statistics: nDPI Memory (once): 221.15 KB Flow Memory (per flow): 2.94 KB ``` Now: ``` nDPI Memory statistics: nDPI Memory (once): 231.71 KB Flow Memory (per flow): 1008 B <--------- ``` i.e. memory usage per flow has been reduced by 66%, dropping below the psychological threshold of 1 KB. To further reduce this value, we probably need to look into #1279: let's fight this battle another day.
* HTTP proxy improvementLuca Deri2021-11-25
|
* Reworked HTTP protocol dissection including HTTP proxy and HTTP connectLuca Deri2021-11-25
|
* Rework how hostname/SNI info is saved (#1330)Ivan Nardi2021-11-24
| | | | | | | | | | | | | | | | | | | | | | | | | | | Looking at `struct ndpi_flow_struct` the two bigger fields are `host_server_name[240]` (mainly for HTTP hostnames and DNS domains) and `protos.tls_quic.client_requested_server_name[256]` (for TLS/QUIC SNIs). This commit aims to reduce `struct ndpi_flow_struct` size, according to two simple observations: 1) maximum one of these two fields is used for each flow. So it seems safe to merge them; 2) even if hostnames/SNIs might be very long, in practice they are rarely longer than a fews tens of bytes. So, using a (single) large buffer is a waste of memory for all kinds of flows. If we need to truncate the name, we keep the *last* characters, easing domain matching. Analyzing some real traffic, it seems safe to assume that the vast majority of hostnames/SNIs is shorter than 80 bytes. Hostnames/SNIs are always converted to lowercase. Attention was given so as to be sure that unit-tests outputs are not affected by this change. Because of a bug, TLS/QUIC SNI were always truncated to 64 bytes (the *first* 64 ones): as a consequence, there were some "Suspicious DGA domain name" and "TLS Certificate Mismatch" false positives.
* Fix writes to `flow->protos` union fields (#1354)Ivan Nardi2021-11-15
| | | | | | | | | | | | | | | | | | | | | | | | | We can write to `flow->protos` only after a proper classification. This issue has been found in Kerberos, DHCP, HTTP, STUN, IMO, FTP, SMTP, IMAP and POP code. There are two kinds of fixes: * write to `flow->protos` only if a final protocol has been detected * move protocol state out of `flow->protos` The hard part is to find, for each protocol, the right tradeoff between memory usage and code complexity. Handle Kerberos like DNS: if we find a request, we set the protocol and an extra callback to further parsing the reply. For all the other protocols, move the state out of `flow->protos`. This is an issue only for the FTP/MAIL stuff. Add DHCP Class Identification value to the output of ndpiReader and to the Jason serialization. Extend code coverage of fuzz tests. Close #1343 Close #1342
* Add detection of OCSP (#1370)Ivan Nardi2021-11-11
| | | | | | | | | | This protocol is detected via HTTP Content-Type header. Until 89d548f9, nDPI had a dedicated automa (`content_automa`) to classify a HTTP flow according to this header. Since then, this automa has been useless because it is always empty. Re-enable it to match only a string seems overkilling. Remove all `content_automa` leftovers.
* Fix broken fuzz_process_packet fuzzer by adding a call to ↵Toni2021-10-18
| | | | | | | | | | | | ndpi_finalize_initialization(). (#1334) * fixed several memory errors (heap-overflow, unitialized memory, etc) * ability to build fuzz_process_packet with a main() allowing to replay crash data generated with fuzz_process_packet by LLVMs libfuzzer * temporarily disable fuzzing if `tests/do.sh` executed with env FUZZY_TESTING_ENABLED=1 Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Reworked flow risks asignmentLuca Deri2021-10-16
| | | | Added esceptions for windows update and binary application transfer risk
* Remove `struct ndpi_packet_struct` from `struct ndpi_flow_struct` (#1319)Ivan Nardi2021-10-05
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There are no real reasons to embed `struct ndpi_packet_struct` (i.e. "packet") in `struct ndpi_flow_struct` (i.e. "flow"). In other words, we can avoid saving dissection information of "current packet" into the "flow" state, i.e. in the flow management table. The nDPI detection module processes only one packet at the time, so it is safe to save packet dissection information in `struct ndpi_detection_module_struct`, reusing always the same "packet" instance and saving a huge amount of memory. Bottom line: we need only one copy of "packet" (for detection module), not one for each "flow". It is not clear how/why "packet" ended up in "flow" in the first place. It has been there since the beginning of the GIT history, but in the original OpenDPI code `struct ipoque_packet_struct` was embedded in `struct ipoque_detection_module_struct`, i.e. there was the same exact situation this commit wants to achieve. Most of the changes in this PR are some boilerplate to update something like "flow->packet" into something like "module->packet" throughout the code. Some attention has been paid to update `ndpi_init_packet()` since we need to reset some "packet" fields before starting to process another packet. There has been one important change, though, in ndpi_detection_giveup(). Nothing changed for the applications/users, but this function can't access "packet" anymore. The reason is that this function can be called "asynchronously" with respect to the data processing, i.e in context where there is no valid notion of "current packet"; for example ndpiReader calls it after having processed all the traffic, iterating the entire session table. Mining LRU stuff seems a bit odd (even before this patch): probably we need to rethink it, as a follow-up.
* Remove `detected_protocol_stack` field from `ndpi_packet_struct` (#1317)Ivan Nardi2021-09-29
| | | | | | | | | | | | | This field is an exact copy of `ndpi_flow_struct->detected_protocol_stack[2]`: * at the very beginning of packet dissection, the value saved in `flow->detected_protocol_stack` is copied in `packet->detected_protocol_stack` (via `ndpi_detection_process_packet()` -> `ndpi_init_packet_header()`) * every time we update `flow->detected_protocol_stack` we update `packet->detected_protocol_stack` too (via `ndpi_int_change_protocol()` -> `ndpi_int_change_packet_protocol()`) These two fields are always in sync: keeping the same value in two different places is useless.
* A (tiny) effort to reduce memory usage per flow (#1304)Ivan Nardi2021-09-18
| | | | | | | Follow-up of 22241a1d Only trivial changes: * remove completely unused fields * remove fields only written (but never read) * CSGO protocol only handles UDP traffic
* Added new risk for clear text credentialsLuca Deri2021-09-10
|
* Compile everything with "-W -Wall -Wno-unused-parameter" flags (#1276)Ivan Nardi2021-08-20
| | | | | | | | | | | | | | | | Fix all the warnings. Getting rid of "-Wno-unused-parameter" is quite complex because some parameters usage depends on compilation variable (i.e. `--enable-debug-messages`). The "-Werror" flag has been added only in Travis builds to avoid breaking the builds to users using uncommon/untested OS/compiler/enviroment. Tested on: * x86_64; Ubuntu 20.04; gcc 7,8,9,10,11; clang 7,8,9,10,11,12 * x86_64; CentOS 7.7; gcc 4.8.5 (with "--disable-gcrypt" flag) * Raspberry 4; Debian 10.10; gcc 8.3.0
* Skip whitespaces between HTTP method and URL. (#1271)Toni2021-08-08
| | | | | * be less case-restrictive, RFC2616 wants it that way Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Improved RTSP detection and fixed HTTP false-positive. Fixes #1229. (#1266)Toni2021-07-31
| | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Reworked flow risk implementationLuca Deri2021-07-23
|
* Code cleanup. (#1246)Vitaly Lavrov2021-07-16
| | | | ndpi_utils.c: use ndpi_malloc,ndpi_calloc,ndpi_free genshin_impact.c, git.c, hpvirtgrp.c, http.c, z3950.c: removed "#include stdlib.h"
* Fix for #1230 (#1235)Vitaly Lavrov2021-07-07
| | | | | | | | | | | | | | | * Revert "Fix return value of ndpi_match_string_subprotocol() (#1230)" This reverts commit 58665e93a98d014b53d131b2481ccab074efc9ff. * Checking the return code after calling ndpi_match_string_subprotocol() ndpi_api.h: Description of the returned error codes for the ndpi_match_string_subprotocol() function. If the ndpi_match_string_subprotocol() function returned an error, then return NDPI_PROTOCOL_UNKNOWN. http: The "Content-type" header is only checked if it is not empty.
Powered by cgit, Themed by Ted Yin.