| Commit message (Collapse) | Author | Age |
|
|
| |
Useful with asymmetric traffic with (D)TLS <= 1.2
|
|
|
|
|
|
|
|
| |
Tell "Advertised" ALPN list from "Negotiated" ALPN; the former is
extracted from the CH, the latter from the SH.
Add some entries to the known ALPN list.
Fix printing of "TLS Supported Versions" field.
|
|
|
|
|
|
|
|
| |
In some rare cases, it is possible to sub-classify the flow via ALPN
matching. This is particularly usefull for asymmetric traffic where the
Client Hello doens't have the SNI.
For the time being there is only one rule, about ANYDESK.
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
TLS classification usually stops after processing *server* certificates
(if any). That means, that *client* certificate, if present, is usually
ignored.
However in some corner cases (i.e. unidirectional traffic) we might end
up processing client certificate and exposing its metadata: the issue is
that the application will think that this metadata are about the server
and not about the client.
So, for the time being, always ignore client certificate processing.
As a future work, we might find an efficient way to process and export both
certificates.
|
|
|
|
|
|
| |
Note that current code access `certificate_processed` state even before
setting the protocol classification, so this piece of information can't
be saved in `flow->protos` union.
|
|
|
|
|
|
|
|
| |
0 as size value disable the cache.
The diffs in unit tests are due to the fact that some lookups are
performed before the first insert: before this change these lookups
weren't counted because the cache was not yet initialized, now they are.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Basically:
* "classification by-ip" (i.e. `flow->guessed_protocol_id_by_ip` is
NEVER returned in the protocol stack (i.e.
`flow->detected_protocol_stack[]`);
* if the application is interested into such information, it can access
`ndpi_protocol->protocol_by_ip` itself.
There are mainly 4 points in the code that set the "classification
by-ip" in the protocol stack: the generic `ndpi_set_detected_protocol()`/
`ndpi_detection_giveup()` functions and the HTTP/STUN dissectors.
In the unit tests output, a print about `ndpi_protocol->protocol_by_ip`
has been added for each flow: the huge diff of this commit is mainly due
to that.
Strictly speaking, this change is NOT an API/ABI breakage, but there are
important differences in the classification results. For examples:
* TLS flows without the initial handshake (or without a matching
SNI/certificate) are simply classified as `TLS`;
* similar for HTTP or QUIC flows;
* DNS flows without a matching request domain are simply classified as
`DNS`; we don't have `DNS/Google` anymore just because the server is
8.8.8.8 (that was an outrageous behaviour...);
* flows previusoly classified only "by-ip" are now classified as
`NDPI_PROTOCOL_UNKNOWN`.
See #1425 for other examples of why adding the "classification by-ip" in
the protocol stack is a bad idea.
Please, note that IPV6 is not supported :( (long standing issue in nDPI) i.e.
`ndpi_protocol->protocol_by_ip` wil be always `NDPI_PROTOCOL_UNKNOWN` for
IPv6 flows.
Define `NDPI_CONFIDENCE_MATCH_BY_IP` has been removed.
Close #1687
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The field `flow->guessed_host_protocol_id` is set at the beginning of
the flow analysis and it represents the "classification by ip" of the flow
itself.
This field should never be changed. Dissectors which want to provide an
"hint" about the classification, should update `flow->guessed_protocol_id`
instead. Such "hint" is useless if the dissector set the "extra-dissection"
data-path.
Rename such field to `guessed_protocol_id_by_ip` to better describe its
role.
Preliminary work necessary for #1687
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The host automa is used for two tasks:
* protocol sub-classification (obviously);
* DGA evaluation: the idea is that if a domain is present in this
automa, it can't be a DGA, regardless of its format/name.
In most dissectors both checks are executed, i.e. the code is something
like:
```
ndpi_match_host_subprotocol(..., flow->host_server_name, ...);
ndpi_check_dga_name(..., flow->host_server_name,...);
```
In that common case, we can perform only one automa lookup: if we check the
sub-classification before the DGA, we can avoid the second lookup in
the DGA function itself.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A lot of protocols provide the feature to upgrade their plain text
connections to an encrypted one, via some kind of "STARTTLS" command.
Add generic code to support this extension, and allow dissection of the
entire TLS handshake.
As examples, SMTP, POP, IMAP and FTP dissectors have been updated.
Since this feature requires to process more packets per flow, add the
possibility to disable it.
Fix some log messages.
Slight improvement on TCP sequence number tracking.
As a side effect, this commit fix also a memory leak found by
oss-fuzzer
```
==108966==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 22 byte(s) in 1 object(s) allocated from:
#0 0x55f8b367a0be in malloc (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader_with_main+0x5480be) (BuildId: 94debacb4a6784c30420ab748c8bf3cc59621063)
#1 0x55f8b36e1345 in ndpi_malloc_wrapper /home/ivan/svnrepos/nDPI/example/reader_util.c:321:10
#2 0x55f8b379c7d2 in ndpi_malloc /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:212:25
#3 0x55f8b379cb18 in ndpi_strdup /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:279:13
#4 0x55f8b386ce46 in processClientServerHello /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:2153:34
#5 0x55f8b385ebf7 in processTLSBlock /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:867:5
#6 0x55f8b39e708c in ndpi_extra_search_mail_smtp_tcp /home/ivan/svnrepos/nDPI/src/lib/protocols/mail_smtp.c:422:9
#7 0x55f8b37e636c in ndpi_process_extra_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5884:9
#8 0x55f8b37edc05 in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:6276:5
#9 0x55f8b3701ffc in packet_processing /home/ivan/svnrepos/nDPI/example/reader_util.c:1619:31
#10 0x55f8b36faf14 in ndpi_workflow_process_packet /home/ivan/svnrepos/nDPI/example/reader_util.c:2189:10
#11 0x55f8b36b6a50 in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:107:7
```
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=50765
|
|
|
|
|
|
|
|
| |
See 95e16872.
After c0732eda, we can safely remove the protocol list from
`ndpi_process_extra_packet()`.
The field `flow->check_extra_packets` is redundant; remove it.
|
|
|
|
|
|
|
|
| |
* fixed autoconf CFLAGS/LDFLAGS MSAN issue which could lead to build errors
* introduced portable version of gmtime_r aka ndpi_gmtime_r
* do as most as possible of the serialization work in ndpi_utils.c
* use flow2json in ndpiReader
Signed-off-by: lns <matzeton@googlemail.com>
|
|
|
|
|
|
|
| |
If we have seen Application Data blocks in both directions, it means we
are after the handshake. At that point, extra dissection is useless.
Useful feature with TLS mid-sessions, i.e. sessions without initial
packets.
|
|
|
|
|
|
| |
The big change in TLS code is to allow "master" protocols other than
TLS/DTLS, like SMTPS, POPS and IMAPS.
This change will allow, in a future, a proper and complete TLS dissection
for all these protocols with "STARTTLS"-like messages.
|
|
|
|
|
|
|
| |
Move the prottocol specific logic into the proper dissector code, where
it belongs.
Next step: remove that list of protocols. Long goal: remove this
function altogether...
|
|
|
|
|
|
|
|
| |
* TLS: cosmetic changes
* TLS: improve reassembler
We might need to contemporary re-order messages from both directions:
use one buffer per direction.
|
|
|
|
| |
Add an helper to exclude a generic protocol
|
| |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Label SMTP w/ STARTTLS as SMTPS *and* dissect TLS clho.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Revert "SMTP with STARTTLS is now identified as SMTPS"
This reverts commit 52d987b603f49d996b4060f43265d1cf43c3c482.
* Revert "Compilation fix"
This reverts commit c019946f601bf3b55f64f78841a0d696e6c0bfc5.
* Sync unit tests.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
|
| |
|
|
|
|
|
|
|
| |
#1630. (#1637)
* FTP needs to get updated as well as it has similiar STARTTLS semantics -> follow-up
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
|
|
|
|
|
|
|
|
|
| |
u_int8_t ndpi_check_issuerdn_risk_exception(struct ndpi_detection_module_struct *ndpi_str, char *issuerDN);
Added new API function for checking nDPI-configured exceptions
u_int8_t ndpi_check_flow_risk_exception(struct ndpi_detection_module_struct *ndpi_str,
u_int num_params,
ndpi_risk_params **params);
|
|
|
|
| |
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
|
| |
|
| |
|
|
|
|
|
|
|
| |
* syslog: removed unnecessary/unreliable printable string check
* added `ndpi_isalnum()`
* splitted `ndpi_is_printable_string()` into `ndpi_is_printable_buffer()` and `ndpi_normalize_printable_string()`
Signed-off-by: lns <matzeton@googlemail.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Proper fix for the error already reported in 9040bc74
```
Uninitialized bytes in __interceptor_strlen at offset 3 inside [0x7ffc7a147390, 4)
==111876==WARNING: MemorySanitizer: use-of-uninitialized-value
#0 0x55e3e4f32e5b in ndpi_strdup /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:268:13
#1 0x55e3e4ef7391 in ndpi_set_risk /home/ivan/svnrepos/nDPI/src/lib/ndpi_utils.c:2254:12
#2 0x55e3e5022fdf in processClientServerHello /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:1523:8
#3 0x55e3e503af44 in processTLSBlock /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:865:5
#4 0x55e3e50397cd in ndpi_search_tls_tcp /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:1024:2
#5 0x55e3e503570c in ndpi_search_tls_wrapper /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:2453:5
#6 0x55e3e4f84a6a in check_ndpi_detection_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5150:6
#7 0x55e3e4f85778 in check_ndpi_tcp_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5198:12
#8 0x55e3e4f851e1 in ndpi_check_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5217:12
#9 0x55e3e4f96c7a in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:6076:15
#10 0x55e3e4ed91ef in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:29:5
#11 0x55e3e4eda27e in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:101:17
#12 0x7f5cb3146082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16
#13 0x55e3e4e5140d in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet_with_main+0xa340d) (BuildId: 0c02c433e039970dd13a60382b94dd5a8e19f625)
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
```
==300852==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7f108951f060 at pc 0x5641db0ee78c bp 0x7fff3b10b910 sp 0x7fff3b10b0d0
WRITE of size 116 at 0x7f108951f060 thread T0
#0 0x5641db0ee78b in __interceptor_strncpy (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet_with_main+0x64e78b) (BuildId: 23cb34bbaf8ac11eb97563bbdc12e29ead9fb0fa)
#1 0x5641db28efad in tlsCheckUncommonALPN /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:1224:7
#2 0x5641db27ec76 in processClientServerHello /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:1533:6
#3 0x5641db295677 in processTLSBlock /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:865:5
#4 0x5641db2935bb in ndpi_search_tls_tcp /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:1024:2
#5 0x5641db28f692 in ndpi_search_tls_wrapper /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:2446:5
#6 0x5641db1d87ce in check_ndpi_detection_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5150:6
#7 0x5641db1d91e4 in check_ndpi_tcp_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5198:12
#8 0x5641db1d8e87 in ndpi_check_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5217:12
#9 0x5641db1eb4a7 in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:6076:15
#10 0x5641db140b45 in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:29:5
#11 0x5641db14130b in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:101:17
#12 0x7f108bcab082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16
#13 0x5641db07f46d in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet_with_main+0x5df46d) (BuildId: 23cb34bbaf8ac11eb97563bbdc12e29ead9fb0fa)
Address 0x7f108951f060 is located in stack of thread T0 at offset 96 in frame
#0 0x5641db28ec4f in tlsCheckUncommonALPN /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:1204
This frame has 1 object(s):
[32, 96) 'str' (line 1218) <== Memory access at offset 96 overflows this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
(longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-buffer-overflow (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet_with_main+0x64e78b) (BuildId: 23cb34bbaf8ac11eb97563bbdc12e29ead9fb0fa) in __interceptor_strncpy
Shadow bytes around the buggy address:
```
Avoid zeroing the entire string.
Found by oss-fuzzer
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=47730
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
```
Uninitialized bytes in __interceptor_strlen at offset 0 inside [0x7ffc987443a0, 7)
==282918==WARNING: MemorySanitizer: use-of-uninitialized-value
#0 0x5651277e001b in ndpi_strdup /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:268:13
#1 0x5651277a45c1 in ndpi_set_risk /home/ivan/svnrepos/nDPI/src/lib/ndpi_utils.c:2254:12
#2 0x5651278cfaa6 in processClientServerHello /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:1515:8
#3 0x5651278e7834 in processTLSBlock /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:865:5
#4 0x5651278e60bd in ndpi_search_tls_tcp /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:1024:2
#5 0x5651278e1ffc in ndpi_search_tls_wrapper /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:2445:5
#6 0x565127831c2a in check_ndpi_detection_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5150:6
#7 0x565127832938 in check_ndpi_tcp_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5198:12
#8 0x5651278323a1 in ndpi_check_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5217:12
#9 0x565127843e3a in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:6076:15
#10 0x56512778641f in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:29:5
#11 0x5651277874ae in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:101:17
#12 0x7f4c46794082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16
#13 0x5651276fe40d in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet_with_main+0xa340d)
```
Found by oss-fuzzer
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=47728
|
|
|
| |
This way, the CI results are consistent
|
|
|
|
| |
Code cleanup
|
| |
|
|
|
|
|
| |
See: https://github.com/ntop/nDPI/runs/6643914510?check_suite_focus=true
Convert al the `MIN(a,b)` calls to `ndpi_min(a,b)`
|
| |
|
| |
|
|
|
| |
Signed-off-by: lns <matzeton@googlemail.com>
|
|
|
|
|
|
| |
* #1532 did fx TLS appdata detection only partially
* use flow->l4.tcp.tls.message.buffer_used instead of packet->payload
Signed-off-by: lns <matzeton@googlemail.com>
|
|
|
| |
Signed-off-by: lns <matzeton@googlemail.com>
|
| |
|
|
|
|
|
|
|
| |
* Removed Visual Studio leftovers. Maintaining an autotools project with VS integration requires some additional overhead.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
Signed-off-by: lns <matzeton@googlemail.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
```
protocols/tls.c:650:54: runtime error: member access within null pointer of type 'const struct ndpi_tcphdr'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior protocols/tls.c:650:54 in
protocols/tls.c:650:54: runtime error: load of null pointer of type 'const u_int16_t' (aka 'const unsigned short')
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior protocols/tls.c:650:54 in
AddressSanitizer:DEADLYSIGNAL
=================================================================
==47401==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x55f7a61b661c bp 0x7f38190f91b0 sp 0x7f38190f70e0 T1)
==47401==The signal is caused by a READ memory access.
==47401==Hint: address points to the zero page.
#0 0x55f7a61b661c in processCertificateElements /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:650:41
#1 0x55f7a61ac3cc in processCertificate /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:792:7
#2 0x55f7a61d34e1 in processTLSBlock /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:846:13
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Found by oss-fuzz:
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=44280
```
==263603==WARNING: MemorySanitizer: use-of-uninitialized-value
#0 0x592478 in ndpi_is_printable_string ndpi/src/lib/ndpi_utils.c:2200:9
#1 0x5b047c in processCertificateElements ndpi/src/lib/protocols/tls.c:400:7
#2 0x5ac880 in processCertificate ndpi/src/lib/protocols/tls.c:790:7
#3 0x5c3a32 in processTLSBlock ndpi/src/lib/protocols/tls.c:844:13
#4 0x5c2c61 in ndpi_search_tls_tcp ndpi/src/lib/protocols/tls.c:973:2
#5 0x5c117d in ndpi_search_tls_wrapper ndpi/src/lib/protocols/tls.c:2367:5
#6 0x552a50 in check_ndpi_detection_func ndpi/src/lib/ndpi_main.c:4792:6
```
|
|
|
| |
See #1312
|
|
|
|
|
|
| |
Detected by oss-fuzz:
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=43823
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=43921
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=43925
|
|
|
|
| |
Added ndpi_set_tls_cert_expire_days() API call to modify the number of days for triggering the above alert that by default is set to 30 days
|
|
|
|
|
|
|
|
|
|
|
| |
The check for grease was too broad and filtered some valid values.
In particular, the value 257 was skipped because it matched the previous check.
This has been discovered while parsing tests/pcap/443-firefox.pcap
expected ja3:
771,4865-4867-4866-49195-49199-52393-52392-49196-49200-49162-49161-49171-49172-51-57-47-53-10,0-23-65281-10-11-35-16-5-51-43-13-45-28-21,29-23-24-25-256-257,0
previously generated ja3:
771,4865-4867-4866-49195-49199-52393-52392-49196-49200-49162-49161-49171-49172-51-57-47-53-10,0-23-65281-10-11-35-16-5-51-43-13-45-28-21,29-23-24-25-256,0
Signed-off-by: Patrick Havelange <patrick.havelange_ext@softathome.com>
|
| |
|
|
|
|
|
|
|
| |
Detected by oss-fuzz:
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=26880
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=26906
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=43782
https://oss-fuzz.com/testcase-detail/6334089358082048
|
|
|
|
| |
Detected by oss-fuzz:
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=43705
|