aboutsummaryrefslogtreecommitdiff
path: root/src/lib/protocols/tls.c
Commit message (Collapse)AuthorAge
...
* Update copyrightAlfredo Cardigliano2022-01-03
|
* A final(?) effort to reduce memory usage per flow (#1389)Ivan Nardi2021-12-22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Remove some unused fields and re-organize other ones. In particular: * Update the parameters of `ndpi_ssl_version2str()` function * Zattoo, Thunder: these timestamps aren't really used. * Ftp/mail: these protocols are dissected only over TCP. * Attention must be paid to TLS.Bittorrent flows to avoid invalid read/write to `flow->protos.bittorrent.hash` field. This is the last(?) commit of a long series (see 22241a1d, 227e586e, 730c2360, a8ffcd8b) aiming to reduce library memory consumption. Before, at nDPI 4.0 (more precisly, at a6b10cf7, because memory stats were wrong until that commit): ``` nDPI Memory statistics: nDPI Memory (once): 221.15 KB Flow Memory (per flow): 2.94 KB ``` Now: ``` nDPI Memory statistics: nDPI Memory (once): 231.71 KB Flow Memory (per flow): 1008 B <--------- ``` i.e. memory usage per flow has been reduced by 66%, dropping below the psychological threshold of 1 KB. To further reduce this value, we probably need to look into #1279: let's fight this battle another day.
* Improve/add several protocols (#1383)Ivan Nardi2021-12-18
| | | | | | | | | | | | | | | | | | | Improve Microsoft, GMail, Likee, Whatsapp, DisneyPlus and Tiktok detection. Add Vimeo, Fuze, Alibaba and Firebase Crashlytics detection. Try to differentiate between Messenger/Signal standard flows (i.e chat) and their VOIP (video)calls (like we already do for Whatsapp and Snapchat). Add a partial list of some ADS/Tracking stuff. Fix Cassandra, Radius and GTP false positives. Fix DNS, Syslog and SIP false negatives. Improve GTP (sub)classification: differentiate among GTP-U, GTP_C and GTP_PRIME. Fix 3 LGTM warnings.
* TLS: add support for IPV6 in Subject Alt Names field (#1385)Ivan Nardi2021-12-18
|
* TLS: fix usage of certificate cache (#1384)Ivan Nardi2021-12-18
|
* Fixed issue that prevented alt certificate names to be fully detected when ↵Luca Deri2021-12-07
| | | | ipAddress and rfc822Name were specified in certificates
* Rework how hostname/SNI info is saved (#1330)Ivan Nardi2021-11-24
| | | | | | | | | | | | | | | | | | | | | | | | | | | Looking at `struct ndpi_flow_struct` the two bigger fields are `host_server_name[240]` (mainly for HTTP hostnames and DNS domains) and `protos.tls_quic.client_requested_server_name[256]` (for TLS/QUIC SNIs). This commit aims to reduce `struct ndpi_flow_struct` size, according to two simple observations: 1) maximum one of these two fields is used for each flow. So it seems safe to merge them; 2) even if hostnames/SNIs might be very long, in practice they are rarely longer than a fews tens of bytes. So, using a (single) large buffer is a waste of memory for all kinds of flows. If we need to truncate the name, we keep the *last* characters, easing domain matching. Analyzing some real traffic, it seems safe to assume that the vast majority of hostnames/SNIs is shorter than 80 bytes. Hostnames/SNIs are always converted to lowercase. Attention was given so as to be sure that unit-tests outputs are not affected by this change. Because of a bug, TLS/QUIC SNI were always truncated to 64 bytes (the *first* 64 ones): as a consequence, there were some "Suspicious DGA domain name" and "TLS Certificate Mismatch" false positives.
* Fix writes to `flow->protos` union fields (#1354)Ivan Nardi2021-11-15
| | | | | | | | | | | | | | | | | | | | | | | | | We can write to `flow->protos` only after a proper classification. This issue has been found in Kerberos, DHCP, HTTP, STUN, IMO, FTP, SMTP, IMAP and POP code. There are two kinds of fixes: * write to `flow->protos` only if a final protocol has been detected * move protocol state out of `flow->protos` The hard part is to find, for each protocol, the right tradeoff between memory usage and code complexity. Handle Kerberos like DNS: if we find a request, we set the protocol and an extra callback to further parsing the reply. For all the other protocols, move the state out of `flow->protos`. This is an issue only for the FTP/MAIL stuff. Add DHCP Class Identification value to the output of ndpiReader and to the Jason serialization. Extend code coverage of fuzz tests. Close #1343 Close #1342
* TLS: fix two warnings (#1365)Ivan Nardi2021-11-02
| | | | | | | | Disable unit tests on CI for big-endian target. We know we have multiple issues on big-endian architectures (see #1312) and so the unit tests always fail there. Ignore this error for the time being and let the CI pass if we don't have other issues. Remove an unused automa definition
* Updated test resultsLuca Deri2021-11-02
|
* Fixed certificate mismatch for TLS flows with no client hello observedLuca Deri2021-11-02
|
* Little change in TLS debug tracesLuca Deri2021-10-28
|
* Fixed TLS certificate dissection of subjectAltNameLuca Deri2021-10-27
|
* Minor code cleanupLuca Deri2021-10-27
|
* Avoid overwriting valid protocol in `ndpi_detection_giveup` (#1360)Ivan Nardi2021-10-27
| | | | | | | | | | | | | | | | | We should avoid updating any valid protocol in `ndpi_detection_giveup`; we should try to find a proper classification only if the flow is still completely unclassified. For example in the attached pcap there is a valid TLS session, recognized as such by TLS dissector. However, the `ndpi_detection_giveup`function updates it to "HTTP/TLS" (!?) simply because the server port is 80. Note that the real issue is not the wrong classification, but the wrong access to `flow->protos` union. If we already set some fields of `flow->protos` and we change the protocol in `ndpi_detection_giveup`, we might end up freeing some invalid pointers in `ndpi_free_flow_data` (no wonder this issue has been found while fuzzing #1354) Fix GIT and TLS dissectors (issues found by CI fuzzer)
* Detect invalid characters in text and set a risk. Fixes #1347. (#1363)Toni2021-10-26
| | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* TLS: fix a heap-buffer-overflow (#1356)Ivan Nardi2021-10-22
| | | | Revert of c3d1c697 Error reproducible with the attached pcap and valgrind
* Fixed heap-buffer-overflow in TLS dissectorLuca Deri2021-10-20
|
* Fix broken fuzz_process_packet fuzzer by adding a call to ↵Toni2021-10-18
| | | | | | | | | | | | ndpi_finalize_initialization(). (#1334) * fixed several memory errors (heap-overflow, unitialized memory, etc) * ability to build fuzz_process_packet with a main() allowing to replay crash data generated with fuzz_process_packet by LLVMs libfuzzer * temporarily disable fuzzing if `tests/do.sh` executed with env FUZZY_TESTING_ENABLED=1 Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Fix compilation with clang-13 or if some debug macros are enabled (#1326)Ivan Nardi2021-10-06
|
* TLS obsolete protocol is set when TLS < 1.2 (used to be 1.1)Luca Deri2021-10-05
|
* Remove `struct ndpi_packet_struct` from `struct ndpi_flow_struct` (#1319)Ivan Nardi2021-10-05
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There are no real reasons to embed `struct ndpi_packet_struct` (i.e. "packet") in `struct ndpi_flow_struct` (i.e. "flow"). In other words, we can avoid saving dissection information of "current packet" into the "flow" state, i.e. in the flow management table. The nDPI detection module processes only one packet at the time, so it is safe to save packet dissection information in `struct ndpi_detection_module_struct`, reusing always the same "packet" instance and saving a huge amount of memory. Bottom line: we need only one copy of "packet" (for detection module), not one for each "flow". It is not clear how/why "packet" ended up in "flow" in the first place. It has been there since the beginning of the GIT history, but in the original OpenDPI code `struct ipoque_packet_struct` was embedded in `struct ipoque_detection_module_struct`, i.e. there was the same exact situation this commit wants to achieve. Most of the changes in this PR are some boilerplate to update something like "flow->packet" into something like "module->packet" throughout the code. Some attention has been paid to update `ndpi_init_packet()` since we need to reset some "packet" fields before starting to process another packet. There has been one important change, though, in ndpi_detection_giveup(). Nothing changed for the applications/users, but this function can't access "packet" anymore. The reason is that this function can be called "asynchronously" with respect to the data processing, i.e in context where there is no valid notion of "current packet"; for example ndpiReader calls it after having processed all the traffic, iterating the entire session table. Mining LRU stuff seems a bit odd (even before this patch): probably we need to rethink it, as a follow-up.
* TLS: avoid zeroing large structures (#1300)Ivan Nardi2021-09-16
| | | | | | | | | | Zeroing large structures (i.e. size > KB) is quite costly (from a CPU point of view): we can safely avoid doing that for a couple of big structures. Standard and Valgrind tests have been diverging quite a lot: it is time to re-sync them. Use the same script and enable Valgrind via an enviroment variable: NDPI_TESTS_VALGRIND=1 ./tests/do.sh
* Hangout: fix usage of STUN function (#1299)Ivan Nardi2021-09-13
|
* Fix access to some TLS fields in flow structure (#1277)Ivan Nardi2021-08-20
| | | | Fields 'tls.hello_processed` and `tls.subprotocol_detected` are used by QUIC (i.e UDP...), too.
* Compile everything with "-W -Wall -Wno-unused-parameter" flags (#1276)Ivan Nardi2021-08-20
| | | | | | | | | | | | | | | | Fix all the warnings. Getting rid of "-Wno-unused-parameter" is quite complex because some parameters usage depends on compilation variable (i.e. `--enable-debug-messages`). The "-Werror" flag has been added only in Travis builds to avoid breaking the builds to users using uncommon/untested OS/compiler/enviroment. Tested on: * x86_64; Ubuntu 20.04; gcc 7,8,9,10,11; clang 7,8,9,10,11,12 * x86_64; CentOS 7.7; gcc 4.8.5 (with "--disable-gcrypt" flag) * Raspberry 4; Debian 10.10; gcc 8.3.0
* Fixed some invalid TLS guessesLuca Deri2021-08-17
|
* Added TLS fatal alert flow riskLuca Deri2021-08-07
|
* Implemented ALPN automa for checking uncommon ALPNsLuca Deri2021-07-24
|
* Reworked flow risk implementationLuca Deri2021-07-23
|
* Implemented function to retrieve flow information. #1253 (#1254)Toni2021-07-23
| | | | | * fixed [h]euristic typo Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Added risk: TLS_EXTENSION_SUSPICIOUS (#1252)Toni2021-07-19
| | | | | | * validates client/server hello TLS extensions * inspects content for some extensions Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Fixed TLS certificate threshold (#1248)pacant2021-07-16
| | | Co-authored-by: pacant <a.pace97@outlook.com>
* Code cleanupLuca Deri2021-07-14
|
* TLS Risks - Certificate Validity Too Long (#1239)pacant2021-07-14
| | | | | | | | | | | * Added flow risk: TLS certificate too long * Added flow risk: TLS certificate too long * Date for TLS limit added * TLS certificate check fixed Co-authored-by: pacant <a.pace97@outlook.com>
* TLS: fix a memory error in JA3 code (#1227)Ivan Nardi2021-07-05
| | | | protocols/tls.c:1856:5: runtime error: index 256 out of bounds for type 'char [256]' SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior protocols/tls.c:1856:5
* Fixed off-by-one memory error for TLS-JA3. (#1222)Toni2021-06-29
| | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Restored 32 bit value in ndpi_match_string_value()Luca Deri2021-06-21
|
* Code review. (#1205)Vitaly Lavrov2021-06-15
| | | | | | | | | | | | | | | The common actions required to call the ac_automata_search() function have been moved to the ndpi_match_string_common function. This made it possible to simplify the ndpi_match_string, ndpi_match_string_protocol_id, ndpi_match_string_value, ndpi_match_custom_category, ndpi_match_string_subprotocol, ndpi_match_bigram, ndpi_match_trigram functions. Using u_int16_t type for protocol identifiers when working with the ahocorasick library (changes src/include/ndpi_api.h.in and src/include/ndpi_typedefs.h). Reworked "finalization" of all AC_AUTOMATA_t structures. Changing the order of fields in the ndpi_call_function_struct structure reduces the size of the ndpi_detection_module_struct structure by 10 kB (for x86_64).
* TLS: improve check for common ALPNs (#1191)Ivan Nardi2021-06-01
| | | Facebook is still using its own ALPN for HTTP2 as well
* Added TLS check to avoid crash with UDP-based trafficLuca2021-05-25
|
* Improved TLS browser detection heuristicsLuca Deri2021-05-19
|
* Added risk/score dump (ndpiReader -h)Luca Deri2021-05-18
| | | | Added ndpi_dump_risks_score() API score
* Added further checksLuca Deri2021-05-15
|
* Added TLS certifiacate cachingLuca Deri2021-05-15
| | | | Added Fortigate protocol
* Added browser TLS heuristicLuca Deri2021-05-13
|
* Implemented heuristic to detect Safari and Firefox TLS browsingLuca Deri2021-05-13
|
* Improved SSL certificate name wildcard handling and risk. #1182 (#1183)Toni2021-05-11
| | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* TLS: fix extraction for TLS signature algorithms (#1180)Ivan Nardi2021-05-09
| | | | | | | | | | ``` ==69562==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6100009000fb at pc 0x7f41882003a7 bp 0x7f4183cfbfc0 sp 0x7f4183cfb768 READ of size 32 at 0x6100009000fb thread T1 #0 0x7f41882003a6 in __interceptor_memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:827 #1 0x560b2d7462a1 in processClientServerHello protocols/tls.c:1647 #2 0x560b2d73be6a in processTLSBlock protocols/tls.c:712 #3 0x560b2d73e61f in ndpi_search_tls_udp protocols/tls.c:968 ```
* TLS: fix another use-of-uninitialized-value error in ClientHello parsing (#1179)Ivan Nardi2021-05-09
| | | | | | | | | | | | Error detected with valgrind. ==13127== Conditional jump or move depends on uninitialised value(s) ==13127== at 0x483EF58: strlen (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so) ==13127== by 0x1A93B6: ndpi_strdup (ndpi_main.c:159) ==13127== by 0x1C07CC: processClientServerHello (tls.c:1678) ==13127== by 0x1C0C4C: processTLSBlock (tls.c:712) ==13127== by 0x1C0C4C: ndpi_search_tls_tcp.part.0 (tls.c:849) See also 8c3674e9
Powered by cgit, Themed by Ted Yin.