aboutsummaryrefslogtreecommitdiff
path: root/example/reader_util.c
Commit message (Collapse)AuthorAge
...
* A final(?) effort to reduce memory usage per flow (#1389)Ivan Nardi2021-12-22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Remove some unused fields and re-organize other ones. In particular: * Update the parameters of `ndpi_ssl_version2str()` function * Zattoo, Thunder: these timestamps aren't really used. * Ftp/mail: these protocols are dissected only over TCP. * Attention must be paid to TLS.Bittorrent flows to avoid invalid read/write to `flow->protos.bittorrent.hash` field. This is the last(?) commit of a long series (see 22241a1d, 227e586e, 730c2360, a8ffcd8b) aiming to reduce library memory consumption. Before, at nDPI 4.0 (more precisly, at a6b10cf7, because memory stats were wrong until that commit): ``` nDPI Memory statistics: nDPI Memory (once): 221.15 KB Flow Memory (per flow): 2.94 KB ``` Now: ``` nDPI Memory statistics: nDPI Memory (once): 231.71 KB Flow Memory (per flow): 1008 B <--------- ``` i.e. memory usage per flow has been reduced by 66%, dropping below the psychological threshold of 1 KB. To further reduce this value, we probably need to look into #1279: let's fight this battle another day.
* Rework how hostname/SNI info is saved (#1330)Ivan Nardi2021-11-24
| | | | | | | | | | | | | | | | | | | | | | | | | | | Looking at `struct ndpi_flow_struct` the two bigger fields are `host_server_name[240]` (mainly for HTTP hostnames and DNS domains) and `protos.tls_quic.client_requested_server_name[256]` (for TLS/QUIC SNIs). This commit aims to reduce `struct ndpi_flow_struct` size, according to two simple observations: 1) maximum one of these two fields is used for each flow. So it seems safe to merge them; 2) even if hostnames/SNIs might be very long, in practice they are rarely longer than a fews tens of bytes. So, using a (single) large buffer is a waste of memory for all kinds of flows. If we need to truncate the name, we keep the *last* characters, easing domain matching. Analyzing some real traffic, it seems safe to assume that the vast majority of hostnames/SNIs is shorter than 80 bytes. Hostnames/SNIs are always converted to lowercase. Attention was given so as to be sure that unit-tests outputs are not affected by this change. Because of a bug, TLS/QUIC SNI were always truncated to 64 bytes (the *first* 64 ones): as a consequence, there were some "Suspicious DGA domain name" and "TLS Certificate Mismatch" false positives.
* Fix writes to `flow->protos` union fields (#1354)Ivan Nardi2021-11-15
| | | | | | | | | | | | | | | | | | | | | | | | | We can write to `flow->protos` only after a proper classification. This issue has been found in Kerberos, DHCP, HTTP, STUN, IMO, FTP, SMTP, IMAP and POP code. There are two kinds of fixes: * write to `flow->protos` only if a final protocol has been detected * move protocol state out of `flow->protos` The hard part is to find, for each protocol, the right tradeoff between memory usage and code complexity. Handle Kerberos like DNS: if we find a request, we set the protocol and an extra callback to further parsing the reply. For all the other protocols, move the state out of `flow->protos`. This is an issue only for the FTP/MAIL stuff. Add DHCP Class Identification value to the output of ndpiReader and to the Jason serialization. Extend code coverage of fuzz tests. Close #1343 Close #1342
* Reduce memory used by `ndpiReader` (#1371)Ivan Nardi2021-11-11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | `ndpiReader` is only an example, aiming to show nDPI capabilities and integration, without any claim about performances. Nonetheless its memory usage per flow is *huge*, limiting the kinds of traces that we can test on a "normal" hardware (example: scan attacks). The key reason of that behaviour is that we preallocate all the memory needed for *all* the available features. Try to reduce memory usage simply allocating some structures only when they are really needed. Most significant example: JOY algorithms. This way we should use a lot less memory in the two most common user-cases: * `ndpiReader` invoked without any particular flag (i.e `ndpiReader -i $FILENAME_OR_IFACE`) * internal unit tests Before (on x86_64): ``` struct ndpi_flow_info { [...] /* size: 7320, cachelines: 115, members: 72 */ ``` After: ``` struct ndpi_flow_info { [...] /* size: 2128, cachelines: 34, members: 75 */ ```
* Fix broken fuzz_process_packet fuzzer by adding a call to ↵Toni2021-10-18
| | | | | | | | | | | | ndpi_finalize_initialization(). (#1334) * fixed several memory errors (heap-overflow, unitialized memory, etc) * ability to build fuzz_process_packet with a main() allowing to replay crash data generated with fuzz_process_packet by LLVMs libfuzzer * temporarily disable fuzzing if `tests/do.sh` executed with env FUZZY_TESTING_ENABLED=1 Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Fix parsing of ipv6 packets with extension headersNardi Ivan2021-09-19
| | | | | | | | Decoding of ipv6 traffic with extension headers was completely broken, since the beginning of the L4 header was always set to a wrong value. Handle the ipv6 fragments in the same way as the ipv4 ones: keep the first one and drop the others.
* Compile everything with "-W -Wall -Wno-unused-parameter" flags (#1276)Ivan Nardi2021-08-20
| | | | | | | | | | | | | | | | Fix all the warnings. Getting rid of "-Wno-unused-parameter" is quite complex because some parameters usage depends on compilation variable (i.e. `--enable-debug-messages`). The "-Werror" flag has been added only in Travis builds to avoid breaking the builds to users using uncommon/untested OS/compiler/enviroment. Tested on: * x86_64; Ubuntu 20.04; gcc 7,8,9,10,11; clang 7,8,9,10,11,12 * x86_64; CentOS 7.7; gcc 4.8.5 (with "--disable-gcrypt" flag) * Raspberry 4; Debian 10.10; gcc 8.3.0
* Fixed Mingw64 build, SonerCloud-CI and more. (#1273)Toni2021-08-18
| | | | | | | | | | | | | * Added ARM build and unit test run for SonarCloud-CI. Signed-off-by: Toni Uhlig <matzeton@googlemail.com> * Fixed Mingw64 build. * adapted to SonarCloud-CI workflow * removed broken and incomplete Windows example (tested on VS2017/VS2019) * removed unnecessary include (e.g. pthread.h for the library which does not make use of it) Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Defined more standard ethernet protocol typesLuca Deri2021-08-09
|
* Added protocol definesLuca Deri2021-08-09
|
* Implemented ALPN automa for checking uncommon ALPNsLuca Deri2021-07-24
|
* Reworked flow risk implementationLuca Deri2021-07-23
|
* Implemented function to retrieve flow information. #1253 (#1254)Toni2021-07-23
| | | | | * fixed [h]euristic typo Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Implementation of flow risk eception (work in progress)Luca Deri2021-07-22
|
* ndpiReader: add statistics about nDPI performance (#1240)Ivan Nardi2021-07-13
| | | | | | | The goal is to have a (roughly) idea about how many packets nDPI needs to properly classify a flow. Log this information (and guessed flows number too) during unit tests, to keep track of improvements/regressions across commits.
* Reworked human readeable string search in flowsLuca Deri2021-05-17
| | | | Removed fragment manager code
* Check datalink during fuzzing to prevent console / logfile spam. See #1175 ↵Toni2021-05-09
| | | | | for more information. (#1177) Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Initial work towards detection via TLS of browser typesLuca2021-05-06
|
* Check for common ALPNs and set a flow risk if not known. (#1175)Toni2021-04-27
| | | | | | * Increased risk bitmask to 64bit (instead of 32bit). * Removed annoying "Unknown datalink" error message for fuzzers. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Added flow risk to wireshark dissectionLuca Deri2021-04-26
|
* Fix detunneling of GTP-U traffic (#1168)Ivan Nardi2021-04-18
| | | | | Fuzzing #1161 exposed some (completely unrelated) issues on GTP-U detunneling code. (see https://github.com/ntop/nDPI/actions/runs/719882047)
* Fixed invalid guess statsLuca Deri2021-03-30
|
* ndpiReader: print an error msg if we found an unsupported datalink type (#1157)Ivan Nardi2021-03-23
|
* Reworked extendal dependency across testing toolsLuca Deri2021-03-14
|
* Added experiemntal JA3+ implementation that can be used with -z i ndpiReaderLuca Deri2021-03-09
|
* Add support for Snapchat voip calls (#1147)Ivan Nardi2021-03-06
| | | | | | | | | * Add support for Snapchat voip calls Snapchat multiplexes some of its audio/video real time traffic with QUIC sessions. The peculiarity of these sessions is that they are Q046 and don't have any SNI. * Fix tests with libgcrypt disabled
* Fixed memory leaks caused by conditional free'ing for some TLS connec… (#1132)Toni2021-02-10
| | | | | | | | | | | | | | | * Fixed memory leaks caused by conditional free'ing for some TLS connections. * Members of tls_quic struct should also free'd if the detected master protocol is IMAPS / POPS / SMTPS / etc. Signed-off-by: Toni Uhlig <matzeton@googlemail.com> * Prevent reader_util.c from exit()'ing if maximum flow count reached. This confuses the fuzzer. * Improved fuzz/Makefile.am to use LDADD for ../example/libndpiReader.a instead of LDFLAGS. That way, fuzz_ndpi_reader re-links to ../example/libndpiReader.a if something changed there. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* RSI enhancementsLuca Deri2021-02-05
|
* Improved debug messageLuca Deri2021-02-03
|
* Cleaned up tls/quic datatypesLuca Deri2021-01-21
|
* Reworked TLS fingerprint calcolationLuca Deri2021-01-21
| | | | Modified TLS memory free
* (C) UpdateLuca Deri2021-01-07
|
* Split HTTP request from response Content-Type. Request Content-Type should ↵Luca Deri2021-01-06
| | | | be present with POSTs and not with other methods such as GET
* Restored QUIC statsLuca Deri2020-12-30
|
* Fixed output when tLS (nad not QUIC) is usedLuca Deri2020-12-28
|
* Fixes bug introduced by https://github.com/ntop/nDPI/pull/1085Luca Deri2020-12-12
|
* Fix minimum packet length condition (#1087)Zied Aouini2020-12-12
|
* Added --ignore-vlanid / -I to exclude VLAN ids for flow hash calculation. ↵Toni2020-12-11
| | | | | #1073 (#1085) Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Support raw IPv4 / IPv6 pcap packet processing. (#1053)Toni2020-11-09
| | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Cosmetic changesLuca Deri2020-11-03
|
* Moved global in reader_util.cLuca Deri2020-10-27
|
* Added -D flag for detecting DoH in the wildLuca Deri2020-10-26
| | | | Removed heuristic from CiscoVPN as it leads to false positives
* Various improvemement when using ndpi_pref_enable_tls_block_dissection:Luca Deri2020-10-24
| | | | | | application data TLS blocks are now ignored when exchanged before - the end of certificate negotiation (up to TLS 1.2) - change cipher
* Fix parsing of DLT_PPP datalink type (#1042)Ivan Nardi2020-10-21
|
* CAPWAP tunnel decoding fix (#1038)Zied Aouini2020-10-21
| | | | | * Fix CAPWAP processing. * Update result.
* Fix CAPWAP handling.aouinizied2020-10-13
|
* Added ndpi_quick_16_byte_hashLuca2020-10-05
| | | | Warning fix
* Warning fixLuca Deri2020-10-03
|
* Reworked MDNS dissector that is not based on the DNS dissectorLuca Deri2020-09-17
|
* http: create a common function to parse User Agent fieldNardi Ivan2020-09-08
| | | | Prepare the code to handle UA information from flows other than HTTP
Powered by cgit, Themed by Ted Yin.