aboutsummaryrefslogtreecommitdiff
path: root/src/lib/ndpi_utils.c
Commit message (Collapse)AuthorAge
* Serializing empty `risk blocks' pollutes the resulting string.fix/json-empty-risk-serializationToni Uhlig2022-01-26
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Update copyrightAlfredo Cardigliano2022-01-03
|
* Added support for Log4J/Log4Shell detection in nDPI via a new flow risk ↵Luca Deri2021-12-23
| | | | named NDPI_POSSIBLE_EXPLOIT
* A final(?) effort to reduce memory usage per flow (#1389)Ivan Nardi2021-12-22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Remove some unused fields and re-organize other ones. In particular: * Update the parameters of `ndpi_ssl_version2str()` function * Zattoo, Thunder: these timestamps aren't really used. * Ftp/mail: these protocols are dissected only over TCP. * Attention must be paid to TLS.Bittorrent flows to avoid invalid read/write to `flow->protos.bittorrent.hash` field. This is the last(?) commit of a long series (see 22241a1d, 227e586e, 730c2360, a8ffcd8b) aiming to reduce library memory consumption. Before, at nDPI 4.0 (more precisly, at a6b10cf7, because memory stats were wrong until that commit): ``` nDPI Memory statistics: nDPI Memory (once): 221.15 KB Flow Memory (per flow): 2.94 KB ``` Now: ``` nDPI Memory statistics: nDPI Memory (once): 231.71 KB Flow Memory (per flow): 1008 B <--------- ``` i.e. memory usage per flow has been reduced by 66%, dropping below the psychological threshold of 1 KB. To further reduce this value, we probably need to look into #1279: let's fight this battle another day.
* Make serialize risk and proto not dependant on any flow. (#1377)Toni2021-12-04
| | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Rework how hostname/SNI info is saved (#1330)Ivan Nardi2021-11-24
| | | | | | | | | | | | | | | | | | | | | | | | | | | Looking at `struct ndpi_flow_struct` the two bigger fields are `host_server_name[240]` (mainly for HTTP hostnames and DNS domains) and `protos.tls_quic.client_requested_server_name[256]` (for TLS/QUIC SNIs). This commit aims to reduce `struct ndpi_flow_struct` size, according to two simple observations: 1) maximum one of these two fields is used for each flow. So it seems safe to merge them; 2) even if hostnames/SNIs might be very long, in practice they are rarely longer than a fews tens of bytes. So, using a (single) large buffer is a waste of memory for all kinds of flows. If we need to truncate the name, we keep the *last* characters, easing domain matching. Analyzing some real traffic, it seems safe to assume that the vast majority of hostnames/SNIs is shorter than 80 bytes. Hostnames/SNIs are always converted to lowercase. Attention was given so as to be sure that unit-tests outputs are not affected by this change. Because of a bug, TLS/QUIC SNI were always truncated to 64 bytes (the *first* 64 ones): as a consequence, there were some "Suspicious DGA domain name" and "TLS Certificate Mismatch" false positives.
* Fix writes to `flow->protos` union fields (#1354)Ivan Nardi2021-11-15
| | | | | | | | | | | | | | | | | | | | | | | | | We can write to `flow->protos` only after a proper classification. This issue has been found in Kerberos, DHCP, HTTP, STUN, IMO, FTP, SMTP, IMAP and POP code. There are two kinds of fixes: * write to `flow->protos` only if a final protocol has been detected * move protocol state out of `flow->protos` The hard part is to find, for each protocol, the right tradeoff between memory usage and code complexity. Handle Kerberos like DNS: if we find a request, we set the protocol and an extra callback to further parsing the reply. For all the other protocols, move the state out of `flow->protos`. This is an issue only for the FTP/MAIL stuff. Add DHCP Class Identification value to the output of ndpiReader and to the Jason serialization. Extend code coverage of fuzz tests. Close #1343 Close #1342
* Serialize additional information stored in the flow struct. (#1362)Toni2021-10-27
| | | | | | | * Changed function signature of ndpi_flow2json (removed unused vlan_id; API break) * Serialize NTP information. * Improved QUIC serialization. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Detect invalid characters in text and set a risk. Fixes #1347. (#1363)Toni2021-10-26
| | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Fixed cleartext protocol assignment (#1357)Ivan Nardi2021-10-25
|
* Fix some invalid memory reads (#1350)Ivan Nardi2021-10-19
| | | | | | | | `ndpi_detection_giveup()` (and any functions called by it) can't access `ndpi_detection_module_struct->packet` anymore since 730c236. Sync unit tests results Close #1348
* Fixed cleartext protocol assignmentLuca Deri2021-10-16
|
* Cleaned up code moving specific includes in files their are using it. Thi ↵Luca Deri2021-10-11
| | | | prevents ndpi_config.h to be included everywhere in apps using nDPI that might leade to #define redefinitions after the latest changes
* TLS obsolete protocol is set when TLS < 1.2 (used to be 1.1)Luca Deri2021-10-05
|
* Remove `struct ndpi_packet_struct` from `struct ndpi_flow_struct` (#1319)Ivan Nardi2021-10-05
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There are no real reasons to embed `struct ndpi_packet_struct` (i.e. "packet") in `struct ndpi_flow_struct` (i.e. "flow"). In other words, we can avoid saving dissection information of "current packet" into the "flow" state, i.e. in the flow management table. The nDPI detection module processes only one packet at the time, so it is safe to save packet dissection information in `struct ndpi_detection_module_struct`, reusing always the same "packet" instance and saving a huge amount of memory. Bottom line: we need only one copy of "packet" (for detection module), not one for each "flow". It is not clear how/why "packet" ended up in "flow" in the first place. It has been there since the beginning of the GIT history, but in the original OpenDPI code `struct ipoque_packet_struct` was embedded in `struct ipoque_detection_module_struct`, i.e. there was the same exact situation this commit wants to achieve. Most of the changes in this PR are some boilerplate to update something like "flow->packet" into something like "module->packet" throughout the code. Some attention has been paid to update `ndpi_init_packet()` since we need to reset some "packet" fields before starting to process another packet. There has been one important change, though, in ndpi_detection_giveup(). Nothing changed for the applications/users, but this function can't access "packet" anymore. The reason is that this function can be called "asynchronously" with respect to the data processing, i.e in context where there is no valid notion of "current packet"; for example ndpiReader calls it after having processed all the traffic, iterating the entire session table. Mining LRU stuff seems a bit odd (even before this patch): probably we need to rethink it, as a follow-up.
* Updated descriptionLuca Deri2021-10-05
|
* Reworked bitmap codeLuca Deri2021-09-27
|
* Added API for handling compressed bitmapsLuca Deri2021-09-26
| | | | | | | | | | | | | | ndpi_bitmap* ndpi_bitmap_alloc(); void ndpi_bitmap_free(ndpi_bitmap* b); u_int64_t ndpi_bitmap_cardinality(ndpi_bitmap* b); void ndpi_bitmap_set(ndpi_bitmap* b, u_int32_t value); void ndpi_bitmap_unset(ndpi_bitmap* b, u_int32_t value); bool ndpi_bitmap_isset(ndpi_bitmap* b, u_int32_t value); void ndpi_bitmap_clear(ndpi_bitmap* b); size_t ndpi_bitmap_serialize(ndpi_bitmap* b, char **buf); ndpi_bitmap* ndpi_bitmap_deserialize(char *buf); based on https://github.com/RoaringBitmap/CRoaring
* Improve CI (#1303)Zied Aouini2021-09-22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Improve CI pipeline * Fix branch name. * Fix branch name. * Fix libgcrypt configuration. * Update build.yml * Move to Github Actions instead of Travis CI. * Fix mingw on ubuntu bionic. * Reactivate cross compile on Ubuntu Bionic. * Switch to single line steps. * Add several compilers versions * Minor fix. * Fix build all and delete cxx * Fix RCE detection. * Fix PCRE configuration. * Add condition on PCRE test pcap. * Update WebattackRCE.pcap.out * Add missing SUBST. * Delete WebattackRCE.pcap.out * Update WebAttackRCE result. * Fix typo. * Extend jobs with pcre+msan+maxminddb. * Fix code inpector warnings. * Delete .appveyor.yml
* Progetto esame Gestione di Reti - Debora Cerretini (#1290)deboracerretini2021-09-17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload Co-authored-by: Luca Deri <lucaderi@users.noreply.github.com>
* Added new risk for clear text credentialsLuca Deri2021-09-10
|
* Fix access to some TLS fields in flow structure (#1277)Ivan Nardi2021-08-20
| | | | Fields 'tls.hello_processed` and `tls.subprotocol_detected` are used by QUIC (i.e UDP...), too.
* Compile everything with "-W -Wall -Wno-unused-parameter" flags (#1276)Ivan Nardi2021-08-20
| | | | | | | | | | | | | | | | Fix all the warnings. Getting rid of "-Wno-unused-parameter" is quite complex because some parameters usage depends on compilation variable (i.e. `--enable-debug-messages`). The "-Werror" flag has been added only in Travis builds to avoid breaking the builds to users using uncommon/untested OS/compiler/enviroment. Tested on: * x86_64; Ubuntu 20.04; gcc 7,8,9,10,11; clang 7,8,9,10,11,12 * x86_64; CentOS 7.7; gcc 4.8.5 (with "--disable-gcrypt" flag) * Raspberry 4; Debian 10.10; gcc 8.3.0
* Clode cleanup (after last merge)Luca Deri2021-08-08
|
* Added entropy calculation to check for suspicious (encrypted) payload. (#1270)Toni2021-08-08
| | | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com> Co-authored-by: Luca Deri <lucaderi@users.noreply.github.com>
* Added TLS fatal alert flow riskLuca Deri2021-08-07
|
* Added ability to report whether a protocol is encryptedLuca Deri2021-08-07
| | | | | | | | | Added new API calls - u_int8_t ndpi_is_valid_protoId(u_int16_t protoId); Cleaned up code and organized in a function. - u_int8_t ndpi_is_encrypted_proto(struct ndpi_detection_module_struct *ndpi_str, ndpi_protocol proto); Check whether a protocol is encrypted or not based on master and app protocols
* Improved risk detection mask algorithmLuca Deri2021-07-26
|
* Implemented ALPN automa for checking uncommon ALPNsLuca Deri2021-07-24
|
* Risk check improvementLuca Deri2021-07-24
|
* Reworked flow risk implementationLuca Deri2021-07-23
|
* Implemented function to retrieve flow information. #1253 (#1254)Toni2021-07-23
| | | | | * fixed [h]euristic typo Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Implementation of flow risk eception (work in progress)Luca Deri2021-07-22
|
* Added risk: TLS_EXTENSION_SUSPICIOUS (#1252)Toni2021-07-19
| | | | | | * validates client/server hello TLS extensions * inspects content for some extensions Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Code cleanup. (#1246)Vitaly Lavrov2021-07-16
| | | | ndpi_utils.c: use ndpi_malloc,ndpi_calloc,ndpi_free genshin_impact.c, git.c, hpvirtgrp.c, http.c, z3950.c: removed "#include stdlib.h"
* TLS Risks - Certificate Validity Too Long (#1239)pacant2021-07-14
| | | | | | | | | | | * Added flow risk: TLS certificate too long * Added flow risk: TLS certificate too long * Date for TLS limit added * TLS certificate check fixed Co-authored-by: pacant <a.pace97@outlook.com>
* ahoсorasick. Code review. Part 2. (#1236)Vitaly Lavrov2021-07-12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Simplified the process of adding lines to AC_AUTOMATA_t. Use the ndpi_string_to_automa() function to add patterns with domain names. For other cases can use ndpi_add_string_value_to_automa(). ac_automata_feature(ac_automa, AC_FEATURE_LC) allows adding and compare data in a case insensitive manner. For mandatory pattern comparison from the end of the line, the "ac_pattern.rep.at_end=1" flag is used. This eliminated unnecessary conversions to lowercase and adding "$" for end-of-line matching in domain name patterns. ac_match_handler() has been renamed ac_domain_match_handler() and has been greatly simplified. ac_domain_match_handler() looks for the template with the highest domain level. For special cases it is possible to manually specify the domain level. Added test for checking ambiguous domain names like: - short.weixin.qq.com is QQ, not Wechat - instagram.faae1-1.fna.fbcdn.net is Instagram, not Facebook If you specify a NULL handler when creating the AC_AUTOMATA_t structure, then a pattern with the maximum length that satisfies the search conditions will be found (exact match, from the beginning of the string, from the end of the string, or a substring). Added debugging for ac_automata_search. To do this, you need to enable debugging globally using ac_automata_enable_debug(1) and enable debugging in the AC_AUTOMATA_t structure using ac_automata_name("name", AC_FEATURE_DEBUG). The search will display "name" and a list of matching patterns. Running "AHO_DEBUG=1 ndpiReader ..." will show the lines that were searched for templates and which templates were found. The ac_automata_dump() prototype has been changed. Now it outputs data to a file. If it is specified as NULL, then the output will be directed to stdout. If you need to get data as a string, then use open_memstream(). Added the ability to run individual tests via the do.sh script
* Removed ht_hash as it is not used anymore. (#1220)Toni2021-06-29
| | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Speed and memory size optimisation (#1214)Vitaly Lavrov2021-06-23
| | | | | | | Removed bigram_automata, impossible_bigram_automata, trigram_automata. The ahocorasick structure is replaced with a bitmap. The bitmap size for ndpi_en_bigram is 176 bytes. The bitmap size for ndpi_en_trigram is 2201 bytes. On the test machine, the test execution time was reduced from 27.3 seconds to 24.7 (9%).
* Warnign fixLuca Deri2021-06-03
|
* Reworked ndpi flow risk score adding client and server scoreLuca2021-06-01
|
* Fix warningsAlfredo Cardigliano2021-05-25
|
* Improved TLS browser detection heuristicsLuca Deri2021-05-19
|
* Added risk/score dump (ndpiReader -h)Luca Deri2021-05-18
| | | | Added ndpi_dump_risks_score() API score
* Updated API for ndpi_risk2score()Luca Deri2021-05-17
| | | | Added ndpi_get_upper_proto() API call
* Reworked human readeable string search in flowsLuca Deri2021-05-17
| | | | Removed fragment manager code
* Implemented heuristic to detect Safari and Firefox TLS browsingLuca Deri2021-05-13
|
* Implemented flow score in Wireshark integrationLuca Deri2021-05-10
|
* Fix some warnings (#1181)Ivan Nardi2021-05-09
| | | | | | | | | | | | | | | | | | | ``` In file included from protocols/fasttrack.c:29: ../include/ndpi_api.h:1504:3: warning: type qualifiers ignored on function return type [-Wignored-qualifiers] 1504 | const ndpi_risk_severity ndpi_risk2severity(ndpi_risk_enum risk); | ^~~~~ In file included from protocols/amazon_video.c:28: ../include/ndpi_api.h:1504:3: warning: type qualifiers ignored on function return type [-Wignored-qualifiers] 1504 | const ndpi_risk_severity ndpi_risk2severity(ndpi_risk_enum risk); | ^~~~~ ... ndpi_utils.c: In function ‘ndpi_risk2severity’: ndpi_utils.c:1834:1: warning: control reaches end of non-void function [-Wreturn-type] 1834 | } | ^ ```
* Added ndpi_risk2severity() API callLuca2021-05-02
|
Powered by cgit, Themed by Ted Yin.