aboutsummaryrefslogtreecommitdiff
path: root/src/lib/ndpi_utils.c
Commit message (Collapse)AuthorAge
* Cleaned up code moving specific includes in files their are using it. Thi ↵Luca Deri2021-10-11
| | | | prevents ndpi_config.h to be included everywhere in apps using nDPI that might leade to #define redefinitions after the latest changes
* TLS obsolete protocol is set when TLS < 1.2 (used to be 1.1)Luca Deri2021-10-05
|
* Remove `struct ndpi_packet_struct` from `struct ndpi_flow_struct` (#1319)Ivan Nardi2021-10-05
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There are no real reasons to embed `struct ndpi_packet_struct` (i.e. "packet") in `struct ndpi_flow_struct` (i.e. "flow"). In other words, we can avoid saving dissection information of "current packet" into the "flow" state, i.e. in the flow management table. The nDPI detection module processes only one packet at the time, so it is safe to save packet dissection information in `struct ndpi_detection_module_struct`, reusing always the same "packet" instance and saving a huge amount of memory. Bottom line: we need only one copy of "packet" (for detection module), not one for each "flow". It is not clear how/why "packet" ended up in "flow" in the first place. It has been there since the beginning of the GIT history, but in the original OpenDPI code `struct ipoque_packet_struct` was embedded in `struct ipoque_detection_module_struct`, i.e. there was the same exact situation this commit wants to achieve. Most of the changes in this PR are some boilerplate to update something like "flow->packet" into something like "module->packet" throughout the code. Some attention has been paid to update `ndpi_init_packet()` since we need to reset some "packet" fields before starting to process another packet. There has been one important change, though, in ndpi_detection_giveup(). Nothing changed for the applications/users, but this function can't access "packet" anymore. The reason is that this function can be called "asynchronously" with respect to the data processing, i.e in context where there is no valid notion of "current packet"; for example ndpiReader calls it after having processed all the traffic, iterating the entire session table. Mining LRU stuff seems a bit odd (even before this patch): probably we need to rethink it, as a follow-up.
* Updated descriptionLuca Deri2021-10-05
|
* Reworked bitmap codeLuca Deri2021-09-27
|
* Added API for handling compressed bitmapsLuca Deri2021-09-26
| | | | | | | | | | | | | | ndpi_bitmap* ndpi_bitmap_alloc(); void ndpi_bitmap_free(ndpi_bitmap* b); u_int64_t ndpi_bitmap_cardinality(ndpi_bitmap* b); void ndpi_bitmap_set(ndpi_bitmap* b, u_int32_t value); void ndpi_bitmap_unset(ndpi_bitmap* b, u_int32_t value); bool ndpi_bitmap_isset(ndpi_bitmap* b, u_int32_t value); void ndpi_bitmap_clear(ndpi_bitmap* b); size_t ndpi_bitmap_serialize(ndpi_bitmap* b, char **buf); ndpi_bitmap* ndpi_bitmap_deserialize(char *buf); based on https://github.com/RoaringBitmap/CRoaring
* Improve CI (#1303)Zied Aouini2021-09-22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Improve CI pipeline * Fix branch name. * Fix branch name. * Fix libgcrypt configuration. * Update build.yml * Move to Github Actions instead of Travis CI. * Fix mingw on ubuntu bionic. * Reactivate cross compile on Ubuntu Bionic. * Switch to single line steps. * Add several compilers versions * Minor fix. * Fix build all and delete cxx * Fix RCE detection. * Fix PCRE configuration. * Add condition on PCRE test pcap. * Update WebattackRCE.pcap.out * Add missing SUBST. * Delete WebattackRCE.pcap.out * Update WebAttackRCE result. * Fix typo. * Extend jobs with pcre+msan+maxminddb. * Fix code inpector warnings. * Delete .appveyor.yml
* Progetto esame Gestione di Reti - Debora Cerretini (#1290)deboracerretini2021-09-17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload Co-authored-by: Luca Deri <lucaderi@users.noreply.github.com>
* Added new risk for clear text credentialsLuca Deri2021-09-10
|
* Fix access to some TLS fields in flow structure (#1277)Ivan Nardi2021-08-20
| | | | Fields 'tls.hello_processed` and `tls.subprotocol_detected` are used by QUIC (i.e UDP...), too.
* Compile everything with "-W -Wall -Wno-unused-parameter" flags (#1276)Ivan Nardi2021-08-20
| | | | | | | | | | | | | | | | Fix all the warnings. Getting rid of "-Wno-unused-parameter" is quite complex because some parameters usage depends on compilation variable (i.e. `--enable-debug-messages`). The "-Werror" flag has been added only in Travis builds to avoid breaking the builds to users using uncommon/untested OS/compiler/enviroment. Tested on: * x86_64; Ubuntu 20.04; gcc 7,8,9,10,11; clang 7,8,9,10,11,12 * x86_64; CentOS 7.7; gcc 4.8.5 (with "--disable-gcrypt" flag) * Raspberry 4; Debian 10.10; gcc 8.3.0
* Clode cleanup (after last merge)Luca Deri2021-08-08
|
* Added entropy calculation to check for suspicious (encrypted) payload. (#1270)Toni2021-08-08
| | | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com> Co-authored-by: Luca Deri <lucaderi@users.noreply.github.com>
* Added TLS fatal alert flow riskLuca Deri2021-08-07
|
* Added ability to report whether a protocol is encryptedLuca Deri2021-08-07
| | | | | | | | | Added new API calls - u_int8_t ndpi_is_valid_protoId(u_int16_t protoId); Cleaned up code and organized in a function. - u_int8_t ndpi_is_encrypted_proto(struct ndpi_detection_module_struct *ndpi_str, ndpi_protocol proto); Check whether a protocol is encrypted or not based on master and app protocols
* Improved risk detection mask algorithmLuca Deri2021-07-26
|
* Implemented ALPN automa for checking uncommon ALPNsLuca Deri2021-07-24
|
* Risk check improvementLuca Deri2021-07-24
|
* Reworked flow risk implementationLuca Deri2021-07-23
|
* Implemented function to retrieve flow information. #1253 (#1254)Toni2021-07-23
| | | | | * fixed [h]euristic typo Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Implementation of flow risk eception (work in progress)Luca Deri2021-07-22
|
* Added risk: TLS_EXTENSION_SUSPICIOUS (#1252)Toni2021-07-19
| | | | | | * validates client/server hello TLS extensions * inspects content for some extensions Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Code cleanup. (#1246)Vitaly Lavrov2021-07-16
| | | | ndpi_utils.c: use ndpi_malloc,ndpi_calloc,ndpi_free genshin_impact.c, git.c, hpvirtgrp.c, http.c, z3950.c: removed "#include stdlib.h"
* TLS Risks - Certificate Validity Too Long (#1239)pacant2021-07-14
| | | | | | | | | | | * Added flow risk: TLS certificate too long * Added flow risk: TLS certificate too long * Date for TLS limit added * TLS certificate check fixed Co-authored-by: pacant <a.pace97@outlook.com>
* ahoсorasick. Code review. Part 2. (#1236)Vitaly Lavrov2021-07-12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Simplified the process of adding lines to AC_AUTOMATA_t. Use the ndpi_string_to_automa() function to add patterns with domain names. For other cases can use ndpi_add_string_value_to_automa(). ac_automata_feature(ac_automa, AC_FEATURE_LC) allows adding and compare data in a case insensitive manner. For mandatory pattern comparison from the end of the line, the "ac_pattern.rep.at_end=1" flag is used. This eliminated unnecessary conversions to lowercase and adding "$" for end-of-line matching in domain name patterns. ac_match_handler() has been renamed ac_domain_match_handler() and has been greatly simplified. ac_domain_match_handler() looks for the template with the highest domain level. For special cases it is possible to manually specify the domain level. Added test for checking ambiguous domain names like: - short.weixin.qq.com is QQ, not Wechat - instagram.faae1-1.fna.fbcdn.net is Instagram, not Facebook If you specify a NULL handler when creating the AC_AUTOMATA_t structure, then a pattern with the maximum length that satisfies the search conditions will be found (exact match, from the beginning of the string, from the end of the string, or a substring). Added debugging for ac_automata_search. To do this, you need to enable debugging globally using ac_automata_enable_debug(1) and enable debugging in the AC_AUTOMATA_t structure using ac_automata_name("name", AC_FEATURE_DEBUG). The search will display "name" and a list of matching patterns. Running "AHO_DEBUG=1 ndpiReader ..." will show the lines that were searched for templates and which templates were found. The ac_automata_dump() prototype has been changed. Now it outputs data to a file. If it is specified as NULL, then the output will be directed to stdout. If you need to get data as a string, then use open_memstream(). Added the ability to run individual tests via the do.sh script
* Removed ht_hash as it is not used anymore. (#1220)Toni2021-06-29
| | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Speed and memory size optimisation (#1214)Vitaly Lavrov2021-06-23
| | | | | | | Removed bigram_automata, impossible_bigram_automata, trigram_automata. The ahocorasick structure is replaced with a bitmap. The bitmap size for ndpi_en_bigram is 176 bytes. The bitmap size for ndpi_en_trigram is 2201 bytes. On the test machine, the test execution time was reduced from 27.3 seconds to 24.7 (9%).
* Warnign fixLuca Deri2021-06-03
|
* Reworked ndpi flow risk score adding client and server scoreLuca2021-06-01
|
* Fix warningsAlfredo Cardigliano2021-05-25
|
* Improved TLS browser detection heuristicsLuca Deri2021-05-19
|
* Added risk/score dump (ndpiReader -h)Luca Deri2021-05-18
| | | | Added ndpi_dump_risks_score() API score
* Updated API for ndpi_risk2score()Luca Deri2021-05-17
| | | | Added ndpi_get_upper_proto() API call
* Reworked human readeable string search in flowsLuca Deri2021-05-17
| | | | Removed fragment manager code
* Implemented heuristic to detect Safari and Firefox TLS browsingLuca Deri2021-05-13
|
* Implemented flow score in Wireshark integrationLuca Deri2021-05-10
|
* Fix some warnings (#1181)Ivan Nardi2021-05-09
| | | | | | | | | | | | | | | | | | | ``` In file included from protocols/fasttrack.c:29: ../include/ndpi_api.h:1504:3: warning: type qualifiers ignored on function return type [-Wignored-qualifiers] 1504 | const ndpi_risk_severity ndpi_risk2severity(ndpi_risk_enum risk); | ^~~~~ In file included from protocols/amazon_video.c:28: ../include/ndpi_api.h:1504:3: warning: type qualifiers ignored on function return type [-Wignored-qualifiers] 1504 | const ndpi_risk_severity ndpi_risk2severity(ndpi_risk_enum risk); | ^~~~~ ... ndpi_utils.c: In function ‘ndpi_risk2severity’: ndpi_utils.c:1834:1: warning: control reaches end of non-void function [-Wreturn-type] 1834 | } | ^ ```
* Added ndpi_risk2severity() API callLuca2021-05-02
|
* Check for common ALPNs and set a flow risk if not known. (#1175)Toni2021-04-27
| | | | | | * Increased risk bitmask to 64bit (instead of 32bit). * Removed annoying "Unknown datalink" error message for fuzzers. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Add vxlan enum to ndpi_packet_tunnelAlfredo Cardigliano2021-04-21
|
* Added NDPI_DESKTOP_OR_FILE_SHARING_SESSION flow riskLuca Deri2021-04-11
|
* Improved DGA detection with trigrams. Disadvantage: slower startup timeLuca Deri2021-03-03
| | | | | Reworked Tor dissector embedded in TLS (fixes #1141) Removed false positive on HTTP User-Agent
* DTLS: improve support (#1146)Ivan Nardi2021-03-02
| | | | | | | * DTLS: add some pcap tests * DTLS: fix parsing of Client/Server Helllo message * DTLS: add parsing of server certificates
* Added NDPI_MALICIOUS_SHA1 flow risk. (#1142)Toni2021-02-26
| | | | | | * An external file which contains known malicious SSL certificate SHA-1 hashes can be loaded via ndpi_load_malicious_sha1_file(...) Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Added protocol breed to JSON serializer. (#1137)Toni2021-02-25
| | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Modified JA3 fingerprint messageLuca Deri2021-02-24
|
* Added NDPI_MALICIOUS_JA3 flow riskLuca Deri2021-02-22
| | | | Added ndpi_load_malicious_ja3_file() API call
* Removed unused NDPI_RISKY_COUNTRYLuca Deri2021-02-21
|
* Fixes due to the fragment mananegr codeLuca Deri2021-02-18
|
* Added new risks (future use)Luca Deri2021-02-16
| | | | | | - NDPI_RISKY_ASN - NDPI_RISKY_DOMAIN - NDPI_RISKY_COUNTRY
Powered by cgit, Themed by Ted Yin.