aboutsummaryrefslogtreecommitdiff
path: root/src/include/ndpi_api.h.in
Commit message (Collapse)AuthorAge
* Moved `NDPI_API_VERSION` to `src/include/ndpi_define.h.in` where it belongs.Toni Uhlig2022-09-21
| | | | | | | That way, we can make `src/include/ndpi_api.h` independent of autoconf. MSBuild will like that. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Avoid useless host automa lookup (#1724)Ivan Nardi2022-09-05
| | | | | | | | | | | | | | | | | | | The host automa is used for two tasks: * protocol sub-classification (obviously); * DGA evaluation: the idea is that if a domain is present in this automa, it can't be a DGA, regardless of its format/name. In most dissectors both checks are executed, i.e. the code is something like: ``` ndpi_match_host_subprotocol(..., flow->host_server_name, ...); ndpi_check_dga_name(..., flow->host_server_name,...); ``` In that common case, we can perform only one automa lookup: if we check the sub-classification before the DGA, we can avoid the second lookup in the DGA function itself.
* Add support to opportunistic TLSNardi Ivan2022-09-04
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A lot of protocols provide the feature to upgrade their plain text connections to an encrypted one, via some kind of "STARTTLS" command. Add generic code to support this extension, and allow dissection of the entire TLS handshake. As examples, SMTP, POP, IMAP and FTP dissectors have been updated. Since this feature requires to process more packets per flow, add the possibility to disable it. Fix some log messages. Slight improvement on TCP sequence number tracking. As a side effect, this commit fix also a memory leak found by oss-fuzzer ``` ==108966==ERROR: LeakSanitizer: detected memory leaks Direct leak of 22 byte(s) in 1 object(s) allocated from: #0 0x55f8b367a0be in malloc (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader_with_main+0x5480be) (BuildId: 94debacb4a6784c30420ab748c8bf3cc59621063) #1 0x55f8b36e1345 in ndpi_malloc_wrapper /home/ivan/svnrepos/nDPI/example/reader_util.c:321:10 #2 0x55f8b379c7d2 in ndpi_malloc /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:212:25 #3 0x55f8b379cb18 in ndpi_strdup /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:279:13 #4 0x55f8b386ce46 in processClientServerHello /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:2153:34 #5 0x55f8b385ebf7 in processTLSBlock /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:867:5 #6 0x55f8b39e708c in ndpi_extra_search_mail_smtp_tcp /home/ivan/svnrepos/nDPI/src/lib/protocols/mail_smtp.c:422:9 #7 0x55f8b37e636c in ndpi_process_extra_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5884:9 #8 0x55f8b37edc05 in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:6276:5 #9 0x55f8b3701ffc in packet_processing /home/ivan/svnrepos/nDPI/example/reader_util.c:1619:31 #10 0x55f8b36faf14 in ndpi_workflow_process_packet /home/ivan/svnrepos/nDPI/example/reader_util.c:2189:10 #11 0x55f8b36b6a50 in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:107:7 ``` See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=50765
* Support serialization of double-precision floating-point numbers. Fixes #1702.lns2022-08-24
| | | | | Signed-off-by: lns <matzeton@googlemail.com> Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Modified definition of hll_addLuca Deri2022-08-17
|
* Improved nDPI JSON serialization. (#1689)Toni2022-08-02
| | | | | | | | * fixed autoconf CFLAGS/LDFLAGS MSAN issue which could lead to build errors * introduced portable version of gmtime_r aka ndpi_gmtime_r * do as most as possible of the serialization work in ndpi_utils.c * use flow2json in ndpiReader Signed-off-by: lns <matzeton@googlemail.com>
* Patricia tree, Ahocarasick automa, LRU cache: add statistics (#1683)Ivan Nardi2022-07-29
| | | | | | | | | | Add (basic) internal stats to the main data structures used by the library; they might be usefull to check how effective these structures are. Add an option to `ndpiReader` to dump them; enabled by default in the unit tests. This new option enables/disables dumping of "num dissectors calls" values, too (see b4cb14ec).
* Revert "Patricia tree, Ahocarasick automa, LRU cache: add statistics ↵Ivan Nardi2022-07-29
| | | | | (#1677)" (#1682) This reverts commit bb83899985c25097341b947c2c535f56254a075c.
* Patricia tree, Ahocarasick automa, LRU cache: add statistics (#1677)Ivan Nardi2022-07-29
| | | | | | | | Add (basic) internal stats to the main data structures used by the library; they might be usefull to check how effective these structures are. Add an option to `ndpiReader` to dump them; disabled by default to avoid too much fuss with the unit tests.
* Add support for flow client/server information (#1671)Ivan Nardi2022-07-24
| | | | | | | | | | | | | | | | | | In a lot of places in ndPI we use *packet* source/dest info (address/port/direction) when we are interested in *flow* client/server info, instead. Add basic logic to autodetect this kind of information. nDPI doesn't perform any "flow management" itself but this task is delegated to the external application. It is then likely that the application might provide more reliable hints about flow client/server direction and about the TCP handshake presence: in that case, these information might be (optionally) passed to the library, disabling the internal "autodetect" logic. These new fields have been used in some LRU caches and in the "guessing" algorithm. It is quite likely that some other code needs to be updated.
* Updated ndpi_check_flow_risk_exceptions() signatureLuca Deri2022-07-04
|
* Cleaned-up issuer DN check code addingLuca Deri2022-07-04
| | | | | | | | | u_int8_t ndpi_check_issuerdn_risk_exception(struct ndpi_detection_module_struct *ndpi_str, char *issuerDN); Added new API function for checking nDPI-configured exceptions u_int8_t ndpi_check_flow_risk_exception(struct ndpi_detection_module_struct *ndpi_str, u_int num_params, ndpi_risk_params **params);
* bins: add support for 64bit bins (#1626)Ivan Nardi2022-07-03
|
* Replaced nDPI's internal hashmap with uthash. (#1602)Toni2022-06-17
| | | Signed-off-by: lns <matzeton@googlemail.com>
* Added ability to return risk info in JSON format in ndpi_get_flow_risk_info()Luca2022-05-30
|
* Updated tests resultsLuca Deri2022-05-30
| | | | Code cleanup
* Added ndpi_get_flow_error_code() API callLuca Deri2022-05-02
| | | | Fixed typo
* Added ability to store custom category file in patricia treeLuca Deri2022-04-26
|
* Add ndpi_json_string_escape to the APIAlfredo Cardigliano2022-04-26
|
* Added ndpi_find_outliers() API call using Z-ScoreLuca Deri2022-04-04
|
* Provide some API functions for convenience. (#1456)Toni2022-02-25
| | | | | * Extended JSON serializsation: risk, risk score, confidence Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Invalid prototupe fixLuca Deri2022-02-10
|
* Added ndpi_serialize_string_string_len() APi callLuca Deri2022-02-08
| | | | Fixed CSV string serialization
* Add commentAlfredo Cardigliano2022-02-03
|
* Remove `struct ndpi_id_struct` (#1427)Ivan Nardi2022-01-30
| | | | | | | | | | | | | | | | | | | | | | | | | | | Remove the last uses of `struct ndpi_id_struct`. That code is not really used and it has not been updated for a very long time: see #1279 for details. Correlation among flows is achieved via LRU caches. This change allows to further reduce memory consumption (see also 91bb77a8). At nDPI 4.0 (more precisly, at a6b10cf, because memory stats were wrong until that commit): ``` nDPI Memory statistics: nDPI Memory (once): 221.15 KB Flow Memory (per flow): 2.94 KB ``` Now: ``` nDPI Memory statistics: nDPI Memory (once): 235.27 KB Flow Memory (per flow): 688 B <-------- ``` i.e. memory usage per flow has been reduced by 77%. Close #1279
* Added NDPI_TLS_CERTIFICATE_ABOUT_TO_EXPIRE flow riskLuca Deri2022-01-26
| | | | Added ndpi_set_tls_cert_expire_days() API call to modify the number of days for triggering the above alert that by default is set to 30 days
* H323: fix a use-after-poison error (#1412)Ivan Nardi2022-01-17
| | | | | | | Detected by oss-fuzz See: https://oss-fuzz.com/testcase-detail/6730505580576768 Fix a function prototype Update a unit test results
* Added performance tests toolsLuca Deri2022-01-16
|
* Added the ability to specify trusted issueDN often used in companies to ↵Luca Deri2022-01-13
| | | | | | | | | | | self-signed certificates This allows to avoid triggering alerts for trusted albeit private certificate issuers. Extended the example/protos.txt with the new syntax for specifying trusted issueDN. Example: trusted_issuer_dn:"CN=813845657003339838, O=Code42, OU=TEST, ST=MN, C=US"
* Add a "confidence" field about the reliability of the classification. (#1395)Ivan Nardi2022-01-11
| | | | | | | | | | | | | As a general rule, the higher the confidence value, the higher the "reliability/precision" of the classification. In other words, this new field provides an hint about "how" the flow classification has been obtained. For example, the application may want to ignore classification "by-port" (they are not real DPI classifications, after all) or give a second glance at flows classified via LRU caches (because of false positives). Setting only one value for the confidence field is a bit tricky: more work is probably needed in the next future to tweak/fix/improve the logic.
* A final(?) effort to reduce memory usage per flow (#1389)Ivan Nardi2021-12-22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Remove some unused fields and re-organize other ones. In particular: * Update the parameters of `ndpi_ssl_version2str()` function * Zattoo, Thunder: these timestamps aren't really used. * Ftp/mail: these protocols are dissected only over TCP. * Attention must be paid to TLS.Bittorrent flows to avoid invalid read/write to `flow->protos.bittorrent.hash` field. This is the last(?) commit of a long series (see 22241a1d, 227e586e, 730c2360, a8ffcd8b) aiming to reduce library memory consumption. Before, at nDPI 4.0 (more precisly, at a6b10cf7, because memory stats were wrong until that commit): ``` nDPI Memory statistics: nDPI Memory (once): 221.15 KB Flow Memory (per flow): 2.94 KB ``` Now: ``` nDPI Memory statistics: nDPI Memory (once): 231.71 KB Flow Memory (per flow): 1008 B <--------- ``` i.e. memory usage per flow has been reduced by 66%, dropping below the psychological threshold of 1 KB. To further reduce this value, we probably need to look into #1279: let's fight this battle another day.
* Make serialize risk and proto not dependant on any flow. (#1377)Toni2021-12-04
| | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Added example for finding similarities in RRDs using nDPI statistical APIsLuca Deri2021-12-04
|
* Added ndpi_serializer_skip_header() serialization APILuca Deri2021-11-26
|
* Add detection of OCSP (#1370)Ivan Nardi2021-11-11
| | | | | | | | | | This protocol is detected via HTTP Content-Type header. Until 89d548f9, nDPI had a dedicated automa (`content_automa`) to classify a HTTP flow according to this header. Since then, this automa has been useless because it is always empty. Re-enable it to match only a string seems overkilling. Remove all `content_automa` leftovers.
* Serialize additional information stored in the flow struct. (#1362)Toni2021-10-27
| | | | | | | * Changed function signature of ndpi_flow2json (removed unused vlan_id; API break) * Serialize NTP information. * Improved QUIC serialization. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Fix QUIC log and remove SoulSeek leftovers after b97dc6ba (#1351)Ivan Nardi2021-10-19
| | | | | Update .gitignore file Fix a function prototype Close #1349
* Implemented ndpi_ses_fitting() and ndpi_des_fitting()Luca Deri2021-10-12
| | | | for comuting the best alpha/beta values for exponential smoothing
* Added -a <num> to ndpiReader for generating OPNsense configurationLuca Deri2021-10-04
| | | | See https://github.com/ntop/opnsense
* Compilation fixed on CentOS 7Luca Deri2021-09-27
| | | | Bitmap APi changes
* Added API for handling compressed bitmapsLuca Deri2021-09-26
| | | | | | | | | | | | | | ndpi_bitmap* ndpi_bitmap_alloc(); void ndpi_bitmap_free(ndpi_bitmap* b); u_int64_t ndpi_bitmap_cardinality(ndpi_bitmap* b); void ndpi_bitmap_set(ndpi_bitmap* b, u_int32_t value); void ndpi_bitmap_unset(ndpi_bitmap* b, u_int32_t value); bool ndpi_bitmap_isset(ndpi_bitmap* b, u_int32_t value); void ndpi_bitmap_clear(ndpi_bitmap* b); size_t ndpi_bitmap_serialize(ndpi_bitmap* b, char **buf); ndpi_bitmap* ndpi_bitmap_deserialize(char *buf); based on https://github.com/RoaringBitmap/CRoaring
* Compile everything with "-W -Wall -Wno-unused-parameter" flags (#1276)Ivan Nardi2021-08-20
| | | | | | | | | | | | | | | | Fix all the warnings. Getting rid of "-Wno-unused-parameter" is quite complex because some parameters usage depends on compilation variable (i.e. `--enable-debug-messages`). The "-Werror" flag has been added only in Travis builds to avoid breaking the builds to users using uncommon/untested OS/compiler/enviroment. Tested on: * x86_64; Ubuntu 20.04; gcc 7,8,9,10,11; clang 7,8,9,10,11,12 * x86_64; CentOS 7.7; gcc 4.8.5 (with "--disable-gcrypt" flag) * Raspberry 4; Debian 10.10; gcc 8.3.0
* Added ability to report whether a protocol is encryptedLuca Deri2021-08-07
| | | | | | | | | Added new API calls - u_int8_t ndpi_is_valid_protoId(u_int16_t protoId); Cleaned up code and organized in a function. - u_int8_t ndpi_is_encrypted_proto(struct ndpi_detection_module_struct *ndpi_str, ndpi_protocol proto); Check whether a protocol is encrypted or not based on master and app protocols
* TypoAlfredo Cardigliano2021-07-23
|
* Fix prototypeAlfredo Cardigliano2021-07-23
|
* Implemented function to retrieve flow information. #1253 (#1254)Toni2021-07-23
| | | | | * fixed [h]euristic typo Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Implementation of flow risk eception (work in progress)Luca Deri2021-07-22
|
* Fix for #1230 (#1235)Vitaly Lavrov2021-07-07
| | | | | | | | | | | | | | | * Revert "Fix return value of ndpi_match_string_subprotocol() (#1230)" This reverts commit 58665e93a98d014b53d131b2481ccab074efc9ff. * Checking the return code after calling ndpi_match_string_subprotocol() ndpi_api.h: Description of the returned error codes for the ndpi_match_string_subprotocol() function. If the ndpi_match_string_subprotocol() function returned an error, then return NDPI_PROTOCOL_UNKNOWN. http: The "Content-type" header is only checked if it is not empty.
* Updated prototypeLuca Deri2021-07-05
|
* Speed and memory size optimisation (#1214)Vitaly Lavrov2021-06-23
| | | | | | | Removed bigram_automata, impossible_bigram_automata, trigram_automata. The ahocorasick structure is replaced with a bitmap. The bitmap size for ndpi_en_bigram is 176 bytes. The bitmap size for ndpi_en_trigram is 2201 bytes. On the test machine, the test execution time was reduced from 27.3 seconds to 24.7 (9%).
Powered by cgit, Themed by Ted Yin.