aboutsummaryrefslogtreecommitdiff
path: root/src/include/ndpi_api.h
Commit message (Collapse)AuthorAge
* Add (kind of) support for loading a list of JA4C malicious fingerprints (#2678)Ivan Nardi2025-01-14
| | | | | | | | | It might be usefull to be able to match traffic against a list of suspicious JA4C fingerprints Use the same code/logic/infrastructure used for JA3C (note that we are going to remove JA3C...) See: #2551
* Update `flow->flow_multimedia_types` to a bitmask (#2625)Ivan Nardi2024-11-25
| | | In the same flow, we can have multiple multimedia types
* SIP: extract some basic metadataIvan Nardi2024-11-12
|
* Added HTTP credentials extractionLuca Deri2024-10-31
|
* Added ndpi_str_endswith()Luca Deri2024-10-28
|
* Improved TCP fingepring calculationLuca Deri2024-10-18
| | | | Adde basidc OS detection based on TCP fingerprint
* Added u_int8_t ndpi_is_public_ipv4(u_int32_t a /* host byte order */);Luca Deri2024-10-13
|
* Added -N option for dumping/restoring the DNS cache (when enabled)Luca Deri2024-10-10
| | | | Example ndpiReader -i en0 --cfg=dpi.address_cache_size,32768 -N /tmp/a
* Added new API calls for serializing/restoring the DNS cacheLuca Deri2024-10-10
| | | | | - bool ndpi_address_cache_dump(struct ndpi_address_cache *cache, char *path, u_int32_t epoch_now); - u_int32_t ndpi_address_cache_restore(struct ndpi_address_cache *cache, char *path, u_int32_t epoch_now);
* Implemented (disabled by default) DNS host cache. You can set the cache size ↵Luca Deri2024-10-07
| | | | | | | | | | as follows: ndpiReader --cfg=dpi.address_cache_size,1000 -i <pcap>.pcap In the above example the cache has up to 1000 entries. In jcase ndpiReader exports data in JSON, the cache hostname (if found) is exported in the field server_hostname
* Let the library returning the packet direction calculated internally (#2572)Ivan Nardi2024-09-27
| | | wireshark, lua: add basic analysis of possible obfuscated flows
* Add some heuristics to detect encrypted/obfuscated/proxied TLS flows (#2553)Ivan Nardi2024-09-24
| | | | | | | | | | | | Based on the paper: "Fingerprinting Obfuscated Proxy Traffic with Encapsulated TLS Handshakes". See: https://www.usenix.org/conference/usenixsecurity24/presentation/xue-fingerprinting Basic idea: * the packets/bytes distribution of a TLS handshake is quite unique * this fingerprint is still detectable if the handshake is encrypted/proxied/obfuscated All heuristics are disabled by default.
* buffer lenghtt is now returned by ndpi_quick_encrypt() and ndpi_quick_deecrypt()Luca Deri2024-09-24
|
* Added new API callsLuca Deri2024-09-24
| | | | | u_int ndpi_hex2bin(u_char *out, u_int out_len, u_char* in, u_int in_len); u_int ndpi_bin2hex(u_char *out, u_int out_len, u_char* in, u_int in_len);
* Added ndpi_quick_encrypt() ndpi_quick_decrypt() APi calls (#2568)Luca Deri2024-09-24
| | | | | * Added ndpi_quick_encrypt() ndpi_quick_decrypt(0 APi calls based on AES * Added aes.c
* Implemented ndpi_strrstr()Luca Deri2024-09-19
| | | | Fixed bug in ndpi_get_host_domain
* Updated ndpi_serialize_flow_fingerprint API signatureLuca2024-09-17
|
* Reworked fingerprint export now in JSONLuca2024-09-16
|
* Implemented algorithms for K-Nearest Neighbor Search (KNN) (#2554)Luca Deri2024-09-10
| | | | | | | | | | | | | | | | * Extended API with functions for vector similarity based on KD-trees https://en.wikipedia.org/wiki/K-d_tree ndpi_kd_tree* ndpi_kd_create(u_int num_dimensions); void ndpi_kd_free(ndpi_kd_tree *tree); void ndpi_kd_clear(ndpi_kd_tree *tree); bool ndpi_kd_insert(ndpi_kd_tree *tree, const double *data_vector, void *user_data); ndpi_kd_tree_result *ndpi_kd_nearest(ndpi_kd_tree *tree, const double *data_vector); u_int32_t ndpi_kd_num_results(ndpi_kd_tree_result *res); bool ndpi_kd_result_end(ndpi_kd_tree_result *res); double* ndpi_kd_result_get_item(ndpi_kd_tree_result *res, double **user_data); bool ndpi_kd_result_next(ndpi_kd_tree_result *res); void ndpi_kd_result_free(ndpi_kd_tree_result *res); double ndpi_kd_distance(double *a1, double *b2, u_int num_dimensions);
* Added print_ndpi_address_port in nDPi APILuca Deri2024-08-27
|
* Added ndpi_risk2code and ndpi_code2riskLuca Deri2024-08-27
|
* CommentsLuca Deri2024-08-26
|
* Added in APILuca Deri2024-08-26
| | | | | | | | | | | | bool ndpi_is_proto(ndpi_master_app_protocol proto, u_int16_t p); bool ndpi_is_proto_unknown(ndpi_master_app_protocol proto); bool ndpi_is_proto_equals(ndpi_master_app_protocol to_check, ndpi_master_app_protocol to_match, bool exact_match_only); u_int16_t ndpi_get_proto_by_name(struct ndpi_detection_module_struct *ndpi_mod, const char *name); char* ndpi_get_proto_by_id(struct ndpi_detection_module_struct *ndpi_mod, u_int id); extern ndpi_master_app_protocol ndpi_get_protocol_by_name(struct ndpi_detection_module_struct *ndpi_str, const char *name); Removed (duplicate of ndpi_get_proto_by_name) int ndpi_get_protocol_id(struct ndpi_detection_module_struct *ndpi_mod, char *proto);
* Endian-independent implementation of IEEE 802.3 CRC32 (#2529)Vladimir Gavrilov2024-08-25
|
* Add infrastructure for explicit support of Fist Packet Classification (#2488)Ivan Nardi2024-07-03
| | | | | Let's start with some basic helpers and with FPC based on flow addresses. See: #2322
* Remove duplicate API for `ndpi_serialize_uint32_double` in ndpi_api.h (#2480)Anshul Thakur2024-06-26
| | | Co-authored-by: Anshul Thakur <athakur@cdot.in>
* Replace ndpi_strnstr() implementation with an optimal one (#2447)Vladimir Gavrilov2024-05-22
|
* Cleaned up APILuca Deri2024-05-17
| | | | | | | | | Removed - int ndpi_load_ipv4_ptree_file(ndpi_ptree_t *tree, const char *path, u_int16_t protocol_id); - int ndpi_load_ipv6_ptree_file(ndpi_ptree_t *tree, const char *path, u_int16_t protocol_id); Added (it supports both IPv4 and v6) + int ndpi_load_ptree_file(ndpi_ptree_t *tree, const char *path, u_int16_t protocol_id);
* Added optimized memmem/strlcpy version (#2424)Toni2024-05-08
| | | | | * credits goes to Vladimir Gavrilov Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Reworked code and added two new API callsLuca Deri2024-05-07
| | | | | - int ndpi_load_ipv4_ptree_file(ndpi_ptree_t *tree, const char *path, u_int16_t protocol_id); - int ndpi_load_ipv6_ptree_file(ndpi_ptree_t *tree, const char *path, u_int16_t protocol_id);
* Fix a warningNardi Ivan2024-04-20
| | | | | | ``` nDPI/PcDebug64/src/include/ndpi_api.h:1970:3: error: function declaration isn’t a prototype [-Werror=strict-prototypes] ```
* Add strlcpy implementation (#2395)0x41CEA552024-04-19
|
* Domain Classification Improvements (#2396)Luca Deri2024-04-18
| | | | | | | | | | | | | | | | | | | * Added size_t ndpi_compress_str(const char * in, size_t len, char * out, size_t bufsize); size_t ndpi_decompress_str(const char * in, size_t len, char * out, size_t bufsize); used to compress short strings such as domain names. This code is based on https://github.com/Ed-von-Schleck/shoco * Major code rewrite for ndpi_hash and ndpi_domain_classify * Improvements to make sure custom categories are loaded and enabled * Fixed string encoding * Extended SalesForce/Cloudflare domains list
* Renamed ndpi_fill_ip6_protocol_category in ndpi_fill_ipv6_protocol_categoryLuca Deri2024-04-11
|
* Add memmem() implementation (#2378)Vladimir Gavrilov2024-04-10
| | | | | | | * Add memmem() implementation * Fix build * Add fix to avoid too many memcmp calls
* Disable `-Wno-unused-parameter -Wno-unused-function`. (#2358)Toni2024-04-03
| | | | | * unused parameters and functions pollute the code and decrease readability Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Added support for roaring bitmap v3 (#2355)Luca Deri2024-03-25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Integrated RoaringBitmap v3 * Renamed ndpi_bitmap64 ro ndpi_bitmap64_fuse * Fixes to ndpi_bitmap for new roaring library * Fixes for bitmap serialization * Fixed format * Warning fix * Conversion fix * Warning fix * Added check for roaring v3 support * Updated file name * Updated path * Uses clang-9 (instead of clang-7) for builds * Fixed fuzz_ds_bitmap64_fuse * Fixes nDPI printf handling * Disabled printf * Yet another printf fix * Cleaup * Fx for compiling on older platforms * Fixes for old compilers * Initialization changes * Added compiler check * Fixes for old compilers * Inline function is not static inline * Added missing include
* LRU cache: move to 64 bits long keys (#2346)Ivan Nardi2024-03-15
| | | | | | Tradeoff between key comparison efficiency (i.e. no `memcmp`) and key length. At least in the ipv4 cases, we have no more different entries with the same key.
* ahocorasick: improve matching with subdomains (#2331)Ivan Nardi2024-03-06
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The basic idea is to have the following logic: * pattern "DOMAIN" matches the domain itself (i.e exact match) *and* any subdomains (i.e. "ANYTHING.DOMAIN") * pattern "DOMAIN." matches *also* any strings for which is a prefix [please, note that this kind of match is handy but it is quite dangerous...] * pattern "-DOMAIN" matches *also* any strings for which is a postfix Examples: * pattern "wikipedia.it": * "wikipiedia.it" -> OK * "foo.wikipedia.it -> OK * "foowikipedia.it -> NO MATCH * "wikipedia.it.com -> NO MATCH * pattern "wikipedia.": * "wikipedia.it" -> OK * "foo.wikipedia.it -> OK * "foowikipedia.it -> NO MATCH * "wikipedia.it.com -> OK * pattern "-wikipedia.it": * "wikipedia.it" -> NO MATCH * "foo.wikipedia.it -> NO MATCH * "0001-wikipedia.it -> OK * "foo.0001-wikipedia.it -> OK Bottom line: * exact match * prefix with "." (always, implicit) * prefix with "-" (only if esplicitly set) * postfix with "." (only if esplicitly set) That means that the patterns cannot start with '.' anymore. Close #2330
* Fixes exception handling glitchLuca Deri2024-02-22
|
* Allow multiple `struct ndpi_detection_module_struct` to share some state (#2271)Ivan Nardi2024-02-01
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add the concept of "global context". Right now every instance of `struct ndpi_detection_module_struct` (we will call it "local context" in this description) is completely independent from each other. This provide optimal performances in multithreaded environment, where we pin each local context to a thread, and each thread to a specific CPU core: we don't have any data shared across the cores. Each local context has, internally, also some information correlating **different** flows; something like: ``` if flow1 (PeerA <-> Peer B) is PROTOCOL_X; then flow2 (PeerC <-> PeerD) will be PROTOCOL_Y ``` To get optimal classification results, both flow1 and flow2 must be processed by the same local context. This is not an issue at all in the far most common scenario where there is only one local context, but it might be impractical in some more complex scenarios. Create the concept of "global context": multiple local contexts can use the same global context and share some data (structures) using it. This way the data correlating multiple flows can be read/write from different local contexts. This is an optional feature, disabled by default. Obviously data structures shared in a global context must be thread safe. This PR updates the code of the LRU implementation to be, optionally, thread safe. Right now, only the LRU caches can be shared; the other main structures (trees and automas) are basically read-only: there is little sense in sharing them. Furthermore, these structures don't have any information correlating multiple flows. Every LRU cache can be shared, independently from the others, via `ndpi_set_config(ndpi_struct, NULL, "lru.$CACHE_NAME.scope", "1")`. It's up to the user to find the right trade-off between performances (i.e. without shared data) and classification results (i.e. with some shared data among the local contexts), depending on the specific traffic patterns and on the algorithms used to balance the flows across the threads/cores/local contexts. Add some basic examples of library initialization in `doc/library_initialization.md`. This code needs libpthread as external dependency. It shouldn't be a big issue; however a configure flag has been added to disable global context support. A new CI job has been added to test it. TODO: we should need to find a proper way to add some tests on multithreaded enviroment... not an easy task... *** API changes *** If you are not interested in this feature, simply add a NULL parameter to any `ndpi_init_detection_module()` calls.
* Provide a u64 wrapper for `ndpi_set_config()` (#2292)Toni2024-01-30
| | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* fuzz: extend fuzzing coverage (#2281)Ivan Nardi2024-01-24
|
* config: follow-up (#2268)Ivan Nardi2024-01-20
| | | | | | Some changes in the parameters names. Add a fuzzer to fuzz the configuration file format. Add the infrastructure to configuratin callbacks. Add an helper to map LRU cache indexes to names.
* config: allow configuration of guessing algorithmsNardi Ivan2024-01-18
|
* config: move debug/log configuration to the new APINardi Ivan2024-01-18
|
* config: configure TLS certificate expiration with the new APINardi Ivan2024-01-18
|
* config: remove `enum ndpi_prefs`Nardi Ivan2024-01-18
|
* config: remove `ndpi_set_detection_preferences()`Nardi Ivan2024-01-18
|
* config: move cfg of aggressiviness and opportunistic TLS to the new APINardi Ivan2024-01-18
|
Powered by cgit, Themed by Ted Yin.