aboutsummaryrefslogtreecommitdiff
path: root/src/lib/third_party/include
Commit message (Collapse)AuthorAge
* Domain Classification Improvements (#2396)Luca Deri2024-04-18
| | | | | | | | | | | | | | | | | | | * Added size_t ndpi_compress_str(const char * in, size_t len, char * out, size_t bufsize); size_t ndpi_decompress_str(const char * in, size_t len, char * out, size_t bufsize); used to compress short strings such as domain names. This code is based on https://github.com/Ed-von-Schleck/shoco * Major code rewrite for ndpi_hash and ndpi_domain_classify * Improvements to make sure custom categories are loaded and enabled * Fixed string encoding * Extended SalesForce/Cloudflare domains list
* Added support for roaring bitmap v3 (#2355)Luca Deri2024-03-25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Integrated RoaringBitmap v3 * Renamed ndpi_bitmap64 ro ndpi_bitmap64_fuse * Fixes to ndpi_bitmap for new roaring library * Fixes for bitmap serialization * Fixed format * Warning fix * Conversion fix * Warning fix * Added check for roaring v3 support * Updated file name * Updated path * Uses clang-9 (instead of clang-7) for builds * Fixed fuzz_ds_bitmap64_fuse * Fixes nDPI printf handling * Disabled printf * Yet another printf fix * Cleaup * Fx for compiling on older platforms * Fixes for old compilers * Initialization changes * Added compiler check * Fixes for old compilers * Inline function is not static inline * Added missing include
* New ndpi_sha256() nDPI API call (#2230)Luca Deri2023-12-23
| | | | | * Added ndpi_sha256.c to the Windows project * Added ndpi_sha256() nDPI API call
* Removed type redefinitionLuca Deri2023-12-23
|
* Changes to avoid type redefinition on windowsLuca Deri2023-12-23
|
* Implements JA4 Support (#2191)Luca Deri2023-12-22
|
* Move from PCRE to PCRE2 (#2134)Christian Marangi2023-11-01
| | | | | | | | | | Move from PCRE to PCRE2. PCRE is EOL and won't receive any security updates anymore. Convert to PCRE2 by converting any function PCRE2 new API. Also update every entry in github workflows and README to point to the new configure flag. (--with-pcre2) Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
* version of dirent.c that is liked by both VC++ and MinGWLuca Deri2023-10-05
|
* Windows code reworkLuca Deri2023-10-05
|
* Windows compilation fixesLuca Deri2023-10-05
|
* Added printf/fprintf replacement for some internal modules. (#1974)Toni2023-09-26
| | | | | | * logging is instead redirected to `ndpi_debug_printf` Signed-off-by: lns <matzeton@googlemail.com> Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* fuzz: extend fuzzing coverageNardi Ivan2023-09-16
|
* Fix some errors found by fuzzers (#2078)Ivan Nardi2023-09-10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fix compilation on Windows. "dirent.h" file has been taken from https://github.com/tronkko/dirent/ Fix Python bindings Fix some warnings with x86_64-w64-mingw32-gcc: ``` protocols/dns.c: In function ‘ndpi_search_dns’: protocols/dns.c:775:41: error: cast from pointer to integer of different size [-Werror=pointer-to-int-cast] 775 | unsigned long first_element_len = (unsigned long)dot - (unsigned long)_hostname; | ^ protocols/dns.c:775:62: error: cast from pointer to integer of different size [-Werror=pointer-to-int-cast] 775 | unsigned long first_element_len = (unsigned long)dot - (unsigned long)_hostname; | ``` ``` In file included from ndpi_bitmap64.c:31: third_party/include/binaryfusefilter.h: In function ‘binary_fuse8_hash’: third_party/include/binaryfusefilter.h:160:32: error: left shift count >= width of type [-Werror=shift-count-overflow] 160 | uint64_t hh = hash & ((1UL << 36) - 1); ``` ``` In function ‘ndpi_match_custom_category’, inlined from ‘ndpi_fill_protocol_category.part.0’ at ndpi_main.c:7056:16: ndpi_main.c:3419:3: error: ‘strncpy’ specified bound depends on the length of the source argument [-Werror=stringop-overflow=] 3419 | strncpy(buf, name, name_len); ```
* Win fixLuca Deri2023-09-08
|
* Compilation fixes for old compilersLuca Deri2023-09-05
|
* Added ndpi_bitmap64 supportLuca Deri2023-09-05
|
* Fix compilation on Windows (#2072)Ivan Nardi2023-08-14
|
* Reworked ndpi_filter_xxx implementation using compressed bitmapsLuca Deri2023-08-14
|
* TypoLuca Deri2023-08-12
|
* ARM compilation fixLuca Deri2023-08-12
|
* Minor initialization fixesLuca Deri2023-08-11
|
* Added new API calls for implementing Bloom-filter like data structuresLuca Deri2023-08-11
| | | | | | | ndpi_filter* ndpi_filter_alloc(uint32_t elements_number); bool ndpi_filter_add(ndpi_filter *f, uint64_t value); bool ndpi_filter_contains(ndpi_filter *f, uint64_t value); void ndpi_filter_free(ndpi_filter *f);
* Fix some memory errors triggered by allocation failures (#1995)Ivan Nardi2023-05-29
| | | | | | | | | | | | Some low hanging fruits found using nallocfuzz. See: https://github.com/catenacyber/nallocfuzz See: https://github.com/google/oss-fuzz/pull/9902 Most of these errors are quite trivial to fix; the only exception is the stuff in the uthash. If the insertion fails (because of an allocation failure), we need to avoid some memory leaks. But the only way to check if the `HASH_ADD_*` failed, is to perform a new lookup: a bit costly, but we don't use that code in any critical data-path.
* Improved debug output. (#1951)Toni2023-04-21
| | | | | | * try to get rid of some `printf(..)`s as they do not belong to a shared library * replaced all `exit(..)`s with `abort()`s to indicate an abnormal process termination Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* fuzz: add fuzzer to test internal gcrypt code (#1920)Ivan Nardi2023-04-04
|
* Update libinjection code (#1918)Ivan Nardi2023-04-04
| | | | | | | | | | Update libinjection code to the current master https://github.com/libinjection/libinjection/commit/7e4b74e824dc3f0623ba4894df2a0d817351ec39 The goal is to finally fix #1820 See: https://github.com/libinjection/libinjection/issues/33 Update the corpus of the libinjection fuzzers Close #1820
* Update roaring bitmap codeLuca Deri2023-02-12
|
* Add some fuzzers to test other data structures. (#1870)Ivan Nardi2023-01-25
| | | | | | | Start using a dictionary for fuzzing (see: https://llvm.org/docs/LibFuzzer.html#dictionaries). Remove some dead code. Fuzzing with debug enabled is not usually a great idea (from performance POV). Keep the code since it might be useful while debugging.
* Add some fuzzers to test algorithms and data structures (#1852)Ivan Nardi2023-01-17
| | | Fix some issues found with these new fuzzers
* Extend CodeQL configuration and fix two warnings (#1816)Ivan Nardi2022-12-02
| | | | | | | | | ``` Call to `memset` may be deleted Comparison of narrow type with wide type in loop condition ``` See https://www.usenix.org/system/files/conference/usenixsecurity17/sec17-yang.pdf for the new portable function `secure_memzero`.
* Fix some warningsNardi Ivan2022-09-05
| | | | | | | | | | | | | | | | | | | | | Ubuntu-20.04, clang-16 (nightly build) ``` Making all in src/lib protocols/smpp.c:70:17: warning: variable 'pdu_c' set but not used [-Wunused-but-set-variable] u_int16_t pdu_c = 1; ^ 1 warning generated. third_party/src/ahocorasick.c:173:20: warning: implicit truncation from 'int' to a one-bit wide bit-field changes value from 1 to -1 [-Wsingle-bit-bitfield-constant-conversion] thiz->root->root = 1; ^ ~ third_party/src/ahocorasick.c:336:15: warning: implicit truncation from 'int' to a one-bit wide bit-field changes value from 1 to -1 [-Wsingle-bit-bitfield-constant-conversion] n->ff = 1; ^ ~ third_party/src/ahocorasick.c:716:21: warning: implicit truncation from 'int' to a one-bit wide bit-field changes value from 1 to -1 [-Wsingle-bit-bitfield-constant-conversion] node->final = 1; [...] ```
* Modified definition of hll_addLuca Deri2022-08-17
|
* Patricia tree, Ahocarasick automa, LRU cache: add statistics (#1683)Ivan Nardi2022-07-29
| | | | | | | | | | Add (basic) internal stats to the main data structures used by the library; they might be usefull to check how effective these structures are. Add an option to `ndpiReader` to dump them; enabled by default in the unit tests. This new option enables/disables dumping of "num dissectors calls" values, too (see b4cb14ec).
* Revert "Patricia tree, Ahocarasick automa, LRU cache: add statistics ↵Ivan Nardi2022-07-29
| | | | | (#1677)" (#1682) This reverts commit bb83899985c25097341b947c2c535f56254a075c.
* Patricia tree, Ahocarasick automa, LRU cache: add statistics (#1677)Ivan Nardi2022-07-29
| | | | | | | | Add (basic) internal stats to the main data structures used by the library; they might be usefull to check how effective these structures are. Add an option to `ndpiReader` to dump them; disabled by default to avoid too much fuss with the unit tests.
* ahocorasick: fix char/uchar bug (#1597)Ivan Nardi2022-07-20
| | | | Porting of https://github.com/vel21ripn/nDPI/commit/06e2967d0c26ab214683a2f5565f4012fd523537 Close #1506
* Replaced nDPI's internal hashmap with uthash. (#1602)Toni2022-06-17
| | | Signed-off-by: lns <matzeton@googlemail.com>
* Fixed msys2 build warnings and re-activated CI Mingw64 build.fix/windows-msys2Toni Uhlig2022-04-14
| | | | | | | * Removed Visual Studio leftovers. Maintaining an autotools project with VS integration requires some additional overhead. Signed-off-by: Toni Uhlig <matzeton@googlemail.com> Signed-off-by: lns <matzeton@googlemail.com>
* Fixed a bug for BE architectures (#1478)Vitaly Lavrov2022-03-05
| | | Fixed a bug in the internal implementation of libgcrypt for bigendian architectures
* Drop support for non-gcrypt builds. (#1469)Toni2022-03-02
| | | | | | | | * As there is now a builtin, lightweight libgcrypt there is no need to disable tls-clho decryption. * It is still possible to use a host libgcrypt with `--with-local-libgcrypt'. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Internal crypto: increase size of authentication buffer (#1468)Ivan Nardi2022-03-02
| | | | | | | Some QUIC flows are not properly decoded while using internal crypto code: the authentication buffer is too small. The new value (like the old one) is arbitrary. Close #1463
* Added lightweight implementation of libgcrypt. (#1444)Vitaly Lavrov2022-02-20
| | | | | | | | | | | | | | | | | | | | | Implementation borrowed from the https://github.com/ARMmbed/mbedtls.git project (v3.1.0) Speed testing (Xeon(R) CPU E3-1230 V2 @ 3.30GHz): gcrypt-gnu Test md 2897 ms enc 2777 ms dec 942 ms gcrypt-int Test md 3668 ms enc 1312 ms dec 2836 ms gcrypt-int-noaesni Test md 3652 ms enc 1916 ms dec 4458 ms gcrypt-gnu-nonopt Test md 3763 ms enc 4978 ms dec 3999 ms gcrypt-gnu-nonopt - libgcrypt compiled without hardware acceleration --disable-padlock-support --disable-aesni-support \ --disable-shaext-support --disable-pclmul-support \ --disable-sse41-support --disable-drng-support \ --disable-avx-support --disable-avx2-support \ --disable-neon-support --disable-arm-crypto-support \ --disable-ppc-crypto-support --disable-amd64-as-feature-detection
* Reworked bitmap codeLuca Deri2021-09-27
|
* Added API for handling compressed bitmapsLuca Deri2021-09-26
| | | | | | | | | | | | | | ndpi_bitmap* ndpi_bitmap_alloc(); void ndpi_bitmap_free(ndpi_bitmap* b); u_int64_t ndpi_bitmap_cardinality(ndpi_bitmap* b); void ndpi_bitmap_set(ndpi_bitmap* b, u_int32_t value); void ndpi_bitmap_unset(ndpi_bitmap* b, u_int32_t value); bool ndpi_bitmap_isset(ndpi_bitmap* b, u_int32_t value); void ndpi_bitmap_clear(ndpi_bitmap* b); size_t ndpi_bitmap_serialize(ndpi_bitmap* b, char **buf); ndpi_bitmap* ndpi_bitmap_deserialize(char *buf); based on https://github.com/RoaringBitmap/CRoaring
* Win fixLuca Deri2021-08-03
|
* Implementation of flow risk eception (work in progress)Luca Deri2021-07-22
|
* ahoсorasick. Code review. Part 2. (#1236)Vitaly Lavrov2021-07-12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Simplified the process of adding lines to AC_AUTOMATA_t. Use the ndpi_string_to_automa() function to add patterns with domain names. For other cases can use ndpi_add_string_value_to_automa(). ac_automata_feature(ac_automa, AC_FEATURE_LC) allows adding and compare data in a case insensitive manner. For mandatory pattern comparison from the end of the line, the "ac_pattern.rep.at_end=1" flag is used. This eliminated unnecessary conversions to lowercase and adding "$" for end-of-line matching in domain name patterns. ac_match_handler() has been renamed ac_domain_match_handler() and has been greatly simplified. ac_domain_match_handler() looks for the template with the highest domain level. For special cases it is possible to manually specify the domain level. Added test for checking ambiguous domain names like: - short.weixin.qq.com is QQ, not Wechat - instagram.faae1-1.fna.fbcdn.net is Instagram, not Facebook If you specify a NULL handler when creating the AC_AUTOMATA_t structure, then a pattern with the maximum length that satisfies the search conditions will be found (exact match, from the beginning of the string, from the end of the string, or a substring). Added debugging for ac_automata_search. To do this, you need to enable debugging globally using ac_automata_enable_debug(1) and enable debugging in the AC_AUTOMATA_t structure using ac_automata_name("name", AC_FEATURE_DEBUG). The search will display "name" and a list of matching patterns. Running "AHO_DEBUG=1 ndpiReader ..." will show the lines that were searched for templates and which templates were found. The ac_automata_dump() prototype has been changed. Now it outputs data to a file. If it is specified as NULL, then the output will be directed to stdout. If you need to get data as a string, then use open_memstream(). Added the ability to run individual tests via the do.sh script
* Removed ht_hash as it is not used anymore. (#1220)Toni2021-06-29
| | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Win changesLuca Deri2021-06-08
|
* New version of the ahocorasick library (#1200)Vitaly Lavrov2021-06-07
| | | | | | | | | | | | | | | | The new version is about 25% faster with -O2 and 45% faster with -O3. No recursion is used (smaller stack size required). Uses less memory (by valgrind info) bigram: - original 1796 allocs, 247864 bytes allocated - new 1232 allocs, 158880 bytes allocated host_match: - original 18038 allocs, 3004576 bytes allocated - new 6861 allocs, 396624 bytes allocated The function ac_automata_search() is thread safe. Optional case-insensitive comparison. Matching at the beginning and at the end of the string is supported. One code file and one header file.
Powered by cgit, Themed by Ted Yin.