aboutsummaryrefslogtreecommitdiff
path: root/fuzz
Commit message (Collapse)AuthorAge
...
* fuzz: fix signed-integer-overflow (#1822)Ivan Nardi2022-12-10
| | | | | | | | | | ``` fuzz_ndpi_reader.c:33:29: runtime error: signed integer overflow: 214013 * 24360337 cannot be represented in type 'int' #0 0x4c1cf7 in fastrand ndpi/fuzz/fuzz_ndpi_reader.c:33:29 #1 0x4c1cf7 in malloc_wrapper ndpi/fuzz/fuzz_ndpi_reader.c:38:11 #2 0x523057 in ndpi_malloc ndpi/src/lib/ndpi_main.c:220:25 ``` Found by oss-fuzz See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=54112
* fuzz: add a new fuzzer testing memory allocation failures (#1818)Ivan Nardi2022-12-06
| | | | | | | | | Try to fuzz error paths triggered by allocation errors. Fix some errors already found by this new fuzzer. Basic idea taken from: https://github.com/harfbuzz/harfbuzz/pull/2566/files `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` is a standard define used to (not)compile specific code in fuzzing builds. See: https://llvm.org/docs/LibFuzzer.html
* Add support for LTO and Gold linker (#1812)Ivan Nardi2022-12-05
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit add (optional) support for Link-Time-Optimization and Gold linker. This is the first, mandatory step needed to make nDPI compliant with "introspector" sanitizer requirements in OSS-Fuzz: see https://github.com/google/oss-fuzz/issues/8939 Gold linker is not supported by Windows and by macOS, so this feature is disabled by default. It has been enable in CI in two linux targets ("latest" gcc and clang). Fix some warnings triggered by LTO. The changes in `src/lib/ndpi_serializer.c` seams reasonable. However, the change in `tests/unit/unit.c` is due to the following warning, which seems to be a false positive. ``` unit.c: In function ‘serializerUnitTest’: ndpi_serializer.c:2258:13: error: ‘MEM[(struct ndpi_private_serializer *)&deserializer].buffer.size’ may be used uninitialized in this function [-Werror=maybe-uninitialized] unit.c:67:31: note: ‘MEM[(struct ndpi_private_serializer *)&deserializer].buffer.size’ was declared here 67 | ndpi_serializer serializer, deserializer; | ^ ndpi_serializer.c:2605:10: error: ‘MEM[(struct ndpi_private_serializer *)&deserializer].status.buffer.size_used’ may be used uninitialized in this function [-Werror=maybe-uninitialized] unit.c:67:31: note: ‘MEM[(struct ndpi_private_serializer *)&deserializer].status.buffer.size_used’ was declared here 67 | ndpi_serializer serializer, deserializer; ``` Since this warning is triggered only with an old version of gcc and `tests/unit/unit.c` is used only during the tests, the easiest fix has been applied. Some (unknown to me) combinations of OS and compiler trigger the following warnings at linker time (with sanitizer and gold linker) ``` /usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_load1_asm' /usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_load2_asm' /usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_load4_asm' /usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_load8_asm' /usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_load16_asm' /usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_store1_asm' /usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_store2_asm' /usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_store4_asm' [..] ``` I have not found any references to this kind of message, with the only exception of https://sourceware.org/bugzilla/show_bug.cgi?id=25975 which seems to suggest that these messages can be safely ignored. In any case, the compilation results are sound. Fix `clean` target in the Makefile in the `example` directory. In OSS-Fuzz enviroments, `fuzz_ndpi_reader` reports a strange link error (as always, when the gold linker is involved...). It's come out that the culprit was the `tempnam` function: the code has been changed to use `tmpfile` instead. No sure why... :( Fuzzing target `fuzz_ndpi_reader.c` doesn't use `libndpiReader.a` anymore: this way we can use `--with-only-libndpi` flag on Oss-Fuzz builds as workaround for the "missing dependencies errors" described in https://github.com/google/oss-fuzz/issues/8939
* Remove some useless fuzzing target (#1813)Ivan Nardi2022-12-01
| | | | | We don't need specific targets to reproduce fuzzing issues. After all, calling `./fuzz/fuzz_process_packet_with_main $ARTIFACT_FILE` is equivalento to `./fuzz/fuzz_process_packet $ARTIFACT_FILE`
* Add a new fuzzer for QUIC (#1800)Ivan Nardi2022-12-01
| | | | | | | | | | | | QUIC packets are encrypted/obfuscated; that means that we need to decrypt them before parsing the real (TLS) message. Fuzzing is not effective here, since a random buffer is hardly a valid encrypted QUIC packet. Add a new fuzzer, testing *decrypted* QUIC packets. Add a basic corpus. Fix a few bugs already found by this fuzzer.
* Add basic profiling of memory allocations on data-path (#1789)Ivan Nardi2022-10-28
| | | | | | | | | | | | | | | The goal is to have an idea of the memory allocation sizes performed in the **library data-path**, i.e. excluding init/deinit phases and all the allocations made by the application itself. In other words, how much memory is needed per-flow, by nDPI, other than `struct ndpi_flow_struct`? It works only on single-thread configurations. It is not enabled by default (in the unit tests) since different canfiguration options (example: `--enable-pcre`) lead to diffferent results. See: #1781
* Fix compilation and sync utests resultsNardi Ivan2022-10-04
|
* Improved nDPI JSON serialization. (#1689)Toni2022-08-02
| | | | | | | | * fixed autoconf CFLAGS/LDFLAGS MSAN issue which could lead to build errors * introduced portable version of gmtime_r aka ndpi_gmtime_r * do as most as possible of the serialization work in ndpi_utils.c * use flow2json in ndpiReader Signed-off-by: lns <matzeton@googlemail.com>
* Add support for flow client/server information (#1671)Ivan Nardi2022-07-24
| | | | | | | | | | | | | | | | | | In a lot of places in ndPI we use *packet* source/dest info (address/port/direction) when we are interested in *flow* client/server info, instead. Add basic logic to autodetect this kind of information. nDPI doesn't perform any "flow management" itself but this task is delegated to the external application. It is then likely that the application might provide more reliable hints about flow client/server direction and about the TCP handshake presence: in that case, these information might be (optionally) passed to the library, disabling the internal "autodetect" logic. These new fields have been used in some LRU caches and in the "guessing" algorithm. It is quite likely that some other code needs to be updated.
* Do not interfere with CFLAGS/LDFLAGS env anymore. (#1659)Toni2022-07-13
| | | | | | | * CI fixes * some build systems do not like that (e.g. OpenWrt) * fixed some rrdtool related build warnings/errors Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Fix some debug messages (#1583)Ivan Nardi2022-06-05
| | | Increase max number of flows handled during fuzzing
* Replaced ndpiReader's libjson-c support with libnDPI's internal ↵Toni2022-05-07
| | | | | | | | | | | serialization interface. (#1535) * Fixes #1528 * Serialization Interface should also fuzzed * libjson-c may only be used in the unit test to verify the internal serialization interface * Serialization Interface supports tlv(broken), csv and json * Unit test does work again and requires libjson-c Signed-off-by: lns <matzeton@googlemail.com>
* fuzz: purge old sessions (#1451)Ivan Nardi2022-02-21
| | | | | | | | At every fuzz iteration (i.e for every trace file): * keep the same ndpi context (`ndpi_init_detection_module` is very slow); * reset the flow table, otherwise it grows indefinitely. This change should fix the "out-of-memory" errors reported by oss-fuzz.
* fuzz: make fuzz_ndpi_reader faster (#1446)Ivan Nardi2022-02-20
| | | | | | | | | | | | | | | Initialize ndpi_workflow_init context only once. On a quite old notebook, before: ``` $ ./fuzz/fuzz_ndpi_reader -max_total_time="${MAX_TOTAL_TIME:-360}" -print_pcs=1 -workers="${FUZZY_WORKERS:-0}" -jobs="${FUZZY_JOBS:-0}" ./tests/pcap/ [...] Done 3256 runs in 361 second(s) ``` after: ``` Done 5032771 runs in 361 second(s) <----------- ~1400X ``` oss-fuzz will be happy!
* Remove `struct ndpi_id_struct` (#1427)Ivan Nardi2022-01-30
| | | | | | | | | | | | | | | | | | | | | | | | | | | Remove the last uses of `struct ndpi_id_struct`. That code is not really used and it has not been updated for a very long time: see #1279 for details. Correlation among flows is achieved via LRU caches. This change allows to further reduce memory consumption (see also 91bb77a8). At nDPI 4.0 (more precisly, at a6b10cf, because memory stats were wrong until that commit): ``` nDPI Memory statistics: nDPI Memory (once): 221.15 KB Flow Memory (per flow): 2.94 KB ``` Now: ``` nDPI Memory statistics: nDPI Memory (once): 235.27 KB Flow Memory (per flow): 688 B <-------- ``` i.e. memory usage per flow has been reduced by 77%. Close #1279
* Fix two use-of-uninitialized-value errors (#1398)Ivan Nardi2022-01-12
| | | | | | | Found by oss-fuzz: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=40269 https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=41432 Fix fuzz compilation (follow-up of f5545a80)
* Fix writes to `flow->protos` union fields (#1354)Ivan Nardi2021-11-15
| | | | | | | | | | | | | | | | | | | | | | | | | We can write to `flow->protos` only after a proper classification. This issue has been found in Kerberos, DHCP, HTTP, STUN, IMO, FTP, SMTP, IMAP and POP code. There are two kinds of fixes: * write to `flow->protos` only if a final protocol has been detected * move protocol state out of `flow->protos` The hard part is to find, for each protocol, the right tradeoff between memory usage and code complexity. Handle Kerberos like DNS: if we find a request, we set the protocol and an extra callback to further parsing the reply. For all the other protocols, move the state out of `flow->protos`. This is an issue only for the FTP/MAIL stuff. Add DHCP Class Identification value to the output of ndpiReader and to the Jason serialization. Extend code coverage of fuzz tests. Close #1343 Close #1342
* Fix broken fuzz_process_packet fuzzer by adding a call to ↵Toni2021-10-18
| | | | | | | | | | | | ndpi_finalize_initialization(). (#1334) * fixed several memory errors (heap-overflow, unitialized memory, etc) * ability to build fuzz_process_packet with a main() allowing to replay crash data generated with fuzz_process_packet by LLVMs libfuzzer * temporarily disable fuzzing if `tests/do.sh` executed with env FUZZY_TESTING_ENABLED=1 Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Compile everything with "-W -Wall -Wno-unused-parameter" flags (#1276)Ivan Nardi2021-08-20
| | | | | | | | | | | | | | | | Fix all the warnings. Getting rid of "-Wno-unused-parameter" is quite complex because some parameters usage depends on compilation variable (i.e. `--enable-debug-messages`). The "-Werror" flag has been added only in Travis builds to avoid breaking the builds to users using uncommon/untested OS/compiler/enviroment. Tested on: * x86_64; Ubuntu 20.04; gcc 7,8,9,10,11; clang 7,8,9,10,11,12 * x86_64; CentOS 7.7; gcc 4.8.5 (with "--disable-gcrypt" flag) * Raspberry 4; Debian 10.10; gcc 8.3.0
* Fixed uninitialized memory use. (#1198)Vitaly Lavrov2021-06-07
| | | The pointer "header" must be initialized before first call pcap_next_ex().
* Check datalink during fuzzing to prevent console / logfile spam. See #1175 ↵Toni2021-05-09
| | | | | for more information. (#1177) Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Compilation fixLuca Deri2021-04-26
|
* Fixed mispelled API callLuca Deri2021-02-22
|
* Fixed memory leaks caused by conditional free'ing for some TLS connec… (#1132)Toni2021-02-10
| | | | | | | | | | | | | | | * Fixed memory leaks caused by conditional free'ing for some TLS connections. * Members of tls_quic struct should also free'd if the detected master protocol is IMAPS / POPS / SMTPS / etc. Signed-off-by: Toni Uhlig <matzeton@googlemail.com> * Prevent reader_util.c from exit()'ing if maximum flow count reached. This confuses the fuzzer. * Improved fuzz/Makefile.am to use LDADD for ../example/libndpiReader.a instead of LDFLAGS. That way, fuzz_ndpi_reader re-links to ../example/libndpiReader.a if something changed there. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Fix a memory leak on fuzzing code (#1036)Hosein Ghahremanzadeh2020-10-21
| | | After allocation of pcap_buffer it is necessary to free it
* Add (optional) dependency on external libraries: libgcrypt and libgpg-errorNardi Ivan2020-08-21
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | To support QUIC payload and header decryption, it is necessary to choose an external crypto library to handle the low-level crypto stuff. Since we will use some Wireshark code, it is quite natural to choose the same library used by Wireshark itself: libgcrypt. More precisely, we will use libgcrypt and libgpg-error. Both libraries have LGPL license, so there should be no issue from this point of view. These libraries are not required to build nDPI, and their usage is optional: nDPI will keep working (and compiling) even if they are not available. However, without them, QUIC sub-classification is next to impossible. The configure flag "--disable-gcrypt" forces the build system to ignore these libraries. libgpg-error is only used for debug to have meaningful error messages and its usage is trivial. The same cannot be said for libgcrypt because its initialization is a significant issue. The rest of this commit message try explaining how libgcrypt is initialized. According to the documentation https://gnupg.org/documentation/manuals/gcrypt/Initializing-the-library.html https://gnupg.org/documentation/manuals/gcrypt/Multi_002dThreading.html#Multi_002dThreading libgcrypt must be initialized before using it, but such initialization should be performed by the actual application and not by any library. Forcing the users to proper initialize libgcrypt in their own code seems unreasonable: most people using nDPI might be complete unaware of any crypto stuff and update each and every one application linking to nDPI with specific libgcrypt code should be out of question, anyway. Fortunately, it seems a workaround exists to initialize libgcrypt in a library https://lists.gnupg.org/pipermail/gcrypt-devel/2003-August/000458.html Therefore, we could provide a wrapper to this initialization stuff in a nDPI function. Unfortunately nDPI API lacks a global init function that must be called only once, before any other functions. We could add it, but that would be a major API break. AFAIK, ndpi_init_detection_module() might be called multiple times, for example to create multiple independent dpi engines in the same program. The proposed solution is to (optionally) initialize libgcrypt in ndpi_init_detection_module() anyway: * if the actual application doesn't directly use libgcrypt and only calls ndpi_init_detection_module() once, everything is formally correct and it should work out of the box [by far the most common user case]; * if the actual application already uses libgcrypt directly, it already performs the required initialization. In this case the ndpi_prefs.ndpi_dont_init_libgcrypt flag should be passed to ndpi_init_detection_module() to avoid further initializations. The only scenario not supported by this solution is when the application is unaware of libgcrypt and calls ndpi_init_detection_module() multiple times concurrently. But this scenario should be uncommon. A completely different option should be to switch to another crypto library, with a huge impact on the QUIC dissector code. Bottom line: crypto is hard, using libgcrypt is complex and the proposed initialization, even if not perfect, should cover the most frequent user cases and should work, for the time being. If anyone has some suggestions...
* Added memory checksLuca Deri2020-08-02
|
* Removed csv_fp as external symbol. Instead passing csv_fp through as argument.Toni Uhlig2020-07-08
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Merge pull request #940 from lnslbrty/fix/small-and-optimistic-improvmentsLuca Deri2020-07-06
|\ | | | | Fixed CodeInspector issues.
| * Fixed CodeInspector issues.Toni Uhlig2020-07-05
| | | | | | | | | | | | | | | | | | * Added compiler search list for AC_PROG_CC, AM_PROG_CC_C_O, AC_PROG_CXX, AC_PROG_CC_STDC for Mac OS X only The list rendered the AX_CHECK_COMPILE_FLAG([-fsanitize=fuzzer]) useless as it did use clang for AX_CHECK_COMPILE_FLAG but gcc during the compile process. Seems broken somehow. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* | Added fuzz_*_LDADD to give Automake the opportunity to do some dependency ↵Toni Uhlig2020-06-29
|/ | | | | | tracking. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Improved fuzz_ndpi_reader which supports now SMP/MT w/o race-coniditions.Toni Uhlig2020-06-29
| | | | | | | ./tests/do.sh can supports SMP/MT via environment variables. Removed -fno-sanitize=shift as well, was fixed by 317d3ffd. Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Re-Added corpus zipfile generation required for google-oss fuzzerToni Uhlig2020-06-27
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* fixed fuzzing tests in a way that ./tests/do.sh is now able to use corpus ↵Toni Uhlig2020-06-27
| | | | | | *.pcap files from ./tests/pcap Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Disabled unused codeLuca Deri2020-06-25
|
* Libfuzz fixesLuca2020-02-02
|
* Added missing additiona libsLuca2020-02-02
|
* Modifies fuzz target to fix leaksPhilippe Antoine2020-01-31
|
* Added memory boundary check in HTTP dissectorLuca Deri2020-01-24
|
* Debug compilation flag for fuzz targetsPhilippe Antoine2020-01-24
| | | | And adding include file to fuzz_ndpi_reader
* Generates automatically corpus for fuzz targetPhilippe Antoine2020-01-22
|
* New fuzz target : fuzz_ndpi_readerPhilippe Antoine2020-01-14
|
* Fixed new nDPi APILuca Deri2019-11-12
|
* Adds fuzz targetPhilippe Antoine2019-10-31
And configur option enable-fuzztargets
Powered by cgit, Themed by Ted Yin.