aboutsummaryrefslogtreecommitdiff
path: root/fuzz/fuzz_ndpi_reader.c
Commit message (Collapse)AuthorAge
* fuzz: extend fuzzing coverage (#2696)Ivan Nardi2025-01-23
|
* Add (kind of) support for loading a list of JA4C malicious fingerprints (#2678)Ivan Nardi2025-01-14
| | | | | | | | | It might be usefull to be able to match traffic against a list of suspicious JA4C fingerprints Use the same code/logic/infrastructure used for JA3C (note that we are going to remove JA3C...) See: #2551
* fuzz: improve fuzzing coverage (#2642)Ivan Nardi2024-12-11
| | | Updtae pl7m code (Fix swap-direction mutation)
* fuzz: extend fuzzing coverage (#2626)Ivan Nardi2024-11-20
|
* Add configuration of TCP fingerprint computation (#2598)Ivan Nardi2024-10-18
| | | Extend configuration of raw format of JA4C fingerprint
* Add monitoring capability (#2588)Ivan Nardi2024-10-14
| | | | | | | | | | | | | Allow nDPI to process the entire flows and not only the first N packets. Usefull when the application is interested in some metadata spanning the entire life of the session. As initial step, only STUN flows can be put in monitoring. See `doc/monitoring.md` for further details. This feature is disabled by default. Close #2583
* Added addr_dump_path definitionLuca Deri2024-10-10
|
* Add some heuristics to detect encrypted/obfuscated/proxied TLS flows (#2553)Ivan Nardi2024-09-24
| | | | | | | | | | | | Based on the paper: "Fingerprinting Obfuscated Proxy Traffic with Encapsulated TLS Handshakes". See: https://www.usenix.org/conference/usenixsecurity24/presentation/xue-fingerprinting Basic idea: * the packets/bytes distribution of a TLS handshake is quite unique * this fingerprint is still detectable if the handshake is encrypted/proxied/obfuscated All heuristics are disabled by default.
* fuzz: fix compilationNardi Ivan2024-09-17
|
* Add an heuristic to detect encrypted/obfuscated OpenVPN flows (#2547)Ivan Nardi2024-09-16
| | | | | | | | | | | | Based on the paper: "OpenVPN is Open to VPN Fingerprinting" See: https://www.usenix.org/conference/usenixsecurity22/presentation/xue-diwen Basic idea: * the distribution of the first byte of the messages (i.e. the distribution of the op-codes) is quite unique * this fingerprint might be still detectable even if the OpenVPN packets are somehow fully encrypted/obfuscated The heuristic is disabled by default.
* fuzz: fix compilationNardi Ivan2024-09-16
|
* fuzz: pl7m: add a custom mutator for better fuzzing of pcap files (#2483)Ivan Nardi2024-06-27
| | | | | | | | | | | Pl7m is a custom mutator (used for structure aware fuzzing) for network traffic packet captures (i.e. pcap files). The output of the mutator is always a valid pcap file, containing the same flows/sessions of the input file. That's it: the mutator only changes the packet payload after the TCP/UDP header, keeping all the original L2/L3 information (IP addresses and L4 ports). See: https://github.com/IvanNardi/pl7m
* wireshark: lua: export some metadataNardi Ivan2024-06-25
| | | | | | | | | | | | | | | Export some metadata (for the moment, SNI and TLS fingerprints) to Wireshark/tshark via extcap. Note that: * metadata are exported only once per flow * metadata are exported (all together) when nDPI stopped processing the flow Still room for a lot of improvements! In particular: * we need to add some boundary checks (if we are going to export other attributes) * we should try to have a variable length trailer
* fuzz: improve fuzzing coverage (#2474)Ivan Nardi2024-06-17
| | | | | | Remove some code never triggered AFP: the removed check is included in the following one MQTT: fix flags extraction
* RTP/STUN: look for STUN packets after RTP/RTCP classification (#2465)Ivan Nardi2024-06-07
| | | | | | | | | | After a flow has been classified as RTP or RTCP, nDPI might analyse more packets to look for STUN/DTLS packets, i.e. to try to tell if this flow is a "pure" RTP/RTCP flow or if the RTP/RTCP packets are multiplexed with STUN/DTLS. Useful for proper (sub)classification when the beginning of the flows are not captured or if there are lost packets in the the captured traffic. Disabled by default
* fuzzing: extend fuzzing coverage (#2371)Ivan Nardi2024-04-05
|
* Allow multiple `struct ndpi_detection_module_struct` to share some state (#2271)Ivan Nardi2024-02-01
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add the concept of "global context". Right now every instance of `struct ndpi_detection_module_struct` (we will call it "local context" in this description) is completely independent from each other. This provide optimal performances in multithreaded environment, where we pin each local context to a thread, and each thread to a specific CPU core: we don't have any data shared across the cores. Each local context has, internally, also some information correlating **different** flows; something like: ``` if flow1 (PeerA <-> Peer B) is PROTOCOL_X; then flow2 (PeerC <-> PeerD) will be PROTOCOL_Y ``` To get optimal classification results, both flow1 and flow2 must be processed by the same local context. This is not an issue at all in the far most common scenario where there is only one local context, but it might be impractical in some more complex scenarios. Create the concept of "global context": multiple local contexts can use the same global context and share some data (structures) using it. This way the data correlating multiple flows can be read/write from different local contexts. This is an optional feature, disabled by default. Obviously data structures shared in a global context must be thread safe. This PR updates the code of the LRU implementation to be, optionally, thread safe. Right now, only the LRU caches can be shared; the other main structures (trees and automas) are basically read-only: there is little sense in sharing them. Furthermore, these structures don't have any information correlating multiple flows. Every LRU cache can be shared, independently from the others, via `ndpi_set_config(ndpi_struct, NULL, "lru.$CACHE_NAME.scope", "1")`. It's up to the user to find the right trade-off between performances (i.e. without shared data) and classification results (i.e. with some shared data among the local contexts), depending on the specific traffic patterns and on the algorithms used to balance the flows across the threads/cores/local contexts. Add some basic examples of library initialization in `doc/library_initialization.md`. This code needs libpthread as external dependency. It shouldn't be a big issue; however a configure flag has been added to disable global context support. A new CI job has been added to test it. TODO: we should need to find a proper way to add some tests on multithreaded enviroment... not an easy task... *** API changes *** If you are not interested in this feature, simply add a NULL parameter to any `ndpi_init_detection_module()` calls.
* example: rework code between `ndpiReader.c` and `reader_util.c` (#2273)Ivan Nardi2024-01-22
|
* config: follow-up (#2268)Ivan Nardi2024-01-20
| | | | | | Some changes in the parameters names. Add a fuzzer to fuzz the configuration file format. Add the infrastructure to configuratin callbacks. Add an helper to map LRU cache indexes to names.
* config: allow configuration of guessing algorithmsNardi Ivan2024-01-18
|
* config: move debug/log configuration to the new APINardi Ivan2024-01-18
|
* config: remove `enum ndpi_prefs`Nardi Ivan2024-01-18
|
* config: remove `ndpi_set_detection_preferences()`Nardi Ivan2024-01-18
|
* STUN: rework extra dissection (#2202)Ivan Nardi2023-12-11
| | | | | | | Keep looking for RTP packets but remove the monitoring concept. We will re-introduce a more general concept of "flow in monitoring state" later. The function was disabled by default. Some configuration knobs will be provided when/if #2190 is merged.
* fuzz: extend fuzzing coverage (#2205)Ivan Nardi2023-12-11
|
* TLS: remove JA3+ fingerprints. (#2192)Ivan Nardi2023-12-05
| | | See: #2191
* STUN: fix detection of DTLS (#2187)Ivan Nardi2023-11-30
| | | | | | | | | | | | | | | | | | | | | | | | | | | Fix a memory leak ``` ==97697==ERROR: LeakSanitizer: detected memory leaks Direct leak of 16 byte(s) in 1 object(s) allocated from: #0 0x55a6967cfa7e in malloc (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader+0x701a7e) (BuildId: c7124999fa1ccc54346fa7bd536d8eab88c3ea01) #1 0x55a696972ab5 in ndpi_malloc /home/ivan/svnrepos/nDPI/src/lib/ndpi_memory.c:60:25 #2 0x55a696972da0 in ndpi_strdup /home/ivan/svnrepos/nDPI/src/lib/ndpi_memory.c:113:13 #3 0x55a696b7658d in processClientServerHello /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:2394:46 #4 0x55a696b86e81 in processTLSBlock /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:897:5 #5 0x55a696b80649 in ndpi_search_tls_udp /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:1262:11 #6 0x55a696b67a57 in ndpi_search_tls_wrapper /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:2751:5 #7 0x55a696b67758 in switch_to_tls /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:1408:3 #8 0x55a696c47810 in stun_search_again /home/ivan/svnrepos/nDPI/src/lib/protocols/stun.c:422:4 #9 0x55a6968a22af in ndpi_process_extra_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:7247:9 #10 0x55a6968acd6f in ndpi_internal_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:7746:5 #11 0x55a6968aba3f in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:8013:22 #12 0x55a69683d30e in packet_processing /home/ivan/svnrepos/nDPI/fuzz/../example/reader_util.c:1723:31 #13 0x55a69683d30e in ndpi_workflow_process_packet /home/ivan/svnrepos/nDPI/fuzz/../example/reader_util.c:2440:10 #14 0x55a69680f08f in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:135:7 [...] SUMMARY: AddressSanitizer: 16 byte(s) leaked in 1 allocation(s). ``` Found by oss-fuzzer See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=64564
* fuzz: extend fuzzing coverageNardi Ivan2023-10-15
|
* fuzz: extend coverage (#2073)Ivan Nardi2023-08-20
|
* fuzz: extend fuzzing coverage (#2040)Ivan Nardi2023-07-11
| | | | | | | | | Some notes: * libinjection: according to https://github.com/libinjection/libinjection/issues/44, it seems NULL characters are valid in the input string; * RTP: `rtp_get_stream_type()` is called only for RTP packets; if you want to tell RTP from RTCP you should use `is_rtp_or_rtcp()`; * TLS: unnecessary check; we already make the same check just above, at the beginning of the `while` loop
* ndpiReader: improve printing of payload statistics (#1989)Ivan Nardi2023-05-29
| | | | | Add a basic unit test Fix an endianess issue
* Add an heuristic to detect/ignore some anomalous TCP ACK packets (#1948)Ivan Nardi2023-04-25
| | | | | | | | | | | | | | | | | | | In some networks, there are some anomalous TCP flows where the smallest ACK packets have some kind of zero padding. It looks like the IP and TCP headers in those frames wrongly consider the 0x00 Ethernet padding bytes as part of the TCP payload. While this kind of packets is perfectly valid per-se, in some conditions they might be treated by the TCP reassembler logic as (partial) overlaps, deceiving the classification engine. Add an heuristic to detect these packets and to ignore them, allowing correct detection/classification. This heuristic is configurable. Default value: * in the library, it is disabled * in `ndpiReader` and in the fuzzers, it is enabled (to ease testing) Credit to @vel21ripn for the initial patch. Close #1946
* fuzz: extend fuzzers coverage (#1952)Ivan Nardi2023-04-25
|
* fuzz: add a new fuzzer triggering the payload analyzer function(s) (#1926)Ivan Nardi2023-04-04
|
* fuzz: extend fuzz coverage (#1888)Ivan Nardi2023-02-16
|
* fuzz: some improvements and add two new fuzzers (#1881)Ivan Nardi2023-02-09
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Remove `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` define from `fuzz/Makefile.am`; it is already included by the main configure script (when fuzzing). Add a knob to force disabling of AESNI optimizations: this way we can fuzz also no-aesni crypto code. Move CRC32 algorithm into the library. Add some fake traces to extend fuzzing coverage. Note that these traces are hand-made (via scapy/curl) and must not be used as "proof" that the dissectors are really able to identify this kind of traffic. Some small updates to some dissectors: CSGO: remove a wrong rule (never triggered, BTW). Any UDP packet starting with "VS01" will be classified as STEAM (see steam.c around line 111). Googling it, it seems right so. XBOX: XBOX only analyses UDP flows while HTTP only TCP ones; therefore that condition is false. RTP, STUN: removed useless "break"s Zattoo: `flow->zattoo_stage` is never set to any values greater or equal to 5, so these checks are never true. PPStream: `flow->l4.udp.ppstream_stage` is never read. Delete it. TeamSpeak: we check for `flow->packet_counter == 3` just above, so the following check `flow->packet_counter >= 3` is always false.
* fuzz: fix memory allocation failure logic (#1867)Ivan Nardi2023-01-20
| | | | | We *do* want to have some allocation errors. Fix some related bugs Fix: 29be01ef
* fuzz: add fuzzer testing nDPI (initial) configurations (#1830)Ivan Nardi2022-12-23
| | | | | | | | | | | | | | | | | The goal of this fuzzer is to test init and deinit of the library, with different configurations. In details: * random memory allocation failures, even during init phase * random `ndpi_init_prefs` parameter of `ndpi_init_detection_module()` * random LRU caches sizes * random bitmask of enabled protocols * random parameters of `ndpi_set_detection_preferences()` * random initialization of opportunistic TLS * random load/don't load of configuration files This new fuzzer is a C++ file, because it uses `FuzzedDataProvider` class (see https://github.com/google/fuzzing/blob/master/docs/split-inputs.md). Note that the (existing) fuzzers need to be linked with C++ compiler anyway, so this new fuzzer doesn't add any new requirements.
* fuzz: some enhancements (#1827)Ivan Nardi2022-12-10
| | | | | | | | | | Load some custom configuration (like in the unit tests) and factorize some (fuzzing) common code. There is no way to pass file paths to the fuzzers as parameters. The safe solution seems to be to load them from the process working dir. Anyway, missing file is not a blocking error. Remove some dead code (found looking at the coverage report)
* fuzz: fix signed-integer-overflow (#1822)Ivan Nardi2022-12-10
| | | | | | | | | | ``` fuzz_ndpi_reader.c:33:29: runtime error: signed integer overflow: 214013 * 24360337 cannot be represented in type 'int' #0 0x4c1cf7 in fastrand ndpi/fuzz/fuzz_ndpi_reader.c:33:29 #1 0x4c1cf7 in malloc_wrapper ndpi/fuzz/fuzz_ndpi_reader.c:38:11 #2 0x523057 in ndpi_malloc ndpi/src/lib/ndpi_main.c:220:25 ``` Found by oss-fuzz See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=54112
* fuzz: add a new fuzzer testing memory allocation failures (#1818)Ivan Nardi2022-12-06
| | | | | | | | | Try to fuzz error paths triggered by allocation errors. Fix some errors already found by this new fuzzer. Basic idea taken from: https://github.com/harfbuzz/harfbuzz/pull/2566/files `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` is a standard define used to (not)compile specific code in fuzzing builds. See: https://llvm.org/docs/LibFuzzer.html
* Add support for LTO and Gold linker (#1812)Ivan Nardi2022-12-05
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit add (optional) support for Link-Time-Optimization and Gold linker. This is the first, mandatory step needed to make nDPI compliant with "introspector" sanitizer requirements in OSS-Fuzz: see https://github.com/google/oss-fuzz/issues/8939 Gold linker is not supported by Windows and by macOS, so this feature is disabled by default. It has been enable in CI in two linux targets ("latest" gcc and clang). Fix some warnings triggered by LTO. The changes in `src/lib/ndpi_serializer.c` seams reasonable. However, the change in `tests/unit/unit.c` is due to the following warning, which seems to be a false positive. ``` unit.c: In function ‘serializerUnitTest’: ndpi_serializer.c:2258:13: error: ‘MEM[(struct ndpi_private_serializer *)&deserializer].buffer.size’ may be used uninitialized in this function [-Werror=maybe-uninitialized] unit.c:67:31: note: ‘MEM[(struct ndpi_private_serializer *)&deserializer].buffer.size’ was declared here 67 | ndpi_serializer serializer, deserializer; | ^ ndpi_serializer.c:2605:10: error: ‘MEM[(struct ndpi_private_serializer *)&deserializer].status.buffer.size_used’ may be used uninitialized in this function [-Werror=maybe-uninitialized] unit.c:67:31: note: ‘MEM[(struct ndpi_private_serializer *)&deserializer].status.buffer.size_used’ was declared here 67 | ndpi_serializer serializer, deserializer; ``` Since this warning is triggered only with an old version of gcc and `tests/unit/unit.c` is used only during the tests, the easiest fix has been applied. Some (unknown to me) combinations of OS and compiler trigger the following warnings at linker time (with sanitizer and gold linker) ``` /usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_load1_asm' /usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_load2_asm' /usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_load4_asm' /usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_load8_asm' /usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_load16_asm' /usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_store1_asm' /usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_store2_asm' /usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_store4_asm' [..] ``` I have not found any references to this kind of message, with the only exception of https://sourceware.org/bugzilla/show_bug.cgi?id=25975 which seems to suggest that these messages can be safely ignored. In any case, the compilation results are sound. Fix `clean` target in the Makefile in the `example` directory. In OSS-Fuzz enviroments, `fuzz_ndpi_reader` reports a strange link error (as always, when the gold linker is involved...). It's come out that the culprit was the `tempnam` function: the code has been changed to use `tmpfile` instead. No sure why... :( Fuzzing target `fuzz_ndpi_reader.c` doesn't use `libndpiReader.a` anymore: this way we can use `--with-only-libndpi` flag on Oss-Fuzz builds as workaround for the "missing dependencies errors" described in https://github.com/google/oss-fuzz/issues/8939
* Remove some useless fuzzing target (#1813)Ivan Nardi2022-12-01
| | | | | We don't need specific targets to reproduce fuzzing issues. After all, calling `./fuzz/fuzz_process_packet_with_main $ARTIFACT_FILE` is equivalento to `./fuzz/fuzz_process_packet $ARTIFACT_FILE`
* Add basic profiling of memory allocations on data-path (#1789)Ivan Nardi2022-10-28
| | | | | | | | | | | | | | | The goal is to have an idea of the memory allocation sizes performed in the **library data-path**, i.e. excluding init/deinit phases and all the allocations made by the application itself. In other words, how much memory is needed per-flow, by nDPI, other than `struct ndpi_flow_struct`? It works only on single-thread configurations. It is not enabled by default (in the unit tests) since different canfiguration options (example: `--enable-pcre`) lead to diffferent results. See: #1781
* Fix compilation and sync utests resultsNardi Ivan2022-10-04
|
* Improved nDPI JSON serialization. (#1689)Toni2022-08-02
| | | | | | | | * fixed autoconf CFLAGS/LDFLAGS MSAN issue which could lead to build errors * introduced portable version of gmtime_r aka ndpi_gmtime_r * do as most as possible of the serialization work in ndpi_utils.c * use flow2json in ndpiReader Signed-off-by: lns <matzeton@googlemail.com>
* Fix some debug messages (#1583)Ivan Nardi2022-06-05
| | | Increase max number of flows handled during fuzzing
* Replaced ndpiReader's libjson-c support with libnDPI's internal ↵Toni2022-05-07
| | | | | | | | | | | serialization interface. (#1535) * Fixes #1528 * Serialization Interface should also fuzzed * libjson-c may only be used in the unit test to verify the internal serialization interface * Serialization Interface supports tlv(broken), csv and json * Unit test does work again and requires libjson-c Signed-off-by: lns <matzeton@googlemail.com>
* fuzz: purge old sessions (#1451)Ivan Nardi2022-02-21
| | | | | | | | At every fuzz iteration (i.e for every trace file): * keep the same ndpi context (`ndpi_init_detection_module` is very slow); * reset the flow table, otherwise it grows indefinitely. This change should fix the "out-of-memory" errors reported by oss-fuzz.
* fuzz: make fuzz_ndpi_reader faster (#1446)Ivan Nardi2022-02-20
| | | | | | | | | | | | | | | Initialize ndpi_workflow_init context only once. On a quite old notebook, before: ``` $ ./fuzz/fuzz_ndpi_reader -max_total_time="${MAX_TOTAL_TIME:-360}" -print_pcs=1 -workers="${FUZZY_WORKERS:-0}" -jobs="${FUZZY_JOBS:-0}" ./tests/pcap/ [...] Done 3256 runs in 361 second(s) ``` after: ``` Done 5032771 runs in 361 second(s) <----------- ~1400X ``` oss-fuzz will be happy!
Powered by cgit, Themed by Ted Yin.