aboutsummaryrefslogtreecommitdiff
path: root/fuzz
Commit message (Collapse)AuthorAge
* fuzz: extend fuzzing coverage (#2696)Ivan Nardi2025-01-23
|
* TLS: remove JA3C (#2679)Ivan Nardi2025-01-14
| | | | | | | | Last step of removing JA3C fingerprint Remove some duplicate tests: testing with ja4c/ja3s disabled is already performed by `disable_metadata_and_flowrisks` configuration. Close:#2551
* Add (kind of) support for loading a list of JA4C malicious fingerprints (#2678)Ivan Nardi2025-01-14
| | | | | | | | | It might be usefull to be able to match traffic against a list of suspicious JA4C fingerprints Use the same code/logic/infrastructure used for JA3C (note that we are going to remove JA3C...) See: #2551
* fuzz: add 2 new fuzzers for KD-trees and Ball-trees (#2670)Ivan Nardi2025-01-13
|
* Add the ability to enable/disable every specific flow risks (#2653)Ivan Nardi2025-01-06
|
* fuzz: improve fuzzing coverage (#2642)Ivan Nardi2024-12-11
| | | Updtae pl7m code (Fix swap-direction mutation)
* Update `flow->flow_multimedia_types` to a bitmask (#2625)Ivan Nardi2024-11-25
| | | In the same flow, we can have multiple multimedia types
* fuzz: extend fuzzing coverage (#2626)Ivan Nardi2024-11-20
|
* SIP: extract some basic metadataIvan Nardi2024-11-12
|
* Unify ndpi debug logging to always use a u16 protocol id (#2613)Toni2024-11-11
| | | | | * fixes SonarCloud complaint Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* fuzz: improve coverage (#2612)Ivan Nardi2024-11-01
| | | Add fuzzer to test `ndpi_quick_encrypt()` and `ndpi_quick_decrypt()`
* Add configuration of TCP fingerprint computation (#2598)Ivan Nardi2024-10-18
| | | Extend configuration of raw format of JA4C fingerprint
* Add monitoring capability (#2588)Ivan Nardi2024-10-14
| | | | | | | | | | | | | Allow nDPI to process the entire flows and not only the first N packets. Usefull when the application is interested in some metadata spanning the entire life of the session. As initial step, only STUN flows can be put in monitoring. See `doc/monitoring.md` for further details. This feature is disabled by default. Close #2583
* fuzz: fix fuzzing (#2586)Ivan Nardi2024-10-10
|
* Added addr_dump_path definitionLuca Deri2024-10-10
|
* fuzz: try to be a little bit faster (#2578)Ivan Nardi2024-09-30
| | | See: 9d07cf281
* Add enable/disable guessing using client IP/port (#2569)Liam Wilson2024-09-27
| | | | | | | | Add configurable options for whether to include client port or client IP in the flow's protocol guesses. This defaults to include both client port/IP if the protocol is not guessed with the server IP/port. This is intended for when flow direction detection is enabled, so we know that sport = client port, dport = server port.
* Add some heuristics to detect encrypted/obfuscated/proxied TLS flows (#2553)Ivan Nardi2024-09-24
| | | | | | | | | | | | Based on the paper: "Fingerprinting Obfuscated Proxy Traffic with Encapsulated TLS Handshakes". See: https://www.usenix.org/conference/usenixsecurity24/presentation/xue-fingerprinting Basic idea: * the packets/bytes distribution of a TLS handshake is quite unique * this fingerprint is still detectable if the handshake is encrypted/proxied/obfuscated All heuristics are disabled by default.
* Allow IP guess before port in ndpi_detection_giveup (#2562)Liam Wilson2024-09-20
| | | | Add dpi.guess_ip_before_port which when enabled uses classification by-ip before classification by-port.
* fuzz: fix compilationNardi Ivan2024-09-17
|
* fuzz: try to be a little bit faster (#2559)Ivan Nardi2024-09-17
| | | | | | | Some fuzzers don't really need a real and complete local context. Try to avoid setting it up, creating a simpler fake version with only the features really needed. That is a kind of experiment: if it works, we can extend the same logic to other fuzzers
* Add an heuristic to detect encrypted/obfuscated OpenVPN flows (#2547)Ivan Nardi2024-09-16
| | | | | | | | | | | | Based on the paper: "OpenVPN is Open to VPN Fingerprinting" See: https://www.usenix.org/conference/usenixsecurity22/presentation/xue-diwen Basic idea: * the distribution of the first byte of the messages (i.e. the distribution of the op-codes) is quite unique * this fingerprint might be still detectable even if the OpenVPN packets are somehow fully encrypted/obfuscated The heuristic is disabled by default.
* fuzz: fix compilationNardi Ivan2024-09-16
|
* OpenVPN, Wireguard: improve sub-classificationNardi Ivan2024-09-05
| | | | | | | | Allow sub-classification of OpenVPN/Wireguard flows using their server IP. That is useful to detect the specific VPN application/app used. At the moment, the supported protocols are: Mullvad, NordVPN, ProtonVPN. This feature is configurable.
* fuzz: improve fuzzing coverage (#2535)Ivan Nardi2024-09-03
| | | | Updtae pl7m code (fix a Use-of-uninitialized-value error and add GTP support)
* HTTP, QUIC, TLS: allow to disable sub-classification (#2533)Ivan Nardi2024-09-03
|
* fuzz: fix compilation (#2532)Ivan Nardi2024-08-26
|
* Endian-independent implementation of IEEE 802.3 CRC32 (#2529)Vladimir Gavrilov2024-08-25
|
* Compilation fixesLuca Deri2024-08-24
|
* Fix `verify_dist_tarball.sh` after latest release (#2519)Ivan Nardi2024-08-07
| | | | | | | | | | Moving from 4.8 to 4.10 (and so, from 4.9 to 4.11 for development builds) made some paths one character longer; that triggers an error with tar when running `verify_dist_tarball.sh` script: ``` tar: libndpi-4.11.0/fuzz/corpus/fuzz_filecfg_config/flow_risk.anonymous_subscriber.list.protonvpn.load.txt: file name is too long (max 99); not dumped ``` As a quick fix, reduce the length of that file name.
* FPC: small improvements (#2512)Ivan Nardi2024-07-22
| | | | Add printing of fpc_dns statistics and add a general cconfiguration option. Rework the code to be more generic and ready to handle other logics.
* fuzzing: improve coverage (#2495)Ivan Nardi2024-07-12
| | | | | | | | | | | | | | Fix detection of WebDAV and Gnutella (over HTTP) Fix detection of z3950 Add two fuzzers to test `ndpi_memmem()` and `ndpi_strnstr()` Remove some dead code: * RTP: the same exact check is performed at the very beginning of the function * MQTT: use a better helper to exclude the protocol * Colletd: `ndpi_hostname_sni_set()` never fails Update pl7m code (fix a Use-of-uninitialized-value error)
* Add infrastructure for explicit support of Fist Packet Classification (#2488)Ivan Nardi2024-07-03
| | | | | Let's start with some basic helpers and with FPC based on flow addresses. See: #2322
* fuzz: improve fuzzers using pl7m (#2486)Ivan Nardi2024-06-29
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | For some unclear reasons, fuzzers using pl7m create huge corpus, triggering OOM in oss-fuzz runs (where the memory RSS limit is set to 2560Mb). Example: ``` ==25340== ERROR: libFuzzer: out-of-memory (used: 2564Mb; limit: 2560Mb) To change the out-of-memory limit use -rss_limit_mb=<N> Live Heap Allocations: 2364004039 bytes in 133791 chunks; quarantined: 60662293 bytes in 3664 chunks; 176432 other chunks; total chunks: 313887; showing top 95% (at most 8 unique contexts) 1285841683 byte(s) (54%) in 2956 allocation(s) #0 0x56f814ef4bde in __interceptor_malloc /src/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:69:3 #1 0x56f814e04416 in operator new(unsigned long) cxa_noexception.cpp:0 #2 0x56f814de6b2d in assign<unsigned char *, 0> /work/llvm-stage2/runtimes/runtimes-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64/include/c++/v1/vector:1443:3 #3 0x56f814de6b2d in operator= /work/llvm-stage2/runtimes/runtimes-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64/include/c++/v1/vector:1412:9 #4 0x56f814de6b2d in fuzzer::InputCorpus::AddToCorpus(std::__Fuzzer::vector<unsigned char, std::__Fuzzer::allocator<unsigned char>> const&, unsigned long, bool, bool, bool, std::__Fuzzer::chrono::duration<long long, std::__Fuzzer::ratio<1l, 1000000l>>, std::__Fuzzer::vector<unsigned int, std::__Fuzzer::allocator<unsigned int>> const&, fuzzer::DataFlowTrace const&, fuzzer::InputInfo const*) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerCorpus.h:221:10 #5 0x56f814de60e5 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool, bool*) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:539:16 #6 0x56f814de7df2 in fuzzer::Fuzzer::ReadAndExecuteSeedCorpora(std::__Fuzzer::vector<fuzzer::SizedFile, std::__Fuzzer::allocator<fuzzer::SizedFile>>&) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:829:7 #7 0x56f814de8127 in fuzzer::Fuzzer::Loop(std::__Fuzzer::vector<fuzzer::SizedFile, std::__Fuzzer::allocator<fuzzer::SizedFile>>&) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:867:3 #8 0x56f814dd6736 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:914:6 #9 0x56f814e02c62 in main /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10 #10 0x7fa11e2c3082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/libc-start.c:308:16 1031350683 byte(s) (43%) in 2468 allocation(s) #0 0x56f814ef4bde in __interceptor_malloc /src/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:69:3 #1 0x56f814e04416 in operator new(unsigned long) cxa_noexception.cpp:0 #2 0x56f814de6b2d in assign<unsigned char *, 0> /work/llvm-stage2/runtimes/runtimes-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64/include/c++/v1/vector:1443:3 #3 0x56f814de6b2d in operator= /work/llvm-stage2/runtimes/runtimes-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64/include/c++/v1/vector:1412:9 #4 0x56f814de6b2d in fuzzer::InputCorpus::AddToCorpus(std::__Fuzzer::vector<unsigned char, std::__Fuzzer::allocator<unsigned char>> const&, unsigned long, bool, bool, bool, std::__Fuzzer::chrono::duration<long long, std::__Fuzzer::ratio<1l, 1000000l>>, std::__Fuzzer::vector<unsigned int, std::__Fuzzer::allocator<unsigned int>> const&, fuzzer::DataFlowTrace const&, fuzzer::InputInfo const*) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerCorpus.h:221:10 #5 0x56f814de60e5 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool, bool*) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:539:16 #6 0x56f814de7635 in fuzzer::Fuzzer::MutateAndTestOne() /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:760:19 #7 0x56f814de8425 in fuzzer::Fuzzer::Loop(std::__Fuzzer::vector<fuzzer::SizedFile, std::__Fuzzer::allocator<fuzzer::SizedFile>>&) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:905:5 #8 0x56f814dd6736 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:914:6 #9 0x56f814e02c62 in main /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10 #10 0x7fa11e2c3082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/libc-start.c:308:16 ``` See: https://oss-fuzz.com/testcase-detail/4717811415449600 See: https://oss-fuzz.com/testcase-detail/6164130982068224 Let's *try* the following workaround: set the parameter `-max-len` to 512K, to force the engine to not genereate inputs (i.e. pcap files...) larger than 512K. Right now the value used is 1MB, i.e the default, because we have file larger than 1MB in the initial seeds (i.e. `/tests/pcaps/*`). Let's hope than having smaller files lead to smaller corpus... Update pl7m code (fix a Use-of-uninitialized-value error)
* fuzz: pl7m: add a custom mutator for better fuzzing of pcap files (#2483)Ivan Nardi2024-06-27
| | | | | | | | | | | Pl7m is a custom mutator (used for structure aware fuzzing) for network traffic packet captures (i.e. pcap files). The output of the mutator is always a valid pcap file, containing the same flows/sessions of the input file. That's it: the mutator only changes the packet payload after the TCP/UDP header, keeping all the original L2/L3 information (IP addresses and L4 ports). See: https://github.com/IvanNardi/pl7m
* wireshark: lua: export some metadataNardi Ivan2024-06-25
| | | | | | | | | | | | | | | Export some metadata (for the moment, SNI and TLS fingerprints) to Wireshark/tshark via extcap. Note that: * metadata are exported only once per flow * metadata are exported (all together) when nDPI stopped processing the flow Still room for a lot of improvements! In particular: * we need to add some boundary checks (if we are going to export other attributes) * we should try to have a variable length trailer
* fuzz: improve fuzzing coverage (#2474)Ivan Nardi2024-06-17
| | | | | | Remove some code never triggered AFP: the removed check is included in the following one MQTT: fix flags extraction
* Zoom: remove "stun_zoom" LRU cacheNardi Ivan2024-06-17
| | | | | Since 070a0908b we are able to detect P2P calls directly from the packet content, without any correlation among flows
* RTP/STUN: look for STUN packets after RTP/RTCP classification (#2465)Ivan Nardi2024-06-07
| | | | | | | | | | After a flow has been classified as RTP or RTCP, nDPI might analyse more packets to look for STUN/DTLS packets, i.e. to try to tell if this flow is a "pure" RTP/RTCP flow or if the RTP/RTCP packets are multiplexed with STUN/DTLS. Useful for proper (sub)classification when the beginning of the flows are not captured or if there are lost packets in the the captured traffic. Disabled by default
* Zoom: faster detection of P2P flows (#2467)Ivan Nardi2024-06-07
|
* Small fixes after API cleanup done in c63446e59 (#2449)Ivan Nardi2024-05-20
|
* Fix/improve fuzzing (#2426)Ivan Nardi2024-05-08
|
* Remove "zoom" cache (#2420)Ivan Nardi2024-05-06
| | | | | | | | | This cache was added in b6b4967aa, when there was no real Zoom support. With 63f349319, a proper identification of multimedia stream has been added, making this cache quite useless: any improvements on Zoom classification should be properly done in Zoom dissector. Tested for some months with a few 10Gbits links of residential traffic: the cache pretty much never returned a valid hit.
* fuzz: fix shoco fuzzer (#2405)Ivan Nardi2024-04-22
| | | | | | | | | | | | | | | | | | | | | | | | ``` ==22779==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7f0900701020 at pc 0x555bcd2a6f02 bp 0x7ffe3ba5e790 sp 0x7ffe3ba5e788 READ of size 1 at 0x7f0900701020 thread T0 #0 0x555bcd2a6f01 in shoco_decompress /home/ivan/svnrepos/nDPI/src/lib/third_party/src/shoco.c:184:26 #1 0x555bcd2a4018 in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_alg_shoco.cpp:18:3 #2 0x555bcd1aa816 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) (/home/ivan/svnrepos/nDPI/fuzz/fuzz_alg_shoco+0x4f816) (BuildId: c54d1c32163c9937e06f62127348ae6bd26d9309) #3 0x555bcd193be8 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) (/home/ivan/svnrepos/nDPI/fuzz/fuzz_alg_shoco+0x38be8) (BuildId: c54d1c32163c9937e06f62127348ae6bd26d9309) #4 0x555bcd1996fa in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (/home/ivan/svnrepos/nDPI/fuzz/fuzz_alg_shoco+0x3e6fa) (BuildId: c54d1c32163c9937e06f62127348ae6bd26d9309) #5 0x555bcd1c3c92 in main (/home/ivan/svnrepos/nDPI/fuzz/fuzz_alg_shoco+0x68c92) (BuildId: c54d1c32163c9937e06f62127348ae6bd26d9309) #6 0x7f090257a082 in __libc_start_main /build/glibc-e2p3jK/glibc-2.31/csu/../csu/libc-start.c:308:16 #7 0x555bcd18e96d in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_alg_shoco+0x3396d) (BuildId: c54d1c32163c9937e06f62127348ae6bd26d9309) Address 0x7f0900701020 is located in stack of thread T0 at offset 4128 in frame #0 0x555bcd2a3d97 in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_alg_shoco.cpp:5 This frame has 4 object(s): [32, 4128) 'out' (line 9) <== Memory access at offset 4128 overflows this variable [4256, 8352) 'orig' (line 9) ``` Found by oss-fuzzer. See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=68211
* fuzz: improvements (#2400)Ivan Nardi2024-04-20
| | | | | Create the zip file with all the traces only once. Add a new fuzzer to test "shoco" compression algorithm
* Domain Classification Improvements (#2396)Luca Deri2024-04-18
| | | | | | | | | | | | | | | | | | | * Added size_t ndpi_compress_str(const char * in, size_t len, char * out, size_t bufsize); size_t ndpi_decompress_str(const char * in, size_t len, char * out, size_t bufsize); used to compress short strings such as domain names. This code is based on https://github.com/Ed-von-Schleck/shoco * Major code rewrite for ndpi_hash and ndpi_domain_classify * Improvements to make sure custom categories are loaded and enabled * Fixed string encoding * Extended SalesForce/Cloudflare domains list
* STUN: fix boundary checks on attribute list parsing (#2387)Ivan Nardi2024-04-12
| | | | | Restore all unit tests. Add some configuration knobs. Fix the endianess.
* STUN: improve extraction of Mapped-Address metadata (#2370)Ivan Nardi2024-04-08
| | | | | | | | | | | | | Enable parsing of Mapped-Address attribute for all STUN flows: that means that STUN classification might require more packets. Add a configuration knob to enable/disable this feature. Note that we can have (any) STUN metadata also for flows *not* classified as STUN (because of DTLS). Add support for ipv6. Restore the correct extra dissection logic for Telegram flows.
* fuzzing: extend fuzzing coverage (#2371)Ivan Nardi2024-04-05
|
* Disable `-Wno-unused-parameter -Wno-unused-function`. (#2358)Toni2024-04-03
| | | | | * unused parameters and functions pollute the code and decrease readability Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
Powered by cgit, Themed by Ted Yin.