aboutsummaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAge
* Update all IP lists (#2515)Ivan Nardi2024-08-02
| | | | | The `suffix_id` is simply an incremental index (see `ndpi_load_domain_suffixes`), so its value might changes every time we update the public suffix list.
* Enhanced PrimeVideo detectionLuca Deri2024-07-30
|
* Enhanced ookla tracingLuca Deri2024-07-29
|
* Improved ICMP malformed packet risk descriptionLuca Deri2024-07-25
|
* FPC: add DPI information (#2514)Ivan Nardi2024-07-23
| | | | If the flow is classified (via DPI) after the first packet, we should use this information as FPC
* Add OpenWire support (#2513)Vladimir Gavrilov2024-07-22
|
* FPC: small improvements (#2512)Ivan Nardi2024-07-22
| | | | Add printing of fpc_dns statistics and add a general cconfiguration option. Rework the code to be more generic and ready to handle other logics.
* FPC: add DNS correlation (#2497)mmanoj2024-07-22
| | | | | | | | | Use DNS information to get a better First Packet Classification. See: #2322 --------- Co-authored-by: Nardi Ivan <nardi.ivan@gmail.com>
* Performed some grammar and typo fixes (#2511)Petr2024-07-19
|
* ipaddr2list.py, ndpi2timeline.py: reformatted (#2509)Petr2024-07-18
|
* ndpi_strncasestr: optimization, fixes, tests (#2507)Petr2024-07-18
|
* shell: reformatted, fixed inspections, typos (#2506)Petr2024-07-18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Reformatted shell scripts according to [ShellCheck](https://github.com/koalaman/shellcheck/). I. Most common changes: 1. https://github.com/koalaman/shellcheck/wiki/SC2086 `$var` → `"$var"` Note: this isn't always necessary and I've been careful not to substitute where it wasn't necessary in meaning. 2. https://github.com/koalaman/shellcheck/wiki/SC2006 `` `command` `` → `$(command)` 3. https://github.com/koalaman/shellcheck/wiki/SC2004 `$(( $a + $b ))` → `$(( a + b ))` 4. https://github.com/koalaman/shellcheck/wiki/SC2164 `cd "$dir"` → `cd "$dir" || exit 1` 5. https://github.com/koalaman/shellcheck/wiki/SC2166 `[ check1 -o check2 ]` → `[ check1 ] || [ check2 ]` 6. https://github.com/koalaman/shellcheck/wiki/SC2002 `cat "${file}" | wc -c` → `< "${file}" wc -c` Note: this looks a bit uglier but works faster. II. Some special changes: 1. In file `utils/common.sh`: https://github.com/koalaman/shellcheck/wiki/SC2112 This script is interpreted by `sh`, not by `bash`, but uses the keyword `function`. So I replaced `#!/usr/bin/env sh` to `#!/usr/bin/env bash`. 2. After that I thought of replacing all shebangs to `#!/usr/bin/env bash` for consistency and cross-platform compatibility, especially since most of the files already use bash. 3. But in cases when it was `#!/bin/sh -e` or `#!/bin/bash -eu` another problem appears: https://github.com/koalaman/shellcheck/wiki/SC2096 So I decided to make all shebangs look uniform: ``` #!/usr/bin/env bash set -e (or set -eu) (if needed) ``` 4. In file `tests/ossfuzz.sh`: https://github.com/koalaman/shellcheck/wiki/SC2162 `read i` → `read -r i` Note: I think that there is no need in special treatment for backslashes, but I could be wrong. 5. In file `tests/do.sh.in`: https://github.com/koalaman/shellcheck/wiki/SC2035 `ls *.*cap*` → `ls -- *.*cap*` 6. In file `utils/verify_dist_tarball.sh`: https://github.com/koalaman/shellcheck/wiki/SC2268 `[ "x${TARBALL}" = x ]` → `[ -z "${TARBALL}" ]` 7. In file `utils/check_symbols.sh`: https://github.com/koalaman/shellcheck/wiki/SC2221 `'[ndpi_utils.o]'|'[ndpi_memory.o]'|'[roaring.o]')` → `'[ndpi_utils.o]'|'[ndpi_memory.o]')` 8. In file `autogen.sh`: https://github.com/koalaman/shellcheck/wiki/SC2145 `echo "./configure $@"` → `echo "./configure $*"` https://github.com/koalaman/shellcheck/wiki/SC2068 `./configure $@` → `./configure "$@"` III. `LIST6_MERGED` and `LIST_MERGED6` There were typos with this variables in files `utils/aws_ip_addresses_download.sh`, `utils/aws_ip_addresses_download.sh` and `utils/microsoft_ip_addresses_download.sh` where variable `LIST6_MERGED` was defined, but `LIST_MERGED6` was removed by `rm`. I changed all `LIST_MERGED6` to `LIST6_MERGED`. Not all changes are absolutely necessary, but some may save you from future bugs.
* smpp: fix parsing of Generic Nack message (#2496)Ivan Nardi2024-07-18
|
* Add Nano (XNO) protocol support (#2508)Vladimir Gavrilov2024-07-18
|
* Added ClickHouse protocolLuca2024-07-17
|
* python: reformatted, fixed bugs (#2504)Petr2024-07-17
|
* .gitignore: reformatted, added patterns for IDEs, for deb packages and for ↵Petr2024-07-16
| | | | test results (#2503)
* Add HLS support (#2502)Vladimir Gavrilov2024-07-16
|
* Refactor ndpi_strnstr to use ndpi_memmem (#2500)Vladimir Gavrilov2024-07-15
|
* ndpi_memmem: optimized, fixed bug, added tests (#2499)Petr2024-07-15
|
* Optimize performance of ndpi_strnstr() and possible bugfix (#2494)Petr2024-07-15
|
* Fixed mistake in shebang (SC1113) (#2498)Petr2024-07-15
|
* fuzzing: improve coverage (#2495)Ivan Nardi2024-07-12
| | | | | | | | | | | | | | Fix detection of WebDAV and Gnutella (over HTTP) Fix detection of z3950 Add two fuzzers to test `ndpi_memmem()` and `ndpi_strnstr()` Remove some dead code: * RTP: the same exact check is performed at the very beginning of the function * MQTT: use a better helper to exclude the protocol * Colletd: `ndpi_hostname_sni_set()` never fails Update pl7m code (fix a Use-of-uninitialized-value error)
* Improve detection of Cloudflare WARP traffic (#2491)Ivan Nardi2024-07-04
| | | See: #2484
* Add infrastructure for explicit support of Fist Packet Classification (#2488)Ivan Nardi2024-07-03
| | | | | Let's start with some basic helpers and with FPC based on flow addresses. See: #2322
* Minor fix in CI action (#2489)Ivan Nardi2024-07-03
|
* Reduce snaplen of some traces (#2490)Ivan Nardi2024-07-03
| | | | | | To avoid the following error with some old libpcap versions: ``` ERROR: could not open pcap file: invalid file capture length 524288, bigger than maximum of 262144 ```
* Add detection of Twitter bot (#2487)Ivan Nardi2024-07-03
| | | Update the global list of crawlers ips
* Make the CI faster (#2475)Ivan Nardi2024-07-01
| | | | | | | | | | | | | | | | | | | | | | | | | | | Without the `-fsanitize-memory-track-origins` flag, MSAN job is ~30% faster. Since this flag is useful only while debugging (and not to simply discover memory issues), avoid it on the CI. Note that, by default it is still enabled by default. Right now, MingW runs on *every* ubuntu builds: limit it only to the standard matrix (i.e. ubuntu 20.04, 22.04, 24.04 with default configuration), without any sanitizers (note that MingW doesn't support *san anyway). armhf job is by far the longest job in the CI: remove asan configuration to make it faster. Note that we already have a lot of different jobs (on x86_64) with some sanitizers, and that the other 2 jobs on arm/s390x don't have asan support anyway. If we really, really want a job with arm + asan we can add it as a async/scheduled job. Remove an old workaround for ubuntu jobs Avoid installing packages needed only for the documentation About `check_symbols.sh` script: even if uses the compiled library/objects, it basicaly only checks if we are using, in the source code, same functions that we shoudn't. We don't need to perform the same kind of check so many times..
* Modified separator from , (comma) to | (pipe) as some fields such as the ↵Luca Deri2024-07-01
| | | | HTTP user agent as sometimes they contain commas and create parsing problems
* tunnelbear: improve detection over wireguard (#2485)Ivan Nardi2024-07-01
| | | See #2484
* Improve detection of Twitter/X (#2482)Ivan Nardi2024-07-01
|
* Add detection of OpenAI ChatGPT bots (#2481)Ivan Nardi2024-07-01
|
* fuzz: improve fuzzers using pl7m (#2486)Ivan Nardi2024-06-29
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | For some unclear reasons, fuzzers using pl7m create huge corpus, triggering OOM in oss-fuzz runs (where the memory RSS limit is set to 2560Mb). Example: ``` ==25340== ERROR: libFuzzer: out-of-memory (used: 2564Mb; limit: 2560Mb) To change the out-of-memory limit use -rss_limit_mb=<N> Live Heap Allocations: 2364004039 bytes in 133791 chunks; quarantined: 60662293 bytes in 3664 chunks; 176432 other chunks; total chunks: 313887; showing top 95% (at most 8 unique contexts) 1285841683 byte(s) (54%) in 2956 allocation(s) #0 0x56f814ef4bde in __interceptor_malloc /src/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:69:3 #1 0x56f814e04416 in operator new(unsigned long) cxa_noexception.cpp:0 #2 0x56f814de6b2d in assign<unsigned char *, 0> /work/llvm-stage2/runtimes/runtimes-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64/include/c++/v1/vector:1443:3 #3 0x56f814de6b2d in operator= /work/llvm-stage2/runtimes/runtimes-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64/include/c++/v1/vector:1412:9 #4 0x56f814de6b2d in fuzzer::InputCorpus::AddToCorpus(std::__Fuzzer::vector<unsigned char, std::__Fuzzer::allocator<unsigned char>> const&, unsigned long, bool, bool, bool, std::__Fuzzer::chrono::duration<long long, std::__Fuzzer::ratio<1l, 1000000l>>, std::__Fuzzer::vector<unsigned int, std::__Fuzzer::allocator<unsigned int>> const&, fuzzer::DataFlowTrace const&, fuzzer::InputInfo const*) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerCorpus.h:221:10 #5 0x56f814de60e5 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool, bool*) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:539:16 #6 0x56f814de7df2 in fuzzer::Fuzzer::ReadAndExecuteSeedCorpora(std::__Fuzzer::vector<fuzzer::SizedFile, std::__Fuzzer::allocator<fuzzer::SizedFile>>&) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:829:7 #7 0x56f814de8127 in fuzzer::Fuzzer::Loop(std::__Fuzzer::vector<fuzzer::SizedFile, std::__Fuzzer::allocator<fuzzer::SizedFile>>&) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:867:3 #8 0x56f814dd6736 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:914:6 #9 0x56f814e02c62 in main /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10 #10 0x7fa11e2c3082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/libc-start.c:308:16 1031350683 byte(s) (43%) in 2468 allocation(s) #0 0x56f814ef4bde in __interceptor_malloc /src/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:69:3 #1 0x56f814e04416 in operator new(unsigned long) cxa_noexception.cpp:0 #2 0x56f814de6b2d in assign<unsigned char *, 0> /work/llvm-stage2/runtimes/runtimes-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64/include/c++/v1/vector:1443:3 #3 0x56f814de6b2d in operator= /work/llvm-stage2/runtimes/runtimes-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64/include/c++/v1/vector:1412:9 #4 0x56f814de6b2d in fuzzer::InputCorpus::AddToCorpus(std::__Fuzzer::vector<unsigned char, std::__Fuzzer::allocator<unsigned char>> const&, unsigned long, bool, bool, bool, std::__Fuzzer::chrono::duration<long long, std::__Fuzzer::ratio<1l, 1000000l>>, std::__Fuzzer::vector<unsigned int, std::__Fuzzer::allocator<unsigned int>> const&, fuzzer::DataFlowTrace const&, fuzzer::InputInfo const*) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerCorpus.h:221:10 #5 0x56f814de60e5 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool, bool*) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:539:16 #6 0x56f814de7635 in fuzzer::Fuzzer::MutateAndTestOne() /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:760:19 #7 0x56f814de8425 in fuzzer::Fuzzer::Loop(std::__Fuzzer::vector<fuzzer::SizedFile, std::__Fuzzer::allocator<fuzzer::SizedFile>>&) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:905:5 #8 0x56f814dd6736 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:914:6 #9 0x56f814e02c62 in main /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10 #10 0x7fa11e2c3082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/libc-start.c:308:16 ``` See: https://oss-fuzz.com/testcase-detail/4717811415449600 See: https://oss-fuzz.com/testcase-detail/6164130982068224 Let's *try* the following workaround: set the parameter `-max-len` to 512K, to force the engine to not genereate inputs (i.e. pcap files...) larger than 512K. Right now the value used is 1MB, i.e the default, because we have file larger than 1MB in the initial seeds (i.e. `/tests/pcaps/*`). Let's hope than having smaller files lead to smaller corpus... Update pl7m code (fix a Use-of-uninitialized-value error)
* fuzz: pl7m: add a custom mutator for better fuzzing of pcap files (#2483)Ivan Nardi2024-06-27
| | | | | | | | | | | Pl7m is a custom mutator (used for structure aware fuzzing) for network traffic packet captures (i.e. pcap files). The output of the mutator is always a valid pcap file, containing the same flows/sessions of the input file. That's it: the mutator only changes the packet payload after the TCP/UDP header, keeping all the original L2/L3 information (IP addresses and L4 ports). See: https://github.com/IvanNardi/pl7m
* Remove duplicate API for `ndpi_serialize_uint32_double` in ndpi_api.h (#2480)Anshul Thakur2024-06-26
| | | Co-authored-by: Anshul Thakur <athakur@cdot.in>
* wireshark: lua: export some metadataNardi Ivan2024-06-25
| | | | | | | | | | | | | | | Export some metadata (for the moment, SNI and TLS fingerprints) to Wireshark/tshark via extcap. Note that: * metadata are exported only once per flow * metadata are exported (all together) when nDPI stopped processing the flow Still room for a lot of improvements! In particular: * we need to add some boundary checks (if we are going to export other attributes) * we should try to have a variable length trailer
* wireshark: lua: filtering and trailer dissection work with tshark, tooNardi Ivan2024-06-25
| | | | | | | | | | | | | ``` ivan@ivan-Latitude-E6540:~/$ tshark -C "nDPI extcap" -i ndpi -o extcap.ndpi.i:/home/ivan/svnrepos/nDPI/tests/pcap/anydesk.pcapng -Y "ndpi.protocol.name contains DNS" Capturing on 'nDPI interface: ndpi' 62 22635386.425683 192.168.1.187 DNS.AnyDesk 192.168.1.1 128 Standard query 0xec22 A relay-3185a847.net.anydesk.com 63 22635386.439540 192.168.1.1 DNS.AnyDesk 192.168.1.187 144 Standard query response 0xec22 A relay-3185a847.net.anydesk.com A 37.61.223.15 64 22635386.721277 192.168.1.187 DNS.AnyDesk 192.168.1.1 128 Standard query 0xea89 A relay-9b6827f2.net.anydesk.com 65 22635386.732444 192.168.1.1 DNS.AnyDesk 192.168.1.187 144 Standard query response 0xea89 A relay-9b6827f2.net.anydesk.com A 138.199.36.115 4 packets captured ```
* wireshark: lua: fix DNS dissectionNardi Ivan2024-06-25
| | | | | Not sure when we (or Wireshark, or Lua...) broke it, but we can't call tonumber() on Bool variables.
* wireshark: extcap: restore filtering mechanismNardi Ivan2024-06-25
|
* wireshark: lua: latest Wireshark versions correctly handle 64 bit maskNardi Ivan2024-06-25
|
* wireshark: lua: minor improvementsNardi Ivan2024-06-25
| | | | | * Use a proper TVB to parse the nDPI trailer * Fix some flow risks definitions
* Improved logic for checking invalid DNS queriesLuca Deri2024-06-23
|
* Added default port mappings to ndpiReader help -H (#2477)Mark Jeffery2024-06-19
| | | Close #2125
* Zoom: fix detection of screen sharing (#2476)Ivan Nardi2024-06-17
|
* fuzz: improve fuzzing coverage (#2474)Ivan Nardi2024-06-17
| | | | | | Remove some code never triggered AFP: the removed check is included in the following one MQTT: fix flags extraction
* Sync unit tests resultsNardi Ivan2024-06-17
|
* Add Ripe Atlas probe protocol. (#2473)Toni2024-06-17
| | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Zoom: harden RTP/RTCP detectionNardi Ivan2024-06-17
|
* Zoom: remove "stun_zoom" LRU cacheNardi Ivan2024-06-17
| | | | | Since 070a0908b we are able to detect P2P calls directly from the packet content, without any correlation among flows
Powered by cgit, Themed by Ted Yin.