aboutsummaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAge
...
* fuzz: add a new fuzzer testing memory allocation failures (#1818)Ivan Nardi2022-12-06
| | | | | | | | | Try to fuzz error paths triggered by allocation errors. Fix some errors already found by this new fuzzer. Basic idea taken from: https://github.com/harfbuzz/harfbuzz/pull/2566/files `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` is a standard define used to (not)compile specific code in fuzzing builds. See: https://llvm.org/docs/LibFuzzer.html
* Fix compilation (#1819)Ivan Nardi2022-12-05
|
* Updated test resultsLuca Deri2022-12-05
|
* Exported HTTP server in metadataLuca Deri2022-12-05
|
* thread_index may by negative. (#1814)Gowa20172022-12-05
| | | | | | | | * thread_index may by negative. Like: 192.168.8.155:55848 --> 183.3.224.139 * reader thread index also need to uint32
* Add support for LTO and Gold linker (#1812)Ivan Nardi2022-12-05
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit add (optional) support for Link-Time-Optimization and Gold linker. This is the first, mandatory step needed to make nDPI compliant with "introspector" sanitizer requirements in OSS-Fuzz: see https://github.com/google/oss-fuzz/issues/8939 Gold linker is not supported by Windows and by macOS, so this feature is disabled by default. It has been enable in CI in two linux targets ("latest" gcc and clang). Fix some warnings triggered by LTO. The changes in `src/lib/ndpi_serializer.c` seams reasonable. However, the change in `tests/unit/unit.c` is due to the following warning, which seems to be a false positive. ``` unit.c: In function ‘serializerUnitTest’: ndpi_serializer.c:2258:13: error: ‘MEM[(struct ndpi_private_serializer *)&deserializer].buffer.size’ may be used uninitialized in this function [-Werror=maybe-uninitialized] unit.c:67:31: note: ‘MEM[(struct ndpi_private_serializer *)&deserializer].buffer.size’ was declared here 67 | ndpi_serializer serializer, deserializer; | ^ ndpi_serializer.c:2605:10: error: ‘MEM[(struct ndpi_private_serializer *)&deserializer].status.buffer.size_used’ may be used uninitialized in this function [-Werror=maybe-uninitialized] unit.c:67:31: note: ‘MEM[(struct ndpi_private_serializer *)&deserializer].status.buffer.size_used’ was declared here 67 | ndpi_serializer serializer, deserializer; ``` Since this warning is triggered only with an old version of gcc and `tests/unit/unit.c` is used only during the tests, the easiest fix has been applied. Some (unknown to me) combinations of OS and compiler trigger the following warnings at linker time (with sanitizer and gold linker) ``` /usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_load1_asm' /usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_load2_asm' /usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_load4_asm' /usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_load8_asm' /usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_load16_asm' /usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_store1_asm' /usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_store2_asm' /usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_store4_asm' [..] ``` I have not found any references to this kind of message, with the only exception of https://sourceware.org/bugzilla/show_bug.cgi?id=25975 which seems to suggest that these messages can be safely ignored. In any case, the compilation results are sound. Fix `clean` target in the Makefile in the `example` directory. In OSS-Fuzz enviroments, `fuzz_ndpi_reader` reports a strange link error (as always, when the gold linker is involved...). It's come out that the culprit was the `tempnam` function: the code has been changed to use `tmpfile` instead. No sure why... :( Fuzzing target `fuzz_ndpi_reader.c` doesn't use `libndpiReader.a` anymore: this way we can use `--with-only-libndpi` flag on Oss-Fuzz builds as workaround for the "missing dependencies errors" described in https://github.com/google/oss-fuzz/issues/8939
* CI: GitHub is updating Ubuntu runners (#1817)Ivan Nardi2022-12-03
| | | | | | | | | | | GitHub is moving `ubuntu-latest` to `ubuntu-22.04`: update our dependencies. See: https://github.blog/changelog/2022-11-09-github-actions-ubuntu-latest-workflows-will-use-ubuntu-22-04/ This is the reason of the recent random failures in CI. Update "newest" tested gcc to gcc-12. Fix a memory error introduced in 557bbcfc5a5165c9eb43bbdd78435796239cd3c9
* Make LRU caches ipv6 aware (#1810)Ivan Nardi2022-12-03
| | | | | | | | | | | | | Simplest solution, keeping the existing cache data structure TLS certificate cache is used for DTLS traffic, too. Note that Ookla cache already works with ipv6 flows. TODO: * make the key/hashing more robust (extending the key size?) * update bittorrent cache too. That task is quite difficult because ntopng uses a public function (`ndpi_guess_undetected_protocol()`) intrinsically ipv4 only...
* Updated tests resultsLuca Deri2022-12-03
|
* Added check for empty/missing user-agent in HTTPLuca Deri2022-12-03
|
* Improved HTTP header parsing skipping lines that won't matchLuca Deri2022-12-03
|
* Extend CodeQL configuration and fix two warnings (#1816)Ivan Nardi2022-12-02
| | | | | | | | | ``` Call to `memset` may be deleted Comparison of narrow type with wide type in loop condition ``` See https://www.usenix.org/system/files/conference/usenixsecurity17/sec17-yang.pdf for the new portable function `secure_memzero`.
* Add CodeQL workflow (#1815)Alvaro Muñoz2022-12-02
|
* Remove some useless fuzzing target (#1813)Ivan Nardi2022-12-01
| | | | | We don't need specific targets to reproduce fuzzing issues. After all, calling `./fuzz/fuzz_process_packet_with_main $ARTIFACT_FILE` is equivalento to `./fuzz/fuzz_process_packet $ARTIFACT_FILE`
* Add a new fuzzer for QUIC (#1800)Ivan Nardi2022-12-01
| | | | | | | | | | | | QUIC packets are encrypted/obfuscated; that means that we need to decrypt them before parsing the real (TLS) message. Fuzzing is not effective here, since a random buffer is hardly a valid encrypted QUIC packet. Add a new fuzzer, testing *decrypted* QUIC packets. Add a basic corpus. Fix a few bugs already found by this fuzzer.
* Fix classification "by-port" of QUIC flows (#1785)Ivan Nardi2022-11-22
| | | | | | | | | nDPI is able to properly classify QUIC flows only if it elaborates the very first packets of the flow. The protocol list in `is_udp_guessable_protocol()` is basically a list of protocols which can be detected from *any* packets in the flow. Rename such function to `is_udp_not_guessable_protocol()`: the name is still quite cryptic, but at least not plainly wrong
* GitHub Actions: fix a warning (#1807)Ivan Nardi2022-11-22
| | | | | | | ``` The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/ ```
* TLS: be sure to always set `ssl_version` field (#1806)Ivan Nardi2022-11-22
| | | Useful with asymmetric traffic with (D)TLS <= 1.2
* Added support for Linux Cooked Capture v2Luca Deri2022-11-16
|
* DNS: fix an infinite loop, due to an integer overflow (#1804)Ivan Nardi2022-11-15
| | | | Found by sydr-fuzz Close #1803
* Improved Teamspeak3 License/Weblist detection. (#1802)Toni2022-11-15
| | | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com> Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Improve export/print of L4 protocol information (#1799)Ivan Nardi2022-11-13
| | | Close #1797
* GitHub Actions: update to Node.js 16 actions (#1801)Ivan Nardi2022-11-13
| | | | | | Fix warnings on recent CI results; example: https://github.com/ntop/nDPI/actions/runs/3455588082 See: https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/
* Merge pull request #1796 from IvanNardi/issue-1792Toni2022-11-07
|\ | | | | RTP: remove a superfluous check
| * RTP: remove a superfluous checkNardi Ivan2022-11-07
| | | | | | | | | | | | | | | | Static analyzer complains about dereferencing `packet->udp` before checking. Since this function is called only with UDP flows, remove the check. Close: #1792
* | Fix json export of ipv6 addressesNardi Ivan2022-11-07
|/ | | | | | | The "string" buffer was to short; better start using `INET6_ADDRSTRLEN` as reported in the man page of `inet_ntop`. Close: #1794
* Fix supported versions.Zied Aouini2022-10-31
|
* Add a note about required Python version.Zied Aouini2022-10-31
|
* Add basic profiling of memory allocations on data-path (#1789)Ivan Nardi2022-10-28
| | | | | | | | | | | | | | | The goal is to have an idea of the memory allocation sizes performed in the **library data-path**, i.e. excluding init/deinit phases and all the allocations made by the application itself. In other words, how much memory is needed per-flow, by nDPI, other than `struct ndpi_flow_struct`? It works only on single-thread configurations. It is not enabled by default (in the unit tests) since different canfiguration options (example: `--enable-pcre`) lead to diffferent results. See: #1781
* Removed unused variableLuca Deri2022-10-26
|
* Fix typoAlfredo Cardigliano2022-10-26
|
* CleanupLuca Deri2022-10-26
|
* TypoLuca Deri2022-10-26
|
* Improved AESNI checkLuca Deri2022-10-26
|
* Hangout: avoid useless lookups in the protocol tree (#1755)Ivan Nardi2022-10-25
| | | | | | | We already performed exactly these lookups in the generic code to populate `flow->guessed_protocol_id_by_ip`: use it! This code probably needs a deeper review, since it is basicaly a simple matching on ip + port.
* TLS: improve handling of ALPN(s) (#1784)Ivan Nardi2022-10-25
| | | | | | | | Tell "Advertised" ALPN list from "Negotiated" ALPN; the former is extracted from the CH, the latter from the SH. Add some entries to the known ALPN list. Fix printing of "TLS Supported Versions" field.
* Fix memory corruption on TCP flows (#1786)Ivan Nardi2022-10-25
|
* IRC: fix memory access (#1787)Ivan Nardi2022-10-25
|
* Sync unit tests results (#1783)Ivan Nardi2022-10-22
|
* Enhanced HTTP numeric IP checkLuca Deri2022-10-22
|
* Add Elasticsearch protocol dissector. (#1782)Toni2022-10-21
| | | | | | | * all credits goes to @verzulli Signed-off-by: Toni Uhlig <matzeton@googlemail.com> Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Add Munin protocol dissector.Toni Uhlig2022-10-20
| | | | | | * all credits goes to @verzulli Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* TLS: allow sub-classification via ALPNNardi Ivan2022-10-20
| | | | | | | | In some rare cases, it is possible to sub-classify the flow via ALPN matching. This is particularly usefull for asymmetric traffic where the Client Hello doens't have the SNI. For the time being there is only one rule, about ANYDESK.
* Sync unit tests resultsNardi Ivan2022-10-19
|
* Updated FortiClient default portsLuca Deri2022-10-19
|
* Reworked severities adding critical and emergency, thus shifting values downLuca Deri2022-10-19
|
* TLS: explicit ignore client certificate (#1776)Ivan Nardi2022-10-18
| | | | | | | | | | | | | TLS classification usually stops after processing *server* certificates (if any). That means, that *client* certificate, if present, is usually ignored. However in some corner cases (i.e. unidirectional traffic) we might end up processing client certificate and exposing its metadata: the issue is that the application will think that this metadata are about the server and not about the client. So, for the time being, always ignore client certificate processing. As a future work, we might find an efficient way to process and export both certificates.
* Fix a use-of-uninitialized-value error on PCRE codeNardi Ivan2022-10-17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is likely a false positive, triggered by the fact that libpcre is usually compiled without MASAN support. It it was a real error, ASAN would complain loudly with a invalid-free error at the end of the same function. ``` ==83793==WARNING: MemorySanitizer: use-of-uninitialized-value #0 0x562296111174 in ndpi_compile_rce_regex /home/ivan/svnrepos/nDPI/src/lib/ndpi_utils.c:1631:3 #1 0x5622960e3e4a in ndpi_is_rce_injection /home/ivan/svnrepos/nDPI/src/lib/ndpi_utils.c:1636:5 #2 0x5622960de7cd in ndpi_validate_url /home/ivan/svnrepos/nDPI/src/lib/ndpi_utils.c:1741:12 #3 0x5622960dae45 in ndpi_dpi2json /home/ivan/svnrepos/nDPI/src/lib/ndpi_utils.c:1362:29 #4 0x5622960e2751 in ndpi_flow2json /home/ivan/svnrepos/nDPI/src/lib/ndpi_utils.c:1512:10 #5 0x562296033b0f in process_ndpi_collected_info /home/ivan/svnrepos/nDPI/example/reader_util.c:1310:9 #6 0x5622960501f9 in packet_processing /home/ivan/svnrepos/nDPI/example/reader_util.c:1659:2 #7 0x562296045aef in ndpi_workflow_process_packet /home/ivan/svnrepos/nDPI/example/reader_util.c:2202:10 #8 0x562295e85374 in ndpi_process_packet /home/ivan/svnrepos/nDPI/example/ndpiReader.c:3937:7 #9 0x7f1235053466 (/lib/x86_64-linux-gnu/libpcap.so.0.8+0x23466) (BuildId: b84c893ea2516d6fb2c1c6726b1fe93b3be78f61) #10 0x7f1235041f67 in pcap_loop (/lib/x86_64-linux-gnu/libpcap.so.0.8+0x11f67) (BuildId: b84c893ea2516d6fb2c1c6726b1fe93b3be78f61) #11 0x562295e53139 in runPcapLoop /home/ivan/svnrepos/nDPI/example/ndpiReader.c:4060:15 #12 0x562295e51e7f in processing_thread /home/ivan/svnrepos/nDPI/example/ndpiReader.c:4130:3 #13 0x7f1234e53608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8 #14 0x7f1234d2f132 in __clone /build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Uninitialized value was created by an allocation of 'pcreErrorStr' in the stack frame #0 0x5622961108a6 in ndpi_compile_rce_regex /home/ivan/svnrepos/nDPI/src/lib/ndpi_utils.c:1603:3 ```
* Add some popular streaming applicationsNardi Ivan2022-10-17
| | | | | | | A simply back-porting from https://github.com/vel21ripn/nDPI/pull/142; full credits to @ChrisNelson-CyberReef. Close #1716
* Updated protocol dissector serialisation code. (#1767)Toni2022-10-15
| | | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com> Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
Powered by cgit, Themed by Ted Yin.