| Commit message (Collapse) | Author | Age |
|
|
|
|
|
|
|
|
| |
Extend internal unit tests to handle multiple configurations.
As some examples, add tests about:
* disabling some protocols
* disabling Ookla aggressiveness
Every configurations data is stored in a dedicated directory under
`tests\cfgs`
|
|
|
|
|
|
|
|
|
|
| |
RFC 6066 3: "Literal IPv4 and IPv6 addresses are not permitted in
"HostName"."
Don't set this risk if we have a valid sub-classification (example:
via certificate)
Since a similar risk already exists for HTTP hostnames, reuse it, with a
more generic name.
|
|
|
|
|
|
| |
DPI (#1891)
Average values are already printed, but this change should ease to
identify regressions/improvements.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
ubuntu-latest s390x CI doesn't like snapshot bigger than 262144 bytes.
Fix an error found by fuzz CI
```
=================================================================
==55399==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000008 (pc 0x561e41cb684d bp 0x7ffd54ce3650 sp 0x7ffd54ce3520 T0)
==55399==The signal is caused by a READ memory access.
==55399==Hint: address points to the zero page.
#0 0x561e41cb684d in ndpi_network_ptree_match /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:2321:41
#1 0x561e41d30879 in ndpi_guess_undetected_protocol /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:7767:8
#2 0x561e41ca804d in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_config.cpp:104:5
#3 0x561e41bb96a0 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) (/home/ivan/svnrepos/nDPI/fuzz/fuzz_config+0x4726a0) (BuildId: d4741c753aafe7c0df2681a592b7df16b38240e9)
#4 0x561e41ba3c2f in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) (/home/ivan/svnrepos/nDPI/fuzz/fuzz_config+0x45cc2f) (BuildId: d4741c753aafe7c0df2681a592b7df16b38240e9)
#5 0x561e41ba96f6 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (/home/ivan/svnrepos/nDPI/fuzz/fuzz_config+0x4626f6) (BuildId: d4741c753aafe7c0df2681a592b7df16b38240e9)
#6 0x561e41bd22e2 in main (/home/ivan/svnrepos/nDPI/fuzz/fuzz_config+0x48b2e2) (BuildId: d4741c753aafe7c0df2681a592b7df16b38240e9)
#7 0x7f94f0e5c082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16
#8 0x561e41b9eb0d in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_config+0x457b0d) (BuildId: d4741c753aafe7c0df2681a592b7df16b38240e9)
```
|
| |
|
| |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
These protocols:
* have been addeded in the OpenDPI era
* have never been updated since then
* we don't have any pcap examples [*]
If (and it is a big if...) some of these protocols are still somehow
used and if someone is still interested in them, we can probably
re-add them starting from scratch (because the current detection
rules are probably outdated)
Protocols removed: DIRECT_DOWNLOAD_LINK, APPLEJUICE, DIRECTCONNECT,
OPENFT, FASTTRACK, SHOUTCAST, THUNDER, AYIYA, STEALTHNET, FIESTA,
FLORENSIA, AIMINI, SOPCAST
PPSTREAM dissector works (...) only on UDP.
[*]: with do have an AIMINI test pcap but it was some trivial http
traffic detected only by hostname matching, on domains no more
available...
|
|
|
|
| |
Update the documentation, hinting how to test/update *all* the unit
tests.
|
|
|
|
|
|
|
|
| |
Tuya IoTOS Embedded Wi-Fi and BLE SDK for bk7231n. Used by many "smart"
devices such as LED light strips, bulbs, etc.
Signed-off-by: Darryl Sokoloski <darryl@sokoloski.ca>
Signed-off-by: Darryl Sokoloski <darryl@sokoloski.ca>
|
|
|
|
| |
See: "Enabling Passive Measurement of Zoom Performance in Production Networks"
https://dl.acm.org/doi/pdf/10.1145/3517745.3561414
|
| |
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
GitHub is moving `ubuntu-latest` to `ubuntu-22.04`: update our
dependencies.
See: https://github.blog/changelog/2022-11-09-github-actions-ubuntu-latest-workflows-will-use-ubuntu-22-04/
This is the reason of the recent random failures in CI.
Update "newest" tested gcc to gcc-12.
Fix a memory error introduced in 557bbcfc5a5165c9eb43bbdd78435796239cd3c9
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Basically:
* "classification by-ip" (i.e. `flow->guessed_protocol_id_by_ip` is
NEVER returned in the protocol stack (i.e.
`flow->detected_protocol_stack[]`);
* if the application is interested into such information, it can access
`ndpi_protocol->protocol_by_ip` itself.
There are mainly 4 points in the code that set the "classification
by-ip" in the protocol stack: the generic `ndpi_set_detected_protocol()`/
`ndpi_detection_giveup()` functions and the HTTP/STUN dissectors.
In the unit tests output, a print about `ndpi_protocol->protocol_by_ip`
has been added for each flow: the huge diff of this commit is mainly due
to that.
Strictly speaking, this change is NOT an API/ABI breakage, but there are
important differences in the classification results. For examples:
* TLS flows without the initial handshake (or without a matching
SNI/certificate) are simply classified as `TLS`;
* similar for HTTP or QUIC flows;
* DNS flows without a matching request domain are simply classified as
`DNS`; we don't have `DNS/Google` anymore just because the server is
8.8.8.8 (that was an outrageous behaviour...);
* flows previusoly classified only "by-ip" are now classified as
`NDPI_PROTOCOL_UNKNOWN`.
See #1425 for other examples of why adding the "classification by-ip" in
the protocol stack is a bad idea.
Please, note that IPV6 is not supported :( (long standing issue in nDPI) i.e.
`ndpi_protocol->protocol_by_ip` wil be always `NDPI_PROTOCOL_UNKNOWN` for
IPv6 flows.
Define `NDPI_CONFIDENCE_MATCH_BY_IP` has been removed.
Close #1687
|
|
|
|
|
| |
Avoid a double call of `ndpi_guess_host_protocol_id()`.
Some code paths work for ipv4/6 both
Remove some never used code.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The host automa is used for two tasks:
* protocol sub-classification (obviously);
* DGA evaluation: the idea is that if a domain is present in this
automa, it can't be a DGA, regardless of its format/name.
In most dissectors both checks are executed, i.e. the code is something
like:
```
ndpi_match_host_subprotocol(..., flow->host_server_name, ...);
ndpi_check_dga_name(..., flow->host_server_name,...);
```
In that common case, we can perform only one automa lookup: if we check the
sub-classification before the DGA, we can avoid the second lookup in
the DGA function itself.
|
|
|
|
|
|
|
|
|
|
| |
Add (basic) internal stats to the main data structures used by the
library; they might be usefull to check how effective these structures
are.
Add an option to `ndpiReader` to dump them; enabled by default in the
unit tests.
This new option enables/disables dumping of "num dissectors calls"
values, too (see b4cb14ec).
|
| |
|
|
|
| |
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
|
| |
|
| |
|
| |
|
| |
|
|
|
|
| |
Added ability to identify application and network protocols
|
|
|
|
|
|
|
|
| |
GCC analyzer won't complain about possible use-after-free (false positive).
* tests/do.sh prints word diff's only once and not the same over and over again
* sync unit tests
Signed-off-by: lns <matzeton@googlemail.com>
|
| |
|
|
|
| |
This way, the CI results are consistent
|
|
|
|
| |
Code cleanup
|
| |
|
| |
|
| |
|
|
|
|
| |
'strcasestr' is not defined in all enviroments: quicker fix is to use
'ndpi_strncasestr' instead.
|
|
|
|
|
|
|
| |
* Sync utest results
* Fix read-heap-buffer-overflow error reported by CI
See: https://github.com/ntop/nDPI/runs/5055876515?check_suite_focus=true
|
| |
|
|
|
| |
Fix: 7a3aa41a
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
As a general rule, the higher the confidence value, the higher the
"reliability/precision" of the classification.
In other words, this new field provides an hint about "how" the flow
classification has been obtained.
For example, the application may want to ignore classification "by-port"
(they are not real DPI classifications, after all) or give a second
glance at flows classified via LRU caches (because of false positives).
Setting only one value for the confidence field is a bit tricky: more
work is probably needed in the next future to tweak/fix/improve the logic.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We should avoid updating any valid protocol in `ndpi_detection_giveup`; we
should try to find a proper classification only if the flow is still
completely unclassified.
For example in the attached pcap there is a valid TLS session, recognized
as such by TLS dissector. However, the `ndpi_detection_giveup`function
updates it to "HTTP/TLS" (!?) simply because the server port is 80.
Note that the real issue is not the wrong classification, but the
wrong access to `flow->protos` union. If we already set some fields of
`flow->protos` and we change the protocol in `ndpi_detection_giveup`, we
might end up freeing some invalid pointers in `ndpi_free_flow_data`
(no wonder this issue has been found while fuzzing #1354)
Fix GIT and TLS dissectors (issues found by CI fuzzer)
|
|
|
|
|
|
|
|
| |
`ndpi_detection_giveup()` (and any functions called by it) can't access
`ndpi_detection_module_struct->packet` anymore since 730c236.
Sync unit tests results
Close #1348
|
| |
|
| |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Improve CI pipeline
* Fix branch name.
* Fix branch name.
* Fix libgcrypt configuration.
* Update build.yml
* Move to Github Actions instead of Travis CI.
* Fix mingw on ubuntu bionic.
* Reactivate cross compile on Ubuntu Bionic.
* Switch to single line steps.
* Add several compilers versions
* Minor fix.
* Fix build all and delete cxx
* Fix RCE detection.
* Fix PCRE configuration.
* Add condition on PCRE test pcap.
* Update WebattackRCE.pcap.out
* Add missing SUBST.
* Delete WebattackRCE.pcap.out
* Update WebAttackRCE result.
* Fix typo.
* Extend jobs with pcre+msan+maxminddb.
* Fix code inpector warnings.
* Delete .appveyor.yml
|