| Commit message (Collapse) | Author | Age |
|---|
| |
|
|
|
|
|
|
|
| |
Extend internal unit tests to handle multiple configurations.
As some examples, add tests about:
* disabling some protocols
* disabling Ookla aggressiveness
Every configurations data is stored in a dedicated directory under
`tests\cfgs`
|
| |
|
|
|
|
| |
DPI (#1891)
Average values are already printed, but this change should ease to
identify regressions/improvements.
|
| | |
|
| |
|
| |
See a944514d. No flow/classification/metadata have been removed.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
| |
Classification "by-port" should be the last possible effort, *after*
having test all the LRU caches.
Remove some dead code from `ndpi_detection_giveup()`:
`flow->guessed_protocol_id` is never set to any od those voip protocols
and at that point in this function we never have both a master *and* a
application protocols.
Coverage reports (both from unit tests and from fuzzing) confirms that
was dead code.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
No traces and no flows has been removed; only long sessions
has been reduced, keeping only their first packets.
This is quite important in fuzzing systems, since these pcaps are used
as initial seed. There is no value in fuzzing long sessions, because
only the very first packets are really used/processed by nDPI.
Before:
```
du -h tests/pcap/
200M tests/pcap/
```
After:
```
du -h tests/pcap/
98M tests/pcap/
```
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
These protocols:
* have been addeded in the OpenDPI era
* have never been updated since then
* we don't have any pcap examples [*]
If (and it is a big if...) some of these protocols are still somehow
used and if someone is still interested in them, we can probably
re-add them starting from scratch (because the current detection
rules are probably outdated)
Protocols removed: DIRECT_DOWNLOAD_LINK, APPLEJUICE, DIRECTCONNECT,
OPENFT, FASTTRACK, SHOUTCAST, THUNDER, AYIYA, STEALTHNET, FIESTA,
FLORENSIA, AIMINI, SOPCAST
PPSTREAM dissector works (...) only on UDP.
[*]: with do have an AIMINI test pcap but it was some trivial http
traffic detected only by hostname matching, on domains no more
available...
|
| |
|
|
| |
See: "Enabling Passive Measurement of Zoom Performance in Production Networks"
https://dl.acm.org/doi/pdf/10.1145/3517745.3561414
|
| |
|
|
|
|
|
|
| |
0 as size value disable the cache.
The diffs in unit tests are due to the fact that some lookups are
performed before the first insert: before this change these lookups
weren't counted because the cache was not yet initialized, now they are.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Basically:
* "classification by-ip" (i.e. `flow->guessed_protocol_id_by_ip` is
NEVER returned in the protocol stack (i.e.
`flow->detected_protocol_stack[]`);
* if the application is interested into such information, it can access
`ndpi_protocol->protocol_by_ip` itself.
There are mainly 4 points in the code that set the "classification
by-ip" in the protocol stack: the generic `ndpi_set_detected_protocol()`/
`ndpi_detection_giveup()` functions and the HTTP/STUN dissectors.
In the unit tests output, a print about `ndpi_protocol->protocol_by_ip`
has been added for each flow: the huge diff of this commit is mainly due
to that.
Strictly speaking, this change is NOT an API/ABI breakage, but there are
important differences in the classification results. For examples:
* TLS flows without the initial handshake (or without a matching
SNI/certificate) are simply classified as `TLS`;
* similar for HTTP or QUIC flows;
* DNS flows without a matching request domain are simply classified as
`DNS`; we don't have `DNS/Google` anymore just because the server is
8.8.8.8 (that was an outrageous behaviour...);
* flows previusoly classified only "by-ip" are now classified as
`NDPI_PROTOCOL_UNKNOWN`.
See #1425 for other examples of why adding the "classification by-ip" in
the protocol stack is a bad idea.
Please, note that IPV6 is not supported :( (long standing issue in nDPI) i.e.
`ndpi_protocol->protocol_by_ip` wil be always `NDPI_PROTOCOL_UNKNOWN` for
IPv6 flows.
Define `NDPI_CONFIDENCE_MATCH_BY_IP` has been removed.
Close #1687
|
| |
|
|
|
| |
Avoid a double call of `ndpi_guess_host_protocol_id()`.
Some code paths work for ipv4/6 both
Remove some never used code.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The host automa is used for two tasks:
* protocol sub-classification (obviously);
* DGA evaluation: the idea is that if a domain is present in this
automa, it can't be a DGA, regardless of its format/name.
In most dissectors both checks are executed, i.e. the code is something
like:
```
ndpi_match_host_subprotocol(..., flow->host_server_name, ...);
ndpi_check_dga_name(..., flow->host_server_name,...);
```
In that common case, we can perform only one automa lookup: if we check the
sub-classification before the DGA, we can avoid the second lookup in
the DGA function itself.
|
| |
|
|
|
|
|
|
|
|
| |
Add (basic) internal stats to the main data structures used by the
library; they might be usefull to check how effective these structures
are.
Add an option to `ndpiReader` to dump them; enabled by default in the
unit tests.
This new option enables/disables dumping of "num dissectors calls"
values, too (see b4cb14ec).
|
| |
|
|
|
|
|
|
|
|
|
| |
Since e6b332aa, we have proper support for detecting client/server
direction. So Tinc dissector is now able to properly initialize the
cache entry only when needed and not anymore at the SYN time; initializing
that entry for **every** SYN packets was a complete waste of resources.
Since 4896dabb, the various `struct ndpi_call_function_struct`
structures are not more separate objects and therefore comparing them
using only their pointers is bogus: this bug was triggered by this
change because `ndpi_str->callback_buffer_size_tcp_no_payload` is now 0.
|
| |
|
|
|
|
|
| |
Tcp retransmissions should be ignored.
Remove some unused protocol bitmasks.
Update script to download Whatsapp IP list.
|
| |
|
| |
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
| |
This way, the CI results are consistent
|
| |
|
|
| |
Code cleanup
|
| | |
|
| | |
|
| |
|
| |
Fix: 7a3aa41a
|
| |
|
|
| |
been added
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
As a general rule, the higher the confidence value, the higher the
"reliability/precision" of the classification.
In other words, this new field provides an hint about "how" the flow
classification has been obtained.
For example, the application may want to ignore classification "by-port"
(they are not real DPI classifications, after all) or give a second
glance at flows classified via LRU caches (because of false positives).
Setting only one value for the confidence field is a bit tricky: more
work is probably needed in the next future to tweak/fix/improve the logic.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
| |
The goal is to have a (roughly) idea about how many packets nDPI needs
to properly classify a flow.
Log this information (and guessed flows number too) during unit tests,
to keep track of improvements/regressions across commits.
|
| | |
|
| |
|
|
|
|
|
|
| |
Before
Accuracy 66%, Precision 86%, Recall 38%
After
Accuracy 71%, Precision 89%, Recall 49%
|
| |
|
|
|
| |
Reworked Tor dissector embedded in TLS (fixes #1141)
Removed false positive on HTTP User-Agent
|
| | |
|
| | |
|
| |
|
|
| |
not condidered safe/secure
|
| |
|
|
| |
Optimized stddev calculation
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
Packet bins are not printed wehn empty
|
| |
|
|
| |
Added packet lenght distribution bins
|