| Commit message (Collapse) | Author | Age |
|---|
| |
|
|
|
|
|
|
|
| |
Extend internal unit tests to handle multiple configurations.
As some examples, add tests about:
* disabling some protocols
* disabling Ookla aggressiveness
Every configurations data is stored in a dedicated directory under
`tests\cfgs`
|
| |
|
|
|
|
| |
DPI (#1891)
Average values are already printed, but this change should ease to
identify regressions/improvements.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
These protocols:
* have been addeded in the OpenDPI era
* have never been updated since then
* we don't have any pcap examples [*]
If (and it is a big if...) some of these protocols are still somehow
used and if someone is still interested in them, we can probably
re-add them starting from scratch (because the current detection
rules are probably outdated)
Protocols removed: DIRECT_DOWNLOAD_LINK, APPLEJUICE, DIRECTCONNECT,
OPENFT, FASTTRACK, SHOUTCAST, THUNDER, AYIYA, STEALTHNET, FIESTA,
FLORENSIA, AIMINI, SOPCAST
PPSTREAM dissector works (...) only on UDP.
[*]: with do have an AIMINI test pcap but it was some trivial http
traffic detected only by hostname matching, on domains no more
available...
|
| |
|
|
| |
See: "Enabling Passive Measurement of Zoom Performance in Production Networks"
https://dl.acm.org/doi/pdf/10.1145/3517745.3561414
|
| |
|
|
|
|
|
| |
We already performed exactly these lookups in the generic code to
populate `flow->guessed_protocol_id_by_ip`: use it!
This code probably needs a deeper review, since it is basicaly a simple
matching on ip + port.
|
| |
|
|
|
|
|
|
| |
Tell "Advertised" ALPN list from "Negotiated" ALPN; the former is
extracted from the CH, the latter from the SH.
Add some entries to the known ALPN list.
Fix printing of "TLS Supported Versions" field.
|
| |
|
|
| |
Fix: a7c2734b
|
| |
|
|
|
|
|
|
| |
0 as size value disable the cache.
The diffs in unit tests are due to the fact that some lookups are
performed before the first insert: before this change these lookups
weren't counted because the cache was not yet initialized, now they are.
|
| |
|
|
|
| |
DNS flows should have `NDPI_PROTOCOL_CATEGORY_NETWORK` as category,
regardless of the subprotocol (if any).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Basically:
* "classification by-ip" (i.e. `flow->guessed_protocol_id_by_ip` is
NEVER returned in the protocol stack (i.e.
`flow->detected_protocol_stack[]`);
* if the application is interested into such information, it can access
`ndpi_protocol->protocol_by_ip` itself.
There are mainly 4 points in the code that set the "classification
by-ip" in the protocol stack: the generic `ndpi_set_detected_protocol()`/
`ndpi_detection_giveup()` functions and the HTTP/STUN dissectors.
In the unit tests output, a print about `ndpi_protocol->protocol_by_ip`
has been added for each flow: the huge diff of this commit is mainly due
to that.
Strictly speaking, this change is NOT an API/ABI breakage, but there are
important differences in the classification results. For examples:
* TLS flows without the initial handshake (or without a matching
SNI/certificate) are simply classified as `TLS`;
* similar for HTTP or QUIC flows;
* DNS flows without a matching request domain are simply classified as
`DNS`; we don't have `DNS/Google` anymore just because the server is
8.8.8.8 (that was an outrageous behaviour...);
* flows previusoly classified only "by-ip" are now classified as
`NDPI_PROTOCOL_UNKNOWN`.
See #1425 for other examples of why adding the "classification by-ip" in
the protocol stack is a bad idea.
Please, note that IPV6 is not supported :( (long standing issue in nDPI) i.e.
`ndpi_protocol->protocol_by_ip` wil be always `NDPI_PROTOCOL_UNKNOWN` for
IPv6 flows.
Define `NDPI_CONFIDENCE_MATCH_BY_IP` has been removed.
Close #1687
|
| |
|
|
|
| |
Avoid a double call of `ndpi_guess_host_protocol_id()`.
Some code paths work for ipv4/6 both
Remove some never used code.
|
| |
|
|
|
|
|
|
|
| |
Add detection over TCP and fix detection over IPv6.
Rename some variables since Stun dissector is no more "udp-centric".
Stun dissector should always classified the flow as `STUN` or
`STUN/Something`.
Don't touch `flow->guessed_host_protocol_id` field, which should be
always be related to "ip-classification" only.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The host automa is used for two tasks:
* protocol sub-classification (obviously);
* DGA evaluation: the idea is that if a domain is present in this
automa, it can't be a DGA, regardless of its format/name.
In most dissectors both checks are executed, i.e. the code is something
like:
```
ndpi_match_host_subprotocol(..., flow->host_server_name, ...);
ndpi_check_dga_name(..., flow->host_server_name,...);
```
In that common case, we can perform only one automa lookup: if we check the
sub-classification before the DGA, we can avoid the second lookup in
the DGA function itself.
|
| |
|
|
|
|
|
|
|
|
| |
Add (basic) internal stats to the main data structures used by the
library; they might be usefull to check how effective these structures
are.
Add an option to `ndpiReader` to dump them; enabled by default in the
unit tests.
This new option enables/disables dumping of "num dissectors calls"
values, too (see b4cb14ec).
|
| |
|
|
|
|
|
|
|
|
|
| |
Since e6b332aa, we have proper support for detecting client/server
direction. So Tinc dissector is now able to properly initialize the
cache entry only when needed and not anymore at the SYN time; initializing
that entry for **every** SYN packets was a complete waste of resources.
Since 4896dabb, the various `struct ndpi_call_function_struct`
structures are not more separate objects and therefore comparing them
using only their pointers is bogus: this bug was triggered by this
change because `ndpi_str->callback_buffer_size_tcp_no_payload` is now 0.
|
| |
|
|
|
|
|
| |
Tcp retransmissions should be ignored.
Remove some unused protocol bitmasks.
Update script to download Whatsapp IP list.
|
| |
|
|
|
|
| |
Skype detection over TCP has been completely disable since 659f75138 (3
years ago!).
Since that logic was too weak anyway, remove it.
|
| |
|
| |
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
|
| | |
|
| |
|
|
| |
If the extra callabck is not set, calling the extra dissection is only a
waste of resources...
|
| | |
|
| | |
|
| |
|
| |
Signed-off-by: lns <matzeton@googlemail.com>
|
| | |
|
| |
|
|
| |
Added ability to identify application and network protocols
|
| |
|
| |
Signed-off-by: lns <matzeton@googlemail.com>
|
| |
|
| |
Signed-off-by: lns <matzeton@googlemail.com>
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We should have two protocols in classification results only when the
"master" protocol allows some sub-protocols.
Classifications like `AmazonAWS`, `TLS/AmazonAWS`, `DNS/AmazonAWS` are
fine. However classifications like `NTP/Apple`, `BitTorrent/Azure`,
`DNScrypt.AmazonAWS` or `NestLogSink.Google` are misleading.
For example, `ndpiReader`shows `BitTorrent/Azure` flows under `Azure`
statistics; that seems to be wrong or, at least, very misleading.
This is quite important since we have lots of addresses from CDN
operators.
The only drawback of this solution is that right now ICMP traffic is
classified simply as `ICMP`; if we are really interested in ICMP stuff
we can restore the old behaviour later.
|
| |
|
| |
Fix: 7a3aa41a
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
As a general rule, the higher the confidence value, the higher the
"reliability/precision" of the classification.
In other words, this new field provides an hint about "how" the flow
classification has been obtained.
For example, the application may want to ignore classification "by-port"
(they are not real DPI classifications, after all) or give a second
glance at flows classified via LRU caches (because of false positives).
Setting only one value for the confidence field is a bit tricky: more
work is probably needed in the next future to tweak/fix/improve the logic.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
in the future
|
| |
|
|
|
|
|
| |
The goal is to have a (roughly) idea about how many packets nDPI needs
to properly classify a flow.
Log this information (and guessed flows number too) during unit tests,
to keep track of improvements/regressions across commits.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
Packet bins are not printed wehn empty
|
| |
|
|
| |
Added packet lenght distribution bins
|