aboutsummaryrefslogtreecommitdiff
path: root/src/lib/ndpi_main.c
Commit message (Collapse)AuthorAge
* Line app support (#1759)sharonenoch2022-10-01
| | | | | | | | | | | | | | | | | | | * Standard support for LINE app * Added test pcap for LINE app * make check result for LINE app * Make check success as 1kxun has LINE packets * Added the ASN inc file for LINE * Removed extra lines as its effecting make check * Editing the SNI required a new pcap output file for TLS.Line format * Run Configure with --with-pcre --with-maxminddb to enable the generation of h323-overflow.pcap.out Co-authored-by: Sharon Enoch <sharone@amzetta.com>
* Fixed invalid unidirectional traffic alert for unidirectional protocols ↵Luca2022-09-29
| | | | (e.g. sFlow)
* Remove unused codeNardi Ivan2022-09-28
| | | | | | | | | LRU callbacks have been added in 460ff3c7a, but they have never been used and they have never been extended to the other LRU caches. `ndpi_search_tcp_or_udp()` basically returns the classification by port/ip of the flow; calling it from the dissector is useless. The same for TOR detection: ips are checked in the generic code
* Fix value of `ndpi_protocol->protocol_by_ip`Nardi Ivan2022-09-25
| | | | Fix: a7c2734b
* Changed some protocol categories to align with others.Toni Uhlig2022-09-23
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Sizes of LRU caches are now configurableNardi Ivan2022-09-23
| | | | | | | | 0 as size value disable the cache. The diffs in unit tests are due to the fact that some lookups are performed before the first insert: before this change these lookups weren't counted because the cache was not yet initialized, now they are.
* DNS: change category of DNS flowsNardi Ivan2022-09-22
| | | | | DNS flows should have `NDPI_PROTOCOL_CATEGORY_NETWORK` as category, regardless of the subprotocol (if any).
* Fix CI after nBPF integration (#1746)Ivan Nardi2022-09-21
| | | Add one CI job testing nBPF
* Copilation fixLuca Deri2022-09-21
|
* Finalized nBPF support. You can now define custom protocols such asLuca Deri2022-09-21
| | | | | | | | | | (see exaple/protos.txt) nbpf:"host 192.168.1.1 and port 80"@HomeRouter In order to have nBPF support, you need to compile nDPI with it. Just download https://github.com/ntop/PF_RING in the same directory where you have downloaded nDPI and compile PF_RING/userland/nbpf
* Remove classification "by-ip" from protocol stack (#1743)Ivan Nardi2022-09-20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Basically: * "classification by-ip" (i.e. `flow->guessed_protocol_id_by_ip` is NEVER returned in the protocol stack (i.e. `flow->detected_protocol_stack[]`); * if the application is interested into such information, it can access `ndpi_protocol->protocol_by_ip` itself. There are mainly 4 points in the code that set the "classification by-ip" in the protocol stack: the generic `ndpi_set_detected_protocol()`/ `ndpi_detection_giveup()` functions and the HTTP/STUN dissectors. In the unit tests output, a print about `ndpi_protocol->protocol_by_ip` has been added for each flow: the huge diff of this commit is mainly due to that. Strictly speaking, this change is NOT an API/ABI breakage, but there are important differences in the classification results. For examples: * TLS flows without the initial handshake (or without a matching SNI/certificate) are simply classified as `TLS`; * similar for HTTP or QUIC flows; * DNS flows without a matching request domain are simply classified as `DNS`; we don't have `DNS/Google` anymore just because the server is 8.8.8.8 (that was an outrageous behaviour...); * flows previusoly classified only "by-ip" are now classified as `NDPI_PROTOCOL_UNKNOWN`. See #1425 for other examples of why adding the "classification by-ip" in the protocol stack is a bad idea. Please, note that IPV6 is not supported :( (long standing issue in nDPI) i.e. `ndpi_protocol->protocol_by_ip` wil be always `NDPI_PROTOCOL_UNKNOWN` for IPv6 flows. Define `NDPI_CONFIDENCE_MATCH_BY_IP` has been removed. Close #1687
* Initial work towards custom BPF protocol definitionLuca2022-09-16
|
* Add CryNetwork dissector.Toni Uhlig2022-09-16
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Add Syncthing dissector.Toni Uhlig2022-09-16
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Add NATPMP dissector. (#1738)Toni2022-09-16
| | | | | | | Signed-off-by: lns <matzeton@googlemail.com> Signed-off-by: Toni Uhlig <matzeton@googlemail.com> Signed-off-by: lns <matzeton@googlemail.com> Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Remove a case of guessed sub-classificationNardi Ivan2022-09-14
| | | | | | | | | | | | | | | | | | This code is triggered only for "unknown" flows with a valid sni/hostname. Why in that case the guessed classification should be something like `DNS/Subprotocol_depending_on_hostname`? Why DNS as master and not HTTP or TLS or QUIC? Furthermore, I have not been able to trigger a positive match from that lookup. I strongly think that if we had a valid subprotocol, we would have a valid master in the first place. In doubt, remove it completely. As a follow up, we should investigate why some dissectors (the HTTP one, at least) set the sni/hostname field without setting a valid protocol, in the first place. This behaviour seems quite suspicious, if not plainly buggy.
* Dissectors shouldn't update `flow->guessed_host_protocol_id`Nardi Ivan2022-09-14
| | | | | | | | | | | | | | | | The field `flow->guessed_host_protocol_id` is set at the beginning of the flow analysis and it represents the "classification by ip" of the flow itself. This field should never be changed. Dissectors which want to provide an "hint" about the classification, should update `flow->guessed_protocol_id` instead. Such "hint" is useless if the dissector set the "extra-dissection" data-path. Rename such field to `guessed_protocol_id_by_ip` to better describe its role. Preliminary work necessary for #1687
* Fix `ndpi_do_guess()` (#1731)Ivan Nardi2022-09-12
| | | | | Avoid a double call of `ndpi_guess_host_protocol_id()`. Some code paths work for ipv4/6 both Remove some never used code.
* Avoid useless host automa lookup (#1724)Ivan Nardi2022-09-05
| | | | | | | | | | | | | | | | | | | The host automa is used for two tasks: * protocol sub-classification (obviously); * DGA evaluation: the idea is that if a domain is present in this automa, it can't be a DGA, regardless of its format/name. In most dissectors both checks are executed, i.e. the code is something like: ``` ndpi_match_host_subprotocol(..., flow->host_server_name, ...); ndpi_check_dga_name(..., flow->host_server_name,...); ``` In that common case, we can perform only one automa lookup: if we check the sub-classification before the DGA, we can avoid the second lookup in the DGA function itself.
* Fix endianess in IP-Port lookupNardi Ivan2022-09-05
| | | | | Port parameter in `ndpi_network_port_ptree_match()` must be in network order.
* Add support to opportunistic TLSNardi Ivan2022-09-04
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A lot of protocols provide the feature to upgrade their plain text connections to an encrypted one, via some kind of "STARTTLS" command. Add generic code to support this extension, and allow dissection of the entire TLS handshake. As examples, SMTP, POP, IMAP and FTP dissectors have been updated. Since this feature requires to process more packets per flow, add the possibility to disable it. Fix some log messages. Slight improvement on TCP sequence number tracking. As a side effect, this commit fix also a memory leak found by oss-fuzzer ``` ==108966==ERROR: LeakSanitizer: detected memory leaks Direct leak of 22 byte(s) in 1 object(s) allocated from: #0 0x55f8b367a0be in malloc (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader_with_main+0x5480be) (BuildId: 94debacb4a6784c30420ab748c8bf3cc59621063) #1 0x55f8b36e1345 in ndpi_malloc_wrapper /home/ivan/svnrepos/nDPI/example/reader_util.c:321:10 #2 0x55f8b379c7d2 in ndpi_malloc /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:212:25 #3 0x55f8b379cb18 in ndpi_strdup /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:279:13 #4 0x55f8b386ce46 in processClientServerHello /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:2153:34 #5 0x55f8b385ebf7 in processTLSBlock /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:867:5 #6 0x55f8b39e708c in ndpi_extra_search_mail_smtp_tcp /home/ivan/svnrepos/nDPI/src/lib/protocols/mail_smtp.c:422:9 #7 0x55f8b37e636c in ndpi_process_extra_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5884:9 #8 0x55f8b37edc05 in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:6276:5 #9 0x55f8b3701ffc in packet_processing /home/ivan/svnrepos/nDPI/example/reader_util.c:1619:31 #10 0x55f8b36faf14 in ndpi_workflow_process_packet /home/ivan/svnrepos/nDPI/example/reader_util.c:2189:10 #11 0x55f8b36b6a50 in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:107:7 ``` See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=50765
* Improved MGCP dissector. (#1717)Toni2022-08-30
| | | | | | | | * typ0s fixed * dissect endpoint hostnames Signed-off-by: Toni Uhlig <matzeton@googlemail.com> Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Add FastCGI protocol detection. (#1711)Toni2022-08-24
| | | | | | | | * CQL: fixed byte order conversion (BigEndian not LittleEndian) * CQL: increased required successful dissected packets to prevent false-positives Signed-off-by: Toni Uhlig <matzeton@googlemail.com> Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Add Kismet protocol detection. (#1710)Toni2022-08-24
| | | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com> Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Add TiVoConnect dissector. Fixes #1697. (#1699)Toni2022-08-08
| | | | | * added static assert if supported, to complain if the flow struct changes Signed-off-by: lns <matzeton@googlemail.com>
* Further simplification of `ndpi_process_extra_packet()` (#1698)Ivan Nardi2022-08-05
| | | | | | | | See 95e16872. After c0732eda, we can safely remove the protocol list from `ndpi_process_extra_packet()`. The field `flow->check_extra_packets` is redundant; remove it.
* HTTP: improve sub-classification (#1696)Ivan Nardi2022-08-05
| | | | | | | | | | Content-matched sub-protocols (DASH, IPP, MPEGDASH...) shouldn't ovewrite the previous master protocol (if any; usually HTTP). Furthermore. the HTTP dissector shouldn't update the classification (in the extra-dissection code path) if a content-matched sub-protocols has already been found. This commit should address the first part of the changes described in #1687.
* Add Discord dissector. (#1694)Toni2022-08-03
| | | | | * fixed RiotGames false positive Signed-off-by: lns <matzeton@googlemail.com>
* Cosmetic changesLuca Deri2022-08-03
|
* Add Activision dissector. (#1693)Toni2022-08-02
| | | Signed-off-by: lns <matzeton@googlemail.com>
* SMTPS, POPS, IMAPS: fix classification and extra dissection (#1685)Ivan Nardi2022-07-30
| | | | | | The big change in TLS code is to allow "master" protocols other than TLS/DTLS, like SMTPS, POPS and IMAPS. This change will allow, in a future, a proper and complete TLS dissection for all these protocols with "STARTTLS"-like messages.
* Add Softether dissector. (#1679)Toni2022-07-29
| | | Signed-off-by: lns <matzeton@googlemail.com>
* Patricia tree, Ahocarasick automa, LRU cache: add statistics (#1683)Ivan Nardi2022-07-29
| | | | | | | | | | Add (basic) internal stats to the main data structures used by the library; they might be usefull to check how effective these structures are. Add an option to `ndpiReader` to dump them; enabled by default in the unit tests. This new option enables/disables dumping of "num dissectors calls" values, too (see b4cb14ec).
* Revert "Patricia tree, Ahocarasick automa, LRU cache: add statistics ↵Ivan Nardi2022-07-29
| | | | | (#1677)" (#1682) This reverts commit bb83899985c25097341b947c2c535f56254a075c.
* Patricia tree, Ahocarasick automa, LRU cache: add statistics (#1677)Ivan Nardi2022-07-29
| | | | | | | | Add (basic) internal stats to the main data structures used by the library; they might be usefull to check how effective these structures are. Add an option to `ndpiReader` to dump them; disabled by default to avoid too much fuss with the unit tests.
* First step in simplify `ndpi_process_extra_packet()` (#1680)Ivan Nardi2022-07-29
| | | | | | | Move the prottocol specific logic into the proper dissector code, where it belongs. Next step: remove that list of protocols. Long goal: remove this function altogether...
* TINC: avoid processing SYN packets (#1676)Ivan Nardi2022-07-28
| | | | | | | | | | | Since e6b332aa, we have proper support for detecting client/server direction. So Tinc dissector is now able to properly initialize the cache entry only when needed and not anymore at the SYN time; initializing that entry for **every** SYN packets was a complete waste of resources. Since 4896dabb, the various `struct ndpi_call_function_struct` structures are not more separate objects and therefore comparing them using only their pointers is bogus: this bug was triggered by this change because `ndpi_str->callback_buffer_size_tcp_no_payload` is now 0.
* Update the protocol bitmask for some protocols (#1675)Ivan Nardi2022-07-27
| | | | | | | Tcp retransmissions should be ignored. Remove some unused protocol bitmasks. Update script to download Whatsapp IP list.
* Add AVAST dissector. (#1674)Toni2022-07-25
| | | Signed-off-by: lns <matzeton@googlemail.com>
* Improve handling of HTTP-Proxy and HTTP-Connect (#1673)Ivan Nardi2022-07-25
| | | | | | | | | | | | Treat HTTP-Proxy and HTTP-Connect flows like the HTTP ones: print/serialize all the attributes and allow parsing of replies. The line about "1kxun" has been removed to avoid regressions in 1KXUN classification in `tests/pcap/1kxun.pcap`. I haven't fully understod what was happening but the comment at the beginning of `static ndpi_category_match category_match[]` says that we can't have overlaps between `host_match` and `category_match` lists and that is no longer true since 938e89ca. Bottom line: removing this line seems the right thing to do, anyway.
* Add support for flow client/server information (#1671)Ivan Nardi2022-07-24
| | | | | | | | | | | | | | | | | | In a lot of places in ndPI we use *packet* source/dest info (address/port/direction) when we are interested in *flow* client/server info, instead. Add basic logic to autodetect this kind of information. nDPI doesn't perform any "flow management" itself but this task is delegated to the external application. It is then likely that the application might provide more reliable hints about flow client/server direction and about the TCP handshake presence: in that case, these information might be (optionally) passed to the library, disabling the internal "autodetect" logic. These new fields have been used in some LRU caches and in the "guessing" algorithm. It is quite likely that some other code needs to be updated.
* Added AliCloud server access dissector. (#1672)Toni2022-07-23
| | | | Signed-off-by: lns <matzeton@googlemail.com> Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* TLS: improve reassembler (#1669)Ivan Nardi2022-07-22
| | | | | | | | * TLS: cosmetic changes * TLS: improve reassembler We might need to contemporary re-order messages from both directions: use one buffer per direction.
* reader_util: stop processing a flow (#1666)Ivan Nardi2022-07-20
| | | | We should stop processing a flow if all protocols have been excluded or if we have already processed too many packets.
* Introduced risk accountabilityLuca2022-07-12
|
* Keep track of how many dissectors calls we made for each flow (#1657)Ivan Nardi2022-07-11
|
* Remove unsafe access to `flow->protos` union (#1656)Ivan Nardi2022-07-10
| | | | | | We can access `flow->protos` only if we already have set a valid classification. It is quite likely that this code is never trigger, anyway.
* Avoid spurious calls to extra dissection (#1648)Ivan Nardi2022-07-07
| | | | If the extra callabck is not set, calling the extra dissection is only a waste of resources...
* Added Threema Messenger. (#1643)Toni2022-07-06
| | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Added RiotGames ASN update.Toni Uhlig2022-07-06
| | | | | | * updated asn lists Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
Powered by cgit, Themed by Ted Yin.