diff options
| -rw-r--r-- | Makefile.am | 9 | ||||
| -rw-r--r-- | README.md | 2 | ||||
| -rw-r--r-- | doc/FAQ.md | 37 | ||||
| -rw-r--r-- | doc/FAQ.rst | 42 | ||||
| -rw-r--r-- | doc/_static/custom.css | 3 | ||||
| -rw-r--r-- | doc/configuration_parameters.md | 88 | ||||
| -rw-r--r-- | doc/configuration_parameters.rst | 206 | ||||
| -rw-r--r-- | doc/include_FAQ.rst | 2 | ||||
| -rw-r--r-- | doc/include_configuration_parameters.rst | 2 | ||||
| -rw-r--r-- | doc/include_library_initialization.rst | 2 | ||||
| -rw-r--r-- | doc/include_monitoring.rst | 2 | ||||
| -rw-r--r-- | doc/index.rst | 8 | ||||
| -rw-r--r-- | doc/library_initialization.md | 84 | ||||
| -rw-r--r-- | doc/library_initialization.rst | 84 | ||||
| -rw-r--r-- | doc/monitoring.md | 26 | ||||
| -rw-r--r-- | doc/monitoring.rst | 33 |
16 files changed, 374 insertions, 256 deletions
diff --git a/Makefile.am b/Makefile.am index ce28b528c..1aee0ddc5 100644 --- a/Makefile.am +++ b/Makefile.am @@ -15,11 +15,10 @@ EXTRA_DIST = README.md README.fuzzer.md CHANGELOG.md CONTRIBUTING.md \ doc/requirements.txt doc/conf.py doc/flow_risks.rst doc/protocols.rst doc/guide/nDPI_QuickStartGuide.pages \ doc/guide/nDPI_QuickStartGuide.pdf doc/img/logo.png doc/index.rst \ doc/Makefile doc/what_is_ndpi.rst \ - doc/FAQ.md doc/include_FAQ.rst \ - doc/configuration_parameters.md doc/include_configuration_parameters.rst \ - doc/library_initialization.md doc/include_library_initialization.rst \ - doc/monitoring.md doc/include_monitoring.rst \ - doc/_static/custom.css \ + doc/FAQ.rst \ + doc/configuration_parameters.rst \ + doc/library_initialization.rst \ + doc/monitoring.rst \ python/DEV_GUIDE.md python/dev_requirements.txt python/ndpi_example.py python/ndpi/__init__.py \ python/ndpi/ndpi_build.py python/ndpi/ndpi.py python/README.md \ python/requirements.txt python/setup.py python/tests.py \ @@ -7,7 +7,7 @@ nDPI® is an open source LGPLv3 library for deep-packet inspection. -A generic FAQ about nDPI® is available [here](https://github.com/ntop/nDPI/blob/dev/doc/FAQ.md) +A generic FAQ about nDPI is available [here](https://github.com/ntop/nDPI/blob/dev/doc/FAQ.rst) ### How To Compile nDPI diff --git a/doc/FAQ.md b/doc/FAQ.md deleted file mode 100644 index 41efa0843..000000000 --- a/doc/FAQ.md +++ /dev/null @@ -1,37 +0,0 @@ - -# FAQ - -From [blog post](https://www.ntop.org/ndpi/ndpi-internals-and-frequent-questions/) - -**Q**: How nDPI implements protocol detection?\ -**A**: nDPI includes a list of protocol dissectors (364 as of today) that are able to dissect protocols such as WhatsApp or TLS. As soon as a new flow is submitted to nDPI, the library applies in sequence dissectors that can potentially match the protocols (i.e. telnet is a TCP-based protocol, and it will not be considered for UDP flows). We start from the dissector that can most probably match using the port number. This means for traffic on TCP/22 nDPI will start with the SSH dissectors and if not matching continue with the others. Dissection completes as soon as a protocol matches or when none of them matched and in this case the flow will be labelled as Unknown. - -**Q**: What is the nDPI release cycle?\ -**A**: We cut a release approximately every 6-8 months, fixes and improvements are on a daily basis (check the nDPI code on GitHub). - -**Q**: Is nDPI running on all popular platforms?\ -**A**: Yes it runs on Linux, macOS, Windows… and also on not-so-popular ones such as IBM mainframes. We support ARM, Intel, RISC… architectures. - -**Q**: How many packets does nDPI need in order to implement detection?\ -**A**: It depends on the protocol. For UDP-based protocols such as DNS one packet is enough, for more complex protocols such as TLS about 10 packets. For sure if after 15-20 packets nDPI has not detected the application protocol, then the protocol is labelled as Unknown. - -**Q**: Is nDPI detection only based on protocol dissectors?\ -**A**: No, payload inspection is the main technique, but nDPI can also use IP address, ports, TLS certificates, etc., as signatures for protocols. In this case, after detection is complete, nDPI will report if the match was performed on payload inspection or other means (e.g. IP address). - -**Q**: Does nDPI contain list of known IP addresses?\ -**A**: Yes it includes lists of well known IPs such as those provided by Microsoft of Meta for identifying known service. - -**Q**: Can I extend nDPI by defining new protocols with a configuration file?\ -**A**: Yes you can. See this [file](https://github.com/ntop/nDPI/blob/dev/example/protos.txt) as an example for defining new protocols. - -**Q**: Is nDPI able to detect VPNs?\ -**A**: Yes it can detect VPNS such as Tailscale, WireGuard, OpenVPN, FortiClient.. and also in-app VPNs such as UltraSurf or OperaVPN. - -**Q**: Is nDPI able to detect malware and viruses?\ -**A**: It can detect anomalous behaviour that can be caused by a malware, but nDPI is not a signature-based tool, so it does not include signatures for malware A or B. This is because [signature-based tools](https://en.wikipedia.org/wiki/Intrusion_detection_system) have various limitations and resource intensive, whereas nDPI has been designed to be used also in high-speed (100 Gbit+) networks. - -**Q**: Is nDPI able to detect security issues?\ -**A**: Yes it can by means of a technique called [flow risk](https://github.com/ntop/nDPI/blob/dev/doc/flow_risks.rst). It can identify 50+ threats (e.g. a host that is talking with a malware host). - -**Q**: Is nDPI able to block traffic?\ -**A**: No, nDPI is a passive traffic analysis library that does not manipulate packets. You can create applications on top of it for policing (i.e. blocking or shaping) traffic. Examples of such applications are ntopng Edge, nProbe IPS and nProbe Cento. diff --git a/doc/FAQ.rst b/doc/FAQ.rst new file mode 100644 index 000000000..059005bda --- /dev/null +++ b/doc/FAQ.rst @@ -0,0 +1,42 @@ +FAQ +=== + +From `blog post <https://www.ntop.org/ndpi/ndpi-internals-and-frequent-questions/>`_ + +**Q**: How does nDPI implement protocol detection? +**A**: nDPI includes a list of protocol dissectors (364 as of today) that can dissect protocols such as WhatsApp or TLS. As soon as a new flow is submitted to nDPI, the library applies in sequence dissectors that can potentially match the protocols (i.e., Telnet is a TCP-based protocol, and it will not be considered for UDP flows). +nDPI starts with the dissector most likely to match based on the port number (e.g., TCP/22 starts with SSH). Dissection completes when a match is found or if none match, in which case the flow is labeled as Unknown. + +**Q**: What is the nDPI release cycle? +**A**: A release is made approximately every 6–8 months. Fixes and improvements happen daily (see the nDPI code on GitHub). + +**Q**: Is nDPI running on all popular platforms? +**A**: Yes, it runs on Linux, macOS, Windows… and even on less common platforms like IBM mainframes. It supports ARM, Intel, RISC… architectures. + +**Q**: How many packets does nDPI need to perform detection? +**A**: It depends on the protocol. For UDP-based protocols like DNS, one packet may suffice. More complex protocols like TLS require about 10 packets. +If no protocol is detected after 15–20 packets, the protocol is labeled as Unknown. + +**Q**: Is nDPI detection only based on protocol dissectors? +**A**: No. While payload inspection is the primary method, nDPI can also use IP addresses, ports, TLS certificates, etc., as protocol signatures. +After detection, nDPI reports whether matching was based on payload or other means (e.g., IP address). + +**Q**: Does nDPI contain a list of known IP addresses? +**A**: Yes, it includes known IP lists, such as those provided by Microsoft or Meta, to identify known services. + +**Q**: Can I extend nDPI by defining new protocols with a configuration file? +**A**: Yes. See this `example file <https://github.com/ntop/nDPI/blob/dev/example/protos.txt>`_ for defining new protocols. + +**Q**: Can nDPI detect VPNs? +**A**: Yes, it can detect VPNs like Tailscale, WireGuard, OpenVPN, FortiClient, and in-app VPNs such as UltraSurf or OperaVPN. + +**Q**: Can nDPI detect malware and viruses? +**A**: It can detect anomalous behavior caused by malware. However, nDPI is not signature-based, so it does not include specific malware signatures. +This is because `signature-based tools <https://en.wikipedia.org/wiki/Intrusion_detection_system>`_ are limited and resource-intensive, whereas nDPI is designed for high-speed (100 Gbit+) networks. + +**Q**: Can nDPI detect security issues? +**A**: Yes, using a technique called `flow risk <https://github.com/ntop/nDPI/blob/dev/doc/flow_risks.rst>`_. It can detect 50+ threats (e.g., a host communicating with a malware host). + +**Q**: Can nDPI block traffic? +**A**: No. nDPI is a passive traffic analysis library. You can build applications on top of it to block or shape traffic (e.g., ntopng Edge, nProbe IPS, nProbe Cento). + diff --git a/doc/_static/custom.css b/doc/_static/custom.css deleted file mode 100644 index 0b570a74a..000000000 --- a/doc/_static/custom.css +++ /dev/null @@ -1,3 +0,0 @@ -.wy-nav-content { - max-width: 75% !important; -} diff --git a/doc/configuration_parameters.md b/doc/configuration_parameters.md deleted file mode 100644 index 5a5caeb7b..000000000 --- a/doc/configuration_parameters.md +++ /dev/null @@ -1,88 +0,0 @@ - -# Configuration knobs - -List of the supported configuration options: - -| Protocol | Parameter | Default value | Min value | Max value | Description | Notes | -| ------ | ------ | ------ | ------ | ------ | ------ | ------ | -| NULL | "packets_limit_per_flow" | 32 | 0 | 255 | The upper limit on the number of packets per flow that will be subject to DPI, after which classification will be considered complete (0 = no limit) | -| NULL | "flow.direction_detection" | enable | NULL | NULL | Enable/disable internal detection of packet direction (client to server or server to client) | -| NULL | "flow.track_payload" | disable | NULL | NULL | Enable/disable tracking/export of flow payload (i.e. L5/7 data): if enabled, the library exports the first 1024 bytes of payload for each flow | -| NULL | "flow.use_client_ip_in_guess" | enable | NULL | NULL | Use client IP in guesses of flow protocol IDs by IP. | -| NULL | "flow.use_client_port_in_guess" | enable | NULL | NULL | Use client port in guesses of flow protocol IDs. | -| NULL | "tcp_ack_payload_heuristic" | disable | NULL | NULL | In some networks, there are some anomalous TCP flows where the smallest ACK packets have some kind of zero padding. It looks like the IP and TCP headers in those frames wrongly consider the 0x00 Ethernet padding bytes as part of the TCP payload. While this kind of packets is perfectly valid per-se, in some conditions they might be treated by the TCP reassembler logic as (partial) overlaps, deceiving the classification engine. This parameter enable/disable an heuristic to detect these packets and to ignore them, allowing correct detection/classification. See #1946 for other details | -| NULL | "fully_encrypted_heuristic" | disable | NULL | NULL | Enable/disable an heuristic to detect fully encrypted sessions, i.e. flows where every bytes of the payload is encrypted in an attempt to “look like nothing”. This heuristic only analyzes the first packet of the flow. See: https://www.usenix.org/system/files/sec23fall-prepub-234-wu-mingshi.pdf | -| NULL | "libgcrypt.init" | 1 | NULL | NULL | Enable/disable initialization of libgcrypt. When using the external libgcrypt (instead of the internal crypto code) the libgcrypt runtime must be initialized. If, for whatever reasons, the application alread does it, nDPI must be told to skip it. Note that, by default, nDPI uses the crypto code and not libgcrypt: in that case this parameter is ignored | -| NULL | "dpi.compute_entropy" | 1 | NULL | NULL | Enable/disable computation of flow entropy | -| NULL | "fpc" | enable | NULL | NULL | Enable/disable First Packet Classification | -| NULL | "metadata.tcp_fingerprint" | enable | NULL | NULL | Enable/disable computation and export of TCP fingerprint for all TCP flows -| NULL | "metadata.tcp_fingerprint_raw" | disable | NULL | NULL | Enable/disable computation and export of raw TCP fingerprint for all TCP flows -| NULL | "metadata.tcp_fingerprint_format" | 0 | 0 | 1 | Specify the format of the TCP fingerprint. Possible values: 0 = native nDPI format, 1 = MuonOF (https://github.com/sundruid/muonfp) | -| NULL | "dpi.guess_on_giveup" | 0x03 | 0x00 | 0x03 | Tell the library to guess flow classification, if any DPI algorithms/logics fail. The value is a bitmask. Values: 0x0 = disabled; 0x01 = enable guessing by port; 0x02 = enable guessing by ip | -| NULL | "dpi.guess_ip_before_port" | disable | NULL | NULL | Enable/disable guessing by IP first when guessing flow classifcation. Disabled = guess by port first. | -| NULL | "flow_risk.$FLOWRISK_NAME_OR_ID" | enable | NULL | NULL | Enable/disable the specific flow risk. Use "any" as flow risk name if you want to easily enable/disable all flow risks. The names of the flow risks are available at `src/include/ndpi_typedefs.h`: look for `ndpi_risk_shortnames` | -| NULL | "flow_risk.$FLOWRISK_NAME_OR_ID.info" | enable | NULL | NULL | Enable/disable the export of flow risk information, i.e. some strings clarifing some details about the specific flow risk set. Use "any" as flow risk name if you want to easily enable/disable flow info for all flow risks. The names of the flow risks are available at `src/include/ndpi_typedefs.h`: look for `ndpi_risk_shortnames` | -| NULL | "flow_risk_lists.load" | 1 | NULL | NULL | Enable/disable loading of every IP addresses lists used to check any flow risks | -| NULL | "flow_risk.anonymous_subscriber.list.icloudprivaterelay.load" | 1 | NULL | NULL | Enable/disable loading of internal iCouldPrivateRealy IP address list used to check `NDPI_ANONYMOUS_SUBSCRIBER` flow risk | -| NULL | "flow_risk.anonymous_subscriber.list.tor.load" | 1 | NULL | NULL | Enable/disable loading of internal IP address list of TOR exit nodes used to check `NDPI_ANONYMOUS_SUBSCRIBER` flow risk | -| NULL | "flow_risk.crawler_bot.list.load" | 1 | NULL | NULL | Enable/disable loading of internal IP address list used to check `NDPI_HTTP_CRAWLER_BOT` flow risk | -| NULL | "filename.config" | NULL | NULL | NULL | Name of the file containing a list of configuration knobs itself (one per line)!. Useful to configure nDPI via text file instead of via API | -| NULL | "log.level" | 0 | 0 | 3 | Configure the log/debug level. Possible values: 0 = error, 1 = trace, 2 = debug, 3 = extra debug | -| NULL | "lru.$CACHE_NAME.size" | See description | 0 | 16777215 | Set the size (in number of elements) of the specified LRU cache (0 = the cache is disabled). The keyword "$CACHE_NAME" is a placeholder for the cache name and the possible values are: ookla, bittorrent, stun, tls_cert, mining, msteams, fpc_dns, signal. The default value is "32768" for the bittorrent and signal cache and "1024" for all the other caches | -| NULL | "lru.$CACHE_NAME.ttl" | See description | 0 | 16777215 | Set the TTL (in seconds) for the elements of the specified LRU cache (0 = the elements never explicitly expire). The keyword "$CACHE_NAME" is a placeholder for the cache name and the possible values are: ookla, bittorrent, stun, tls_cert, mining, msteams, fpc_dns, signal. The default value is "120" for the ookla cache, "60" for the msteams and fpc_dns caches and "300" for all the other caches | -| NULL | "lru.$CACHE_NAME.scope" | 0 | 0 | 1 | Set the scope of the specified LRU cache (0 = the cache is local, 1 = the cache is global). The keyword "$CACHE_NAME" is a placeholder for the cache name and the possible values are: ookla, bittorrent, stun, tls_cert, mining, msteams, fpc_dns, signal. The global scope con be set only if a global context has been initialized | -| "http" | "metadata.request_content_type" | enable | NULL | NULL | Enable/disable export of Request Content Type header for HTTP flows. | -| "http" | "metadata.referer" | enable | NULL | NULL | Enable/disable export of Referer header for HTTP flows. | -| "http" | "metadata.host" | enable | NULL | NULL | Enable/disable export of Host header for HTTP flows. | -| "http" | "metadata.username" | enable | NULL | NULL | Enable/disable export of (cleartext) username metadata for HTTP flows. | -| "http" | "metadata.password" | enable | NULL | NULL | Enable/disable export of (cleartext) password metadata for HTTP flows. | -| "tls" | "certificate_expiration_threshold" | 30 | 0 | 365 | The threshold (in days) used to trigger the `NDPI_TLS_CERTIFICATE_ABOUT_TO_EXPIRE` flow risk | -| "tls" | "application_blocks_tracking" | disable | NULL | NULL | Enable/disable processing of TLS Application Blocks (post handshake) to extract statistical information about the flow | -| "tls " | "dpi.heuristics", | 0x00 | 0x00 | 0x07 | Enable/disable some heuristics to detect encrypted/obfuscated/proxied TLS flows. The value is a bitmask. Values: 0x0 = disabled; 0x01 = enable basic detection (i.e. encrypted TLS without any encapsulation); 0x02 = enable detection over TLS (i.e. TLS-in-TLS); 0x04 = enable detection over HTTP (i.e. TLS-over-WebSocket). If enabled, some false positives are expected. See: https://www.usenix.org/conference/usenixsecurity24/presentation/xue-fingerprinting | -| "tls " | "dpi.heuristics.max_packets_extra_dissection", | 25 | 0 | 255 | If at least one TLS heuristics is enabled (see `tls,"dpi.heuristics"`, this parameter set the upper limit on the number of packets required/processed for each flow. Higher the value, lower the false positive rate but more packets are required by nDPI for processing. | -| "tls" | "metadata.sha1_fingerprint" | enable | NULL | NULL | Enable/disable computation and export of SHA1 fingerprint for TLS flows. Note that if it is disable, the flow risk `NDPI_MALICIOUS_SHA1_CERTIFICATE` is not checked | -| "tls" | "metadata.versions_supported" | enable | NULL | NULL | Enable/disable export of supported versions metadata for TLS flows | -| "tls" | "metadata.alpn_negotiated" | enable | NULL | NULL | Enable/disable export of negotiated ALPN metadata for TLS flows | -| "tls" | "metadata.cipher" | enable | NULL | NULL | Enable/disable export of negotiated cipher metadata for TLS flows | -| "tls" | "metadata.cert_server_names" | enable | NULL | NULL | Enable/disable export of server names list from certificate for TLS flows | -| "tls" | "metadata.cert_validity" | enable | NULL | NULL | Enable/disable export of certificate validity timestamps for TLS flows | -| "tls" | "metadata.cert_issuer" | enable | NULL | NULL | Enable/disable export of certificate issuer metadata for TLS flows | -| "tls" | "metadata.cert_subject" | enable | NULL | NULL | Enable/disable export of certificaste subject metadata for TLS flows | -| "tls" | "metadata.browser" | enable | NULL | NULL | Enable/disable an heurstic to determine the broswer used to generate this TLS flows | -| "tls" | "metadata.ja3s_fingerprint" | enable | NULL | NULL | Enable/disable computation and export of JA3S fingerprint for TLS flows | -| "tls" | "metadata.ja4c_fingerprint" | enable | NULL | NULL | Enable/disable computation and export of JA4C fingerprint for TLS flows. Note that if it is disable, the flow risk `NDPI_MALICIOUS_FINGERPRINT` is not checked | -| "tls" | "metadata.ja4r_fingerprint" | disable | NULL | NULL | Enable/disable computation and export of JA4C fingerprint for TLS flows also in raw format | -| "tls" | "subclassification" | enable | NULL | NULL | Enable/disable sub-classification of TLS/DTLS flows | -| "quic" | "subclassification" | enable | NULL | NULL | Enable/disable sub-classification of QUIC flows | -| "smtp" | "tls_dissection" | enable | NULL | NULL | Enable/disable dissection of TLS packets in cleartext SMTP flows (because of opportunistic TLS, via STARTTLS msg) | -| "imap" | "tls_dissection" | enable | NULL | NULL | Enable/disable dissection of TLS packets in cleartext IMAP flows (because of opportunistic TLS, via STARTTLS msg) | -| "pop" | "tls_dissection" | enable | NULL | NULL | Enable/disable dissection of TLS packets in cleartext POP flows (because of opportunistic TLS, via STARTTLS msg) | -| "ftp" | "tls_dissection" | enable | NULL | NULL | Enable/disable dissection of TLS packets in cleartext FTP flows (because of opportunistic TLS, via AUTH TLS msg) | -| "sip" | "metadata.attribute.from" | enable | NULL | NULL | Enable/disable extraction of "From" header from SIP flows | -| "sip" | "metadata.attribute.from_imsi" | enable | NULL | NULL | In a SIP flow, if the "From" header contains a valid IMSI, this option enable/disable the extraction of the IMSI itself | -| "sip" | "metadata.attribute.to" | enable | NULL | NULL | Enable/disable extraction of "To" header from SIP flows | -| "sip" | "metadata.attribute.to_imsi" | enable | NULL | NULL | In a SIP flow, if the "To" header contains a valid IMSI, this option enable/disable the extraction of the IMSI itself | -| "stun" | "max_packets_extra_dissection" | 4 | 0 | 255 | After a flow has been classified has STUN, nDPI might analyse more packets to look for a sub-classification or for metadata. This parameter set the upper limit on the number of these packets | -| "stun" | "tls_dissection" | enable | NULL | NULL | Enable/disable dissection of TLS packets multiplexed into STUN flows | -| "stun" | "metadata.attribute.mapped_address" | enable | NULL | NULL | Enable/disable extraction of (xor)-mapped-address attribute for STUN flows. If it is disabled, STUN classification might be significant faster | -| "stun" | "metadata.attribute.response_origin" | enable | NULL | NULL | Enable/disable extraction of response-origin attribute for STUN flows. If it is disabled, STUN classification might be significant faster | -| "stun" | "metadata.attribute.other_address" | enable | NULL | NULL | Enable/disable extraction of other-address attribute for STUN flows. If it is disabled, STUN classification might be significant faster | -| "stun" | "metadata.attribute.relayed_address" | enable | NULL | NULL | Enable/disable extraction of (xor-)relayed-address attribute for STUN flows. If it is disabled, STUN classification might be significant faster | -| "stun" | "metadata.attribute.peer_address" | enable | NULL | NULL | Enable/disable extraction of (xor-)peer-address attribute for STUN flows. If it is disabled, STUN classification might be significant faster; however sub-classification capability might be negatively impacted | -| "bittorrent" | "metadata.hash" | enable | NULL | NULL | Enable/disable extraction of hash metadata for Bittorrent flows. | -| "ssdp" | "metadata" | enable | NULL | NULL | Enable/disable extraction of ALL metadata for SSDP flows. Note that, unlike all others protocols, for SSDP flows you can't enable/disable a specific metadata | -| "dns" | "subclassification" | disable | NULL | NULL | Enable/disable sub-classification of DNS flows (via query/response domain name). | -| "dns" | "process_response" | enable | NULL | NULL | Enable/disable processing of DNS responses. By default, DNS flows are fully classified after the first request/response pair (or after the first response, if the request is missing). If this parameter is disabled, the flows are fully classified after the first packet, i.e. usually after the first request; in that case, some flow risks are not checked and some metadata are not exported | -| "http" | "process_response" | enable | NULL | NULL | Enable/disable processing of HTTP responses. By default, HTTP flows are usually fully classified after the first request/response pair. If this parameter is disabled, the flows are fully classified after the first request (or after the first response, if the request is missing); in that case, some flow risks are not checked and some metadata are not exported | -| "http" | "subclassification" | enable | NULL | NULL | Enable/disable sub-classification of HTTP flows | -| "ookla" | "dpi.aggressiveness", | 0x01 | 0x00 | 0x01 | Detection aggressiveness for Ookla. The value is a bitmask. Values: 0x0 = disabled; 0x01 = enable heuristic for detection over TLS (via Ookla LRU cache) | -| "zoom" | "max_packets_extra_dissection" | 4 | 0 | 255 | After a flow has been classified has Zoom, nDPI might analyse more packets to look for a sub-classification or for metadata. This parameter set the upper limit on the number of these packets | -| "rtp" | "search_for_stun" | disable | NULL | NULL | After a flow has been classified as RTP or RTCP, nDPI might analyse more packets to look for STUN/DTLS packets, i.e. to try to tell if this flow is a "pure" RTP/RTCP flow or if the RTP/RTCP packets are multiplexed with STUN/DTLS. Useful for proper (sub)classification when the beginning of the flows are not captured or if there are lost packets in the the captured traffic. If enabled, nDPI requires more packets to process for each RTP/RTCP flow. | -| "rtp" | "max_packets_extra_dissection" | 32 | 0 | 255 | After a flow has been classified has RTP, nDPI might analyse more packets to look for more metadat. This parameter set the upper limit on the number of these packets | -| "openvpn" | "dpi.heuristics", | 0x00 | 0 | 0x01 | Enable/disable some heuristics to better detect OpenVPN. The value is a bitmask. Values: 0x0 = disabled; 0x01 = enable heuristic based on op-code frequency. If enabled, some false positives are expected. See: https://www.usenix.org/conference/usenixsecurity22/presentation/xue-diwen | -| "openvpn" | "dpi.heuristics.num_messages", | 10 | 0 | 255 | If at least one OpenVPN heuristics is enabled (see `openvpn,"dpi.heuristics"`, this parameter set the maximum number of OpenVPN messages required for each flow. Note that an OpenVPN message may be splitted into multiple (TCP/UDP) packets and that a (TCP/UDP) packet may contains multiple OpenVPN messages. Higher the value, lower the false positive rate but more packets are required by nDPI for processing. | -| "openvpn" | "subclassification_by_ip" | enable | NULL | NULL | Enable/disable sub-classification of OpenVPN flows using server IP. Useful to detect the specific VPN application/app. At the moment, this knob allows to identify: Mullvad, NordVPN. | -| "wireguard" | "subclassification_by_ip" | enable | NULL | NULL | Enable/disable sub-classification of Wireguard flows using server IP. Useful to detect the specific VPN application/app. At the moment, this knob allows to identify: Mullvad, NordVPN. | -| $PROTO_NAME | "log" | disable | NULL | NULL | Enable/disable logging/debug for specific protocol. Use "any" as protocol name if you want to easily enable/disable logging/debug for all protocols | -| $PROTO_NAME | "ip_list.load" | enable | NULL | NULL | Enable/disable loading of internal list of IP addresses (used for (sub)classification) specific to that protocol. Use "any" as protocol name if you want to easily enable/disable all lists. This knob is valid only for the following protocols: Alibaba, Amazon AWS, Apple, Avast, Blizzard, Bloomberg, Cachefly, Canonical, Cloudflare, DigitalOcean, Discord, Disney+, Dropbox, Edgecast, EpicGames, Ethereum, Facebook, Github, Google, Google Cloud, GoTo, Hotspot Shield, Hulu, Line, Microsoft 365, Microsoft Azure, Microsoft One Drive, Microsoft Outlook, Microsoft Teams, Mullvad, Netflix, NordVPN, Nvidia, OpenDNS, RiotGames, Roblox, Steam, SurfSharkVPN, Teamviewer, Telegram, Tencent, Threema, TOR, Twitch, Twitter, VK, Yandex, Yandex Cloud, Webex, Whatsapp, Zoom | -| $PROTO_NAME | "monitoring" | disable | NULL | NULL | Enable/disable monitoring state for this specific protocol. Use "any" as protocol name if you want to easily enable/disable monitoring feature for all protocols. This knob is valid only for the following protocols: Stun. Monitoring allows nDPI to process the entire flow (i.e. all its packets), without any limits. See doc/monitoring.md for further details | diff --git a/doc/configuration_parameters.rst b/doc/configuration_parameters.rst new file mode 100644 index 000000000..f91da312e --- /dev/null +++ b/doc/configuration_parameters.rst @@ -0,0 +1,206 @@ +Configuration Knobs +=================== + +List of the supported configuration options: + ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| Protocol | Parameter | Default value | Min value | Max value | Description | ++==============+===============================================================+=================+============+============+===================================================================================================================================================================+ +| NULL | "packets_limit_per_flow" | 32 | 0 | 255 | The upper limit on the number of packets per flow that will be subject to DPI, after which classification will be considered complete (0 = no limit) | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| NULL | "flow.direction_detection" | enable | NULL | NULL | Enable/disable internal detection of packet direction (client to server or server to client) | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| NULL | "flow.track_payload" | disable | NULL | NULL | Enable/disable tracking/export of flow payload (i.e. L5/7 data): if enabled, the library exports the first 1024 bytes of payload for each flow | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| NULL | "flow.use_client_ip_in_guess" | enable | NULL | NULL | Use client IP in guesses of flow protocol IDs by IP. | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| NULL | "flow.use_client_port_in_guess" | enable | NULL | NULL | Use client port in guesses of flow protocol IDs. | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| NULL | "tcp_ack_payload_heuristic" | disable | NULL | NULL | In some networks, there are anomalous TCP flows with 0x00 Ethernet padding bytes treated as TCP payload. Enables heuristic to detect and ignore these. See #1946 | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| NULL | "fully_encrypted_heuristic" | disable | NULL | NULL | Enable heuristic to detect fully encrypted sessions. Analyzes first packet only (see: https://www.usenix.org/system/files/sec23fall-prepub-234-wu-mingshi.pdf). | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| NULL | "libgcrypt.init" | 1 | NULL | NULL | Enable/disable initialization of libgcrypt. Ignored if nDPI is not using external libgcrypt. | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| NULL | "dpi.compute_entropy" | 1 | NULL | NULL | Enable/disable computation of flow entropy. | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| NULL | "fpc" | enable | NULL | NULL | Enable/disable First Packet Classification. | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| NULL | "metadata.tcp_fingerprint" | enable | NULL | NULL | Enable/disable computation and export of TCP fingerprint for all TCP flows. | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| NULL | "metadata.tcp_fingerprint_raw" | disable | NULL | NULL | Enable/disable computation and export of raw TCP fingerprint. | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| NULL | "metadata.tcp_fingerprint_format" | 0 | 0 | 1 | Format of the TCP fingerprint. 0 = native nDPI format, 1 = MuonOF (see: https://github.com/sundruid/muonfp). | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| NULL | "dpi.guess_on_giveup" | 0x03 | 0x00 | 0x03 | Guess flow classification if DPI fails. Bitmask: 0x0 = disabled; 0x01 = port; 0x02 = IP. | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| NULL | "dpi.guess_ip_before_port" | disable | NULL | NULL | Enable/disable guessing by IP first when guessing flow classifcation. Disabled = guess by port first. | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| NULL | "flow_risk.$FLOWRISK_NAME_OR_ID" | enable | NULL | NULL | Enable/disable the specific flow risk. Use "any" as flow risk name if you want to easily enable/disable all flow risks. The names of the flow risks are available | +| | | | | | at `src/include/ndpi_typedefs.h`: look for `ndpi_risk_shortnames` | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| NULL | "flow_risk.$FLOWRISK_NAME_OR_ID.info" | enable | NULL | NULL | Enable/disable the export of flow risk information, i.e. some strings clarifing some details about the specific flow risk set. Use "any" as flow risk name if | +| | | | | | you want to easily enable/disable flow info for all flow risks. The names of the flow risks are available at `src/include/ndpi_typedefs.h`: look for | +| | | | | | `ndpi_risk_shortnames` | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| NULL | "flow_risk_lists.load" | 1 | NULL | NULL | Enable/disable loading of every IP addresses lists used to check any flow risks | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| NULL | "flow_risk.anonymous_subscriber.list.icloudprivaterelay.load" | 1 | NULL | NULL | Enable/disable loading of internal iCouldPrivateRealy IP address list used to check `NDPI_ANONYMOUS_SUBSCRIBER` flow risk | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| NULL | "flow_risk.anonymous_subscriber.list.tor.load" | 1 | NULL | NULL | Enable/disable loading of internal IP address list of TOR exit nodes used to check `NDPI_ANONYMOUS_SUBSCRIBER` flow risk | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| NULL | "flow_risk.crawler_bot.list.load" | 1 | NULL | NULL | Enable/disable loading of internal IP address list used to check `NDPI_HTTP_CRAWLER_BOT` flow risk | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| NULL | "filename.config" | NULL | NULL | NULL | Name of the file containing a list of configuration knobs itself (one per line)!. Useful to configure nDPI via text file instead of via API | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| NULL | "log.level" | 0 | 0 | 3 | Configure the log/debug level. Possible values: 0 = error, 1 = trace, 2 = debug, 3 = extra debug | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| NULL | "lru.$CACHE_NAME.size" | See description | 0 | 16777215 | Set the size (in number of elements) of the specified LRU cache (0 = the cache is disabled). The keyword "$CACHE_NAME" is a placeholder for the cache name and | +| | | | | | the possible values are: ookla, bittorrent, stun, tls_cert, mining, msteams, fpc_dns, signal. The default value is "32768" for the bittorrent and signal cache | +| | | | | | and "1024" for all the other caches | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| NULL | "lru.$CACHE_NAME.ttl" | See description | 0 | 16777215 | Set the TTL (in seconds) for the elements of the specified LRU cache (0 = the elements never explicitly expire). The keyword "$CACHE_NAME" is a placeholder for | +| | | | | | the cache name and the possible values are: ookla, bittorrent, stun, tls_cert, mining, msteams, fpc_dns, signal. The default value is "120" for the ookla cache, | +| | | | | | "60" for the msteams and fpc_dns caches and "300" for all the other caches | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| NULL | "lru.$CACHE_NAME.scope" | 0 | 0 | 1 | Set the scope of the specified LRU cache (0 = the cache is local, 1 = the cache is global). The keyword "$CACHE_NAME" is a placeholder for the cache name and the | +| | | | | | possible values are: ookla, bittorrent, stun, tls_cert, mining, msteams, fpc_dns, signal. The global scope con be set only if a global context has been | +| | | | | | initialized | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "http" | "metadata.request_content_type" | enable | NULL | NULL | Enable/disable export of Request Content Type header for HTTP flows. | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "http" | "metadata.referer" | enable | NULL | NULL | Enable/disable export of Referer header for HTTP flows. | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "http" | "metadata.host" | enable | NULL | NULL | Enable/disable export of Host header for HTTP flows. | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "http" | "metadata.username" | enable | NULL | NULL | Enable/disable export of (cleartext) username metadata for HTTP flows. | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "http" | "metadata.password" | enable | NULL | NULL | Enable/disable export of (cleartext) password metadata for HTTP flows. | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "tls" | "certificate_expiration_threshold" | 30 | 0 | 365 | The threshold (in days) used to trigger the `NDPI_TLS_CERTIFICATE_ABOUT_TO_EXPIRE` flow risk | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "tls" | "application_blocks_tracking" | disable | NULL | NULL | Enable/disable processing of TLS Application Blocks (post handshake) to extract statistical information about the flow | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "tls " | "dpi.heuristics", | 0x00 | 0x00 | 0x07 | Enable/disable some heuristics to detect encrypted/obfuscated/proxied TLS flows. The value is a bitmask. Values: 0x0 = disabled; 0x01 = enable basic detection | +| | | | | | (i.e. encrypted TLS without any encapsulation); 0x02 = enable detection over TLS (i.e. TLS-in-TLS); 0x04 = enable detection over HTTP (i.e. TLS-over-WebSocket). | +| | | | | | If enabled, some false positives are expected. See: https://www.usenix.org/conference/usenixsecurity24/presentation/xue-fingerprinting | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "tls " | "dpi.heuristics.max_packets_extra_dissection", | 25 | 0 | 255 | If at least one TLS heuristics is enabled (see `tls,"dpi.heuristics"`, this parameter set the upper limit on the number of packets required/processed for each | +| | | | | | flow. Higher the value, lower the false positive rate but more packets are required by nDPI for processing. | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "tls" | "metadata.sha1_fingerprint" | enable | NULL | NULL | Enable/disable computation and export of SHA1 fingerprint for TLS flows. Note that if it is disable, the flow risk `NDPI_MALICIOUS_SHA1_CERTIFICATE` is not | +| | | | | | checked | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "tls" | "metadata.versions_supported" | enable | NULL | NULL | Enable/disable export of supported versions metadata for TLS flows | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "tls" | "metadata.alpn_negotiated" | enable | NULL | NULL | Enable/disable export of negotiated ALPN metadata for TLS flows | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "tls" | "metadata.cipher" | enable | NULL | NULL | Enable/disable export of negotiated cipher metadata for TLS flows | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "tls" | "metadata.cert_server_names" | enable | NULL | NULL | Enable/disable export of server names list from certificate for TLS flows | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "tls" | "metadata.cert_validity" | enable | NULL | NULL | Enable/disable export of certificate validity timestamps for TLS flows | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "tls" | "metadata.cert_issuer" | enable | NULL | NULL | Enable/disable export of certificate issuer metadata for TLS flows | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "tls" | "metadata.cert_subject" | enable | NULL | NULL | Enable/disable export of certificaste subject metadata for TLS flows | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "tls" | "metadata.browser" | enable | NULL | NULL | Enable/disable an heurstic to determine the broswer used to generate this TLS flows | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "tls" | "metadata.ja3s_fingerprint" | enable | NULL | NULL | Enable/disable computation and export of JA3S fingerprint for TLS flows | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "tls" | "metadata.ja4c_fingerprint" | enable | NULL | NULL | Enable/disable computation and export of JA4C fingerprint for TLS flows. Note that if it is disable, the flow risk `NDPI_MALICIOUS_FINGERPRINT` is not checked | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "tls" | "metadata.ja4r_fingerprint" | disable | NULL | NULL | Enable/disable computation and export of JA4C fingerprint for TLS flows also in raw format | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "tls" | "subclassification" | enable | NULL | NULL | Enable/disable sub-classification of TLS/DTLS flows | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "quic" | "subclassification" | enable | NULL | NULL | Enable/disable sub-classification of QUIC flows | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "smtp" | "tls_dissection" | enable | NULL | NULL | Enable/disable dissection of TLS packets in cleartext SMTP flows (because of opportunistic TLS, via STARTTLS msg) | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "imap" | "tls_dissection" | enable | NULL | NULL | Enable/disable dissection of TLS packets in cleartext IMAP flows (because of opportunistic TLS, via STARTTLS msg) | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "pop" | "tls_dissection" | enable | NULL | NULL | Enable/disable dissection of TLS packets in cleartext POP flows (because of opportunistic TLS, via STARTTLS msg) | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "ftp" | "tls_dissection" | enable | NULL | NULL | Enable/disable dissection of TLS packets in cleartext FTP flows (because of opportunistic TLS, via AUTH TLS msg) | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "sip" | "metadata.attribute.from" | enable | NULL | NULL | Enable/disable extraction of "From" header from SIP flows | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "sip" | "metadata.attribute.from_imsi" | enable | NULL | NULL | In a SIP flow, if the "From" header contains a valid IMSI, this option enable/disable the extraction of the IMSI itself | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "sip" | "metadata.attribute.to" | enable | NULL | NULL | Enable/disable extraction of "To" header from SIP flows | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "sip" | "metadata.attribute.to_imsi" | enable | NULL | NULL | In a SIP flow, if the "To" header contains a valid IMSI, this option enable/disable the extraction of the IMSI itself | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "stun" | "max_packets_extra_dissection" | 4 | 0 | 255 | After a flow has been classified has STUN, nDPI might analyse more packets to look for a sub-classification or for metadata. This parameter set the upper limit | +| | | | | | on the number of these packets | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "stun" | "tls_dissection" | enable | NULL | NULL | Enable/disable dissection of TLS packets multiplexed into STUN flows | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "stun" | "metadata.attribute.mapped_address" | enable | NULL | NULL | Enable/disable extraction of (xor)-mapped-address attribute for STUN flows. If it is disabled, STUN classification might be significant faster | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "stun" | "metadata.attribute.response_origin" | enable | NULL | NULL | Enable/disable extraction of response-origin attribute for STUN flows. If it is disabled, STUN classification might be significant faster | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "stun" | "metadata.attribute.other_address" | enable | NULL | NULL | Enable/disable extraction of other-address attribute for STUN flows. If it is disabled, STUN classification might be significant faster | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "stun" | "metadata.attribute.relayed_address" | enable | NULL | NULL | Enable/disable extraction of (xor-)relayed-address attribute for STUN flows. If it is disabled, STUN classification might be significant faster | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "stun" | "metadata.attribute.peer_address" | enable | NULL | NULL | Enable/disable extraction of (xor-)peer-address attribute for STUN flows. If it is disabled, STUN classification might be significant faster; however | +| | | | | | sub-classification capability might be negatively impacted | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "bittorrent" | "metadata.hash" | enable | NULL | NULL | Enable/disable extraction of hash metadata for Bittorrent flows. | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "ssdp" | "metadata" | enable | NULL | NULL | Enable/disable extraction of ALL metadata for SSDP flows. Note that, unlike all others protocols, for SSDP flows you can't enable/disable a specific metadata | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "dns" | "subclassification" | disable | NULL | NULL | Enable/disable sub-classification of DNS flows (via query/response domain name). | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "dns" | "process_response" | enable | NULL | NULL | Enable/disable processing of DNS responses. By default, DNS flows are fully classified after the first request/response pair (or after the first response, if the | +| | | | | | request is missing). If this parameter is disabled, the flows are fully classified after the first packet, i.e. usually after the first request; in that case, | +| | | | | | some flow risks are not checked and some metadata are not exported | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "http" | "process_response" | enable | NULL | NULL | Enable/disable processing of HTTP responses. By default, HTTP flows are usually fully classified after the first request/response pair. If this parameter is | +| | | | | | disabled, the flows are fully classified after the first request (or after the first response, if the request is missing); in that case, some flow risks are not | +| | | | | | checked and some metadata are not exported | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "http" | "subclassification" | enable | NULL | NULL | Enable/disable sub-classification of HTTP flows | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "ookla" | "dpi.aggressiveness", | 0x01 | 0x00 | 0x01 | Detection aggressiveness for Ookla. The value is a bitmask. Values: 0x0 = disabled; 0x01 = enable heuristic for detection over TLS (via Ookla LRU cache) | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "zoom" | "max_packets_extra_dissection" | 4 | 0 | 255 | After a flow has been classified has Zoom, nDPI might analyse more packets to look for a sub-classification or for metadata. This parameter set the upper limit | +| | | | | | on the number of these packets | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "rtp" | "search_for_stun" | disable | NULL | NULL | After a flow has been classified as RTP or RTCP, nDPI might analyse more packets to look for STUN/DTLS packets, i.e. to try to tell if this flow is a "pure" | +| | | | | | RTP/RTCP flow or if the RTP/RTCP packets are multiplexed with STUN/DTLS. Useful for proper (sub)classification when the beginning of the flows are not captured | +| | | | | | or if there are lost packets in the the captured traffic. If enabled, nDPI requires more packets to process for each RTP/RTCP flow. | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "rtp" | "max_packets_extra_dissection" | 32 | 0 | 255 | After a flow has been classified has RTP, nDPI might analyse more packets to look for more metadat. This parameter set the upper limit on the number of these | +| | | | | | packets | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "openvpn" | "dpi.heuristics", | 0x00 | 0 | 0x01 | Enable/disable some heuristics to better detect OpenVPN. The value is a bitmask. Values: 0x0 = disabled; 0x01 = enable heuristic based on op-code frequency. | +| | | | | | If enabled, some false positives are expected. See: https://www.usenix.org/conference/usenixsecurity22/presentation/xue-diwen | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "openvpn" | "dpi.heuristics.num_messages", | 10 | 0 | 255 | If at least one OpenVPN heuristics is enabled (see `openvpn,"dpi.heuristics"`, this parameter set the maximum number of OpenVPN messages required for each flow. | +| | | | | | Note that an OpenVPN message may be splitted into multiple (TCP/UDP) packets and that a (TCP/UDP) packet may contains multiple OpenVPN messages. Higher the value,| +| | | | | | lower the false positive rate but more packets are required by nDPI for processing. | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "openvpn" | "subclassification_by_ip" | enable | NULL | NULL | Enable/disable sub-classification of OpenVPN flows using server IP. Useful to detect the specific VPN application/app. At the moment, this knob allows to | +| | | | | | identify: Mullvad, NordVPN. | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| "wireguard" | "subclassification_by_ip" | enable | NULL | NULL | Enable/disable sub-classification of Wireguard flows using server IP. Useful to detect the specific VPN application/app. At the moment, this knob allows to | +| | | | | | identify: Mullvad, NordVPN. | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| $PROTO_NAME | "log" | disable | NULL | NULL | Enable/disable logging/debug for specific protocol. Use "any" as protocol name if you want to easily enable/disable logging/debug for all protocols | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| $PROTO_NAME | "ip_list.load" | enable | NULL | NULL | Enable/disable loading of internal list of IP addresses (used for (sub)classification) specific to that protocol. Use "any" as protocol name if you want to | +| | | | | | easily enable/disable all lists. This knob is valid only for the following protocols: Alibaba, Amazon AWS, Apple, Avast, Blizzard, Bloomberg, Cachefly, Canonical,| +| | | | | | Cloudflare, DigitalOcean, Discord, Disney+, Dropbox, Edgecast, EpicGames, Ethereum, Facebook, Github, Google, Google Cloud, GoTo, Hotspot Shield, Hulu, Line, | +| | | | | | Microsoft 365, Microsoft Azure, Microsoft One Drive, Microsoft Outlook, Microsoft Teams, Mullvad, Netflix, NordVPN, Nvidia, OpenDNS, RiotGames, Roblox, Steam, | +| | | | | | SurfSharkVPN, Teamviewer, Telegram, Tencent, Threema, TOR, Twitch, Twitter, VK, Yandex, Yandex Cloud, Webex, Whatsapp, Zoom | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| $PROTO_NAME | "monitoring" | disable | NULL | NULL | Enable/disable monitoring state for this specific protocol. Use "any" as protocol name if you want to easily enable/disable monitoring feature for all protocols. | +| | | | | | This knob is valid only for the following protocols: Stun. Monitoring allows nDPI to process the entire flow (i.e. all its packets), without any limits. | +| | | | | | See doc/monitoring.md for further details | ++--------------+---------------------------------------------------------------+-----------------+------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + + diff --git a/doc/include_FAQ.rst b/doc/include_FAQ.rst deleted file mode 100644 index 3e6cc12e3..000000000 --- a/doc/include_FAQ.rst +++ /dev/null @@ -1,2 +0,0 @@ -.. include:: FAQ.md -# :parser: myst_parser.sphinx_ diff --git a/doc/include_configuration_parameters.rst b/doc/include_configuration_parameters.rst deleted file mode 100644 index 49eed3e4f..000000000 --- a/doc/include_configuration_parameters.rst +++ /dev/null @@ -1,2 +0,0 @@ -.. include:: configuration_parameters.md -# :parser: myst_parser.sphinx_ diff --git a/doc/include_library_initialization.rst b/doc/include_library_initialization.rst deleted file mode 100644 index 0e284e104..000000000 --- a/doc/include_library_initialization.rst +++ /dev/null @@ -1,2 +0,0 @@ -.. include:: library_initialization.md -# :parser: myst_parser.sphinx_ diff --git a/doc/include_monitoring.rst b/doc/include_monitoring.rst deleted file mode 100644 index 0c2517ab5..000000000 --- a/doc/include_monitoring.rst +++ /dev/null @@ -1,2 +0,0 @@ -.. include:: monitoring.md -# :parser: myst_parser.sphinx_ diff --git a/doc/index.rst b/doc/index.rst index 32a8f30e3..44c8ebaa7 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -8,18 +8,18 @@ nDPI is an open source DPI (Deep Packet Inspection) toolkit for traffic analysis :caption: User's Guide what_is_ndpi - include_FAQ + FAQ .. toctree:: :maxdepth: 2 :caption: Developer's Guide api/library_root + configuration_parameters + library_initialization + monitoring protocols flow_risks - include_configuration_parameters - include_library_initialization - include_monitoring .. toctree:: :caption: Other Products diff --git a/doc/library_initialization.md b/doc/library_initialization.md deleted file mode 100644 index 491d00c93..000000000 --- a/doc/library_initialization.md +++ /dev/null @@ -1,84 +0,0 @@ - -# Library Initialization - -A simple, common example - -``` - -struct ndpi_detection_module_struct *ndpi_struct; -ndpi_cfg_error rc; -int ret; - -ndpi_struct = ndpi_init_detection_module(NULL); -if(!ndpi_struct) { - ERROR; -} - -/* Configuration */ - -rc = ndpi_set_config(ndpi_struct, "tls", "certificate_expiration_threshold", "10"); -if(rc != NDPI_CFG_OK) { - ERROR; -} - -/* Finalization */ -ret = ndpi_finalize_initialization(ndpi_struct); -if(ret != 0) { - ERROR; -} - - -/* Initialization done, now you can feed packets to the library */ - - - -/* Cleanup */ - -ndpi_exit_detection_module(ndpi_struct); - - -``` - -A more complex example, with global context and a shared Oookla LRU cache (all the others caches are local) - -``` - -struct ndpi_global_context *g_ctx; -struct ndpi_detection_module_struct *ndpi_structs[num_local_contexts]; -ndpi_cfg_error rc; -int i, ret; - -g_ctx = ndpi_global_init(); -if(!g_ctx) { - ERROR; -} - -for(i = 0; i < num_local_contexts; i++) { - ndpi_structs[i] = ndpi_init_detection_module(g_ctx); - if(!ndpi_struct[i]) { - ERROR; - } - - rc = ndpi_set_config(ndpi_structs[i], NULL, "lru.ookla.scope", "1"); - if(rc != NDPI_CFG_OK) { - ERROR; - } - - ret = ndpi_finalize_initialization(ndpi_structs[i]); - if(ret != 0) { - ERROR; - } -} - -/* Initialization done */ - -/* Cleanup */ - -for(i = 0; i < num_local_contexts; i++) { - ndpi_exit_detection_module(ndpi_structs[i]); -} - -ndpi_global_deinit(g_ctx); - - -``` diff --git a/doc/library_initialization.rst b/doc/library_initialization.rst new file mode 100644 index 000000000..c374f293d --- /dev/null +++ b/doc/library_initialization.rst @@ -0,0 +1,84 @@ +Library Initialization +======================= + + +# Library Initialization + +A simple, common example + +.. code:: c + + struct ndpi_detection_module_struct *ndpi_struct; + ndpi_cfg_error rc; + int ret; + + ndpi_struct = ndpi_init_detection_module(NULL); + if(!ndpi_struct) { + ERROR; + } + + /* Configuration */ + + rc = ndpi_set_config(ndpi_struct, "tls", "certificate_expiration_threshold", "10"); + if(rc != NDPI_CFG_OK) { + ERROR; + } + + /* Finalization */ + ret = ndpi_finalize_initialization(ndpi_struct); + if(ret != 0) { + ERROR; + } + + + /* Initialization done, now you can feed packets to the library */ + + + + /* Cleanup */ + + ndpi_exit_detection_module(ndpi_struct); + + + +A more complex example, with global context and a shared Oookla LRU cache (all the others caches are local) + +.. code:: c + + struct ndpi_global_context *g_ctx; + struct ndpi_detection_module_struct *ndpi_structs[num_local_contexts]; + ndpi_cfg_error rc; + int i, ret; + + g_ctx = ndpi_global_init(); + if(!g_ctx) { + ERROR; + } + + for(i = 0; i < num_local_contexts; i++) { + ndpi_structs[i] = ndpi_init_detection_module(g_ctx); + if(!ndpi_struct[i]) { + ERROR; + } + + rc = ndpi_set_config(ndpi_structs[i], NULL, "lru.ookla.scope", "1"); + if(rc != NDPI_CFG_OK) { + ERROR; + } + + ret = ndpi_finalize_initialization(ndpi_structs[i]); + if(ret != 0) { + ERROR; + } + } + + /* Initialization done */ + + /* Cleanup */ + + for(i = 0; i < num_local_contexts; i++) { + ndpi_exit_detection_module(ndpi_structs[i]); + } + + ndpi_global_deinit(g_ctx); + diff --git a/doc/monitoring.md b/doc/monitoring.md deleted file mode 100644 index 46c54ce91..000000000 --- a/doc/monitoring.md +++ /dev/null @@ -1,26 +0,0 @@ - -# Monitoring - -nDPI usually needs only a few packets per flow to get full classification and to get all the required metadata/flow_risks. After that point, nDPI stops processing the flow. -However, in some use cases, it might be useful to allow nDPI to process the *entire* flow (i.e. *all* its packets, without any limits). Some examples: -* to extract all the STUN metadata from a STUN flow -* to extract all the request/replay pairs from a DNS flow -In essence, monitoring allows the application to get the same metadata, multiple times, throughout the entire life of the session. - -If monitoring is enabled in a flow: -* structures `ndpi_flow->protos`, `ndpi_flow->http`, `ndpi_flow->stun`,... are populated as usual, usually with the *first* instance of the specific metadata. Nothing changed. -* packet by packet, the new structure `ndpi_flow->monitor` is populated with the metadata of the *current* packet. This information is lost when starting processing the next packet in the same flow; it is the responsibility of the application to get it. - -In other words: -* "flow metadata" is saved in `ndpi_flow->protos`, `ndpi_flow->http`, `ndpi_flow->stun`, regardless of the monitoring feature being enabled or not. These fields are always available -* "(curent) packet metadata" is saved in `ndpi_flow->monitor`, only if monitor is enabled. - -Monitoring must be explicit enabled with something like: `--cfg=stun,monitoring,1`; to enable/disable monitoring for all protocols you can use `--cfg=any,monitoring,1` but only STUN is supported right now. - -Since monitoring processess *all* the flow packets, it might have an impact on performances. - -## Implementation notes - -* Flows move to monitoring state only after extra-dissections end -* The classification doesn't change for flows in monitoring state -* We probably need to improve TCP reassembler to best handle TCP flows in monitoring state diff --git a/doc/monitoring.rst b/doc/monitoring.rst new file mode 100644 index 000000000..53dcbea9c --- /dev/null +++ b/doc/monitoring.rst @@ -0,0 +1,33 @@ +Monitoring +========== + +nDPI usually needs only a few packets per flow to get full classification and to get all the required metadata/flow_risks. After that point, nDPI stops processing the flow. +However, in some use cases, it might be useful to allow nDPI to process the *entire* flow (i.e., *all* its packets, without any limits). Some examples: + +- to extract all the STUN metadata from a STUN flow +- to extract all the request/replay pairs from a DNS flow + +In essence, monitoring allows the application to get the same metadata, multiple times, throughout the entire life of the session. + +If monitoring is enabled in a flow: + +- structures ``ndpi_flow->protos``, ``ndpi_flow->http``, ``ndpi_flow->stun``, ... are populated as usual, usually with the *first* instance of the specific metadata. Nothing changed. +- packet by packet, the new structure ``ndpi_flow->monitor`` is populated with the metadata of the *current* packet. This information is lost when starting processing the next packet in the same flow; it is the responsibility of the application to get it. + +In other words: + +- "flow metadata" is saved in ``ndpi_flow->protos``, ``ndpi_flow->http``, ``ndpi_flow->stun``, regardless of the monitoring feature being enabled or not. These fields are always available +- "(current) packet metadata" is saved in ``ndpi_flow->monitor``, only if monitor is enabled. + +Monitoring must be explicitly enabled with something like: ``--cfg=stun,monitoring,1``. +To enable/disable monitoring for all protocols you can use ``--cfg=any,monitoring,1`` but only STUN is supported right now. + +Since monitoring processes *all* the flow packets, it might have an impact on performances. + +Implementation notes +-------------------- + +- Flows move to monitoring state only after extra-dissections end +- The classification doesn't change for flows in monitoring state +- We probably need to improve TCP reassembler to best handle TCP flows in monitoring state + |