aboutsummaryrefslogtreecommitdiff
path: root/example/reader_util.h
Commit message (Collapse)AuthorAge
* fuzz: purge old sessions (#1451)Ivan Nardi2022-02-21
| | | | | | | | At every fuzz iteration (i.e for every trace file): * keep the same ndpi context (`ndpi_init_detection_module` is very slow); * reset the flow table, otherwise it grows indefinitely. This change should fix the "out-of-memory" errors reported by oss-fuzz.
* Make some protocols more "big-endian" friendly (#1402)Ivan Nardi2022-01-29
| | | See #1312
* Add a "confidence" field about the reliability of the classification. (#1395)Ivan Nardi2022-01-11
| | | | | | | | | | | | | As a general rule, the higher the confidence value, the higher the "reliability/precision" of the classification. In other words, this new field provides an hint about "how" the flow classification has been obtained. For example, the application may want to ignore classification "by-port" (they are not real DPI classifications, after all) or give a second glance at flows classified via LRU caches (because of false positives). Setting only one value for the confidence field is a bit tricky: more work is probably needed in the next future to tweak/fix/improve the logic.
* Improved user agent analysisLuca Deri2022-01-09
|
* Update copyrightAlfredo Cardigliano2022-01-03
|
* Rework how hostname/SNI info is saved (#1330)Ivan Nardi2021-11-24
| | | | | | | | | | | | | | | | | | | | | | | | | | | Looking at `struct ndpi_flow_struct` the two bigger fields are `host_server_name[240]` (mainly for HTTP hostnames and DNS domains) and `protos.tls_quic.client_requested_server_name[256]` (for TLS/QUIC SNIs). This commit aims to reduce `struct ndpi_flow_struct` size, according to two simple observations: 1) maximum one of these two fields is used for each flow. So it seems safe to merge them; 2) even if hostnames/SNIs might be very long, in practice they are rarely longer than a fews tens of bytes. So, using a (single) large buffer is a waste of memory for all kinds of flows. If we need to truncate the name, we keep the *last* characters, easing domain matching. Analyzing some real traffic, it seems safe to assume that the vast majority of hostnames/SNIs is shorter than 80 bytes. Hostnames/SNIs are always converted to lowercase. Attention was given so as to be sure that unit-tests outputs are not affected by this change. Because of a bug, TLS/QUIC SNI were always truncated to 64 bytes (the *first* 64 ones): as a consequence, there were some "Suspicious DGA domain name" and "TLS Certificate Mismatch" false positives.
* Fix writes to `flow->protos` union fields (#1354)Ivan Nardi2021-11-15
| | | | | | | | | | | | | | | | | | | | | | | | | We can write to `flow->protos` only after a proper classification. This issue has been found in Kerberos, DHCP, HTTP, STUN, IMO, FTP, SMTP, IMAP and POP code. There are two kinds of fixes: * write to `flow->protos` only if a final protocol has been detected * move protocol state out of `flow->protos` The hard part is to find, for each protocol, the right tradeoff between memory usage and code complexity. Handle Kerberos like DNS: if we find a request, we set the protocol and an extra callback to further parsing the reply. For all the other protocols, move the state out of `flow->protos`. This is an issue only for the FTP/MAIL stuff. Add DHCP Class Identification value to the output of ndpiReader and to the Jason serialization. Extend code coverage of fuzz tests. Close #1343 Close #1342
* Reduce memory used by `ndpiReader` (#1371)Ivan Nardi2021-11-11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | `ndpiReader` is only an example, aiming to show nDPI capabilities and integration, without any claim about performances. Nonetheless its memory usage per flow is *huge*, limiting the kinds of traces that we can test on a "normal" hardware (example: scan attacks). The key reason of that behaviour is that we preallocate all the memory needed for *all* the available features. Try to reduce memory usage simply allocating some structures only when they are really needed. Most significant example: JOY algorithms. This way we should use a lot less memory in the two most common user-cases: * `ndpiReader` invoked without any particular flag (i.e `ndpiReader -i $FILENAME_OR_IFACE`) * internal unit tests Before (on x86_64): ``` struct ndpi_flow_info { [...] /* size: 7320, cachelines: 115, members: 72 */ ``` After: ``` struct ndpi_flow_info { [...] /* size: 2128, cachelines: 34, members: 75 */ ```
* Implemented function to retrieve flow information. #1253 (#1254)Toni2021-07-23
| | | | | * fixed [h]euristic typo Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* ndpiReader: add statistics about nDPI performance (#1240)Ivan Nardi2021-07-13
| | | | | | | The goal is to have a (roughly) idea about how many packets nDPI needs to properly classify a flow. Log this information (and guessed flows number too) during unit tests, to keep track of improvements/regressions across commits.
* Initial work towards detection via TLS of browser typesLuca2021-05-06
|
* Added flow risk to wireshark dissectionLuca Deri2021-04-26
|
* ndpiReader: print an error msg if we found an unsupported datalink type (#1157)Ivan Nardi2021-03-23
|
* Fixed CPHA missing protocol initializationLuca Deri2021-02-10
| | | | Improved IEC104 and IRC detection
* Fix a warning (#1125)Ivan Nardi2021-02-03
| | | | | | | | | | | | | | | Introduced in 5f7b9d802 reader_util.c: In function ‘process_ndpi_collected_info’: reader_util.c:1148:60: warning: ‘%s’ directive output may be truncated writing up to 255 bytes into a region of size 64 [-Wformat-truncation=] 1148 | sizeof(flow->ssh_tls.client_requested_server_name), "%s", | ^~ reader_util.c:1147:5: note: ‘snprintf’ output between 1 and 256 bytes into a destination of size 64 1147 | snprintf(flow->ssh_tls.client_requested_server_name, | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1148 | sizeof(flow->ssh_tls.client_requested_server_name), "%s", | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1149 | flow->ndpi_flow->protos.tls_quic_stun.tls_quic.client_requested_server_name); | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
* Code cleanup: third party uthash is at the right placeLuca Deri2021-01-20
|
* (C) UpdateLuca Deri2021-01-07
|
* Split HTTP request from response Content-Type. Request Content-Type should ↵Luca Deri2021-01-06
| | | | be present with POSTs and not with other methods such as GET
* Added --ignore-vlanid / -I to exclude VLAN ids for flow hash calculation. ↵Toni2020-12-11
| | | | | #1073 (#1085) Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Added -D flag for detecting DoH in the wildLuca Deri2020-10-26
| | | | Removed heuristic from CiscoVPN as it leads to false positives
* OpenBSD: Introduce pkt_timeval to deal with (bpf_)_timevalAdrian Zgorzałek2020-08-09
| | | | | | Some BSD APIs called in example/ return `struct bpf_timeval`, where nDPI APIs expect `struct timeval`. These two structs, besides having a different name, share the exact same set of fields.
* Removed csv_fp as external symbol. Instead passing csv_fp through as argument.Toni Uhlig2020-07-08
| | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Merge pull request #932 from IvanNardi/logLuca Deri2020-07-07
|\ | | | | Log
| * Incorporated some feedbackNardi Ivan2020-07-01
| |
| * Fix compilation with --enable-debug-messages flagNardi Ivan2020-06-26
| | | | | | | | | | | | | | | | | | | | | | | | NDPI_LOG* macros dereference ndpi_detection_module_struct object which is private to ndpi library (via NDPI_LIB_COMPILATION define). So we can't use them outside the library itself, i.e. in ndpiReader code Therefore, in files in example/, convert all (rare) uses of NDPI_LOG* macros to a new very simple macro, private to ndpiReader program. If necessary, such macro may be improved. According to a comment in ndpi_define.h, each dissector must define its own NDPI_CURRENT_PROTO macro before including ndpi_api.h file
* | Fix memory leak about purged/expired flowsNardi Ivan2020-06-28
| | | | | | | | | | Create an helper to avoid similar errors in the future Fixes: 1a62f4c7
* | Fix (harmless) memory leaks when DPDK is enabledNardi Ivan2020-06-28
|/
* Added ndpi_bin_XXX APILuca Deri2020-06-22
| | | | Added packet lenght distribution bins
* Removed some obsolete protocols (battlefield, oscar, pcanywhere, tvants)Luca Deri2020-06-06
|
* Added support for Encrypted TLS SNI dissectionLuca Deri2020-05-28
| | | | https://datatracker.ietf.org/doc/draft-ietf-tls-sni-encryption/
* Added the ability to detect when a known protocol is using a non-standard portLuca Deri2020-05-10
| | | | Added check to spot executables exchanged via HTTP
* Added TLS issuerDN and subjectDNLuca Deri2020-05-07
|
* ndpiReader: fix memory leak in idle sessions purgingNardi Ivan2020-04-08
|
* Minor fixLuca Deri2020-02-17
|
* Added TLS ALPN supportLuca Deri2020-02-07
|
* Added flow extra info fieldLuca Deri2020-01-10
| | | | Updated tests/results
* Minor cleanupLuca Deri2020-01-05
|
* Fix leaks and sha1 certificate detectionemanuele-f2020-01-02
|
* Renamed TLS requested server nameLuca2020-01-02
|
* Reworked TLS dissectionLuca2020-01-01
|
* Merge pull request #813 from SimoneRicci97/devLuca Deri2019-12-09
|\ | | | | Fixed intrusion detection
| * Fixed intrusion detectionSimone Ricci2019-11-26
| |
* | Implemented telnet password exportLuca Deri2019-11-21
|/
* Added flow duration and goodput in output (-v and -c)Luca Deri2019-11-21
|
* Merge branch 'dev' of https://github.com/ntop/nDPI into devLuca2019-11-05
|\
| * Merge pull request #806 from oleg-umnik/fix_1Luca Deri2019-11-03
| |\ | | | | | | Don't leak memory in live capture mode
| | * Don't leak memory in live capture modeOleg A. Arkhangelsky2019-10-29
| | |
* | | Added tunnelling decapsulationLuca2019-11-05
|/ /
* | Initial work towards HTTP content-type exportLuca2019-10-31
| |
* | Added telnet dissectorLuca2019-10-29
|/ | | | Improved data report
Powered by cgit, Themed by Ted Yin.