aboutsummaryrefslogtreecommitdiff
path: root/example/reader_util.h
Commit message (Collapse)AuthorAge
* QUIC: export QUIC version as metadataNardi Ivan2023-10-11
|
* fuzz: extend fuzzing coverageNardi Ivan2023-10-07
|
* Cleaned up mining datastructureLuca2023-09-27
|
* fuzz: add fuzzers to test reader_util code (#2080)Ivan Nardi2023-09-10
|
* DNS: extract geolocation information, if available (#2065)Ivan Nardi2023-07-31
| | | | | | | The option NSID (RFC5001) is used by Google DNS to report the airport code of the metro where the DNS query is handled. This option is quite rare, but the added overhead in DNS code is pretty much zero for "normal" DNS traffic
* TLS: add basic, basic, detection of Encrypted ClientHello (#2053)Ivan Nardi2023-07-21
|
* added feature to extract filename from http attachment (#2037)Chiara Maggi2023-07-11
| | | | | | | | | | | | | | | | | | | * added feature to extract filename from http attachment * fixed some issues * added check for filename format * added check for filename format * remove an unnecessary print * changed the size from 952 to 960 * modified some test result files * small changes string size * comment removed and mallocs checked
* Simplify the report of streaming multimedia info (#2026)Ivan Nardi2023-06-26
| | | | | The two fields `flow->flow_type` and `flow->protos.rtp.stream_type` are pretty much identical: rename the former in `flow->flow_multimedia_type` and remove the latter.
* Implemented Zoom/Teams stream type detectionLuca Deri2023-06-14
|
* ndpiReader: improve printing of payload statistics (#1989)Ivan Nardi2023-05-29
| | | | | Add a basic unit test Fix an endianess issue
* HTTP: improve extraction of metadata and of flow risks (#1959)Ivan Nardi2023-05-05
|
* ndpiReader: print how many packets (per flow) were needed to perform full ↵Ivan Nardi2023-03-01
| | | | | | DPI (#1891) Average values are already printed, but this change should ease to identify regressions/improvements.
* fuzz: some improvements and add two new fuzzers (#1881)Ivan Nardi2023-02-09
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Remove `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` define from `fuzz/Makefile.am`; it is already included by the main configure script (when fuzzing). Add a knob to force disabling of AESNI optimizations: this way we can fuzz also no-aesni crypto code. Move CRC32 algorithm into the library. Add some fake traces to extend fuzzing coverage. Note that these traces are hand-made (via scapy/curl) and must not be used as "proof" that the dissectors are really able to identify this kind of traffic. Some small updates to some dissectors: CSGO: remove a wrong rule (never triggered, BTW). Any UDP packet starting with "VS01" will be classified as STEAM (see steam.c around line 111). Googling it, it seems right so. XBOX: XBOX only analyses UDP flows while HTTP only TCP ones; therefore that condition is false. RTP, STUN: removed useless "break"s Zattoo: `flow->zattoo_stage` is never set to any values greater or equal to 5, so these checks are never true. PPStream: `flow->l4.udp.ppstream_stage` is never read. Delete it. TeamSpeak: we check for `flow->packet_counter == 3` just above, so the following check `flow->packet_counter >= 3` is always false.
* Some small changes (#1869)Ivan Nardi2023-01-25
| | | | | | | | All dissector callbacks should not be exported by the library; make static some other local functions. The callback logic in `ndpiReader` has never been used. With internal libgcrypt, `gcry_control()` should always return no errors. We can check `categories` length at compilation time.
* Added RTP stream type in flow metadataLuca Deri2022-12-09
|
* Exported HTTP server in metadataLuca Deri2022-12-05
|
* Added support for Linux Cooked Capture v2Luca Deri2022-11-16
|
* Fix json export of ipv6 addressesNardi Ivan2022-11-07
| | | | | | | The "string" buffer was to short; better start using `INET6_ADDRSTRLEN` as reported in the man page of `inet_ntop`. Close: #1794
* TLS: improve handling of ALPN(s) (#1784)Ivan Nardi2022-10-25
| | | | | | | | Tell "Advertised" ALPN list from "Negotiated" ALPN; the former is extracted from the CH, the latter from the SH. Add some entries to the known ALPN list. Fix printing of "TLS Supported Versions" field.
* Added the ability to track the payload via -E and via the new option ↵Luca2022-10-04
| | | | 'ndpi_track_flow_payload'
* Improved NATPMP dissection. (#1745)Toni2022-09-21
| | | | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com> Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Add TiVoConnect dissector. Fixes #1697. (#1699)Toni2022-08-08
| | | | | * added static assert if supported, to complain if the flow struct changes Signed-off-by: lns <matzeton@googlemail.com>
* Improved nDPI JSON serialization. (#1689)Toni2022-08-02
| | | | | | | | * fixed autoconf CFLAGS/LDFLAGS MSAN issue which could lead to build errors * introduced portable version of gmtime_r aka ndpi_gmtime_r * do as most as possible of the serialization work in ndpi_utils.c * use flow2json in ndpiReader Signed-off-by: lns <matzeton@googlemail.com>
* Add Softether dissector. (#1679)Toni2022-07-29
| | | Signed-off-by: lns <matzeton@googlemail.com>
* Patricia tree, Ahocarasick automa, LRU cache: add statistics (#1683)Ivan Nardi2022-07-29
| | | | | | | | | | Add (basic) internal stats to the main data structures used by the library; they might be usefull to check how effective these structures are. Add an option to `ndpiReader` to dump them; enabled by default in the unit tests. This new option enables/disables dumping of "num dissectors calls" values, too (see b4cb14ec).
* Revert "Patricia tree, Ahocarasick automa, LRU cache: add statistics ↵Ivan Nardi2022-07-29
| | | | | (#1677)" (#1682) This reverts commit bb83899985c25097341b947c2c535f56254a075c.
* Patricia tree, Ahocarasick automa, LRU cache: add statistics (#1677)Ivan Nardi2022-07-29
| | | | | | | | Add (basic) internal stats to the main data structures used by the library; they might be usefull to check how effective these structures are. Add an option to `ndpiReader` to dump them; disabled by default to avoid too much fuss with the unit tests.
* Keep track of how many dissectors calls we made for each flow (#1657)Ivan Nardi2022-07-11
|
* Added RiskInfo stringLuca Deri2022-05-30
|
* Replaced ndpiReader's libjson-c support with libnDPI's internal ↵Toni2022-05-07
| | | | | | | | | | | serialization interface. (#1535) * Fixes #1528 * Serialization Interface should also fuzzed * libjson-c may only be used in the unit test to verify the internal serialization interface * Serialization Interface supports tlv(broken), csv and json * Unit test does work again and requires libjson-c Signed-off-by: lns <matzeton@googlemail.com>
* fuzz: purge old sessions (#1451)Ivan Nardi2022-02-21
| | | | | | | | At every fuzz iteration (i.e for every trace file): * keep the same ndpi context (`ndpi_init_detection_module` is very slow); * reset the flow table, otherwise it grows indefinitely. This change should fix the "out-of-memory" errors reported by oss-fuzz.
* Make some protocols more "big-endian" friendly (#1402)Ivan Nardi2022-01-29
| | | See #1312
* Add a "confidence" field about the reliability of the classification. (#1395)Ivan Nardi2022-01-11
| | | | | | | | | | | | | As a general rule, the higher the confidence value, the higher the "reliability/precision" of the classification. In other words, this new field provides an hint about "how" the flow classification has been obtained. For example, the application may want to ignore classification "by-port" (they are not real DPI classifications, after all) or give a second glance at flows classified via LRU caches (because of false positives). Setting only one value for the confidence field is a bit tricky: more work is probably needed in the next future to tweak/fix/improve the logic.
* Improved user agent analysisLuca Deri2022-01-09
|
* Update copyrightAlfredo Cardigliano2022-01-03
|
* Rework how hostname/SNI info is saved (#1330)Ivan Nardi2021-11-24
| | | | | | | | | | | | | | | | | | | | | | | | | | | Looking at `struct ndpi_flow_struct` the two bigger fields are `host_server_name[240]` (mainly for HTTP hostnames and DNS domains) and `protos.tls_quic.client_requested_server_name[256]` (for TLS/QUIC SNIs). This commit aims to reduce `struct ndpi_flow_struct` size, according to two simple observations: 1) maximum one of these two fields is used for each flow. So it seems safe to merge them; 2) even if hostnames/SNIs might be very long, in practice they are rarely longer than a fews tens of bytes. So, using a (single) large buffer is a waste of memory for all kinds of flows. If we need to truncate the name, we keep the *last* characters, easing domain matching. Analyzing some real traffic, it seems safe to assume that the vast majority of hostnames/SNIs is shorter than 80 bytes. Hostnames/SNIs are always converted to lowercase. Attention was given so as to be sure that unit-tests outputs are not affected by this change. Because of a bug, TLS/QUIC SNI were always truncated to 64 bytes (the *first* 64 ones): as a consequence, there were some "Suspicious DGA domain name" and "TLS Certificate Mismatch" false positives.
* Fix writes to `flow->protos` union fields (#1354)Ivan Nardi2021-11-15
| | | | | | | | | | | | | | | | | | | | | | | | | We can write to `flow->protos` only after a proper classification. This issue has been found in Kerberos, DHCP, HTTP, STUN, IMO, FTP, SMTP, IMAP and POP code. There are two kinds of fixes: * write to `flow->protos` only if a final protocol has been detected * move protocol state out of `flow->protos` The hard part is to find, for each protocol, the right tradeoff between memory usage and code complexity. Handle Kerberos like DNS: if we find a request, we set the protocol and an extra callback to further parsing the reply. For all the other protocols, move the state out of `flow->protos`. This is an issue only for the FTP/MAIL stuff. Add DHCP Class Identification value to the output of ndpiReader and to the Jason serialization. Extend code coverage of fuzz tests. Close #1343 Close #1342
* Reduce memory used by `ndpiReader` (#1371)Ivan Nardi2021-11-11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | `ndpiReader` is only an example, aiming to show nDPI capabilities and integration, without any claim about performances. Nonetheless its memory usage per flow is *huge*, limiting the kinds of traces that we can test on a "normal" hardware (example: scan attacks). The key reason of that behaviour is that we preallocate all the memory needed for *all* the available features. Try to reduce memory usage simply allocating some structures only when they are really needed. Most significant example: JOY algorithms. This way we should use a lot less memory in the two most common user-cases: * `ndpiReader` invoked without any particular flag (i.e `ndpiReader -i $FILENAME_OR_IFACE`) * internal unit tests Before (on x86_64): ``` struct ndpi_flow_info { [...] /* size: 7320, cachelines: 115, members: 72 */ ``` After: ``` struct ndpi_flow_info { [...] /* size: 2128, cachelines: 34, members: 75 */ ```
* Implemented function to retrieve flow information. #1253 (#1254)Toni2021-07-23
| | | | | * fixed [h]euristic typo Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* ndpiReader: add statistics about nDPI performance (#1240)Ivan Nardi2021-07-13
| | | | | | | The goal is to have a (roughly) idea about how many packets nDPI needs to properly classify a flow. Log this information (and guessed flows number too) during unit tests, to keep track of improvements/regressions across commits.
* Initial work towards detection via TLS of browser typesLuca2021-05-06
|
* Added flow risk to wireshark dissectionLuca Deri2021-04-26
|
* ndpiReader: print an error msg if we found an unsupported datalink type (#1157)Ivan Nardi2021-03-23
|
* Fixed CPHA missing protocol initializationLuca Deri2021-02-10
| | | | Improved IEC104 and IRC detection
* Fix a warning (#1125)Ivan Nardi2021-02-03
| | | | | | | | | | | | | | | Introduced in 5f7b9d802 reader_util.c: In function ‘process_ndpi_collected_info’: reader_util.c:1148:60: warning: ‘%s’ directive output may be truncated writing up to 255 bytes into a region of size 64 [-Wformat-truncation=] 1148 | sizeof(flow->ssh_tls.client_requested_server_name), "%s", | ^~ reader_util.c:1147:5: note: ‘snprintf’ output between 1 and 256 bytes into a destination of size 64 1147 | snprintf(flow->ssh_tls.client_requested_server_name, | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1148 | sizeof(flow->ssh_tls.client_requested_server_name), "%s", | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1149 | flow->ndpi_flow->protos.tls_quic_stun.tls_quic.client_requested_server_name); | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
* Code cleanup: third party uthash is at the right placeLuca Deri2021-01-20
|
* (C) UpdateLuca Deri2021-01-07
|
* Split HTTP request from response Content-Type. Request Content-Type should ↵Luca Deri2021-01-06
| | | | be present with POSTs and not with other methods such as GET
* Added --ignore-vlanid / -I to exclude VLAN ids for flow hash calculation. ↵Toni2020-12-11
| | | | | #1073 (#1085) Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Added -D flag for detecting DoH in the wildLuca Deri2020-10-26
| | | | Removed heuristic from CiscoVPN as it leads to false positives
Powered by cgit, Themed by Ted Yin.