aboutsummaryrefslogtreecommitdiff
path: root/utils
Commit message (Collapse)AuthorAge
* Improved statisticsLuca Deri2024-10-16
|
* Parser for ndpiReader JSON filesLuca Deri2024-10-15
|
* Add detection of SurfShark VPNNardi Ivan2024-09-05
|
* Add detection of NordVPNNardi Ivan2024-09-05
|
* Fix url for downloading X/Twitter crawler IPs (#2526)Ivan Nardi2024-08-22
|
* Performed some grammar and typo fixes (#2511)Petr2024-07-19
|
* ipaddr2list.py, ndpi2timeline.py: reformatted (#2509)Petr2024-07-18
|
* shell: reformatted, fixed inspections, typos (#2506)Petr2024-07-18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Reformatted shell scripts according to [ShellCheck](https://github.com/koalaman/shellcheck/). I. Most common changes: 1. https://github.com/koalaman/shellcheck/wiki/SC2086 `$var` → `"$var"` Note: this isn't always necessary and I've been careful not to substitute where it wasn't necessary in meaning. 2. https://github.com/koalaman/shellcheck/wiki/SC2006 `` `command` `` → `$(command)` 3. https://github.com/koalaman/shellcheck/wiki/SC2004 `$(( $a + $b ))` → `$(( a + b ))` 4. https://github.com/koalaman/shellcheck/wiki/SC2164 `cd "$dir"` → `cd "$dir" || exit 1` 5. https://github.com/koalaman/shellcheck/wiki/SC2166 `[ check1 -o check2 ]` → `[ check1 ] || [ check2 ]` 6. https://github.com/koalaman/shellcheck/wiki/SC2002 `cat "${file}" | wc -c` → `< "${file}" wc -c` Note: this looks a bit uglier but works faster. II. Some special changes: 1. In file `utils/common.sh`: https://github.com/koalaman/shellcheck/wiki/SC2112 This script is interpreted by `sh`, not by `bash`, but uses the keyword `function`. So I replaced `#!/usr/bin/env sh` to `#!/usr/bin/env bash`. 2. After that I thought of replacing all shebangs to `#!/usr/bin/env bash` for consistency and cross-platform compatibility, especially since most of the files already use bash. 3. But in cases when it was `#!/bin/sh -e` or `#!/bin/bash -eu` another problem appears: https://github.com/koalaman/shellcheck/wiki/SC2096 So I decided to make all shebangs look uniform: ``` #!/usr/bin/env bash set -e (or set -eu) (if needed) ``` 4. In file `tests/ossfuzz.sh`: https://github.com/koalaman/shellcheck/wiki/SC2162 `read i` → `read -r i` Note: I think that there is no need in special treatment for backslashes, but I could be wrong. 5. In file `tests/do.sh.in`: https://github.com/koalaman/shellcheck/wiki/SC2035 `ls *.*cap*` → `ls -- *.*cap*` 6. In file `utils/verify_dist_tarball.sh`: https://github.com/koalaman/shellcheck/wiki/SC2268 `[ "x${TARBALL}" = x ]` → `[ -z "${TARBALL}" ]` 7. In file `utils/check_symbols.sh`: https://github.com/koalaman/shellcheck/wiki/SC2221 `'[ndpi_utils.o]'|'[ndpi_memory.o]'|'[roaring.o]')` → `'[ndpi_utils.o]'|'[ndpi_memory.o]')` 8. In file `autogen.sh`: https://github.com/koalaman/shellcheck/wiki/SC2145 `echo "./configure $@"` → `echo "./configure $*"` https://github.com/koalaman/shellcheck/wiki/SC2068 `./configure $@` → `./configure "$@"` III. `LIST6_MERGED` and `LIST_MERGED6` There were typos with this variables in files `utils/aws_ip_addresses_download.sh`, `utils/aws_ip_addresses_download.sh` and `utils/microsoft_ip_addresses_download.sh` where variable `LIST6_MERGED` was defined, but `LIST_MERGED6` was removed by `rm`. I changed all `LIST_MERGED6` to `LIST6_MERGED`. Not all changes are absolutely necessary, but some may save you from future bugs.
* python: reformatted, fixed bugs (#2504)Petr2024-07-17
|
* Fixed mistake in shebang (SC1113) (#2498)Petr2024-07-15
|
* Add detection of Twitter bot (#2487)Ivan Nardi2024-07-03
| | | Update the global list of crawlers ips
* Remove unused code. (#2450)Toni2024-05-21
| | | | | | | | | * some `#ifdef`ed code dates back to 2019, 2020 and 2021 * some function signatures were still present in `ndpi_main.h` which may cause linker errors for libnDPI dependee's * return an error while trying to serialize a double instead of `fprintf(stderr, ...)` Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* utils: update script to download Cloudflare ipsNardi Ivan2024-02-26
|
* Add a script to download/update the domain suffix list (#2321)Ivan Nardi2024-02-20
|
* Improved Polish gambling sites fetch script. (#2315)Toni2024-02-10
| | | | | * fails quite often in the CI, so ignore potential xmllint error Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Fix some warnings reported by CODESonar (#2227)Ivan Nardi2024-01-12
| | | | | | | | | | | | | | | | | | | Remove some unreached/duplicated code. Add error checking for `atoi()` calls. About `isdigit()` and similar functions. The warning reported is: ``` Negative Character Value help isdigit() is invoked here with an argument of signed type char, but only has defined behavior for int arguments that are either representable as unsigned char or equal to the value of macro EOF(-1). Casting the argument to unsigned char will avoid the undefined behavior. In a number of libc implementations, isdigit() is implemented using lookup tables (arrays): passing in a negative value can result in a read underrun. ``` Switching to our macros fix that. Add a check to `check_symbols.sh` to avoid using the original functions from libc.
* Improved belgium gambling sites regex. (#2184)Toni2023-11-29
| | | Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Ip address list: aggregate Mullvad and Tor lists too (#2154)Ivan Nardi2023-11-21
| | | | Missing from bdb73db1a See #2150
* IP lists: aggregate addresses wherever possible (#2152)Ivan Nardi2023-11-17
| | | See #2150
* Remove duplicate addreess list (#2151)Ivan Nardi2023-11-16
| | | | We are loading the same AS list as GOTO See #2150
* Added malicious sites from the polish cert. (#2121)Toni2023-11-02
| | | | | * added handling of parsing errors Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* IPv6: add support for IPv6 risk tree (#2118)Ivan Nardi2023-10-27
| | | Fix the script to download crawler addressess
* ipv6: add support for ipv6 addresses lists (#2113)Ivan Nardi2023-10-26
|
* Tool for creating bitcoing IP filesLuca Deri2023-10-26
|
* add ethereum protocol dissector. (#2111)Maatuq2023-10-25
| | | | | | | as explained here for bitcoin https://www.ntop.org/guides/nDPI/protocols.html#ndpi-protocol-bitcoin the same is applicable for ethereum. ethereum detection was removed from mining protocol and is now handled separately. Signed-off-by: Mahmoud Maatuq <mahmoudmatook.mm@gmail.com>
* Added printf/fprintf replacement for some internal modules. (#1974)Toni2023-09-26
| | | | | | * logging is instead redirected to `ndpi_debug_printf` Signed-off-by: lns <matzeton@googlemail.com> Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Swap from Aho-Corasick to an experimental/home-grown algorithm that uses a ↵Luca Deri2023-08-29
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | probabilistic approach for handling Internet domain names. For switching back to Aho-Corasick it is necessary to edit ndpi-typedefs.h and uncomment the line // #define USE_LEGACY_AHO_CORASICK [1] With Aho-Corasick $ ./example/ndpiReader -G ./lists/ -i tests/pcap/ookla.pcap | grep Memory nDPI Memory statistics: nDPI Memory (once): 37.34 KB Flow Memory (per flow): 960 B Actual Memory: 33.09 MB Peak Memory: 33.09 MB [2] With the new algorithm $ ./example/ndpiReader -G ./lists/ -i tests/pcap/ookla.pcap | grep Memory nDPI Memory statistics: nDPI Memory (once): 37.31 KB Flow Memory (per flow): 960 B Actual Memory: 7.42 MB Peak Memory: 7.42 MB In essence from ~33 MB to ~7 MB This new algorithm will enable larger lists to be loaded (e.g. top 1M domans https://s3-us-west-1.amazonaws.com/umbrella-static/index.html) In ./lists there are file names that are named as <category>_<string>.list With -G ndpiReader can load all of them at startup
* Changes for supporinng more efficient sub-string matchingLuca Deri2023-08-26
|
* Mullvad VPN service added (based on entry node IP addresses) (#2062)snicket21002023-08-02
|
* ProtonVPN: split the ip list (#2060)Ivan Nardi2023-07-27
| | | | | | | | | | | Use two separate lists: * one for the ingress nodes, which triggers a ProtonVPN classification * one for the egress nodes, which triggers the `NDPI_ANONYMOUS_SUBSCRIBER` risk Add a command line option (to `ndpiReader`) to easily test IP/port matching. Add another example of custom rule.
* Add detection of Roblox games (#2054)Ivan Nardi2023-07-21
|
* tests: restore some old paths as symbolic links (#2050)Ivan Nardi2023-07-16
|
* Included Gambling website data from the Polish `hazard.mf.gov.pl` list (#2041)snicket21002023-07-14
| | | | | | | | | | | | | * Refreshed the Belgium Gambling Site list data Unfortunately some hostnames have been removed from that list, which means they are disappearing from the `ndpi_gambling_match.c.inc` file as well. * build: added `libxml2-utils` (for `xmllint`) * Included Gambling website data from the Polish `hazard.mf.gov.pl` list The list contains over 30k gambling website hostnames as of today.
* Refreshed ASN listsLuca Deri2023-06-13
| | | | Enhanced the Line IP list with https://ipinfo.io/AS23576/125.209.252.0/24 used by line
* ProtonVPN: add basic detection (#2006)Ivan Nardi2023-06-08
|
* Improved helper scripts. (#1986)Toni2023-05-28
| | | | | * added additional (more restrictive) checks Signed-off-by: lns <matzeton@googlemail.com>
* Add support for Epic Games and GeForceNow/Nvidia (#1990)Ivan Nardi2023-05-27
|
* Improve detection of Alibaba flows (#1991)Ivan Nardi2023-05-27
|
* Fixed invalid use of ndpi_free(). Sorry, my fault. (#1988)Toni2023-05-24
| | | | | | | | | | | | | | | * Fixed invalid use of ndpi_free(). Sorry, my fault. Signed-off-by: Toni Uhlig <matzeton@googlemail.com> * Fine tuned symbol check script. * added check for expected syms in modules Signed-off-by: Toni Uhlig <matzeton@googlemail.com> --------- Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Improved missing usage of nDPIs malloc wrapper. Fixes #1978. (#1979)Toni2023-05-20
| | | | | * added CI check Signed-off-by: lns <matzeton@googlemail.com>
* Added scripts to auto generate hostname/SNI *.inc files. (#1984)Toni2023-05-20
| | | | | * add illegal gambling sites (Belgium) Signed-off-by: lns <matzeton@googlemail.com>
* Improve detection of crawlers/bots (#1968)Ivan Nardi2023-05-09
| | | Add support for Facebook crawler
* Refreshed listsLuca Deri2023-05-08
|
* Improve detection of crawler/bot traffic (#1956)Ivan Nardi2023-05-04
|
* DisneyPlus/Hulu ip lists should be auto-generated (#1905)Ivan Nardi2023-03-20
| | | | | | Remove two stale ip lists: 1) these 3 ips are in the Amazon ranges (now)... 2) the Instagram list originated from AS32934, which is now a Facebook AS; see https://github.com/ntop/nDPI/pull/1264/commits/8dabd06301a802dd38616ba8684a1d995783e023
* Add Yandex services detection (#1882)0xA50C1A12023-02-09
| | | | | Add Yandex services detection Add VK and Yandex to the TLS certificate match list
* Add VK detection (#1880)0xA50C1A12023-02-02
|
* Line app support (#1759)sharonenoch2022-10-01
| | | | | | | | | | | | | | | | | | | * Standard support for LINE app * Added test pcap for LINE app * make check result for LINE app * Make check success as 1kxun has LINE packets * Added the ASN inc file for LINE * Removed extra lines as its effecting make check * Editing the SNI required a new pcap output file for TLS.Line format * Run Configure with --with-pcre --with-maxminddb to enable the generation of h323-overflow.pcap.out Co-authored-by: Sharon Enoch <sharone@amzetta.com>
* Add Discord dissector. (#1694)Toni2022-08-03
| | | | | * fixed RiotGames false positive Signed-off-by: lns <matzeton@googlemail.com>
* Update the protocol bitmask for some protocols (#1675)Ivan Nardi2022-07-27
| | | | | | | Tcp retransmissions should be ignored. Remove some unused protocol bitmasks. Update script to download Whatsapp IP list.
Powered by cgit, Themed by Ted Yin.