| Commit message (Collapse) | Author | Age |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Add a simple job with macos-14 on M1.
https://github.blog/changelog/2024-01-30-github-actions-introducing-the-new-m1-macos-runner-available-to-open-source/)
There are some issues with external dependencies (they are installed but
autoconf script doens;t find them) so keep it simple.
On macos-13 it seems that:
* there is no `realpath` program (even if coreutils has been
installed...)
* most of the filesystem is read only (we can't write on /usr/lib).
So I change
```
make install DESTDIR=$(realpath _install)
ls -alhHR _install
```
to
```
DESTDIR=/tmp/ndpi make install
ls -alhHR /tmp/ndpi
```
for all the jobs
Fix a warning on GitHub logs:
```
Node.js 16 actions are deprecated. Please update the following actions
to use Node.js 20: actions/checkout@v3. For more information see:
https://github.blog/changelog/2023-09-22-github-actions-transitioning-from-node-16-to-node-20/.
```
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Add the concept of "global context".
Right now every instance of `struct ndpi_detection_module_struct` (we
will call it "local context" in this description) is completely
independent from each other. This provide optimal performances in
multithreaded environment, where we pin each local context to a thread,
and each thread to a specific CPU core: we don't have any data shared
across the cores.
Each local context has, internally, also some information correlating
**different** flows; something like:
```
if flow1 (PeerA <-> Peer B) is PROTOCOL_X; then
flow2 (PeerC <-> PeerD) will be PROTOCOL_Y
```
To get optimal classification results, both flow1 and flow2 must be
processed by the same local context. This is not an issue at all in the far
most common scenario where there is only one local context, but it might
be impractical in some more complex scenarios.
Create the concept of "global context": multiple local contexts can use
the same global context and share some data (structures) using it.
This way the data correlating multiple flows can be read/write from
different local contexts.
This is an optional feature, disabled by default.
Obviously data structures shared in a global context must be thread safe.
This PR updates the code of the LRU implementation to be, optionally,
thread safe.
Right now, only the LRU caches can be shared; the other main structures
(trees and automas) are basically read-only: there is little sense in
sharing them. Furthermore, these structures don't have any information
correlating multiple flows.
Every LRU cache can be shared, independently from the others, via
`ndpi_set_config(ndpi_struct, NULL, "lru.$CACHE_NAME.scope", "1")`.
It's up to the user to find the right trade-off between performances
(i.e. without shared data) and classification results (i.e. with some
shared data among the local contexts), depending on the specific traffic
patterns and on the algorithms used to balance the flows across the
threads/cores/local contexts.
Add some basic examples of library initialization in
`doc/library_initialization.md`.
This code needs libpthread as external dependency. It shouldn't be a big
issue; however a configure flag has been added to disable global context
support. A new CI job has been added to test it.
TODO: we should need to find a proper way to add some tests on
multithreaded enviroment... not an easy task...
*** API changes ***
If you are not interested in this feature, simply add a NULL parameter to
any `ndpi_init_detection_module()` calls.
|
| |
|
|
|
| |
Change the working directory of `ndpiReader` in the Github Actions so
that it can load the domain suffix list during `domainsUnitTest()`
|
| |
|
|
|
| |
Try using latest gcc and clang versions.
We still care about RHEL7: since handling a RHEL7 runner on GitHub is
quite complex, let try to use a similar version of gcc, at least
|
| |
|
|
|
|
|
|
|
|
| |
Move from PCRE to PCRE2. PCRE is EOL and won't receive any security
updates anymore. Convert to PCRE2 by converting any function PCRE2 new
API.
Also update every entry in github workflows and README to point to the
new configure flag. (--with-pcre2)
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
* Refreshed the Belgium Gambling Site list data
Unfortunately some hostnames have been removed from that list,
which means they are disappearing from the `ndpi_gambling_match.c.inc`
file as well.
* build: added `libxml2-utils` (for `xmllint`)
* Included Gambling website data from the Polish `hazard.mf.gov.pl` list
The list contains over 30k gambling website hostnames as of today.
|
| |
|
|
|
| |
* added CI check
Signed-off-by: lns <matzeton@googlemail.com>
|
| |
|
| |
Add support for Facebook crawler
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
CI duration is quite long: the longest jobs is the "Performance" one.
Try to reduce the overall duration: that job (and some others) will not
be triggered for each PR/commit anymore, but asynchronously, once a day
(this scheduling seems right since the frequency of the PR/commits in
the project).
It should be possibly to trigger them manually, via GUI, anyway.
Remove two identical jobs; we already tests ASAN with 4 different
compilers.
After 9eff0754 it is safe to reduce fuzzing time.
Bottom line: try to have as upper-time of CI tests the duration of the
fuzzing jobs
|
| |
|
| |
See: https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This commit add (optional) support for Link-Time-Optimization and Gold
linker.
This is the first, mandatory step needed to make nDPI compliant with
"introspector" sanitizer requirements in OSS-Fuzz: see
https://github.com/google/oss-fuzz/issues/8939
Gold linker is not supported by Windows and by macOS, so this feature is
disabled by default. It has been enable in CI in two linux targets
("latest" gcc and clang).
Fix some warnings triggered by LTO.
The changes in `src/lib/ndpi_serializer.c` seams reasonable.
However, the change in `tests/unit/unit.c` is due to the following
warning, which seems to be a false positive.
```
unit.c: In function ‘serializerUnitTest’:
ndpi_serializer.c:2258:13: error: ‘MEM[(struct ndpi_private_serializer *)&deserializer].buffer.size’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
unit.c:67:31: note: ‘MEM[(struct ndpi_private_serializer *)&deserializer].buffer.size’ was declared here
67 | ndpi_serializer serializer, deserializer;
| ^
ndpi_serializer.c:2605:10: error: ‘MEM[(struct ndpi_private_serializer *)&deserializer].status.buffer.size_used’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
unit.c:67:31: note: ‘MEM[(struct ndpi_private_serializer *)&deserializer].status.buffer.size_used’ was declared here
67 | ndpi_serializer serializer, deserializer;
```
Since this warning is triggered only with an old version of gcc and
`tests/unit/unit.c` is used only during the tests, the easiest fix has
been applied.
Some (unknown to me) combinations of OS and compiler trigger the
following warnings at linker time (with sanitizer and gold linker)
```
/usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_load1_asm'
/usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_load2_asm'
/usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_load4_asm'
/usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_load8_asm'
/usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_load16_asm'
/usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_store1_asm'
/usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_store2_asm'
/usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_store4_asm'
[..]
```
I have not found any references to this kind of message, with the only
exception of https://sourceware.org/bugzilla/show_bug.cgi?id=25975
which seems to suggest that these messages can be safely ignored.
In any case, the compilation results are sound.
Fix `clean` target in the Makefile in the `example` directory.
In OSS-Fuzz enviroments, `fuzz_ndpi_reader` reports a strange link error
(as always, when the gold linker is involved...).
It's come out that the culprit was the `tempnam` function: the code has
been changed to use `tmpfile` instead. No sure why... :(
Fuzzing target `fuzz_ndpi_reader.c` doesn't use `libndpiReader.a`
anymore: this way we can use `--with-only-libndpi` flag on Oss-Fuzz builds
as workaround for the "missing dependencies errors" described in
https://github.com/google/oss-fuzz/issues/8939
|
| |
|
|
|
|
|
|
|
|
|
| |
GitHub is moving `ubuntu-latest` to `ubuntu-22.04`: update our
dependencies.
See: https://github.blog/changelog/2022-11-09-github-actions-ubuntu-latest-workflows-will-use-ubuntu-22-04/
This is the reason of the recent random failures in CI.
Update "newest" tested gcc to gcc-12.
Fix a memory error introduced in 557bbcfc5a5165c9eb43bbdd78435796239cd3c9
|
| |
|
|
|
|
|
| |
```
The `set-output` command is deprecated and will be disabled soon.
Please upgrade to using Environment Files. For more information see:
https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
```
|
| |
|
|
|
|
| |
Fix warnings on recent CI results; example:
https://github.com/ntop/nDPI/actions/runs/3455588082
See: https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/
|
| |
|
|
|
|
| |
* add CI support via MSBuild
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
|
| |
|
| |
Add one CI job testing nBPF
|
| |
|
|
|
|
|
|
|
|
| |
ubuntu-18.04 is deprecated (ubuntu-latest points to 20.04).
macos-latest points to macos-11, so it makes sense to test macos-12,
too.
About the compilers, the general idea it to test the oldest and the
newest versions easily available: switch to gcc-11 and clang-14.
See: https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners
|
| |
|
|
|
| |
* added static assert if supported, to complain if the flow struct changes
Signed-off-by: lns <matzeton@googlemail.com>
|
| | |
|
| |
|
|
|
|
|
| |
* CI fixes
* some build systems do not like that (e.g. OpenWrt)
* fixed some rrdtool related build warnings/errors
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
|
| |
|
|
|
|
| |
* use -ltcmalloc_and_profiler and try to get rid of LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libprofiler.so
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
|
| |
|
|
|
| |
Signed-off-by: lns <matzeton@googlemail.com>
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
|
| |
|
| |
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
|
| |
|
|
|
|
|
|
| |
GCC analyzer won't complain about possible use-after-free (false positive).
* tests/do.sh prints word diff's only once and not the same over and over again
* sync unit tests
Signed-off-by: lns <matzeton@googlemail.com>
|
| |
|
|
|
|
|
| |
* make check great again (not so much)
* make doc/doc-view
* CI updates
Signed-off-by: lns <matzeton@googlemail.com>
|
| |
|
|
|
| |
* Sync unit tests
Signed-off-by: lns <matzeton@googlemail.com>
|
| |
|
|
|
| |
* Integrated Doxygen documentation into Sphinx
Signed-off-by: lns <matzeton@googlemail.com>
|
| |
|
|
|
|
|
| |
This fixes some build/test issues resulting when using tarballs.
* nDPI uses autotools (especially autoconf) in a wrong way, see #1163
Signed-off-by: lns <matzeton@googlemail.com>
|
| |
|
|
|
| |
* The warning itself looks like a bug
Signed-off-by: lns <matzeton@googlemail.com>
|
| |
|
|
|
|
|
| |
* Removed Visual Studio leftovers. Maintaining an autotools project with VS integration requires some additional overhead.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
Signed-off-by: lns <matzeton@googlemail.com>
|
| | |
|
| | |
|
| |
|
|
| |
packaging and CI integration)
|
| |
|
|
|
| |
* CI will print a warning if ASN/IP addresses changed.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
|
| |
|
|
| |
* Switch fail fast to True.
* Windows CI.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Fixed errors for bigendian platforms in ndpiReader.
All address and port comparisons and hash calculations are done with
endian in mind.
The get_ndpi_flow_info() function searched for an existing flow for the
forward and reverse direction of the packet.
The ndpi_workflow_node_cmp() function looked for a flow regardless of
the packet's direction. This is what led to an error in determining the
direction of transmission of the packet.
Fixed error in "synscan" test: the number of packets in the forward and
reverse direction is incorrectly defined (verified via tcpdump).
Fixed bug with icmp protocol checksum check for big endian platforms.
|
| |
|
|
|
|
| |
* Testing more code in CI environments.
* Added strict option checking for `./configure' in CI environments.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
|
| |
|
|
|
|
|
|
| |
* As there is now a builtin, lightweight libgcrypt
there is no need to disable tls-clho decryption.
* It is still possible to use a host libgcrypt
with `--with-local-libgcrypt'.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
|
| |
|
|
|
| |
* CI job generates a coverage report
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
|
| |
|
|
|
| |
Deleted, probably by mistake, in 406ac7e8
Fix Makefile and add compilation of `rrdtool` in CI tests
|
| |
|
|
|
|
|
|
| |
Disable unit tests on CI for big-endian target. We know we have multiple
issues on big-endian architectures (see #1312) and so the unit tests
always fail there. Ignore this error for the time being and let the CI
pass if we don't have other issues.
Remove an unused automa definition
|
| |
|
|
|
| |
#1324 (#1327)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
|
| |
|
|
|
| |
* using Autotools best-practices to achieve (hopefully) the same result
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
|
| |
|
|
|
|
|
| |
* Add arm64, armhf and s390x as part on CI.
* Minor fix.
* Fix code inspector complaints.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Improve CI pipeline
* Fix branch name.
* Fix branch name.
* Fix libgcrypt configuration.
* Update build.yml
* Move to Github Actions instead of Travis CI.
* Fix mingw on ubuntu bionic.
* Reactivate cross compile on Ubuntu Bionic.
* Switch to single line steps.
* Add several compilers versions
* Minor fix.
* Fix build all and delete cxx
* Fix RCE detection.
* Fix PCRE configuration.
* Add condition on PCRE test pcap.
* Update WebattackRCE.pcap.out
* Add missing SUBST.
* Delete WebattackRCE.pcap.out
* Update WebAttackRCE result.
* Fix typo.
* Extend jobs with pcre+msan+maxminddb.
* Fix code inpector warnings.
* Delete .appveyor.yml
|
| |
|
|
| |
Let's try adding a dedicated compilation in GitHub Actions to easily
detect this kind of errors in the future
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
* Added ARM build and unit test run for SonarCloud-CI.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Fixed Mingw64 build.
* adapted to SonarCloud-CI workflow
* removed broken and incomplete Windows example (tested on VS2017/VS2019)
* removed unnecessary include (e.g. pthread.h for the library which does not make use of it)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
|
| |
|
| |
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
|
| |
|