Memory problem ?

I have had memory problems in dnscrypt server-docker images.
Memory usage is growing rapidly.
Come across this:
tikv/rust-prometheus#424
Could prometheus v0.13.0 be the problem?
Compiled encrypted-dns with prometheus v0.12.0 and it looks like memory problem is gone.
This was done on a singleboard computer arm32v7 with 2Gbyte of LPDDR3 RAM.

Do not know what does this, tried on another arm32v7 board. CPU (1-core of 4) usage also goes up to 100%

Can you double check that the issue goes away when downgrading the prometheus crate to v0.12.0?

How did you do that?

The weird thing is that one device works with prometheus v0.12.0 and another does not (release 0.9.1), but both devices work well with encrypted-dns release 0.3.23 and arm32v7/ubuntu:22.04

Build dnscrypt-server-docker, FROM arm32v7/ubuntu:22.04

I have also tried prometheus v0.13.1, but there are also issues.
tikv/rust-prometheus#442

Does it still happen if you don't enable the metrics feature? (disabling it removes prometheus)

Does it still happen if you don't enable the metrics feature? (disabling it removes prometheus)

I have to try.

disabled metrics in encrypted-dns.toml.in and -M option when initializing the docker image.
Memory and CPU usage are growing, and are running in loop.

Ok, so at least we know that this is unrelated to Prometheus.

In case this is not in a dependency, would you be able to pinpoint a specific commit that introduced this?

Version 0.3.23 works.
cargo install --git https://github.com/DNSCrypt/encrypted-dns-server --rev 631ebd4

Version 0.9.0 does not work.
cargo install --git https://github.com/DNSCrypt/encrypted-dns-server --rev a178d25

So there must be something between these versions.

Will probably have to test more to find the specific commit that introduced this.

I think this problem is due to jemallocator.

See what was mentioned earlier.
jemalloc/jemalloc#1497
gnzlbg/jemallocator#168
rust-lang/rust#83152

I have tried with tikv-jemallocator. Compiled fine and without memory problem, but unfortunately no response from encrypted-dns. Only gets TIMEOUT on server in dnscrypt proxy.
tikv-jemallocator supports arm32v7 ?

So, like way too many Rust crates, jemalloc is abandoned, and the maintained fork doesn't work on ARM? Great :(

As an experiment, 748b944 replaces jemalloc with mimalloc. Does in improve things on your end?

The app doesn't seem to compile at all on armv7 :(

error[E0308]: mismatched types
   --> /home/pi/.cargo/registry/src/github.com-1285ae84e5963aae/prometheus-0.13.1/src/process_collector.rs:192:13
    |
190 |     static ref CLK_TCK: i64 = {
    |                         --- expected `i64` because of return type
191 |         unsafe {
192 |             libc::sysconf(libc::_SC_CLK_TCK)
    |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected `i64`, found `i32`

tikv/rust-prometheus#442
I use prometheus v0.12.0
Trying mimalloc.

After removing prometheus' process feature, it works for me on armv7l (raspberry Pi with 32-bit raspbian).

Update.
Has compiled with the latest version of tikv-jemallocator (0.5.0), which was released today and server works excellent.
tikv/jemallocator#26
This issue has been resolved. Thank you so much

Navigation Menu

Repository navigation

Memory problem ? #86

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Memory problem ? #86

Description

Activity

Haraade commented on May 21, 2022

jedisct1 commented on May 21, 2022

Haraade commented on May 21, 2022

Haraade commented on May 21, 2022

jedisct1 commented on May 21, 2022

Haraade commented on May 21, 2022

Haraade commented on May 22, 2022

jedisct1 commented on May 22, 2022

Haraade commented on May 22, 2022

Haraade commented on May 24, 2022

jedisct1 commented on May 24, 2022

jedisct1 commented on May 24, 2022

Haraade commented on May 24, 2022

jedisct1 commented on May 24, 2022

Haraade commented on May 25, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions