c++ - Independent multithreaded processes block simultaneously -

c++ - Independent multithreaded processes block simultaneously -

the scheme linux (gentoo x64), code c++. have daemon application, several instances of run on same machine. application multithreaded itself. time, have been observing unusual delays in performance.

after putting debugging code, came unusual thing when several instances of daemon literally block simultaneously allegedly caused external reason or something. set simple, have sequence this:

log time (t1) lock mutex call c++ std::list::push_back()/pop_back() (i.e. simple math) unlock mutex log time (t2)

from time time, see sequence above running in several independent (!) processes blocks @ step 2 (or probaby @ step 4) excessive time concerning math @ step 3 (for instance, 0.5 - 1.0 seconds). proof, see t2 in logs processes literally same (different in microseconds). looks threads of processes come in section @ relatively different times (i can see 0.5 - 1 seconds difference t1), lock in mutex, , unlock @ same time having allegedly spent unreasonable amount of time in lock according log (t2 - t1 difference). looks creepy me.

the manifestation of issue relatively rare, 1 time 5-10 minutes under moderate load. no ntp time shifts logged within test (that first thought actually). if ntp, there not actual delays in service, wrong times in log.

where start? start tuning scheduler? can theoretically block entire multithreaded process in linux?

run programme with:

valgrind --tool=helgrind ./your_program

you find more issues expect.

valgrind (helgrind) give detailed scenario of threaded application, nowadays must before deployment.

c++ linux-kernel mutex blocking scheduler