Our system had a memory footprint growing gradually.After doing too many debugging with the profilers, didnt reach the exact point of issue.Now after verifying random things on the system, it got cornered to the websockets we used.
Those sockets had a lot of unread messages in its queue. Memory usage was directly proportional to the number of messages. By clearing up the messages in queue, there was a huge memory reclaimed.
Problem:
Tested OS version: CentOS 7.5
I tried checking the memory occupied by the sockets using '/proc/net/sockstat'the mem column showed the memory was ~300mb
Total memory for the recv-Q ~6mb measured using netstat -tunp (treated the numbers in recv-Q as bytes)
But when i cleaned up the unread messages, I got ~1.5gb of memory reclaimed. (using free command)
Anything else to be checked to get the right memory usage for sockets?
Is that an unwanted memory usage done by linux? How to debug the memory used by sockets further?
Why linux tools like top isnt listing the memory usage for sockets? It shows us the memory for processes, cache and buffers, but why not sockets.
Additional details:Changing the memory allocator to jemalloc didnt stop this memory growth. So it is not an issue associated with glibc.
=================================================================
Edited Info: After doing some work with the test application
Converted our problem into a simple test program and ran it in servers with different kernel versions.
The test program: 5000 sockets and 4 incoming messages(3 bytes per message) to that socket every minute. Also did some work on using the ss -tm to clearly understand the buffer memory behavior.
Machine 1: Kernel: 2.6.32/proc/sys/net/core/rmem_max=124928
At start: Free mem: 2.5gbFor every incoming message, mem in ss -tm grew by 512 bytes per socket.At some point, there was a sudden drop in memory usage by sockets.
Before memory drop:
free -m : free memory: 1.1G
sockstat: TCP: inuse 6 orphan 1 tw 161 alloc 5265 mem 114138
ss -tm : mem:(r112784,w0,f1904,t0)
After memory drop:
free -m free memory: 2.3G
sockstat TCP: inuse 6 orphan 1 tw 157 alloc 5266 mem 8042
ss -tm mem:(r9528,w0,f952,t0)
Values in recv-Q was constantly increasing with the expected values.
It was the point where "r" value reached approx equal to core/rmem_maxSeemed like a compaction process happened there.
Machine 2: Kernel: 3.10.0/proc/sys/net/core/rmem_max=212992
Here I expected that memory will get dropped at ~212992. But this machine had the upgraded version ss which showed the rb=367360 size itself. So waited for the exact compaction process to happen.
At start:
ss -tm : skmem:(r0,rb367360,t0,tb87040,f53248,w0,o0,bl0)
sockstat: TCP: inuse 4 orphan 0 tw 97 alloc 5042 mem 4992
Here also the memory kept on increasing in the expected rate. There was memory drop at a particular point of time.
At memory drop point 1:Before memory drop:
free : free memory: 2.1gb
sockstat : TCP: inuse 4 orphan 0 tw 89 alloc 5097 mem 354398
ss -tm : skmem:(r290560,rb367360,t0,tb87040,f256,w0,o0,bl0)
After memory drop:
free : free memory: 3.1gb
sockstat : TCP: inuse 4 orphan 0 tw 93 alloc 5099 mem 187542
coming to ss -tm, saw a different behavior this time:
50% of the sockets had compacted values,
skmem:(r4352,rb367360,t0,tb87040,f3840,w0,o0,bl0)
and the remaining had actual values (not compacted)
skmem:(r291072,rb367360,t0,tb87040,f3840,w0,o0,bl0)
So compaction happened soon before "r" value reached "rb"
Next, waited till the "r" value reaches "rb"
Memory drop point 2There the next point of memory drop happened. All the socket buffers were compacted.(except 100 sockets) and huge memory was reclaimed.
=================================================================
Now my understanding:
The actual problem we faced in our servers: The memory footprint was growing continuously and machine started using swap space and got slowed down. Now after running the test programs, I understood that the free space available in our servers were not enough to hold till the point of compaction.
My Questions:
Is this compaction a inbuilt behavior of socket buffers?
If yes, when will that happen, in Machine 2, i had different experience than the one in machine 1? Which value should be tuned to bring the compaction early?
"mem" value in sockstat and sum of "r" value in ss add up to give the total memory occupied by the socket? Or they are same values listed by different tools.
(As per my tests, I see (mem value sockstat + skmem buffer value) equals the memory getting freed up.)