The actual Memory Usage on Linux

Hi,

It has been quite long since I have posted here. The main reason was the lots of work in the office and preparing for my next AWS Certification, which was AWS Solutions Architect Professional. I will write a short note on the exam details and what I did to clear it, later on.

What was happening ?

Coming back to this topic, I had setup Cloudwatch alerts for Memory Usage and Disk Usage and then alert the stakeholders when the usage breaches the threshold set. For me it was set to 90% for both the metrics. But upon observing closely, the alerts for memory usage were quite frequent, say 2 times per 5 minutes. This meant that something was definitely wrong and I needed to recheck the script deployed, fast.

Upon inspection, my previous script was calculating the memory from free -m command, using the first column:

1
2
3
4
             total       used       free     shared    buffers     cached
Mem: 7984 5423 2561 0 385 3130
-/+ buffers/cache: 1907 6077
Swap: 0 0 0

The calculation was as follows:

1
Memory Usage Percentage = (Used/Total) * 100

This comes out to be 67.92%.

Now I had used a custom script ps_mem.py LINK to calculate the approximate memory usage:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
[ec2-user@ip-172-31-X-Y tmp]$ sudo ./ps_mem.py
Private + Shared = RAM used Program

144.0 KiB + 58.0 KiB = 202.0 KiB agetty
168.0 KiB + 44.5 KiB = 212.5 KiB rngd
196.0 KiB + 68.0 KiB = 264.0 KiB atd
272.0 KiB + 76.5 KiB = 348.5 KiB irqbalance
444.0 KiB + 164.5 KiB = 608.5 KiB rpcbind
580.0 KiB + 140.5 KiB = 720.5 KiB auditd
628.0 KiB + 98.5 KiB = 726.5 KiB dbus-daemon
680.0 KiB + 115.0 KiB = 795.0 KiB crond
532.0 KiB + 305.0 KiB = 837.0 KiB mingetty (6)
780.0 KiB + 125.5 KiB = 905.5 KiB init
356.0 KiB + 569.5 KiB = 925.5 KiB mysqld_safe
856.0 KiB + 254.5 KiB = 1.1 MiB rpc.statd
636.0 KiB + 570.5 KiB = 1.2 MiB bash
492.0 KiB + 885.5 KiB = 1.3 MiB udevd (3)
1.1 MiB + 421.0 KiB = 1.5 MiB ntpd
1.0 MiB + 560.0 KiB = 1.5 MiB sudo
1.0 MiB + 547.5 KiB = 1.6 MiB dhclient (2)
2.4 MiB + 580.5 KiB = 3.0 MiB cupsd
1.4 MiB + 2.9 MiB = 4.3 MiB sshd (3)
3.4 MiB + 1.2 MiB = 4.6 MiB sendmail.sendmail (2)
4.7 MiB + 129.0 KiB = 4.8 MiB rsyslogd
9.9 MiB + 14.5 KiB = 9.9 MiB docker-containerd
34.7 MiB + 108.0 KiB = 34.8 MiB dockerd
43.5 MiB + 732.0 KiB = 44.2 MiB mysqld
1.4 GiB + 2.0 MiB = 1.4 GiB java (2)
---------------------------------
1.5 GiB
=================================
Warning: Swap is not reported by this system.

As per the output above, the actual memory usage is about 1.5 GB or 1536 MB, which is 19.23%.


Calculating the memory usage, properly

I tried various flavours of linux, such as Ubuntu, RHEL, CentOS and Amazon Linux, and found variations of the commands such as:

  • free -m
  • cat /proc/meminfo

However, 1 specific command vmstat was giving the output almost properly among all the linux flavours. So I decided to go with it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# vmstat -s
8176604 total memory
5555796 used memory
4379188 active memory
745036 inactive memory
2620808 free memory
394516 buffer memory
3206024 swap cache
0 total swap
0 used swap
0 free swap
664123 non-nice user cpu ticks
45 nice user cpu ticks
138166 system cpu ticks
75768535 idle cpu ticks
39822 IO-wait cpu ticks
0 IRQ cpu ticks
1913 softirq cpu ticks
12845 stolen cpu ticks
5794205 pages paged in
10155760 pages paged out
0 pages swapped in
0 pages swapped out
57218586 interrupts
101894616 CPU context switches
1515624326 boot time
589690 forks

In the above output, we are looking for:

  • total memory
  • free memory
  • buffer memory
  • swap cache, or simply cache.

Command 1:

1
2
# Gathering info
vmstat -s | sed -e 's/^[ \t]*//' | cut -d' ' -f1,3- | tr -s ' ' ',' | grep -e 'total,memory' -e 'free,memory' -e 'buffer,memory' -e 'swap,cache' | awk -F, '{print $1/1024","$2" "$3}'

Output:

1
2
3
4
7984.96,total memory
2553.59,free memory
386.012,buffer memory
3131.41,swap cache

With some mathematical formulas, I am able to calculate the near accurate (actual) memory usage on the system.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# Total Memory
MemTotal="$(cat /tmp/mem-usage.csv | grep -e 'total memory' | awk -F, '{print $1}')"
# Free Memory
MemFree="$(cat /tmp/mem-usage.csv | grep -e 'free memory' | awk -F, '{print $1}')"
# Buffers
Buffers="$(cat /tmp/mem-usage.csv | grep -e 'buffer memory' | awk -F, '{print $1}')"
# Cached
Cached="$(cat /tmp/mem-usage.csv | grep -e 'swap cache' | awk -F, '{print $1}')"
# Calculating total free memory (Free + Buffers + cached)
TotalMemFree="$(bc -l <<< $MemFree+$Buffers+$Cached)"
# Calculating total memory used (Total Memory - Total Free Memory)
TotalMemUsed="$(bc -l <<< $MemTotal-$TotalMemFree)"
# Percentage Used
mem="$(bc <<< "scale=2; $TotalMemUsed*100/$MemTotal")"
echo "$mem"

Conclusion

Till now, this is the best method I have come up with, to calculate the memory usage of the linux boxes. If you have any other method to calculate the memory usage, do let me know in the comments boxes.

The Memory and Disk Usage script is available on my Github account.

amazon web services aws cloudwatch ubuntu ec2 centos rhel amazon linux RAM Memory