When tuning a site's performance, a system administrator needs certain tools to measure and monitor how the site is doing under increasing load, as well as identify where bottlenecks may be.

Linux system performance monitoring tools

On Linux, there are several tools available to you to do this. We describe what we have found as the most helpful in this section.

top

Top is a nice little program that provides real time information on the system level. It provides how much CPU is available (idle), as well as how much is used by the system, and how much by user programs. It also provides the load averages, memory utilization, swapping and other information.

Top is interactive, and hence it provides a snapshot of the system at the moment it is being viewed. In other words it does not provide history, beyond the latter two figures of the load average.

You can set certain options as the time interval to refresh the screen (default 3 seconds), as well as what columns to sort by. Just press the ? key, and you will get a help screen.

Note that we have seen top report overinflated figures for CPU and load under certain circumstances, such as when a system had two Xeon processors, but with hyperthreading, they appeared as 4 CPUs to the Linux.

Ubuntu Edgy 6.10, lightly loaded system running several Drupal sites, using Apache 2.0.55, PHP 5.1.6, and APC 3.0.13. 

top - 12:47:23 up 22 days, 17 min,  1 user,  load average: 0.05, 0.09, 0.09
Tasks:  61 total,   1 running,  59 sleeping,   1 stopped,   0 zombie
Cpu(s):  3.4%us,  0.2%sy,  0.0%ni, 96.4%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:    393380k total,   366376k used,    27004k free,        0k buffers
Swap:   802808k total,    37184k used,   765624k free,    68568k cached
PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
23236 www-data  16   0 62756  17m 4132 S    7  4.5   1:05.17 apache2
2125 mysql     16   0 59152  46m 3124 S    0 12.2  28:28.53 mysqld
3624 www-data  16   0  2252 1120  860 R    0  0.3   0:00.08 top
1 root      16   0  1636  376  332 S    0  0.1   0:00.38 init
2 root      RT   0     0    0    0 S    0  0.0   0:11.49 migration/0
3 root      34  19     0    0    0 S    0  0.0   0:00.04 ksoftirqd/0
4 root      RT   0     0    0    0 S    0  0.0   0:00.01 watchdog/0

Ubuntu Edgy 6.10, relatively busy server running just one Drupal site, using using Apache 2.0.55, PHP 5.1.6, and eAccelerator 0.9.5.

Note that there is about 5 MB less usage per Apache process than APC (17 MB vs. 12 MB as per the RES column). 

top - 12:40:59 up 11 days,  4:23,  1 user,  load average: 0.75, 0.51, 0.41
Tasks: 102 total,   1 running, 101 sleeping,   0 stopped,   0 zombie
Cpu(s): 11.8%us,  1.9%sy,  0.0%ni, 86.0%id,  0.1%wa,  0.0%hi,  0.2%si,  0.0%st
Mem:   2075932k total,  1978924k used,    97008k free,    50492k buffers
Swap:  1574360k total,      116k used,  1574244k free,  1590960k cached
PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
13523 mysql     15   0  222m 130m 5176 S    8  6.4   1793:05 mysqld
7906 www-data  15   0 55584  12m 8020 S    4  0.6   0:06.79 apache2
9581 www-data  15   0 55572  12m 7832 S    3  0.6   0:01.87 apache2
8748 www-data  15   0 55296  12m 7984 S    2  0.6   0:03.99 apache2
9584 www-data  15   0 55136  11m 7776 S    1  0.6   0:01.78 apache2
9585 www-data  15   0 55300  12m 7868 S    1  0.6   0:02.47 apache2
10006 www-data  15   0 55416  12m 7772 S    1  0.6   0:01.17 apache2
10015 www-data  15   0 55240  11m 7724 S    1  0.6   0:00.89 apache2
10004 www-data  15   0 54936  11m 7764 S    1  0.6   0:01.06 apache2
10020 www-data  15   0 55300  11m 7720 S    1  0.6   0:00.84 apache2
10032 www-data  15   0 55524  12m 7840 S    1  0.6   0:00.74 apache2
10008 www-data  15   0 55464  12m 7692 S    0  0.6   0:00.60 apache2
10010 www-data  15   0 55468  12m 7848 S    0  0.6   0:01.13 apache2
6608 www-data  15   0 55304  12m 7920 S    0  0.6   0:12.06 apache2
7913 www-data  15   0 55552  12m 8012 S    0  0.6   0:06.75 apache2
8326 www-data  15   0 55336  12m 8396 S    0  0.6   0:06.76 apache2
10021 www-data  15   0 55436  12m 7800 S    0  0.6   0:00.91 apache2
10027 www-data  15   0 55272  11m 7724 S    0  0.6   0:00.62 apache2
10028 www-data  15   0 55284  11m 7736 S    0  0.6   0:00.40 apache2
10033 www-data  15   0 55544  12m 7780 S    0  0.6   0:00.74 apache2
10459 root      16   0  2252 1152  852 R    0  0.1   0:00.01 top
1 root      16   0  1632  536  448 S    0  0.0   0:01.82 init
2 root      RT   0     0    0    0 S    0  0.0   0:00.26 migration/0
3 root      34  19     0    0    0 S    0  0.0   0:00.14 ksoftirqd/0

The following is from a Fedora Core 4 dedicated server running one moderately busy Drupal site. Apache 2.0.54, MySQL 4.1.20, PHP 5.0.4, and eAccelerator 0.9.3.

Note that although they run eAccelerator, they have some other modules enabled that eat up memory (e.g. mod_python). 

 top - 12:47:58 up 6 days,  4:22,  3 users,  load average: 0.10, 0.15, 0.10
Tasks: 128 total,   1 running, 127 sleeping,   0 stopped,   0 zombie
Cpu(s):  3.2% us,  0.7% sy,  0.0% ni, 96.0% id,  0.1% wa,  0.1% hi,  0.1% si
Mem:   1015012k total,   975996k used,    39016k free,    44364k buffers
Swap:  2048276k total,       52k used,  2048224k free,   627996k cached
PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
1781 mysql     16   0 84328  72m 3608 S  3.7  7.3 113:23.58 mysqld
21058 apache    15   0 61424  20m  12m S  1.3  2.1   0:01.57 httpd
21061 apache    15   0 60484  20m  13m S  1.3  2.0   0:01.86 httpd
21055 apache    15   0 59788  19m  12m S  0.2  1.9   0:00.71 httpd
21064 apache    15   0 60168  19m  13m S  0.2  2.0   0:01.35 httpd
21069 apache    15   0 60200  19m  13m S  0.2  2.0   0:01.00 httpd
21054 apache    16   0 60148  21m  14m S  0.1  2.1   0:01.67 httpd
21066 apache    15   0 61408  20m  12m S  0.1  2.1   0:00.89 httpd
21067 apache    15   0 59720  19m  13m S  0.1  2.0   0:00.94 httpd
21068 apache    16   0 61504  21m  13m S  0.1  2.1   0:01.44 httpd
24626 root      16   0  2024 1056  808 R  0.1  0.1   0:00.19 top
1 root      16   0  1740  576  500 S  0.0  0.1   0:00.91 init
2 root      RT   0     0    0    0 S  0.0  0.0   0:00.11 migration/0
3 root      34  19     0    0    0 S  0.0  0.0   0:00.01 ksoftirqd/0

Read more about the Top manual page. Also here, but as always, check the man page for your specific system though

htop

htop is similar to top, but shows individual CPU utilization in a multi-process system. It also uses colors so the output is more friendly than just numnbers. Columns can be sorted by using function keys, and not obscure letter commands like top. Highly recommended.

vmstat

The vmstat utility provides similar information to top, but can give a time dimension.

Here is a sample from a large dedicated server running a busy web site, but during non-peak hours.

Note that the cache is more than 1.5 GB out of 2 GB total memory. This is good since the database can cache lots of data in memory avoiding slower disk I/O. 

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
3  0    116  61288  50396 1589016    0    0     1   111  530 2764  8  1 91  0
0  0    116  64928  50400 1589216    0    0    42   100  446 3017 10  3 87  0
1  0    116  64184  50408 1589276    0    0     2   121  538 3677 16  4 79  1
0  0    116  64208  50408 1589276    0    0     0   268  569 3407 13  5 82  0
0  0    116  64208  50408 1589344    0    0     3   118  518 2853 14  2 84  0
1  0    116  62348  50408 1589412    0    0     3   183  511 3828 20  4 76  0
0  0    116  63092  50408 1589412    0    0     0   130  521 2324 10  2 88  0
0  0    116  71928  50412 1589680    0    0    42    99  399 2931 12  2 85  0
0  0    116  71812  50416 1589948    0    0    54   147  583 3778 13  4 83  0
2  0    116  80492  50416 1590016    0    0     0   130  423 2121  7  2 91  0
0  0    116  88144  50416 1590016    0    0     0   303  422 4090 16  3 81  0
0  0    116  85772  50416 1590016    0    0     0   103  592 3742 15  3 82  0

The following snapshot is from the same site, but during peak hours. Note the occasional rise in the number of processes on the run queue, as well as increased user time and decreased idle time.

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
1  0    124  65856  38620 1546664    0    0     2   288  898 7319 36  6 56  2
0  0    124  60628  38628 1546928    0    0    20   534 1174 7254 40 10 49  1
2  0    124  64168  38640 1547188    0    0    44   290  817 5049 26  6 68  1
1  0    124  63680  38644 1547456    0    0    48   236  975 6968 41  6 52  1
2  0    124  59864  38656 1547920    0    0    69   254 1077 8106 45 11 43  0
5  0    124  59372  38660 1548052    0    0    18   268 1052 6450 37  9 52  2
1  0    124  58084  38672 1548584    0    0    99   236 1080 7249 35  9 54  1
1  0    124  61440  38676 1548716    0    0     0   229  888 7829 36  8 54  1
0  0    124  61304  38680 1548848    0    0     6   226  978 7137 47  9 44  0
2  0    124  70392  38688 1548976    0    0     5   583  939 7389 42  8 48  2
0  0    124  70348  38696 1548968    0    0    14   250  826 7412 41  7 52  0
2  0    124  70040  38700 1549100    0    0     8   228 1100 8181 42  7 50  1
1  0    124  64152  38708 1549296    0    0    22   260 1181 6074 32  6 61  2
0  0    124  56868  38712 1549360    0    0     2   282 1034 7218 44 11 44  1
2  0    124  55116  38716 1549560    0    0     2   243 1087 5628 23  7 69  1

Case study: before and after using vmstat

A case study for how to use vmstat to determine bottlenecks, is a relatively busy Drupal site that runs on two large VPSs, one for Apache/PHP/Drupal, and the other for the database.

The Apache VPS had no issues at all, but the database VPS was severely loaded before tuning.

Here is what vmstat said before tuning.

What is wrong? Many things are wrong, including:

  • High user cpu time (us column) of 70-80%
  • Low idle time (id column) 10-15%
  • Low free memory, combined with a relatively low cache usage
  • Many processes waiting in the run queue (the r column)
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
8  0    560   7408    984 281116    0    0    30   383  147   160 41  6 51  2
8  0    560   7524    984 281116    0    0     0     0 1083 12305 78 11 11  0
9  0    560   7548    984 281116    0    0     5     2 1047 11191 81 11  9  0
9  0    560   6896    984 281636    0    0   194  2426 2600  9378 71 13 10  6
7  0    560   6776    984 282156    0    0    29   170 1262 11739 79 11 10  0
8  0    560   6776    984 282156    0    0     0   449 1176 13187 77  9 13  1
8  0    560   6776    984 282156    0    0     0     0 1112 12271 78  9 13  0
5  0    560   6776    988 282152    0    0     1    83 1239 13453 76  9 14  0
8  0    560   6776    988 282152    0    0     0   472 1156 12706 77 11 12  0
8  0    560   6896    988 282152    0    0     0     1 1084 12147 78 10 12  0
8  0    560   6904    988 282152    0    0     0     2 1122 12272 78  8 14  0
3  1    560   6904    988 282152    0    0     0    36 1007 11444 76 11 13  0
3  0    560   7152    988 282152    0    0     0    59  926 10498 76 13 11  0
8  0    560   7244    988 282152    0    0     0    52 1031 11115 79 10 11  0
9  1    560   7120    988 282152    0    0     0   794 1235 10674 78 11 10  1
8  0    560   6900    984 282156    0    0   423  2464 2535  8873 64 12 17  7
8  0    560   6476    984 282416    0    0    70   526 1510 11515 81 10  9  1
9  0    560   7760    984 281376    0    0   123   534 1404 12478 77 12  9  2
8  0    560   7076    992 282408    0    0    32   228 1309 12573 80 10 10  0
8  0    560   6516    992 283188    0    0    17    71 1197 12585 80 10 10  0

After tuning, here is how the server looks:

  • Low user cpu time (us column) 2-5%
  • High idle time (id column) of 90% or more
  • High free memory, and higher cache usage
  • Almost no processes waiting in the run queue (the r column)
 procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
1  0    560 439660   1552 468008    0    0    28   346   34    65 37  5 56  2
0  0    560 439536   1552 468008    0    0     0   614 1081  1275  1  1 95  3
0  0    560 439164   1552 468008    0    0     0  1192  708   841  2  0 94  3
0  0    560 438668   1552 468008    0    0     0   864 1054  1202  1  0 95  4
0  0    560 438668   1552 468008    0    0     0  1004  863  1036  1  0 94  4
0  0    560 438424   1552 468268    0    0    32   917  942  1137  1  0 95  4
0  0    560 438672   1552 468268    0    0     0  1642  785   930  1  0 95  4
0  0    560 438620   1552 468268    0    0     7   996 1087  1349  1  0 94  4
0  0    560 438248   1552 468268    0    0     0  1336 1144  1436  1  0 95  4
0  0    560 438124   1552 468268    0    0     0   737 1091  1334  1  0 95  3
0  0    560 437752   1552 468268    0    0     0   931 1101  1379  1  0 95  3
0  0    560 437752   1552 468268    0    0     0   622  878  1085  1  0 95  3
0  0    560 437380   1552 468268    0    0     0  1244  938  1147  1  0 95  4
0  0    560 437132   1552 468268    0    0     0  1078  909  1073  1  0 94  4
0  0    560 436140   1552 468268    0    0     0  1310 1110  1296  5  0 90  4
0  0    560 435892   1552 468268    0    0     0   473  857  1056  1  0 96  3
0  0    560 435768   1552 468268    0    0     0  1714 1507  1840  2  0 92  6
0  0    560 435768   1552 468268    0    0     0   653 1287  1625  2  0 94  3

Check the vmstat manual page for more details.

apachetop

Apachetop is a command that watches the access log and displays real time statistics. Because it continually reads the access log, it may put some load on a busy system.

mtop

mtop is a top-like utility for monitoring MySQL. It provides some important statistics, like the number of questions per second, total queries since server started, number of slow queries, ...etc.

More importantly it shows any slow queries or locks that are active at the time, with the exact SQL that is executed. This can help identify the module that caused this query to run.

If you do not have the mtop utility you can still see some information it provides by running the command SHOW FULL PROCESSLIST from a mysql prompt.

mytop

mytop is another top-like tool for monitoring MySQL.

munin

Munin is a system monitoring tool that produces nice easy to understand graphs. It can monitor CPU, memory, Apache, MySQL, network and many other components.

It keeps a history of what is going on in the system by day, week, month and year.

cacti

Cacti is a graphing tool similar to Munin.

vnstat

Vnstat provides network interface statistics in a human readable way.  It can give daily or monthly statistics for each ethernet card, and provide transmit and receive figures for each.

Resources

Drupal performance monitoring tools

At the Drupal level, there may be bottlenecks in a certain module. To get an idea on how much time your pages are taking to be generated, you can use the very useful devel module.

Besides the total time used to generate a page, the module can also provide you with info on how much database I/O is taken, as well as the individual queries executed with how much time each took to execute.

You can also insert timing code in the areas you suspect by calling the functions timer_start(), timer_read() and timer_stop().

Articles

Comments

Visitor (not verified)

Fri, 2009/10/16 - 05:05

I'm working on setting up some environments for performance testing . In the past when doing performance testing in Windows, I've made extensive use of PerfMon to record and view various performance (disk I/O, mem usage, CPU usage, etc) counters over the network.

What tools exist for Linux fedora core 6 that will allow me to do the same thing?

Will Zabbix Tool solve my problem.

Suggestions from your side required

Visitor (not verified)

Wed, 2008/04/02 - 12:51

We use Versiera to manage/monitor our desktops and servers. You can monitor down to the process level and performance information for host stats can also be displayed in graphs over time intervals. The net version is free.

Visitor (not verified)

Sat, 2009/01/17 - 19:56

iptraf - measures the trafic on interfaces
mpstats - shows processor statistcs
mysqladmin - real time statics for mysql like queries per second, query cache utilization, etc...
and of course system monitor for gnome

Regards,
B.

Visitor (not verified)

Thu, 2009/05/21 - 09:28

Hi,

My server hold many HTML static pages and php pages (Drupal site and more). How Can I distinguish , in the "top -c " output , which tasks are PHP?
I want to analyze if some heavy pages come from PHP (Drupal site in my case).

Many thanks ,
Miriam

Is your Drupal or Backdrop CMS site slow?
Is it suffering from server resources shortages?
Is it experiencing outages?
Contact us for Drupal or Backdrop CMS Performance Optimization and Tuning Consulting