If a linux-build-server suddenly starts getting slow, I would divide my approach / troubleshooting into 3 section as follows;
1. System Level troubleshooting
a. RAM related issues
b. Disk Space related Issues
c. Disk I/O read write issues
d. Network Hardware issues
e. Mount issues
f. Too Many process running in the machine
2. Application Level troubleshooting
a. Application is not behaving properly. Hit to Application log file OR application server log file OR web server Log file and try to understand the issues.
b. zombie process issues – Find out if any as such process which is causing the system performance issues.
c. Application Log – depends on the application installed, this can be referred and make use of the experience with the project and troubleshoot.
d. Web Server Log – we can check http, tomcat log as well.
e. Application Server Log – We can see jboss, weblogic logs to see if the application server response/receive time is the issues for slowness.
f. Memory Leak of any application – This is one of well known issues in lunux based server due to bad application coding. Many times this can be resolved either by fixing the code or rebooting. But many other solutions are there to apply.
3. Dependent Services troubleshooting
a. SMTP Response time – SMTP server is not responding faster which is causing delay in response and queue up many processes.
b. Network issues – There are many System performance issues is dependent on network or service which is depends on the network.
c. Firewall related issues
d. Antivirus related issues
Some of the useful commands for troubleshooting are..
1. df –k
2. du –sh
5. ps –eaf | grep
8. tail –f <logfile>
14.ifconfig eth0 | enable | disable
There are various options available for each commands and depends on the need during the troubleshooting, right commands can be applied. To find arguments and their explanation, I usually refer google.com rather than man page.