If a linux-build-server suddenly starts getting slow, I would divide my approach / troubleshooting into 3 section as follows;

1. System Level troubleshooting

a. RAM related issues

b. Disk Space related Issues

c. Disk I/O read write issues

d. Network Hardware issues

e. Mount issues

f. Too Many process running in the machine

2. Application Level troubleshooting

a. Application is not behaving properly. Hit to Application log file OR application server log file OR web server Log file and try to understand the issues.

b. zombie process issues – Find out if any as such process which is causing the system performance issues.

c. Application Log – depends on the application installed, this can be referred and make use of the experience with the project and troubleshoot.

d. Web Server Log – we can check http, tomcat log as well.

e. Application Server Log – We can see jboss, weblogic logs to see if the application server response/receive time is the issues for slowness.

f. Memory Leak of any application – This is one of well known issues in lunux based server due to bad application coding. Many times this can be resolved either by fixing the code or rebooting. But many other solutions are there to apply.

3. Dependent Services troubleshooting

a. SMTP Response time – SMTP server is not responding faster which is causing delay in response and queue up many processes.

b. Network issues – There are many System performance issues is dependent on network or service which is depends on the network.

c. Firewall related issues

d. Antivirus related issues

 Some of the useful commands for troubleshooting are..

1. df –k

2. du –sh

3. top

4. uptime

5. ps –eaf | grep

6. vmstat

7. ping

8. tail –f <logfile>

9. iostat

10.free

11.kill -9

12.mount

13.sar

14.ifconfig eth0 | enable | disable

15.traceroute

16.netstat -r

17.nslookup

18.route

   There are various options available for each commands and depends on the need during the troubleshooting, right commands can be applied. To find arguments and their explanation, I usually refer google.com rather than man page.

The post If a linux-build-server suddenly starts getting slow, what would you check? appeared first on scmGalaxy.

Performance Optimization Checklist of Build Servers

Build Infrastructure Level

1.  Do you really need to build all source code or only the part of code which has changes?

2. Project should be divided into multiple modules/component which can be build independently and should be integrated when it needed.

3. Understand technology and make use of performance related features which will help you. e.g make, visual studio and maven has some feature which help us to compile only code which was changed also, some performance tuning features should be explored.

4. Compile dependent project only there are changes. no need to compile all the time.

5. Distribute application in non-compiled and to be compiled files. e.g src, docs and copy only src to comile. docs files cane be used while creating installer.

6. Plan for disk usage growth upfront.

Application Server level

1. Apache Server optimization and proper log setting
2. Disabling unused modules and component which will help performance.
3. Follow best practices of Module Configuration, Apache Server Configuration etc

 System level

1. Check if any background task/services is running which is consuming cpu/memory and can be  stopped and not relevant.
2. Keep system uptodate
3. Avoid using build server for any other task – If you must edit components or install software, build or edit the components on another machine and copy them over.
4. Use high speen drive for i/o operations

Build Server application level

http://www.cloudbees.com/sites/default/files/whitepapers/7WaysToOptimizeJenkins.pdf

1. Keep log files maintenance and truncate regularly. have limited build records.
2. Move build to document server. not to keep in build server
3. Take advantage of distribute build server using agents/slaves/nodes etc

The post Performance Optimization of Build Server | Performance Optimization Guide appeared first on scmGalaxy.

Hardware Configuration – Hardware Requirements for Build Server

Reference: The Build Master: Microsoft’s Software Configuration Management Best Practices

At a minimum, the build lab should have four machines:

Server that contains the Source Code Control program— This is your product. Do you really want this server residing someplace where you have little control over this box?

Debug build machine for the mainline builds— If you don’t separate your debug and release machines, you will accidentally ship debug binaries, which is not a good thing.

Release build machine for the mainline builds— This is a “golden goose” that turns out the “gold eggs” of your company or group. Treasure this machine like a princess, and guard it like all the king’s fortunes.

Internal release share server— This is one more piece of hardware that stores the “bread and butter” of the group or company. Don’t give up control of this hardware to anyone unless your IT department reports through your development group.

Hardware Requirements
Each machine in the preceding list should meet the following requirements:

Number of processors— This depends on the build tool you use. One is usually sufficient, because few build tools really take advantage of multiple processors.

Processor speed— The lab budget dictates this, but the faster the processor, the better it is.

Amount of installed RAM— Max out the machine. RAM is relatively cheap these days, especially when you consider the performance increase you get. Increasing the RAM is usually the first upgrade done when trying to improve the performance of any computer.

Number of hard drives— A minimum of two drives (or partitions) is preferred:

Drive 1 (C:) is for the operating system and installed applications.

Drive 2 (D:) is for building binaries, release shares, or the source database; the minimum space required is roughly ten times the space needed to build your application.

The split partitions are good because if you ever need to format or blow away a drive due to corruption, only part of the project will be affected. The recovery is much faster and easier.

Hard drive type— This is most likely SCSI, but it could be IDE.

Number of power supplies— If you purchase server class hardware (pizza boxes) that belong in racks, you need to consider how many power supplies to order.

Motherboard BIOS version— This does make a difference. Make sure you note what is being used and standardize on it.

The post Hardware Configuration – What are the Hardware Requirements for Build Server? appeared first on scmGalaxy.