If a linux-build-server suddenly starts getting slow, I would divide my approach / troubleshooting into 3 section as follows;

1. System Level troubleshooting

a. RAM related issues

b. Disk Space related Issues

c. Disk I/O read write issues

d. Network Hardware issues

e. Mount issues

f. Too Many process running in the machine

2. Application Level troubleshooting

a. Application is not behaving properly. Hit to Application log file OR application server log file OR web server Log file and try to understand the issues.

b. zombie process issues – Find out if any as such process which is causing the system performance issues.

c. Application Log – depends on the application installed, this can be referred and make use of the experience with the project and troubleshoot.

d. Web Server Log – we can check http, tomcat log as well.

e. Application Server Log – We can see jboss, weblogic logs to see if the application server response/receive time is the issues for slowness.

f. Memory Leak of any application – This is one of well known issues in lunux based server due to bad application coding. Many times this can be resolved either by fixing the code or rebooting. But many other solutions are there to apply.

3. Dependent Services troubleshooting

a. SMTP Response time – SMTP server is not responding faster which is causing delay in response and queue up many processes.

b. Network issues – There are many System performance issues is dependent on network or service which is depends on the network.

c. Firewall related issues

d. Antivirus related issues

 Some of the useful commands for troubleshooting are..

1. df –k

2. du –sh

3. top

4. uptime

5. ps –eaf | grep

6. vmstat

7. ping

8. tail –f <logfile>

9. iostat

10.free

11.kill -9

12.mount

13.sar

14.ifconfig eth0 | enable | disable

15.traceroute

16.netstat -r

17.nslookup

18.route

   There are various options available for each commands and depends on the need during the troubleshooting, right commands can be applied. To find arguments and their explanation, I usually refer google.com rather than man page.

The post If a linux-build-server suddenly starts getting slow, what would you check? appeared first on scmGalaxy.

Perforce Slow Sync issues

Network latency
Sync Performance issues are not obvious with locally connected hosts, as the network latency is low. However, as latency increases, performance worsens.

To correct the issue, follow these steps on the client machines exhibiting the issue:
http://kb.perforce.com/article/1191/slow-sync-on-remote-windows-clients

Network problems:
How do I determine if slow Perforce response time is caused by network problems?

A network issue can be suspected if Perforce commands run quickly on the local machine but run slowly across a network. You can also compare the lapse time against the usage time, for example, if the information is available in the Perforce logs.

Any of the following can cause slow responses:

1. Misconfigured domain name server (DNS)
2. Misconfigured Windows name server (WINS) or Windows domain
3. Slow network response

1. p4 info

A good initial test is to run the p4 info command. If this command does not respond nearly immediately then there is a network related problem.
The p4 info command uses the P4PORT setting to contact the Perforce Server. If the P4PORT setting uses the hostname of the server machine, a DNS lookup is required to fetch the server IP address. If the DNS server is failing or the network is slow, the lookup process takes time. You can use the IP address directly to avoid the DNS lookup.

2. Hostname vs. IP Address
On a client machine, try using the Perforce Server’s IP address in the P4PORT setting. This avoids the DNS lookup used to convert the hostname to an IP address. Here is a P4PORT example which uses the server hostname:

P4PORT=hostname:1666
Using the IP address directly, the P4PORT setting then looks like this:

P4PORT=1.2.3.4:1666

If you do not have the Perforce Server machine’s IP address handy, you can use the ping command to find it. Here is an example of how you would run the ping command:

3. ping hostnam

The output of the ping command lists the IP address for the hostname.

If the p4 info command responds immediately when you use the IP address in the P4PORT setting, you have a misconfigured DNS.

“p4 info” vs. P4Win

The p4 info command is processed by the Perforce Server. As the Perforce Server compiles the output information for the command, it does a reverse DNS lookup on the client’s IP address. A forward DNS lookup might be fast, while a reverse DNS lookup is slow. One way to determine if a reverse DNS lookup is slow is by using P4Win, the Perforce Visual Client. The “Show Connection Info” operation in P4Win does not perform a reverse DNS lookup.

You can compare the response of P4Win’s “Show Connection Info” with the response from the command line p4 info. If a “Show Connection Info” operation is fast while a p4 info is slow, there is a reverse DNS lookup problem on the Perforce Server machine.

4. The “hosts” file

To work around DNS problems, add hostname IP address entries to the hosts file. This task is typically performed by a Systems Administrator; be sure to follow your company’s standard procedures.

Rather than using an IP address for the P4PORT setting, you can add a hostname IP address entry in your hosts file. The host file can be tricky to find. Here are a few places it can be:

Windows Server 2000 (and later):

C:\Winnt\System32\Drivers\etc\hosts

Unix:

/etc/hosts

The following is an example entry for the hosts file:

1.2.3.4 hostname

If you have determined there is a reverse DNS problem, you need to add an entry for your client hostname in the machine’s hosts file where the Perforce Server resides.

5. Wildcards on Windows

In some cases, p4 commands using unquoted file patterns with a combination of depot syntax or client syntax and wildcards can cause delays.

p4 files //depot/*

p4 files //client/*

The delay occurs because the wild card expansion routines for Windows mistakes the depot name or client name for a network system name. You can prevent the delay by putting double quotes around the file pattern:

p4 files “//depot/*”

Network Topology

Most networks use 100 Mbit technology. The maximum theoretical transfer rate of this type of network is 10 Mbytes per second. In most cases slower transfer rates are experienced. If a network is saturated, transfer rates can drop off significantly, maybe as low as 4 Mbytes per second. In cases of network saturation use of network routers or switches can help.

6. Client on a Network Filesystem

It is possible that the p4 executable itself is on a networked file system that is performing very poorly. To check for slow access to the executable, try running:

p4 -v

The p4 -v command simply prints out the version information and does not attempt to access the network. If you get a slow response, network access to the p4 executable itself may be the problem. Try copying or downloading the p4 executable onto a local filesystem.

7. Server on a Network Filesystem

Using a network filesystem for the Perforce Server must be thought through very carefully. Just as in the scenario above with the Perforce client stored on a network filesystem, data to and from the server application must go through the network twice. The Perforce Server also uses file locking on vital data files. Not all network filesystems have efficient locking implementations and some are buggy.

If the network is saturated and the transfer rate is down to about 4 Mbytes per second, the Perforce Server cannot satisfy client requests. Each client request uses part of the network, and the Perforce Server must share that resource in order to access the network filesystem. A cheap hard drive these days will provide a 20 Mbyte per second transfer rate. A good SCSI hard drive can transfer as much as 160 Mbytes per second. If possible, try to avoid a network filesystem related server bottle neck.

Try and play with “p4 ping” by progressively increasing the message count, receive and transmit lengths (one at a time). If you hit the same wall repeatedly, then this could very well be indicative of some throttling taking place somewhere

8. p4 -z tag info

9. Reference.

http://kb.perforce.com/article/40/isolating-network-problems
http://kb.perforce.com/article/1191/slow-sync-on-remote-windows-clients
http://kb.perforce.com/article/1462/slow-sync-performance-on-windows

The post Perforce Slow Sync issues | Perforce Slow Sync Troubleshooting Guide appeared first on scmGalaxy.