Syncing missing/modified files to a specific version in Perforce

I think this is the 5th time I’ve had to figure this out in the past few years, so this time I’m going to preserve it here for posterity :)

The problem is usually one or more of:

  • You have files missing from your workspace that need replaced.
  • You have locally modified (i.e. not opened for edit) files that need replaced.
  • These files are large in either size or number, and on a slow or remote server, so a complete force-sync is not an appealing option.

Normally this could easily be solved with something like p4 diff -sd/-se //depot/path/.. | p4 -x – sync -f, which finds all the missing/outdated files and then force-syncs them.

The problem with this approach comes when you need the files from a specific revision or label. While you can pass in a label/revision to ‘p4 diff’, the sync part will always retrieve the latest version of a file. To fix this we need to save the list of files we need to a temporary location, then postfix each line with the version we want, then feed that new list back to p4 sync.

But there’s one final catch. If your filenames are local paths instead of Perforce paths, you cannot specify a label, only revisions. E.g. c:\temp\foo.dat@SOME_LABEL will not work, but c:\temp\foo.dat#999 will. Fortunately we can use the pseudo revision ‘have’ for our purposes.

@echo off

REM Could optionally pass these as %1 and %2 from the cmd line
set LABEL=@SOME_LABEL
set BASE_DIR=//depot/path/…

del p4sync.txt

echo Setting ‘have’ to %LABEL% for %BASE_DIR%
REM do a nop sync that sets our ‘have’ version to the label we want
p4 sync -k %BASE_DIR%%LABEL%

echo Finding modified files..
p4 diff -se %BASE_DIR%%LABEL% > p4diff.txt

echo Finding missing files..
p4 diff -sd %BASE_DIR%%LABEL% >> p4diff.txt

echo Building sync commands
REM Append #have to each filename and put them all in a list of files to sync
for /f %%i in (p4diff.txt) Do echo %%i#have>> p4sync.txt

REM Now sync everything
if exist p4sync.txt (
echo Syncing..
p4 -x p4sync.txt sync -f
)else (
echo It seems no files need updated!
)

Voila!

Tagged : / / / / / /

Perforce Slow Sync issues | Perforce Slow Sync Troubleshooting Guide

perforce-slow-sync-issues

Perforce Slow Sync issues

Network latency
Sync Performance issues are not obvious with locally connected hosts, as the network latency is low. However, as latency increases, performance worsens.

To correct the issue, follow these steps on the client machines exhibiting the issue:
http://kb.perforce.com/article/1191/slow-sync-on-remote-windows-clients

Network problems:
How do I determine if slow Perforce response time is caused by network problems?

A network issue can be suspected if Perforce commands run quickly on the local machine but run slowly across a network. You can also compare the lapse time against the usage time, for example, if the information is available in the Perforce logs.

Any of the following can cause slow responses:

  1. Misconfigured domain name server (DNS)
  2. Misconfigured Windows name server (WINS) or Windows domain
  3. Slow network response

1. p4 info

A good initial test is to run the p4 info command. If this command does not respond nearly immediately then there is a network related problem.
The p4 info command uses the P4PORT setting to contact the Perforce Server. If the P4PORT setting uses the hostname of the server machine, a DNS lookup is required to fetch the server IP address. If the DNS server is failing or the network is slow, the lookup process takes time. You can use the IP address directly to avoid the DNS lookup.

2. Hostname vs. IP Address
On a client machine, try using the Perforce Server’s IP address in the P4PORT setting. This avoids the DNS lookup used to convert the hostname to an IP address. Here is a P4PORT example which uses the server hostname:

P4PORT=hostname:1666
Using the IP address directly, the P4PORT setting then looks like this:

P4PORT=1.2.3.4:1666

If you do not have the Perforce Server machine’s IP address handy, you can use the ping command to find it. Here is an example of how you would run the ping command:

3. ping hostnam

The output of the ping command lists the IP address for the hostname.

If the p4 info command responds immediately when you use the IP address in the P4PORT setting, you have a misconfigured DNS.

“p4 info” vs. P4Win

The p4 info command is processed by the Perforce Server. As the Perforce Server compiles the output information for the command, it does a reverse DNS lookup on the client’s IP address. A forward DNS lookup might be fast, while a reverse DNS lookup is slow. One way to determine if a reverse DNS lookup is slow is by using P4Win, the Perforce Visual Client. The “Show Connection Info” operation in P4Win does not perform a reverse DNS lookup.

You can compare the response of P4Win’s “Show Connection Info” with the response from the command line p4 info. If a “Show Connection Info” operation is fast while a p4 info is slow, there is a reverse DNS lookup problem on the Perforce Server machine.

4. The “hosts” file

To work around DNS problems, add hostname IP address entries to the hosts file. This task is typically performed by a Systems Administrator; be sure to follow your company’s standard procedures.

Rather than using an IP address for the P4PORT setting, you can add a hostname IP address entry in your hosts file. The host file can be tricky to find. Here are a few places it can be:

Windows Server 2000 (and later):

C:\Winnt\System32\Drivers\etc\hosts

Unix:

/etc/hosts

The following is an example entry for the hosts file:

1.2.3.4 hostname

If you have determined there is a reverse DNS problem, you need to add an entry for your client hostname in the machine’s hosts file where the Perforce Server resides.

5. Wildcards on Windows

In some cases, p4 commands using unquoted file patterns with a combination of depot syntax or client syntax and wildcards can cause delays.

p4 files //depot/*

p4 files //client/*

The delay occurs because the wild card expansion routines for Windows mistakes the depot name or client name for a network system name. You can prevent the delay by putting double quotes around the file pattern:

p4 files “//depot/*”

Network Topology

Most networks use 100 Mbit technology. The maximum theoretical transfer rate of this type of network is 10 Mbytes per second. In most cases slower transfer rates are experienced. If a network is saturated, transfer rates can drop off significantly, maybe as low as 4 Mbytes per second. In cases of network saturation use of network routers or switches can help.

6. Client on a Network Filesystem

It is possible that the p4 executable itself is on a networked file system that is performing very poorly. To check for slow access to the executable, try running:

p4 -v

The p4 -v command simply prints out the version information and does not attempt to access the network. If you get a slow response, network access to the p4 executable itself may be the problem. Try copying or downloading the p4 executable onto a local filesystem.

7. Server on a Network Filesystem

Using a network filesystem for the Perforce Server must be thought through very carefully. Just as in the scenario above with the Perforce client stored on a network filesystem, data to and from the server application must go through the network twice. The Perforce Server also uses file locking on vital data files. Not all network filesystems have efficient locking implementations and some are buggy.

If the network is saturated and the transfer rate is down to about 4 Mbytes per second, the Perforce Server cannot satisfy client requests. Each client request uses part of the network, and the Perforce Server must share that resource in order to access the network filesystem. A cheap hard drive these days will provide a 20 Mbyte per second transfer rate. A good SCSI hard drive can transfer as much as 160 Mbytes per second. If possible, try to avoid a network filesystem related server bottle neck.

Try and play with “p4 ping” by progressively increasing the message count, receive and transmit lengths (one at a time). If you hit the same wall repeatedly, then this could very well be indicative of some throttling taking place somewhere

8. p4 -z tag info

9. Reference.

http://kb.perforce.com/article/40/isolating-network-problems
http://kb.perforce.com/article/1191/slow-sync-on-remote-windows-clients
http://kb.perforce.com/article/1462/slow-sync-performance-on-windows

Tagged : / / / / / / / / / / / / / /