How to mirror CPAN
There are several ways to mirror CPAN depending upon what you want to achieve.
How do I create a private or offline mirror?
Requirements for a full / public mirror
- Good internet connectivity
- Around 1GB of storage space for just the current modules.
- Around 18GB of storage space for the full mirror.
It's highly recommended that you also subscribe to the announcements-only cpan-mirrors mailing list by emailing cpan-mirrors-subscribe at perl.org.
Only use FTP if these other methods are absolutely impossible. Never mirror with HTTP - you will end up with a million duplicate files in tens of gigabytes.
Which CPAN Mirror should I use?
Please note that after 15 years of service to the Perl community, FUnet is no longer the master CPAN mirror. If you update from funet.fi you should update your mirroring source.
You can also sync from
"tier 1 mirrors"), though you currently might get better
performance from a "local" mirror.
Please limit to once or twice a day. For more frequent updates please see Instant mirroring.
On Unix systems
/usr/bin/rsync -av --delete cpan-rsync.perl.org::CPAN /project/CPAN/
Using 'crontab' you can make rsync run once a day, for example
40 4 * * * sleep $(expr $RANDOM \% 7200); /usr/bin/rsync -a --delete cpan-rsync.perl.org::CPAN /project/CPAN/
The "sleep $(...);" statement makes the command delay up to 2 hours before running rsync; the advantage of this is that you (and everybody else) won't access the mirror at the same time.
Unless you are mirroring to an SSD you might get timeouts using --delete-after when many symlinks are being purged. Using --delete will work properly.
If you have a problem with permissions (files are created with mode
-rw-------), set umask in your cronjob :
40 4 * * * umask 022 ; sleep ... ; /usr/bin/rsync ...
The umask 022 allows rsync to set proper permissions for files and directories.
On Windows systems
C:\Program Files\Rsync\rsync -av --delete cpan-rsync.perl.org::CPAN /project/CPAN/
Using the 'AT' tool, you can schedule rsync to run daily, for example:
AT 20:00 /every:M,T,W,Th,F,S,Su "C:\Program Files\Rsync\rsync -a --delete cpan-rsync.perl.org::CPAN /project/CPAN/"
How do I create a public mirror?
- Consider Instant mirroring, required if you wish to be a tier 1 mirror, or..
- rsync once a day
- Provide (in order of preference) rsync, HTTP and/or FTP public access
- To be added to http://www.cpan.org/SITES.html and mirrors.json please complete the template confirming the public accessible URLs to your mirror: rsync, ftp, http and email it to email@example.com.
"Instant mirroring" keeps your CPAN mirror up-to-date by continuously tracking the CPAN master; picking up the changes from the master, a short time (minutes) after they occur.
Instant mirroring is used for all Tier 1 mirrors (so cpan-rsync.perl.org stays in sync across mirrors).
To use "instant mirroring", you need a special client: "rrr-client" or "iim".
"rrr-client" is part of the File::Rsync::Mirror::Recent
(also known as
rrr) package ; it is the official client, used
on the CPAN master to get updates from PAUSE : the true heart and soul of "all things
perl", see the setup
guide for more details.
"iim" is an alternative for "rrr-client" ; basically it does the same thing, but it is more efficient (on start-up) and has some features that may be helpful to CPAN mirror operators.