Mirror howto: Difference between revisions
Jump to navigation
Jump to search
No edit summary |
m (Update full mirror size in checklist) |
||
| (13 intermediate revisions by 5 users not shown) | |||
| Line 1: | Line 1: | ||
[[Category:HowTo]] | [[Category:HowTo]] | ||
= Qt Project download mirror how-to = | =Qt Project download mirror how-to= | ||
This page explains how to set up mirror server for Qt Project downloads. When your mirror is up and running we are able to redirect clients in your area to your server via our download redirector - http://download.qt.io/. There are only a few steps involved to set up a mirror: | This page explains how to set up mirror server for Qt Project downloads. When your mirror is up and running we are able to redirect clients in your area to your server via our download redirector - http://download.qt.io/. There are only a few steps involved to set up a mirror: | ||
== Resources checklist == | ==Resources checklist== | ||
* '''Bandwidth''' - In general we would like our mirrors to have at least a 50 Mbit/s Internet link. The link itself should be fairly stable, so please don't set up a mirror over a home-use cable or DSL link. However, if you are in a country with limited bandwidth you can disregard this checkpoint. | *'''Bandwidth''' - In general we would like our mirrors to have at least a 50 Mbit/s Internet link. The link itself should be fairly stable, so please don't set up a mirror over a home-use cable or DSL link. However, if you are in a country with limited bandwidth you can disregard this checkpoint. | ||
* '''Disk space''' - Currently, a full mirror of qt.io is about | *'''Disk space''' - Currently, a full mirror of qt.io is about 2.5 TB. It is possible to only mirror parts, but we would recommend mirrors to at least have the "Online" and "Official Releases" tree. | ||
* '''Mirror dedication''' - We would like our pool of mirrors to be stable, so if you only have temporary access to your server and/or bandwidth, we would not recommend that you sign up to be a global mirror. In addition, the mirror admin should be willing to respond to changes to the Qt Project server in a timely fashion, and that the admin will be around for the foreseeable future. | *'''Mirror dedication''' - We would like our pool of mirrors to be stable, so if you only have temporary access to your server and/or bandwidth, we would not recommend that you sign up to be a global mirror. In addition, the mirror admin should be willing to respond to changes to the Qt Project server in a timely fashion, and that the admin will be around for the foreseeable future. | ||
* '''HTTP and rsync access''' - Your mirror needs to be accessible by HTTP and rsync. We use rsync to check the status of the files you are serving, so we don't redirect clients to your mirror for files you don't have or not synced yet. You can enable FTP access for your mirror as well, but it's not mandatory from our point of view. However, HTTP and rsync are. | *'''HTTP and rsync access''' - Your mirror needs to be accessible by HTTP and rsync. We use rsync to check the status of the files you are serving, so we don't redirect clients to your mirror for files you don't have or not synced yet. You can enable FTP access for your mirror as well, but it's not mandatory from our point of view. However, HTTP and rsync are. | ||
== Sync files == | ==Sync files== | ||
As the mirror admin you should also subscribe to the Qt Project [http://lists.qt-project.org/mailman/listinfo/mirrors mirrors] mailing list. This will be the point of contact for all Qt Project mirror admins | |||
To update your mirror you can use a tool called '''rsync'''. When you run it e.g. daily to keep your mirror updated, it will only transfer the files that have changed since the last update. | To update your mirror you can use a tool called '''rsync'''. When you run it e.g. daily to keep your mirror updated, it will only transfer the files that have changed since the last update. | ||
| Line 19: | Line 21: | ||
Rsync is included in most Linux distros, or it can be downloaded from http://rsync.samba.org/. | Rsync is included in most Linux distros, or it can be downloaded from http://rsync.samba.org/. | ||
We have a set of rsync "modules" that can be used to update your mirror. The recommended one is '''qt-all''' which will pull down the complete download tree. This is also the module that will consume most disk space, currently about | We have a set of rsync "modules" that can be used to update your mirror. The recommended one is '''qt-all''' which will pull down the complete download tree. This is also the module that will consume most disk space, currently about 2.5 TB. If you are short on disk space we recommend to at least use the two modules called '''qt-online''' and '''qt-official'''. The first one includes the ''pub/qtproject/online'' tree, has roughly the size of 553 GB, and contains Qt repositories (both stable and development releases) used by Qt online installer. The second one pulls down the ''pub/qtproject/official_releases'' tree, takes up roughly 90 GB of disk space, and contains stable releases of Qt related offline installers and source packages. The third one pulls down the pub/qtproject/development_releases tree, takes up roughly 394 GB of disk space, and contains various alpha, beta and release candidates of Qt. There are other modules available as well, please drop an email the Qt Project Sysadmin at mirrors@qt-project.org if you want a list of the modules. | ||
{| | {| | ||
| Line 25: | Line 27: | ||
|'''Rsync server/module''' | |'''Rsync server/module''' | ||
|- | |- | ||
|All (pub/qtproject/) - | |All (pub/qtproject/) - 2.5 TB | ||
|rsync master.qt.io::qt-all | |rsync master.qt.io::qt-all | ||
|- | |- | ||
|Official Releases (pub/qtproject/official_releases/) - | |Official Releases (pub/qtproject/online/) - 553 GB | ||
|rsync master.qt.io::qt-online | |||
|- | |||
|Official Releases (pub/qtproject/official_releases/) - 90 GB | |||
|rsync master.qt.io::qt-official | |rsync master.qt.io::qt-official | ||
|- | |- | ||
|Development Releases (pub/qtproject/development_releases/) - | |Development Releases (pub/qtproject/development_releases/) - 394 GB | ||
|rsync master.qt.io::qt-development | |rsync master.qt.io::qt-development | ||
|} | |} | ||
| Line 39: | Line 44: | ||
<code>rsync -rlpt --delete master.qt.io::qt-all /data/pub/qtproject</code> | <code>rsync -rlpt --delete master.qt.io::qt-all /data/pub/qtproject</code> | ||
Say you only want to mirror Official | Say you only want to mirror Online and Official Releases: | ||
<code> | <code> | ||
rsync -rlpt --delete master.qt.io::qt- | rsync -rlpt --delete master.qt.io::qt-development /data/pub/qtproject/online | ||
rsync - | rsync -rLpt --delete master.qt.io::qt-official /data/pub/qtproject/official_releases | ||
</code> | </code> | ||
| Line 49: | Line 54: | ||
<code> | <code> | ||
# run each four (4) hours | #run each four (4) hours | ||
0 */4''' * * $HOME/bin/mirror_qtproject.sh | 0 */4''' * * $HOME/bin/mirror_qtproject.sh''' | ||
</code> | </code> | ||
== Inform us == | ==Inform us== | ||
Now you should have a fully working Qt Project mirror up and running. The last thing to do is to inform the Qt Project Sysadmin by email at mirrors@qt-project.org to get listed on the mirrors page. Please include the following information in the email: | |||
*Your name and email address | |||
*Operator | |||
*The HTTP address of the mirror | |||
*The rsync address of the mirror | |||
*The FTP address of the mirror (optional) | |||
Thank you for becoming a Qt Project mirror! | Thank you for becoming a Qt Project mirror! | ||
Latest revision as of 06:57, 7 April 2022
Qt Project download mirror how-to
This page explains how to set up mirror server for Qt Project downloads. When your mirror is up and running we are able to redirect clients in your area to your server via our download redirector - http://download.qt.io/. There are only a few steps involved to set up a mirror:
Resources checklist
- Bandwidth - In general we would like our mirrors to have at least a 50 Mbit/s Internet link. The link itself should be fairly stable, so please don't set up a mirror over a home-use cable or DSL link. However, if you are in a country with limited bandwidth you can disregard this checkpoint.
- Disk space - Currently, a full mirror of qt.io is about 2.5 TB. It is possible to only mirror parts, but we would recommend mirrors to at least have the "Online" and "Official Releases" tree.
- Mirror dedication - We would like our pool of mirrors to be stable, so if you only have temporary access to your server and/or bandwidth, we would not recommend that you sign up to be a global mirror. In addition, the mirror admin should be willing to respond to changes to the Qt Project server in a timely fashion, and that the admin will be around for the foreseeable future.
- HTTP and rsync access - Your mirror needs to be accessible by HTTP and rsync. We use rsync to check the status of the files you are serving, so we don't redirect clients to your mirror for files you don't have or not synced yet. You can enable FTP access for your mirror as well, but it's not mandatory from our point of view. However, HTTP and rsync are.
Sync files
As the mirror admin you should also subscribe to the Qt Project mirrors mailing list. This will be the point of contact for all Qt Project mirror admins
To update your mirror you can use a tool called rsync. When you run it e.g. daily to keep your mirror updated, it will only transfer the files that have changed since the last update.
Rsync is included in most Linux distros, or it can be downloaded from http://rsync.samba.org/.
We have a set of rsync "modules" that can be used to update your mirror. The recommended one is qt-all which will pull down the complete download tree. This is also the module that will consume most disk space, currently about 2.5 TB. If you are short on disk space we recommend to at least use the two modules called qt-online and qt-official. The first one includes the pub/qtproject/online tree, has roughly the size of 553 GB, and contains Qt repositories (both stable and development releases) used by Qt online installer. The second one pulls down the pub/qtproject/official_releases tree, takes up roughly 90 GB of disk space, and contains stable releases of Qt related offline installers and source packages. The third one pulls down the pub/qtproject/development_releases tree, takes up roughly 394 GB of disk space, and contains various alpha, beta and release candidates of Qt. There are other modules available as well, please drop an email the Qt Project Sysadmin at mirrors@qt-project.org if you want a list of the modules.
| Tree | Rsync server/module |
| All (pub/qtproject/) - 2.5 TB | rsync master.qt.io::qt-all |
| Official Releases (pub/qtproject/online/) - 553 GB | rsync master.qt.io::qt-online |
| Official Releases (pub/qtproject/official_releases/) - 90 GB | rsync master.qt.io::qt-official |
| Development Releases (pub/qtproject/development_releases/) - 394 GB | rsync master.qt.io::qt-development |
Here is an example of how to use rsync. This assumes that your mirror files will be in /data/pub/qtproject and that you want a full Qt Project mirror.
rsync -rlpt --delete master.qt.io::qt-all /data/pub/qtproject
Say you only want to mirror Online and Official Releases:
rsync -rlpt --delete master.qt.io::qt-development /data/pub/qtproject/online
rsync -rLpt --delete master.qt.io::qt-official /data/pub/qtproject/official_releases
To keep the files updated and in sync with the master server you can set this up as a cron job that runs e.g. once a day. This example assumes that you put the rsync command in a script called mirror_qtproject.sh. Preferred sync interval is four (4) hours
#run each four (4) hours
0 */4''' * * $HOME/bin/mirror_qtproject.sh'''
Inform us
Now you should have a fully working Qt Project mirror up and running. The last thing to do is to inform the Qt Project Sysadmin by email at mirrors@qt-project.org to get listed on the mirrors page. Please include the following information in the email:
- Your name and email address
- Operator
- The HTTP address of the mirror
- The rsync address of the mirror
- The FTP address of the mirror (optional)
Thank you for becoming a Qt Project mirror!