First, choose which array to put the project:¶

• We have two arrays: /data/ftp/.1 and /data/ftp/.2
• Make sure there is enough disk space for your project
• If you expect the project to use a lot of bandwidth, put it on the less used array (look at iostat)

For local master (account on ftp-osl to upload files):¶

• add trigger permissions in cf.mirror:

  /data/trigger/set/$project mode=2775 owner=$project group=trigger
  

• add files/data/mirror/bin/update-master/$project:

#!/bin/bash

echo "We are master"

For remote mirrors (update via rsync):¶

• add project cronjob to files/etc/cron.d/mirror/update.master
• add rsync line to files/data/mirror/bin/update-master/$project:

#!/bin/bash

rsync -avH --delete projecthost::module/ /data/ftp/.$array/$project/

For all mirrors:¶

• add symlink in cf.mirror:

  /data/ftp/pub/$project -> /data/ftp/.1/$project
  

• if wanted, add vhost config: files/etc/apache2/vhosts.d/mirror/$project.osuosl.org.conf

• commit

For local master:¶

useradd -m $project
mkdir /data/ftp/.$array/$project
chown $project:$project /data/ftp/.$array/$project
ln -sf /data/ftp/pub/$project /home/$project/data

• put ssh keys in /home/$project/.ssh/authorized_keys
• add trigger script /home/$project/trigger-$project

#!/bin/bash

/data/mirror/bin/trigger-set $project

• add /home/$project/README

Your data is in /data/ftp/.$array/$project/
The symlink "data" links there for your convenience.

To signal the server to push your data to the other servers, run the
trigger-$project script.

For remote mirror:¶

• after committing/pushing changes in git, run cfexecd -F -q
• run the update script /data/mirror/bin/run-update <project> --email - this will sync the project, and trigger the slaves to sync with the master

ftp-osl failure¶

Some things to consider:

• This machine acts as master mirror for some projects
• All slaves sync from this host
• While we rebuild ftp-osl, all new syncs will likely be stopped to ensure data integrity

With this in mind, we have to consider a few list of actions. The TL;DR version of what we should probably do is:

• Stop all cronjobs on all ftp hosts to mitigate any potential data loss (simply stopping cron is probably ok)
• Take ftp-osl out of DNS rotation
• Shutdown http/xinetd. We might want to restrict ssh access too
• Notify the outage via hosting list and other methods (may even need a news post on our website)
• Ensure the integrity of the data on the other two slave machines for the same partition (basically make sure each repo is the same size or close to it)
• Designate which slave to use to re-copy data back to ftp-osl, likely chi but nyc might be faster
• Rebuild the disk array and mkfs.xfs it, etc. It’s probably easier to just recreate the array at that point
• Start an rsync from slave node (this will likely take DAYS depending on the load and such).
• Ensure everything looks sane
• Re-enable cronjobs and do some tests
• Add back into rotation
• Send out another announcement

Sync Data from Upstream¶

• /data/mirror/bin/run-update
  • Wrapper for scripts to sync and lock.
  • If you need to force an update to a specific project, please do the following on ftp-osl:

$ /data/mirror/bin/run-update <project name> --email

• /data/mirror/bin/update-master/*
  • Run by run-update to actually sync the data for each tree.
  • Should not be run manually (use run-update instead).

Sync Between Master and Slave¶

This system of triggering runs on a cron job once per minute, so in theory changes to ftp-osl are propagated very quickly out to the other ftp servers.

• /data/mirror/bin/run-update - Same use as for pulling from upstream, but pulls from master.
• /data/mirror/bin/trigger-set
  • Puts a trigger file in /data/trigger/set/tree_name/ on master.
  • The trigger indicates that the master is up-to-date and the slaves can begin pulling data.
• /data/mirror/bin/trigger-run
  • Used by the slave servers to check for new triggers set on the master.
  • If it tries to remove a trigger that has already been removed, will send an error e-mail. It is safe to ignore this e-mail unless a deluge of errors occurs. Network latency is usually the cause.
• /data/mirror/bin/trigger-scan
  • Copies set triggers from master to slaves.
• /usr/local/sbin/mkchroot
  • Creates and sets up chroot used by trigger system.

Control Downloads¶

• /usr/local/sbin/i2-check
  • Check if a particular IP address is on the I2 list.
• /usr/local/sbin/i2-check.pl
  • A much faster version of the bash script.
• /usr/local/sbin/i2-fill
  • Update the mysql database of IP addresses from the I2 list.
• /usr/local/sbin/ip*
  • Manage temporary iptables blacklist.
  • Accepts any number of IPs/netblocks as arguments.
  • Only affects local box.
  • Won’t affect normal firewall; uses mangle table.

Gather Statistics¶

• /data/mirror/bin/check-size
  • Goes through trees to find size of each.
  • Info stored in a log file, processed by fir.
• /data/mirror/bin/gen-header
  • Run manually to update fancy header used on http indexes.
  • Since the output files are in cfengine, need to update those files at the same time.
• /data/mirror/bin/gen-footer
  • Run every minute to update bandwidth bar.
• /usr/local/sbin/check-apt-mirror
  • Runs md5sum on every package in specified ubuntu or debian tree.
  • Takes about half a day to run.
• /usr/local/sbin/logs-* Deprecated??
  • Manage apache and ftp logs instead of using logrotate
  • Run by cron jobs.

ftpsync¶

• /data/mirror/ftpsync/bin/ftpsync
  • syncs files with a 2-stage sync to avoid breaking
• /data/mirror/ftpsync/bin/runmirrors
  • triggers slaves
• /data/mirror/etc/ftpsync-$project.conf
  • config file for ftpsync to sync $project
• /data/mirror/etc/runmirrors-$project.conf
  • config file for runmirrors to push $project
• /data/mirror/etc/$project.mirror
  • config file listing slaves to push
• see http://www.debian.org/mirror/push_mirroring for more info