OpenMPT Subversion repository migration away from SourceForge

[manx]

After the last (and ongoing) major service outage at sourceforge.net, we decided to immediately move our subversion source code repository away from their site and onto our own openmpt.org server (where most other parts of the OpenMPT infrastructure had already been hosted for a long time).

Further information will follow in this thread.

manx (and Saga Musix)

[manx]

Migration of the OpenMPT source repository has been finished.
The new repository root URL is https://source.openmpt.org/svn/openmpt/.
The new URL for viewing the repository from the web is https://source.openmpt.org/browse/openmpt/.
The developement trunk is still located in the trunk/OpenMPT subdirectory.
We did not change the repository UUID.

Repository corruption

SourceForge suffered a major outage of their storage infrastructure. It took them over 1 week to restore the subversion repositories at all. However, what they claimed to be fully restored repositories in fact were not. A couple of revisions right before the SourceForge crash happened had not been restored (in our case, a single one, r5415). Furthermore, even some revisions dating back some weeks had been restored in corrupted state which caused a couple of svn commands to fail on the restored repository (in our case, at least r5389 was affected). This corruption rendered it impossible to do any further full repository backup for us (the backup failed right at that revision).
At the point we had noticed the corruption, we already had another revision committed on top of the corrupted state. The SourceForge subversion stored this as r5415, as the real r5415 that had been committed several days ago before the crash, was missing from the restored repository.

Recovering the repository from the corruption

Since a couple of months, we already had a live subversion mirror running that got updated almost in realtime on every commit. This mirror stayed at the older r5415 even after SourceForge had accepted the inconsistent duplicate.
As taking any full backup of the SourceForge subversion repository turned out to be impossible due to the other corruptions, we decided to take our working and intact backup mirror as the new primary authorative state of the OpenMPT repository.
During the past 2 days, we set up a fully working subversion server with a WebSVN repository browser as well as working commit mailings (using the old SourceForge-hosted mailing list, therefore any subscribers will continue to receive commit messages just fine without any further action on their side) on the openmpt.org server.
This repository will be the primary OpenMPT subversion repository from now on.
The duplicate r5415 on the SourceForge subversion server got re-applyed to the known good state at our new repository as r5416.

Recovering working copies from the corruption

In order to prevent any further damage to any working copies, we removed the complete folder structure from the old SourceForge repository. This causes any subversion client to fail the default svn update command early, without any modifications to the working copy at all.
The corruption means, that any working copy of the old SourceForge subversion server updated to any state (or part of a state) later than 2015-07-16 14:14:00 UTC (r5415 or later) is in ambiguous state.
Subversion is not supposed to be able to deal with such inconsistencies as it only ever uses a repository UUID and revision number tuple to identify a particular state. If these 2 match, it only ever transfers differences from one side to the other which will then fail if the states were not in fact the same.
There are a few possible solutions for your working copies:

• Do a clean checkout from the new repository: If you have any local changes made to your working copy, create a patch from them and re-apply that patch to the new checkout (please refer to the Subversion manual on how to exactly do that). If you have no local changes, just do a new checkout.
• If your working copy is still at r5414 or older, use svn to relocate your working copy to the new location.
  
  
  
  • TortoiseSVN: Select "Relocate ..." from the topmost folder of any working copy and enter https://source.openmpt.org/svn/openmpt as the new location URL.
  • svn command-line client: Navigate to the topmost folder of any working copy and execute svn switch --relocate https://svn.code.sf.net/p/modplug/code/ https://source.openmpt.org/svn/openmpt/ . (note: the  SourceForge URL may need to be slightly different for you, like using http: instead of https: or using svn: or using one of the older paths or hostnames).
• WARNING: This third solution is NOT recommended. If you went unlucky and your working copy is at r5415 or beyond (or ever had been and got updated backwards), you may update backwards to r5414 and then continue with relocating the working copy like explained above, or, staying at r5415 and replacing the .svn folder of the working copy with a .svn folder of a clean checkout of the same path from the new repository at r5415 or 5416. There may turn up changes actually included in those revisions in the diff your working copy contains after having done so. Please do compare with the problematic revisions r5415 and r5416. In this case, it is your responsibility to locally revert or re-apply them as appropriate.
  WARNING: This third solution is NOT recommended. It is by far more fragile than the other two mentioned above. Even just updating backwards using the old repository to a good revision may in fact leave corrupted state in the working copy, as we have no clear picture of the internal state of the subversion repository at SourceForge. However, depending on the amount and kind of local modifications your working copy may have, this can turn out to be less work. In any case, do any verification you can imagine on the resulting working copy and do proper backup before proceeding.

A we did not change the UUID of the repository (which eases transition for working copies), subversion clients may end up caching and confusing inconsistent information about the repository. As we have no complete knowledge of all subversion clients around, we can not give any particular advice here, short of consulting the manual and determining a method to invalidate any such caches that might exist.
A well known client that does this sort of caching is TortoiseSVN for Windows. The affected information is just log messages which will not cause any secondary corruption to actual source code or other data. However, log messages of r5415 to at least r5417 may be affected. The problem is solved by deleting the log cache after having all working copies migrated to the new repository: Go to TortoiseSVN/Settings/LogCaching/CachedRepositories, and delete the cache for both, the old and the new OpenMPT repositories.

[Saga Musix]

It is also worth noting that while the commit notification mailing list will remain active for the forseeable future, it is still powered by Sourceforge and might thus be unstable every now and then (we've had phases where commit mails were delivered with a large delay or not at all). So if you want, you may subscribe to the commit RSS feed instead and get commit notifications into your favourite RSS reader at any interval you like.

[Harbinger]

Is it going to cost the devs or admin more money to keep the sources code and other uploaded files? Will the fundraising drives (so to speak) be at a higher goal?

We're grateful for you keeping the software alive and now that you can't (or don't want to) use Sourceforge, we should help if it's going to incur extra cost to keep the builds and docs safe...

[Saga Musix]

No, the server costs are the same no matter how many different services we run on it, and the current server is more than capable to run this service. All the off-site backups are practically free for the project as well, as they are mirrored on existing infrastructure we already possess. If anyone wants to support this, they can easily create yet another mirror for the repository by means of the svnsync tool.
If anything, this costs us less nerves.