[WIP] Building a CLI for migrating between storage backends#1609
[WIP] Building a CLI for migrating between storage backends#1609arschles wants to merge 3 commits intogomods:mainfrom
Conversation
marwan-at-work
left a comment
marwan-at-work
left a comment
There was a problem hiding this comment.
Nice work :) I think we should make this a bit more robust per the comment below.
Also, how do we feel about making a cmd/athens that has all the tools we need:
athens serverwould run the server (and possibly justathens)athens migratewould run the migration processathens uploadfor if we ever implement direct uploads to athens
| if err != nil { | ||
| return err | ||
| } | ||
| for _, pathParam := range pathParams { |
There was a problem hiding this comment.
moving storages is likely a very timely process, we should do a couple of things:
- Parallelize reads/writes
- Check if the destination storage has the module@version first, instead of always overriding it.
- Further enhancement: make the migration process resumable, though with 2 it should be good enough i think.
There was a problem hiding this comment.
@marwan-at-work for (1), I agree that it should be parallelized . I was waiting to see if the general structure of this looks ok.
If you're ok with my suggestion for a follow-up, I can continue on with parallelizing and doing reads before writes in here. Let me know ✌️
|
Thanks for this PR! Great job! In my opinion having everything in single binary is good idea and I agree for parallelization as migrating my test repository took quite a long time. |
|
@miecio45 I'm glad you like this 😄 @marwan-at-work regarding the I'll just add it to |
3 participants
What is the problem I am trying to address?
It's hard to move modules from one storage backend to another. You have to understand the internals of how Athens lays out zip/mod/info files in each backend and then you have to write a script or manually copy.
How is the fix applied?
I've started a CLI that can migrate between any 2 storages that Athens supports. It uses built-in catalog, get and set methods, so any new storages automatically become supported by the CLI next time it's built.
What GitHub issue(s) does this PR fix or close?
Fixes #1604