Clone all repositories from a github organization

Tomorrow I will attend the first Hackergarten at Paris.
This is an event where active open-source committers can met potential contributors and help them to submit a (first) contribution.
I’ll be here to help those who would like to contribute on JenkinsCI and Apache Maven.
To prepare this event I wanted to prepare a checkout of all sources and a local repository with all required dependencies to avoid to see contributors losing their time by downloading the earth with Maven or checking out Subversion (and to not be blocked if we had some network issues).
For Maven which is hosted on Subversion it is long but easy because we have a special SVN directory using svn:externals to checkout all trunks in one step : https://svn.apache.org/repos/asf/maven/trunks/
But for Jenkins with several thousands of plugins and thus git repositories it is less easy to clone all of them.
I saw that there was everything needed in github APIs but it was late and I didn’t want to start to write a new script (One teacher told me a long long time ago that it was a quality to be idle – do it with less and less you’ll have to maintain).
I asked on jenkinsci IRC channel and thanks to Jørgen P. Tjernø I had my reply quickly.
With Ruby 1.8 (default version on Macos) :

sudo gem install json
curl -s https://github.com/api/v2/json/repos/show/jenkinsci | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read)["repositories"].each {|repo| %x[git clone #{repo["url"]} }'

With Ruby 1.9+, the json library is by default thus you just use :

curl -s https://github.com/api/v2/json/repos/show/jenkinsci | ruby -rjson -e 'JSON.load(STDIN.read)["repositories"].each {|repo| %x[git clone #{repo["url"]} }'

With that I’m able to clone all repositories from jenkinsci github organization.
Tomorrow I’ll just have to launch a command like this to download all dependencies :

find . -name pom.xml -maxdepth 2 -exec mvn -f {} -Dmaven.repo.local=./maven-local-repo dependency:go-offline \;

Update : As there are more than 30 repositories in JenkinsCI Github organization, I had to call the clone command several times by adding ?page=X (X=2,..) after the url https://github.com/api/v2/json/repos/show/jenkinsci to grab all results pages.

VN:F [1.9.13_1145]
Rating: 0.0/10 (0 votes cast)
VN:F [1.9.13_1145]
Rating: -1 (from 1 vote)
This entry was posted in Technologie and tagged , , , , . Bookmark the permalink.

One Response to Clone all repositories from a github organization

  1. Fun for a REST Json API to have pagination enabled by default as the webUI does :-/

    VA:F [1.9.13_1145]
    Rating: 5.0/5 (1 vote cast)
    VA:F [1.9.13_1145]
    Rating: 0 (from 0 votes)

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>