How to clone all repos at once from GitHub?

Question

I have a company GitHub account and I want to back up all of the repositories within, accounting for anything new that might get created for purposes of automation. I was hoping something like this:

git clone git@github.com:company/*.git

or similar would work, but it doesn't seem to like the wildcard there.

Is there a way in Git to clone and then pull everything assuming one has the appropriate permissions?

Good question. And how about keeping them in sync, via pull? Do any of the answers work for pulls? — nealmcb, May 17 '15 at 17:37
We need a python solution, for those of us not so adept at node or ruby ;) Or github should read this and take pity on us and just provide a simple web interface for this.... — nealmcb, May 18 '15 at 03:33

score 131 · Answer 1 · edited Aug 08 '20 at 13:12

131

On Windows and all UNIX/LINUX systems, using Git Bash or any other Terminal, replace YOURUSERNAME by your username and use:

CNTX={users|orgs}; NAME={username|orgname}; PAGE=1
curl "https://api.github.com/$CNTX/$NAME/repos?page=$PAGE&per_page=100" |
  grep -e 'git_url*' |
  cut -d \" -f 4 |
  xargs -L1 git clone

Set CNTX=users and NAME=yourusername, to download all your repositories.
Set CNTX=orgs and NAME=yourorgname, to download all repositories of your organization.

The maximum page-size is 100, so you have to call this several times with the right page number to get all your repositories (set PAGE to the desired page number you want to download).

Here is a shell script that does the above: https://gist.github.com/erdincay/4f1d2e092c50e78ae1ffa39d13fa404e

edited Aug 08 '20 at 13:12

Mike Pennington

38,579
16
126
167

answered Sep 28 '15 at 23:36

Erdinc Ay

2,574
4
22
38

5

Pure bash solution, the most simplest. For your information, this bash code can be executed in almost any *nix enviroment, Linux, Cygwin, Mingw and of course the Gitbash wich is really a terminal emulation like others. – m3nda Oct 05 '15 at 11:32
1

This doesn't work with organizations, so it doesn't directly answer the question. [This answer](https://stackoverflow.com/a/32803025/992887) from Kenorb does handle orgs and works for up to 1000 repos as well - worked better for me. – RichVel Jan 26 '18 at 07:39
how do u use this for private repos of the org – timberlake Mar 09 '18 at 17:46
1

with authentification: curl "https://api.github.com/$CNTX/$NAME/repos?page=$PAGE&per_page=100&access_token=$ACCESS_TOKEN" | grep -e 'git_url*' | cut -d \" -f 4 | xargs -L1 git clone – Yannick Wurm Mar 19 '18 at 22:37
2

Please update answer (Feb-2019): according to GitHub API v3 your curl should go to /orgs/ORGNAME/repos. Also maybe include a link to API v3: https://developer.github.com/v3/ Also for private repos you would need to add curl -u "username", then curl will ask you password once. Otherwise working great! : ))) – Dmitry Shevkoplyas Feb 15 '19 at 21:00
working sample CNTX={orgs}; NAME={officedev}; PAGE=1 curl "https://api.github.com/$CNTX/$NAME/repos?page=$PAGE&per_page=100" | grep -e 'git_url*' | cut -d \" -f 4 | xargs -L1 git clone – Ved Prakash Mar 18 '19 at 06:20
1

UPDATE from dimitry hevkoplyas comment https://stackoverflow.com/questions/19576742/how-to-clone-all-repos-at-once-from-github#comment96219716_32833411. developer.github.com/v3 returns 301 status when try to curl. use this bash command curl -u "{username}" "https://api.github.com/orgs/{org}/repos?page=1&per_page=100" | grep -o 'git@[^"]*' | xargs -L1 git clone works 100% – Tommy May 04 '19 at 09:01
This only grabs the public repos, not the private and internal ones. – wogsland Feb 05 '20 at 12:18
Does this only download public repositories, or private ones too? – stevec Aug 16 '20 at 20:00

score 61 · Accepted Answer · answered Oct 24 '13 at 21:40

61

I don't think it's possible to do it that way. Your best bet is to find and loop through a list of an Organization's repositories using the API.

Try this:

Create an API token by going to Account Settings -> Applications
Make a call to: http://${GITHUB_BASE_URL}/api/v3/orgs/${ORG_NAME}/repos?access_token=${ACCESS_TOKEN}
The response will be a JSON array of objects. Each object will include information about one of the repositories under that Organization. I think in your case, you'll be looking specifically for the ssh_url property.
Then git clone each of those ssh_urls.

It's a little bit of extra work, but it's necessary for GitHub to have proper authentication.

answered Oct 24 '13 at 21:40

Thomas Kelley

9,707
1
34
41

I created the API token and I'm getting output from the call, but I do not see anything referencing what I know of our repositories or the 'ssh_url' string. I suspect I didn't do the call properly. `curl -i https://github.com/api/v3/orgs/company/repos?access_token=` – numb3rs1x Oct 24 '13 at 21:59
Is this a GitHub Enterprise account, or github.com? – Thomas Kelley Oct 24 '13 at 22:05
1

Ah, I misunderstood you. I thought it was en Enterprise account. Instead of `https://github.com/api/v3/`, try `https://api.github.com/`. – Thomas Kelley Oct 24 '13 at 22:08
1

And I'm not sure how your particular company is set up, but if it's a "user" instead of an "organization", then you'll want to use the `/users/${COMPANY}/repos` path instead of `/orgs/${COMPANY}/repos`. – Thomas Kelley Oct 24 '13 at 22:10
You might find this link helpful... It has all the current API paths: http://developer.github.com/v3/ – Thomas Kelley Oct 24 '13 at 22:11
I found what I think is the right call, but I'm not getting all the repos that I can see when I bring up the organization in the browser. This is what I found show some of them: `https://api.github.com/orgs/company/repos?access_token=` – numb3rs1x Oct 25 '13 at 05:04
see my comment below and use whenever you want http://stackoverflow.com/a/32932865/3022613 – muhasturk Oct 04 '15 at 11:20
my jutils command line client supports this: https://github.com/jjYBdx4IL/misc/tree/master/jutils – user1050755 Jun 27 '17 at 21:11
if you prefer using a cli [ghorg](https://github.com/gabrie30/ghorg) will accomplish this task for you – Jay Gabez Jun 16 '19 at 18:26
2

Per GitHub: Please use the Authorization HTTP header instead, as using the `access_token` query parameter is deprecated. If this token is being used by an app you don't have control over, be aware that it may stop working as a result of this deprecation. – BogeyMan Jun 24 '20 at 19:43

kenorb · Answer 3 · 2019-07-04T18:27:49.277

Organisation repositories

To clone all repos from your organisation, try the following shell one-liner:

GHORG=company; curl "https://api.github.com/orgs/$GHORG/repos?per_page=1000" | grep -o 'git@[^"]*' | xargs -L1 git clone

User repositories

Cloning all using Git repository URLs:

GHUSER=CHANGEME; curl "https://api.github.com/users/$GHUSER/repos?per_page=1000" | grep -o 'git@[^"]*' | xargs -L1 git clone

Cloning all using Clone URL:

GHUSER=CHANGEME; curl "https://api.github.com/users/$GHUSER/repos?per_page=1000" | grep -w clone_url | grep -o '[^"]\+://.\+.git' | xargs -L1 git clone

Here is the useful shell function which can be added to user's startup files (using curl + jq):

# Usage: gh-clone-user (user)
gh-clone-user() {
  curl -sL "https://api.github.com/users/$1/repos?per_page=1000" | jq -r '.[]|.clone_url' | xargs -L1 git clone
}

Private repositories

If you need to clone the private repos, you can add Authorization token either in your header like:

-H 'Authorization: token <token>'

or pass it in the param (?access_token=TOKEN), for example:

curl -s "https://api.github.com/users/$GHUSER/repos?access_token=$GITHUB_API_TOKEN&per_page=1000" | grep -w clone_url | grep -o '[^"]\+://.\+.git' | xargs -L1 git clone

Notes:

To fetch only private repositories, add type=private into your query string.
Another way is to use hub after configuring your API key.

See also:

^Hints:
^{- To increase speed, set number of parallel processes by specifying -P parameter for xargs (-P4 = 4 processes).}
^{- If you need to raise the GitHub limits, try authenticating by specifying your API key.}
^{- Add --recursive to recurse into the registered submodules, and update any nested submodules within.}

per_page=1000 maxes out at 100 – aehlke Dec 27 '19 at 18:57 — aehlke, Dec 27 '19 at 18:57

score 22 · Answer 4 · edited Mar 11 '17 at 13:33

22

This gist accomplishes the task in one line on the command line:

curl -s https://api.github.com/orgs/[your_org]/repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["ssh_url"]} ]}'

Replace [your_org] with your organization's name. And set your per_page if necessary.

UPDATE:

As ATutorMe mentioned, the maximum page size is 100, according to the GitHub docs.

If you have more than 100 repos, you'll have to add a page parameter to your url and you can run the command for each page.

curl -s "https://api.github.com/orgs/[your_org]/repos?page=2&per_page=100" | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["ssh_url"]} ]}'

Note: The default per_page parameter is 30.

edited Mar 11 '17 at 13:33

kenorb

118,428
63
588
624

answered Jan 06 '15 at 18:18

seancdavis

2,316
1
19
29

Any idea how this is done for private repos that you have access to? – MichaelGofron Aug 04 '16 at 05:59
second one doesn't work cause the ampersand makes it go to a background task – slashdottir Sep 20 '16 at 23:18
I added &access_token= to the url and it worked perfectly – rmartinus Oct 27 '17 at 06:19
2nd one: page=1 (!) – Yannick Wurm Mar 19 '18 at 22:42
As per comments to other answers the max value for per_page parameter is 100, you can put any bigger number but you will only get 100 repos – Mikhail Chibel Feb 24 '21 at 07:16

score 6 · Answer 5 · answered Jun 26 '14 at 17:37

Go to Account Settings -> Application and create an API key
Then insert the API key, github instance url, and organization name in the script below

#!/bin/bash

# Substitute variables here
ORG_NAME="<ORG NAME>"
ACCESS_TOKEN="<API KEY>"
GITHUB_INSTANCE="<GITHUB INSTANCE>

URL="https://${GITHUB_INSTANCE}/api/v3/orgs/${ORG_NAME}/repos?access_token=${ACCESS_TOKEN}"

curl ${URL} | ruby -rjson -e 'JSON.load(STDIN.read).each {|repo| %x[git clone #{repo["ssh_url"]} ]}'

Save that in a file, chmod u+x the file, then run it.

_{Thanks to Arnaud for the ruby code.}

score 6 · Answer 6 · answered Nov 03 '16 at 23:36

So, I will add my answer too. :) (I found it's simple)

Fetch list (I've used "magento" company):

curl -si https://api.github.com/users/magento/repos | grep ssh_url | cut -d '"' -f4

Use clone_url instead ssh_url to use HTTP access.

So, let's clone them all! :)

curl -si https://api.github.com/users/magento/repos | \
    grep ssh_url | cut -d '"' -f4 | xargs -i git clone {}

If you are going to fetch private repo's - just add GET parameter ?access_token=YOURTOKEN

score 4 · Answer 7 · edited Dec 31 '16 at 01:49

I found a comment in the gist @seancdavis provided to be very helpful, especially because like the original poster, I wanted to sync all the repos for quick access, however the vast majority of which were private.

curl -u [[USERNAME]] -s https://api.github.com/orgs/[[ORGANIZATION]]/repos?per_page=200 |
  ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["ssh_url"]} ]}'

Replace [[USERNAME]] with your github username and [[ORGANIZATION]] with your Github organization. The output (JSON repo metadata) will be passed to a simple ruby script:

# bring in the Ruby json library
require "json"

# read from STDIN, parse into ruby Hash and iterate over each repo
JSON.load(STDIN.read).each do |repo|
  # run a system command (re: "%x") of the style "git clone <ssh_url>"
  %x[git clone #{repo["ssh_url"]} ]
end

This solution worked perfectly for me. Actually all I needed was to clone all of my personal account repos to my new local machine. Very handy for setting up new workstation. Note: to do this I had to change `.../orgs/[[organization]]/repos...` to `.../users/[[username]]/repos...`. Now I can quickly import all my work to different local machines. **THANKS!** — B. Bulpett, Sep 26 '15 at 15:30

score 3 · Answer 8 · edited Aug 11 '15 at 16:20

3

This python one-liner will do what you need. It:

checks github for your available repos

for each, makes a system call to git clone

python -c "import json, urllib, os; [os.system('git clone ' + r['ssh_url']) for r in json.load(urllib.urlopen('https://api.github.com/orgs/<<ORG_NAME>>/repos?per_page=200'))]"

edited Aug 11 '15 at 16:20

Barett

5,226
6
46
53

answered Aug 11 '15 at 09:10

Evren Ozkan

63
5

score 3 · Answer 9 · answered Oct 04 '15 at 11:19

3

I made a script with Python3 and Github APIv3

https://github.com/muhasturk/gitim

Just run

./gitim

answered Oct 04 '15 at 11:19

muhasturk

1,965
15
15

score 3 · Answer 10 · edited Apr 23 '18 at 11:16

3

curl -s https://api.github.com/orgs/[GITHUBORG_NAME]/repos | grep clone_url | awk -F '":' '{ print $2 }' | sed 's/\"//g' | sed 's/,//' | while read line; do git clone "$line"; done

edited Apr 23 '18 at 11:16

halfelf

8,484
13
47
58

answered Apr 23 '18 at 11:07

julianzhang

31
1

2

Please add an explanation to your solution. That way other people with similar issues will be able to understand your solution more easily! – Nander Speerstra Apr 23 '18 at 11:27
1

A detail is are need pass page number like `?page=2`. – Bruno Wego Jun 12 '19 at 00:41

score 3 · Answer 11 · answered Oct 31 '20 at 15:59

Here is a Python solution:

curl -s https://api.github.com/users/org_name/repos?per_page=200 | python -c $'import json, sys, os\nfor repo in json.load(sys.stdin): os.system("git clone " + repo["clone_url"])'

Substitute org_name with the name of the organization or user whose repos you wish to download. In Windows you can run this in Git Bash. In case it cannot find python (not in your PATH etc.), the easiest solution I have found is to replace python with the path to the actual Python executable, for example: /c/ProgramData/Anaconda3/python for an Anaconda installation in Windows 10.

1

Best option! Thank u – Natan Lotério May 04 '21 at 01:59

Antonio Feitosa · Answer 12 · 2019-05-05T00:31:29.667

2

Simple solution:

NUM_REPOS=1000
DW_FOLDER="Github_${NUM_REPOS}_repos"
mkdir ${DW_FOLDER}
cd ${DW_FOLDER}
for REPO in $(curl https://api.github.com/users/${GITHUB_USER}/repos?per_page=${NUM_REPOS} | awk '/ssh_url/{print $2}' | sed 's/^"//g' | sed 's/",$//g') ; do git clone ${REPO} ; done

edited May 05 '19 at 00:31

answered Feb 16 '19 at 15:11

Antonio Feitosa

872
8
8

score 2 · Answer 13 · answered Jun 20 '19 at 12:04

I tried a few of the commands and tools above, but decided they were too much of a hassle, so I wrote another command-line tool to do this, called github-dl.

To use it (assuming you have nodejs installed)

npx github-dl -d /tmp/test wires

This would get a list of all the repo's from wires and write info into the test directory, using the authorisation details (user/pass) you provide on the CLI.

In detail, it

Asks for auth (supports 2FA)
Gets list of repos for user/org through Github API
Does pagination for this, so more than 100 repo's supported

It does not actually clone the repos, but instead write a .txt file that you can pass into xargs to do the cloning, for example:

cd /tmp/test
cat wires-repo-urls.txt | xargs -n2 git clone

# or to pull
cat /tmp/test/wires-repo-urls.txt | xargs -n2 git pull

Maybe this is useful for you; it's just a few lines of JS so should be easy to adjust to your needs

score 1 · Answer 14 · edited Dec 31 '16 at 01:47

1

So, in practice, if you want to clone all repos from the organization FOO which match BAR, you could use the one-liner below, which requires jq and common cli utilities

curl 'https://api.github.com/orgs/FOO/repos?access_token=SECRET' |
  jq '.[] |
  .ssh_url' |
  awk '/BAR/ {print "git clone " $0 " & "}' |
  sh

edited Dec 31 '16 at 01:47

Martin Tournoij

23,567
24
90
127

answered Apr 15 '14 at 16:13

konr

2,465
2
20
35

score 1 · Answer 15 · answered Nov 06 '15 at 14:27

There is also a very useful npm module to do this. It can not only clone, but pull as well (to update data you already have).

You just create config like this:

[{
   "username": "BoyCook",
   "dir": "/Users/boycook/code/boycook",
   "protocol": "ssh"
}]

and do gitall clone for example. Or gitall pull

score 1 · Answer 16 · answered Oct 06 '16 at 11:34

In case anyone looks for a Windows solution, here's a little function in PowerShell to do the trick (could be oneliner/alias if not the fact I need it to work both with and without proxy).

function Unj-GitCloneAllBy($User, $Proxy = $null) {
    (curl -Proxy $Proxy "https://api.github.com/users/$User/repos?page=1&per_page=100").Content 
      | ConvertFrom-Json 
      | %{ $_.clone_url } 
      # workaround git printing to stderr by @wekempf aka William Kempf
      # https://github.com/dahlbyk/posh-git/issues/109#issuecomment-21638678
      | %{ & git clone $_ 2>&1 } 
      | % { $_.ToString() }
}

score 1 · Answer 17 · answered Sep 01 '19 at 17:04

Another shell script with comments that clones all repositories (public and private) from a user:

#!/bin/bash

USERNAME=INSERT_USERNAME_HERE
PASSWORD=INSERT_PASSWORD_HERE

# Generate auth header
AUTH=$(echo -n $USERNAME:$PASSWORD | base64)

# Get repository URLs
curl -iH "Authorization: Basic "$AUTH https://api.github.com/user/repos | grep -w clone_url > repos.txt

# Clean URLs (remove " and ,) and print only the second column
cat repos.txt | tr -d \"\, | awk '{print $2}'  > repos_clean.txt

# Insert username:password after protocol:// to generate clone URLs
cat repos_clean.txt |  sed "s/:\/\/git/:\/\/$USERNAME\:$PASSWORD\@git/g" > repos_clone.txt

while read FILE; do
    git clone $FILE
done <repos_clone.txt

rm repos.txt & rm repos_clone.txt

Madacol · Answer 18 · 2021-01-11T21:04:38.217

Clone all your public and private repos that are not forks:

Create first a Personal token for authentication, make sure it has all the repo permissions

curl -u username:token https://api.github.com/user/repos\?page\=1\&per_page\=100 |
  jq -r 'map(select(.fork == false)) | .[] | .ssh_url' |
  xargs -L1 git clone

Clone your gists:

curl https://api.github.com/users/{username}/gists\?page\=1\&per_page\=100 |
   jq -r ".[] | .git_pull_url +\" '\" + (.files|keys|join(\"__\") + \"'\")" |
   xargs -L1 git clone

This jq command is complex because gists' repo's name are hashes, so that command concatenates all filenames to be the repo's name

You can filter the JSON arbitrarily using `jq`

install: sudo apt-get install jq

In the example above, I filtered out forks using this: curl ... | jq -r 'map(select(.fork == false))' ... -- useful for not cloning repos where you've made casual pull requests

jq supports some very advanced features. man jq is your friend

Github's API urls

Your repos (needs authentication): https://api.github.com/user/repos\?page\=1\&per_page\=100
Any user: https://api.github.com/users/{other_username}/repos\?page\=1\&per_page\=100
Orgs: https://api.github.com/orgs/orgname/repos\?page\=1\&per_page\=100

Github API Docs for repos

score 1 · Answer 19 · answered Feb 22 '20 at 02:52

Create a bash alias/func in your `~/.bashrc file`

I solved this for my team by creating an alias/bash func in my ~/.bashrc file

steps

open a terminal or linux shell and open your ~/.bashrc file:

sudo nano ~/.bashrc

add this function:

CloneAll() {
    # Make the url to the input github organization's repository page.
    ORG_URL="https://api.github.com/orgs/${1}/repos?per_page=200";

    # List of all repositories of that organization (seperated by newline-eol).
    ALL_REPOS=$(curl -s ${ORG_URL} | grep html_url | awk 'NR%2 == 0' \
                | cut -d ':' -f 2-3 | tr -d '",');

    # Clone all the repositories.
    for ORG_REPO in ${ALL_REPOS}; do
        git clone ${ORG_REPO}.git;
    done
}

save and close your ~/.bashrc flile and then close the terminal -- you need to do this or the new func wont initialize:

open new terminal and try it out:

CloneAll <your_github_org_name>

example: if your personal github repo URL is called https://github.com/awesome-async the command would be

CloneAll awesome-async

Important

the per_page=200 at the end of the first variable ORG_URL sets the number of repos that will be cloned, so pay special attention to that:

ORG_URL="https://api.github.com/orgs/${1}/repos?per_page=200";  <---- make sure this is what you want

Hope this helps! :)

Seems max. value for per_page is 100 ... for large orgs added page number as 2nd parameter and it works perfectly for my needs `...repos?page=${2}&per_page=100"; ` — sv3n, Jul 29 '20 at 00:57

score 0 · Answer 20 · edited Sep 20 '16 at 16:13

0

You can get a list of the repositories by using curl and then iterate over said list with a bash loop:

GIT_REPOS=`curl -s curl https://${GITHUB_BASE_URL}/api/v3/orgs/${ORG_NAME}/repos?access_token=${ACCESS_TOKEN} | grep ssh_url | awk -F': ' '{print $2}' | sed -e 's/",//g' | sed -e 's/"//g'`
for REPO in $GIT_REPOS; do
  git clone $REPO
done

edited Sep 20 '16 at 16:13

jotasi

4,477
2
22
45

answered Sep 20 '16 at 14:30

Sriram Mageswaran

1

A.N. · Answer 21 · 2017-08-20T12:09:45.910

You can use open-source tool to clone bunch of github repositories: https://github.com/artiomn/git_cloner

Example:

git_cloner --type github --owner octocat --login user --password user https://my_bitbucket

Use JSON API from api.github.com. You can see the code example in the github documentation: https://developer.github.com/v3/

Or there:

https://github.com/artiomn/git_cloner/blob/master/src/git_cloner/github.py

ubugnu · Answer 22 · 2018-03-21T15:58:01.000

To clone only private repos, given an access key, and given python 3 and requests module installed:

ORG=company; ACCESS_KEY=0000000000000000000000000000000000000000; for i in $(python -c "import requests; print(' '.join([x['ssh_url'] for x in list(filter(lambda x: x['private'] ,requests.get('https://api.github.com/orgs/$ORG/repos?per_page=1000&access_token=$ACCESS_KEY').json()))]))"); do git clone $i; done;

score 0 · Answer 23 · answered Jun 08 '18 at 16:00

A Python3 solution that includes exhaustive pagination via Link Header.

Pre-requisites:

Github API "Personal Access Token"
pip3 install links-from-link-header
hub

import json
import requests
from requests.auth import HTTPBasicAuth
import links_from_header

respget = lambda url: requests.get(url, auth=HTTPBasicAuth('githubusername', 'githubtoken'))

myorgname = 'abc'
nexturl = f"https://api.github.com/orgs/{myorgname}/repos?per_page=100"

while nexturl:
    print(nexturl)
    resp = respget(nexturl)

    linkheads = resp.headers.get('Link', None)
    if linkheads:
        linkheads_parsed = links_from_header.extract(linkheads)
        nexturl = linkheads_parsed.get('next', None)
    else:
        nexturl = None

    respcon = json.loads(resp.content)
    with open('repolist', 'a') as fh:
        fh.writelines([f'{respconi["full_name"]}\n' for respconi in respcon])

Then, you can use xargs or parallel and: cat repolist | parallel -I% hub clone %

score 0 · Answer 24 · edited Jul 16 '18 at 14:42

0

If you have list of repositories in a list like this, then this shell script works:

user="https://github.com/user/"

declare -a arr=("repo1", "repo2")

for i in "${arr[@]}"

do

   echo $user"$i"

   git clone $user"$i"

done

edited Jul 16 '18 at 14:42

Nick

3,359
18
29
40

answered Jul 16 '18 at 13:44

Kasthuri Shravankumar

199
1
9

if there's some private repositories, you can update "user" this way : user="https://user:password@github.com/user/" – Med Ali Difallah May 15 '19 at 15:53

olyanren · Answer 25 · 2019-02-24T14:55:45.827

I created a sample batch script. You can download all private/public repositories from github.com. After a repository is downloaded, it is automatically converted to a zip file.

@echo off
setlocal EnableDelayedExpansion
SET "username=olyanren"
SET "password=G....."
set "mypath=%cd%\"
SET "url=https://%username%:%password%@github.com/%username%/"
FOR /F "tokens=* delims=" %%i in (files.txt) do (
SET repo=%%i
rmdir /s /q !repo!
git clone "!url!!repo!.git"
cd !repo!
echo !mypath!
git archive --format=zip -o "!mypath!!repo!.zip" HEAD
cd ..
)

Note: files.txt file should contain only repository names like:

repository1
repository2

score 0 · Answer 26 · answered May 04 '19 at 09:03

0

Update from May 19

use this bash command for an organization (private repo included)

curl -u "{username}" "https://api.github.com/orgs/{org}/repos?page=1&per_page=100" | grep -o 'git@[^"]*' | xargs -L1 git clone

answered May 04 '19 at 09:03

Tommy

116
8

score 0 · Answer 27 · answered Aug 29 '19 at 14:22

The prevailing answers here don't take into account that the Github API will only return a maximum of 100 repositories despite what you may specify in per_page. If you are cloning a Github org with more than 100 repositories, you will have to follow the paging links in the API response.

I wrote a CLI tool to do just that:

clone-github-org -o myorg

This will clone all repositories in the myorg organization to the current working directory.

score 0 · Answer 28 · answered Dec 03 '19 at 20:50

0

For orgs you have access to with private repos:

curl -u <YOUR_GITHUB_USERNAME> -s https://api.github.com/orgs/<ORG_NAME>/repos?per_page=200 | ruby -rubygems -e ’require “json”; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo[“html_url”]} ]}'

It uses the html_url, so you don't need an access_token just enter your github password when prompted.

answered Dec 03 '19 at 20:50

Flavio

376
1
9

1

Basic authentication using a password to the API is deprecated and will soon no longer work. Visit [deprecating-password-auth] (https://developer.github.com/changes/2020-02-14-deprecating-password-auth/) for more information around suggested workarounds and removal dates. – BogeyMan Jun 24 '20 at 19:19

score 0 · Answer 29 · answered May 15 '20 at 10:45

"""
Clone all public Github Repos

https://developer.github.com/v3/repos/#list-repositories-for-a-user
"""

import urllib.request, base64
import json
import os


def get_urls(username):
    url = f"https://api.github.com/users/{username}/repos?per_page=200"
    request = urllib.request.Request(url)
    result = urllib.request.urlopen(request)
    return json.load(result)


if __name__ == "__main__":
    for r in get_urls("MartinThoma"):
        if not os.path.isdir(r["name"]):
            print(f"Clone {r['name']}...")
            os.system("git clone " + r["ssh_url"])
        else:
            print(f"SKIP {r['name']}...")

score 0 · Answer 30 · answered Jul 25 '20 at 19:16

To clone all your own private and public repos simple generate a new access token with repos access and use this:

(replace with your own access token and username)

for line in $(curl https://api.github.com/user/repos?access_token=ACCESS_TOKEN_HERE  | grep -o "git@github.com:YOUR_USER_NAME/[^ ,\"]\+");do git clone $line;done

This will clone all repos in current folder

This is a little bash program, you can just paste it in the terminal and hit enter

score 0 · Answer 31 · answered Oct 12 '20 at 03:26

You could use a tool like GitHub Archive which allows you to clone/pull public and private personal repos, organization repos, and gists all with one simple tool.

As for automation, you could then set up GitHub Archive to run once a day or once a week for example and it will skip those that are cloned and pull in new changes since the last time it was run of all others.

Source: https://github.com/Justintime50/github-archive

Suman Saurabh · Answer 32 · 2020-12-22T21:43:42.227

0

Here is the windows version using PowerShell

$name="facebook" #either username or org_name

$api_url="https://api.github.com/users/$($name)/repos?per_page=200"

$repos=Invoke-WebRequest -UseBasicParsing -Uri $api_url |ConvertFrom-Json

foreach ($repo in $repos)
{
  Write-Host  "Cloning via SSH URL  $($repo.ssh_url)"
  git clone $repo.ssh_url
}

edited Dec 22 '20 at 21:43

answered Dec 22 '20 at 20:49

Suman Saurabh

151
1
5

score 0 · Answer 33 · answered Mar 09 '21 at 13:17

0

When I want to clone all my repos fast I do:

for i in `echo https://github.com/user/{repox,repoy,repoz,repob}`; do git clone $i; done

answered Mar 09 '21 at 13:17

Nickotine

55
5

How to clone all repos at once from GitHub?

33 Answers33

Organisation repositories

User repositories

Private repositories

Clone all your public and private repos that are not forks:

Clone your gists:

You can filter the JSON arbitrarily using `jq`

Github's API urls

Create a bash alias/func in your `~/.bashrc file`

steps

Important

Linked

Related

How to clone all repos at once from GitHub?

33 Answers33

Organisation repositories

User repositories

Private repositories

Clone all your public and private repos that are not forks:

Clone your gists:

You can filter the JSON arbitrarily using jq

Github's API urls

Create a bash alias/func in your ~/.bashrc file

steps

Important

Linked

Related

You can filter the JSON arbitrarily using `jq`

Create a bash alias/func in your `~/.bashrc file`