Source{d} Community Edition (CE) is missing data from my organizations

There is a number of reasons why source{d} Community Edition (CE) is not showing up all data for the organizations you imported. The most obvious symptom is a repository count lower than it should. Here’s an overview of possible causes and solutions.

Data is still importing

The most common cause is that data is still being imported. The import time varies with the number of repositories, their size, as well as the number of issues and pull requests. You can refresh the repository count chart from time to time. If the counter is still going up, everything is correct. You just need to wait some more time.

Some repositories are empty

The default repository count does not show up empty git repositories. This count is calculated by gitbase from cloned repositories. We currently do not support importing empty repositories.

Empty git repositories, however, will show up in other non-git tables, like the metadata database with issues and pull requests imported from GitHub.

Repository import failed

If refreshing the charts does not show up new data after a few minutes, it is possible that one of the import processes failed. There are multiple causes for this: networking issues, running out of free disk space, etc. There are two separate processes importing data: 1) gitcollector, which clones git repositories and 2) ghsync, which imports GitHub metadata, like issues and pull requests.

Troubleshooting a gitcollector failure

On the command line, run the following commands to retrieve gitcollector logs:

cd $HOME/.sourced/workdirs/__active__
docker-compose logs gitcollector | grep error

Update 2019-07-11: As for source{d} CE v0.14.0-beta.14, logs can also be retrieved with sourced logs [-f] [component-name...]

If there was an error, you should see it in these logs, for example:

time="2019-06-21T16:33:23.205087863Z" level=error msg=failed error="write /tmp/gitcollector-downloader542609248/go-borges264047728/transaction-tmp/objects/pack/tmp_pack_465019151: no space left on device" id=f7a27fbc-492d-4b49-955b-18409500cfdd job=download source="downloader/download.go:90" url="https://github.com/src-d/go-git"

In this case, the error no space left on device indicates that the partition where your Docker daemon stores volumes ran out of space. Once the cause of the error is solved, you can run gitcollector again:

cd $HOME/.sourced/workdirs/__active__
docker-compose run gitcollector

This will run gitcollector again in the foreground. It will go again over all the repositories in your organization, skipping those that were already cloned, and trying to clone again those that could not be cloned before.

Troubleshooting a ghsync failure

ghsync could also fail for similar reasons as gitcollector. You can find errors as follows:

On the command line, run the following commands to retrieve ghsync logs:

cd $HOME/.sourced/workdirs/__active__
docker-compose logs ghsync | grep error

Once the problem is fixed, you can run the process again:

cd $HOME/.sourced/workdirs/__active__
docker-compose run ghsync

Something else?

If none of this solved your problem, or found an error without a clear solution, feel free to reply to this topic.