Where did GitLab.com repos go?

Code Search
,
1

I regularly use sourcegraph.com/search across some large projects on both github.com and gitlab.com. I don’t seem to find any hits for gitlab.com repos anymore? Was this an intentional change or is there an ongoing incident?

2

Hey @souldzin let me dig into that.

1 Like
3

@souldzin

From my team mate:

On-demand cloning is not a thing anymore, all GitLab.com repos were previously synced through that (so they were all most likely severely out of date). For GitHub.com, we have a code host connection that explicitly clones repos, right now all that have more than a certain number of stars.
Since we didn’t have such a config for GitLab.com, they don’t show up any longer. We can totally add gitlab.com repos if we have a list of repos we want to add there…

…Someone contributed an API for star count ordering for projects within a group 6 months ago, maybe they’d be willing to accept a contribution for the global projects endpoint as well :thinking_face: Add support for sorting GET /groups/:id/projects by star count (!145308) · Merge requests · GitLab.org / GitLab · GitLab

I have a contact at GitLab that I will reach out to and ask if they would consider this. That said, I can’t promise that this will happen or when.

What repos where you looking for in particular?

4

Thanks for looking into this @jdorfman!

I have a contact at GitLab that I will reach out to and ask if they would consider this.

I actually work at GitLab, so I might be able to help with things on our end too :slight_smile:. It looks like our GraphQL API would support sorting by stars:

query {
  projects(sort:"stars_desc") {
    pageInfo {
    	startCursor,
      endCursor,
      hasNextPage,
      hasPreviousPage
    },
    nodes {
      fullPath,
      name,
      id,
      webUrl,
      starCount
    }
  }
}

Is that an option for you all? If not, I just opened up a contribution to update our API to accept order_by=stars (I can’t believe that’s not there yet!)

What repos where you looking for in particular?

I am particularly interested in the projects under the gitlab-org group. Would we be able to include all of those projects, regardless of stars?

2 Likes
5

Woah! Awesome! Let me relay this. I think this will be a great win for the open source community, I’m really excited (if you haven’t noticed) =)

1 Like
6

@souldzin, by the way, is there an easy way to add GitLab login to Discourse forums? My initial search didn’t turn up anything.

7

by the way, is there an easy way to add GitLab login to Discourse forums? My initial search didn’t turn up anything.

@jdorfman if you’re talking about from an OAuth perspective, it looks like Discourse doesn’t have GitLab settings out-of-the-box, but forums.gitlab.com is a Disourse server which I think uses the OAuth2 Basic plugin OAuth 2.0 & OpenID Connect Support | Discourse - Civilized Discussion.

  1. On the gitlab.com side, I think you could get by with creating an OAuth Application under a GitLab group (there used to be a sourcegraph group I thought) Configure GitLab as an OAuth 2.0 authentication identity provider | GitLab. According to OAuth2 Basic plugin docs, the Redirect URI for this OAuth Application will be something like https://community.sourcegraph.com/auth/oauth2_basic/callback
  2. Once created, the new GitLab OAuth app metadata can be registered with community.sourcegraph.com through the OAuth2 Basic plugin.

I think :sweat_smile:

8

@souldzin, I haven’t forgotten about this; I just had a few fires to put out. Will follow up asap.

Edit: what’s the status on that merge request? I see it’s still open on your end. That will help us get this moving forward.

What we need is: global projects endpoint

“Someone contributed an API for star count ordering for projects within a group 6 months ago, maybe they’d be willing to accept a contribution for the global projects endpoint as well”

9

Thanks for the update @jdorfman!

Edit: what’s the status on that merge request? I see it’s still open on your end. That will help us get this moving forward.

I’ll look into pushing this forward this week.

There’s been some conflicting priorities at the moment - a situation I’m sure you’re familiar with :sweat_smile:

1 Like
10

Sounds good and yes I know. I know

11

@jdorfman looks like the Merge Request to add ?order_by=star_count to the projects API endpoint has been merged and deployed :tada:

Does this unblock your team from including gitlab.com projects in sourcegraph.com search?

12

@souldzin NICE! let me inform my guy. Will be in touch

13

@jdorfman friendly ping on this. Any update? Is there a GitHub issue I can follow?

14

Hey @souldzin sorry for the delay (was out).

We will look at this - we have [offsite this week] and need to prioritize some more [other] work, but want to still look at this after.

I’ll be on PTO next week but I will create a reminder to keep you updated once I get back.

15

Hey @souldzin

Working on a temp solution. There are others requesting GitLab repos so I’m thinking we just add them one by one for now.

Can you list some top repos you would like me to add? This format would be best:

    {
      "name": "<group>/<name>"
    }

Thanks!

16

@souldzin

Got these indexed in the meantime:

https://sourcegraph.com/gitlab.com/gitlab-org/gitlab
https://sourcegraph.com/gitlab.com/gitlab-org/gitaly
https://sourcegraph.com/gitlab.com/gitlab-org/gitlab-pages
https://sourcegraph.com/gitlab.com/gitlab-org/gitlab-runner
https://sourcegraph.com/gitlab.com/gitlab-org/container-registry
https://sourcegraph.com/gitlab.com/gitlab-org/gitlab-shell
https://sourcegraph.com/gitlab.com/gitlab-org/cluster-integration/gitlab-agent
17

@souldzin

Update:

“Quick update: It’s cycle 13 now but we have decided to not prioritize it for the time being as the was not enough signal to warrant putting in the amount of work required here right now.”

With that said I still want to index as many GitLab repos manually.