X.Org BoD meeting minutes 2020-02-27

Tue Mar 3 04:08:12 UTC 2020

On Mon, Mar 2, 2020 at 2:39 PM Lyude Paul <lyude at redhat.com> wrote:
>
> On Fri, 2020-02-28 at 21:31 +0100, Luc Verhaegen wrote:
> > On Fri, Feb 28, 2020 at 07:29:51PM +0100, Daniel Vetter wrote:
> > > On Fri, Feb 28, 2020 at 7:06 PM Lucas Stach <l.stach at pengutronix.de>
> > > wrote:
> > > > The only trace of a rough ballpark figure for gitlab hosting expenses
> > > > was in some board meeting IRC logs [1] where the number of 30k/year was
> > > > floated. Even board members seemed to be surprised by this number.
> > >
> > > We've had a few threads on board@ but I checked, and we seem to have
> > > indeed dropped the formal vote on this somehow. Was quite a busy time
> > > with xdc19 papers committee and our favorite xdc20 proposal folding,
> > > plus everyone on vacation.
> > >
> > > But that was on my secretary watch and definitely shouldn't have happened
> > > :-(
> >
> > Can this be broken down:
>
> I'm piecing together as much of this as I can here to answer people's
> questions, please note I wouldn't be surprised if I make some mistakes here
>
> > * who decided on the move to gitlab
>
> I'm having trouble finding an answer to this one going through the logs
> (especially since this happened before my time as secretary), but it is worth
> noting that the Gitlab instance (note the use of the word instance, this
> doesn't include the CI runners) was originally sponsored by Gitlab
> themselves. Also, while there's been a few critics there's been plenty of
> interest in the community for the Gitlab instance for quite sometime now.
> Otherwise, I don't think we would be seeing the huge growth that's happening
> on it right now.

(Just pitching in my recollection.. others can feel free to correct me
as I wasn't directly involved.  But from my recollection and double
checking fd.o news, election history, minutes, etc, I believe this is
correct.)

You probably won't find it in BoD minutes, since it pre-dated fd.o
joining xorg.  And IMHO it was the better decision compared to hiring
a full-time admin to maintain fd.o's creaky infrastructure, which
already had a lot of technical debt.

(Also, the question about moving to gitlab is largely a red-herring.
Moving to gitlab hasn't been the problem, as much as the rapid uptake
of CI.. which I might add has been a hugely useful thing!  I'm a big
fan of not having to bisect some breakage every couple weeks when I
have a chance to rebase whatever I have been working on to latest
master!)

> Some actual budget for the CI runners came into the picture a little later.
> The first vote I can find is regarding allocating $8k/year for CI runners for
> mesa. This vote was approved on 2018-10-11, and you can find it in the meeting
> minutes for that day. Discussion of the CI/gitlab stuff doesn't come up in the
> meeting minutes again until 2019-07-18, just like danvet mentioned. Understand
> that the reason it didn't come up in meetings is because there wasn't anything
> to talk about, the growth on gitlab which resulted in this bill _was_ rather
> sudden. I do expect that with volunteers who have the ability to spend more
> time on managing this, we'll notice things like this a lot more quickly in the
> future.
>
> > * who decided on cloud hosting at google
>
> Having trouble finding any answers to this one, but this isn't really what's
> driving up the costs here so much as the fact that we need an admin to
> dedicate more time to this so we can be more efficient with our hosting
> resources.

I believe it was because initially gitlab donated some GCP runner
credits to aid in the migration, and later google OSPO donated
credits.

>From what I understand, the GCP hosting isn't directly the problem.
CI also uses runners donated elsewhere (packet.net, iirc?).  And I
think gstreamer is also hosting some of their own CI runners,
presumably also pulling down artifacts and containers from cloud
storage.  There seems to be some theories about inefficiencies with
the various non GCP runners not having enough storage and
re-downloading containers from GCP cloud storage?  TBH network egress
costs to other providers isn't something I would have anticipated up
front as being a problem.  Maybe it would have been to someone else
with more experience with cloud hosting.. but I don't think that
accurately describes any of the xorg board or fd.o admins.

My $0.02 from the sidelines, there seems to be a lot of arm-chair
quarterbacks in this discussion, and somewhat fewer stepping up to
help analyse logs[1] and help pinpoint where the inefficiencies are.
I'd like to remind people that the xorg BoD are volunteers who are
doing their BoD work on top of fulltime jobs and such.  (I know, I was
one!)  IMO, the board did the right thing: when it became apparent
that CI was projected to exceed the budget, they raised the issue
publicly and determined the point where they need to switch off CI if
something doesn't change.  And they correctly identified that some
help from paid IT admin who knows what they are doing and has time to
dig into these issues would be valuable.  It is easier to criticise.
A more useful thing to do is to volunteer.  That's what keeps this
community running!

BR,
-R

[1] from #freedesktop scrollback, looks like bentiss has been helping
with this, much thanks!

> > * who did not notice running a tab 2.5x the amount of googles
> >   sponsorship. Would 30k of google cloud hosting have had red warning
> >   lights flashing in... august 2019?
>
> yes, it would have and did have red warning lights flashing even earlier then
> that:
>
> https://www.x.org/wiki/BoardOfDirectors/MeetingSummaries/2019/07-18/
>
> and a quote from the IRC logs, also available from that page:
>
> [21:44:55] <danvet_> if I'm right gitlab.fd.o is burning down 30k$/year in google cloud credits
> [21:45:11] <danvet_> and pretty big piles of lent machine time for CI
> [21:45:39] <danvet_> so if we don't collect fd.o sponsors for this then the xdc surplus wont pay for it, not even close
> [21:46:45] <anholt> confirmed that that's about right in cloud costs
> [21:47:04] <anholt> I don't have a good handle on what the mix is for CI vs gitlab.
> [21:47:16] <danvet_> I don't think it's hard to get all that (plus paid admin time)
> [21:47:29] <danvet_> anholt, I think CI is mostly dedicated hw sponsored by companies directly
>
> And since then we've been looking into various sponsorships, including
> renewing the sponsorship we had with Gitlab, and we've also been looking into
> trying to hire an admin so that we could optimize our services so that we
> don't end up with another huge bill like this. The only thing that's changed
> since then was that we made an announcement to make sure we were being open
> about this, and also in order to help speed up the search for sponsors since
> the avenues we originally tried didn't come back to us with anything. The
> meeting minutes and logs for this are and have available.
>
> Furthermore, while Lucas and Daniel are completely right that we should have
> had a formal vote between directors for this, remember there was plenty of
> time for projects to object. Some even did object, as I remember discussing
> this over public email threads pretty extensively before I even ended up on
> the board. But the vast majority of projects were completely on board with it
> from the start and have since been taking advantage of the services we provide
> like CI.
>
> >
> > > Please note that we've just had a very successful xdc18 with a massive
> > > surplus and opted to double the sponsoring amounts for xdc19 (to
> > > rather great success). So 30k is definitely a lot for the old pre-spi
> > > X.org, but the entire point of the SPI merger was to get sponsors on
> > > board for hard money (not just in-kind donations and stuff), and that
> > > worked out really well.
> >
> > So the partially pending fd.o merger (how far along is this anyway?) did
> > not improve transparency and did not increase oversight.
> > How on earth do you convince any sponsor to give their money to x.org
> > now?
> >
> > > 30k wasn't something we could just pay
> > > forever, but easily something we could pay for a year if Google drops
> > > and we'd need to find someone else.
> >
> > And yet the bill is 2.5x that, and that did not happen overnight.
> >
> > Luc Verhaegen.
> >
> --
> Cheers,
>         Lyude Paul (she/her)
>         Associate Software Engineer at Red Hat
>
> _______________________________________________
> members at foundation.x.org: X.Org Foundation Members
> Archives: https://foundation.x.org/cgi-bin/mailman/private/members
> Info: https://foundation.x.org/cgi-bin/mailman/listinfo/members