X.Org BoD meeting minutes 2020-02-27

[Lyude Paul]

On Fri, 2020-02-28 at 21:31 +0100, Luc Verhaegen wrote:
> On Fri, Feb 28, 2020 at 07:29:51PM +0100, Daniel Vetter wrote:
> > On Fri, Feb 28, 2020 at 7:06 PM Lucas Stach <l.stach at pengutronix.de>
> > wrote:
> > > The only trace of a rough ballpark figure for gitlab hosting expenses
> > > was in some board meeting IRC logs [1] where the number of 30k/year was
> > > floated. Even board members seemed to be surprised by this number.
> > 
> > We've had a few threads on board@ but I checked, and we seem to have
> > indeed dropped the formal vote on this somehow. Was quite a busy time
> > with xdc19 papers committee and our favorite xdc20 proposal folding,
> > plus everyone on vacation.
> > 
> > But that was on my secretary watch and definitely shouldn't have happened
> > :-(
> 
> Can this be broken down:

I'm piecing together as much of this as I can here to answer people's
questions, please note I wouldn't be surprised if I make some mistakes here

> * who decided on the move to gitlab

I'm having trouble finding an answer to this one going through the logs
(especially since this happened before my time as secretary), but it is worth
noting that the Gitlab instance (note the use of the word instance, this
doesn't include the CI runners) was originally sponsored by Gitlab 
themselves. Also, while there's been a few critics there's been plenty of
interest in the community for the Gitlab instance for quite sometime now.
Otherwise, I don't think we would be seeing the huge growth that's happening
on it right now.

Some actual budget for the CI runners came into the picture a little later.
The first vote I can find is regarding allocating $8k/year for CI runners for
mesa. This vote was approved on 2018-10-11, and you can find it in the meeting
minutes for that day. Discussion of the CI/gitlab stuff doesn't come up in the
meeting minutes again until 2019-07-18, just like danvet mentioned. Understand
that the reason it didn't come up in meetings is because there wasn't anything
to talk about, the growth on gitlab which resulted in this bill _was_ rather
sudden. I do expect that with volunteers who have the ability to spend more
time on managing this, we'll notice things like this a lot more quickly in the
future.

> * who decided on cloud hosting at google

Having trouble finding any answers to this one, but this isn't really what's
driving up the costs here so much as the fact that we need an admin to
dedicate more time to this so we can be more efficient with our hosting
resources.

> * who did not notice running a tab 2.5x the amount of googles 
>   sponsorship. Would 30k of google cloud hosting have had red warning 
>   lights flashing in... august 2019?

yes, it would have and did have red warning lights flashing even earlier then
that:

https://www.x.org/wiki/BoardOfDirectors/MeetingSummaries/2019/07-18/

and a quote from the IRC logs, also available from that page:

[21:44:55] <danvet_> if I'm right gitlab.fd.o is burning down 30k$/year in google cloud credits
[21:45:11] <danvet_> and pretty big piles of lent machine time for CI
[21:45:39] <danvet_> so if we don't collect fd.o sponsors for this then the xdc surplus wont pay for it, not even close
[21:46:45] <anholt> confirmed that that's about right in cloud costs
[21:47:04] <anholt> I don't have a good handle on what the mix is for CI vs gitlab.
[21:47:16] <danvet_> I don't think it's hard to get all that (plus paid admin time)
[21:47:29] <danvet_> anholt, I think CI is mostly dedicated hw sponsored by companies directly

And since then we've been looking into various sponsorships, including
renewing the sponsorship we had with Gitlab, and we've also been looking into
trying to hire an admin so that we could optimize our services so that we
don't end up with another huge bill like this. The only thing that's changed
since then was that we made an announcement to make sure we were being open
about this, and also in order to help speed up the search for sponsors since
the avenues we originally tried didn't come back to us with anything. The
meeting minutes and logs for this are and have available.

Furthermore, while Lucas and Daniel are completely right that we should have
had a formal vote between directors for this, remember there was plenty of
time for projects to object. Some even did object, as I remember discussing
this over public email threads pretty extensively before I even ended up on
the board. But the vast majority of projects were completely on board with it
from the start and have since been taking advantage of the services we provide
like CI.

> 
> > Please note that we've just had a very successful xdc18 with a massive
> > surplus and opted to double the sponsoring amounts for xdc19 (to
> > rather great success). So 30k is definitely a lot for the old pre-spi
> > X.org, but the entire point of the SPI merger was to get sponsors on
> > board for hard money (not just in-kind donations and stuff), and that
> > worked out really well.
> 
> So the partially pending fd.o merger (how far along is this anyway?) did 
> not improve transparency and did not increase oversight.
> How on earth do you convince any sponsor to give their money to x.org 
> now?
> 
> > 30k wasn't something we could just pay
> > forever, but easily something we could pay for a year if Google drops
> > and we'd need to find someone else.
> 
> And yet the bill is 2.5x that, and that did not happen overnight.
> 
> Luc Verhaegen.
> 
-- 
Cheers,
	Lyude Paul (she/her)
	Associate Software Engineer at Red Hat