Rich0's Gentoo Blog

Gentoo Bug Bounties

with 7 comments

Some may have noticed that the Gentoo Foundation has funded a bug bounty. This is something fairly new for the Foundation, and I wanted to offer some comments on the practice. Please note that while I’d love to see some of these make their way into policy some day, these are nothing more than my own opinion, and I reserve the right to change my opinion as we gain experience.

The recent bug bounty was for bug #418431, which was to address a problem with git-svn which was holding up stabilization of the latest version of git, which is a blocker for the migration of the Portage tree to git.

What follows are some principles for the use of bug bounties and how I think we fared in this particular case. I’d like to see the use of bounties expand, as right now I believe we under-utilize our donations. However, it is important that bounties be used with care as they have the potential to cause harm or be wasteful.

One more upfront note – I supported the git-svn bounty as it was ultimately worded, as did the other Trustees. Looking back I think we could have done things a little differently, but hindsight is always 20/20, and no doubt we’ll continue to learn as we experiment with this further.

1. Bounties Should Be Used Strategically
While the Foundation has money to spend, we aren’t swimming in it, so we can’t use bounties for any little bug that annoys us. Bounties should be reserved for matters where spending a little money has a large impact.

I think we did well here – the git-svn issue was going nowhere either within Gentoo or upstream, but the number of other blockers to the git migration are fairly small and within Gentoo’s control. Getting rid of this issue should open the way towards the git migration, which is of course of strategic importance to Gentoo.

2. The Solution Must Be Sustainable
This might also be stated as “consider the total cost.” Before agreeing to fund this bug there was some due diligence to ensure that upstream would carry forward any patches we generated. The problem was the result of changes on the SVN side, and the solution included some general cleanup and refactoring of code to make git-svn more maintainable upstream. Upstream also expressed an interest in accepting the fix, and it was the opinion of the package maintainer that this would be a one-time fix as a result.

When considering whether a solution is sustainable, we need to think about how we got where we are, and consider whether we’re just going to end up back in the same place again. If the solution won’t be maintainable, then any money spent is wasted unless it truly is a one-time event.

3. Gentoo Can’t Fix It With Volunteer Effort
Gentoo is a community distribution. We have some very talented developers. We can usually fix our own problems, and doing so as a volunteer community effort is usually the healthiest solution.

The sense for git-svn was that this was an upstream problem in a language our maintainers were not comfortable with. The bug languished despite attention by several developers and discussions in other forums. It was felt that offering a bounty would allow targeted expertise to tackle the problem, which otherwise was not of great interest to our community.

A policy to not offer bounties unless a bug has been open for some period of time except in unusual circumstances would be appropriate.

4. Be Ready To Capitalize On the Work
If the work is strategic (see #1), then we ought to have a plan ready for when the bug is closed. Otherwise there really should be no urgency to pay somebody to close the bug and it is basically a pig in a snake (clear the jam, and the problem just moves one step down the chain).

I think the jury is still out on how we’re doing here. I think there is a lot of enthusiasm about git but we could have a bit more organization here. None of this is intended as a slight to those who have been laboring hard to make this work – I hope getting this blocker cleared will inspire more to step up and resolve the other issues. (I won’t say more here as I don’t want to make this about the Git migration.)

5. Define the Problem and Success
A bounty is a contract. At the very least misunderstandings can lead to hurt feelings, and at worst they can lead to HIGHLY contentious, expensive, and distracting legal action. While a 10 page document shouldn’t be necessary for a token expense, any bounty should be very up-front about what exactly is to be done, and how success will be evaluated.

I think we could have done a little better in this regard, but there was some iteration on the wording of the bounty to clarify the “victory conditions.” I think it is important to focus on outcomes – in this case we wanted code that upstream was likely to accept. I’d actually have been happier making upstream acceptance a condition of payment, but the sense was that this would be inevitable but might delay payment unduly. I think the jury is still out on this one. What is important is that we don’t just achieve technical resolution of the bug, but that we fully realize the benefits we had in mind when we funded the bounty.

6. Cover Code Ownership and Licensing
This is a work for hire – we can dictate ownership of the code (yes, I realize that the legalities of this vary internationally, but the US is the only nation that legally recognizes the Gentoo Foundation at the moment, and the US will enforce this insofar as its jurisdiction allows). Per the Gentoo Social Contract, if we’re funding the creation of code, it ought to be free (generally GPL).

This was covered in the git-svn case. We didn’t insist on ownership of copyright, but we did ensure the code was licensed using the upstream license (GPLv2). My feeling is that if the bounty really represents payment for a majority of the work Gentoo should just own the code outright. If the bounty is really a token gesture for what is mostly a volunteer effort I think the author should retain copyright as long as the code is FOSS. In practice it doesn’t matter too much, so I think we should use discretion here.

7. Offers of Bounties Should Be Fair
This topic led to some internal debate, and I think that we can probably do a little better in terms of transparency in the future. The bounty was posted publicly on the bug, and anybody already interested in the bug and on the CC list would of course have gotten notification. In retrospect I think that bounties are a significant enough occasion that perhaps a proposal should be offered for comment on -dev or -nfp and the final version announced on -dev-announce. I think that the way we handled the git-svn case met all legal obligations, but I really want to make sure that the whole community has an opportunity to participate when they come up.

Another potential issue with bounties is that you can only pay one person (unless there is some side agreement to share it), and there can be resentment if work gets done but isn’t reimbursed. This was addressed in the present case by asking anybody working on the bug to state their intent. If a bounty is very large it probably would make sense to go through a more formal bidding process and just award a contract more conventionally.

I think that this last point of fairness is actually the most critical. While messing up on any of the others could cause us to waste a few hundred dollars, getting the fairness bit wrong could literally destroy the community. When you start paying people to do what used to be volunteer work the result can be demoralizing to the community. I think the key is to only do this when the community lacks the ability/desire to do the work itself, and especially when the work lies outside of our core expertise. Paying an accounting firm a reasonable fee to ensure our taxes are filed correctly isn’t viewed with much controversy. We should try to keep bug bounties limited to similar sorts of situations.

Trustees of course have duties both under the bylaws and under US law to properly manage conflicts of interest. These certainly apply to any kind of expenditure of money.

So, what do you think? I’m very open to criticism about how we handled our first bug bounty, and how the community feels about this use of money. As is evident from the Treasurer’s Report at today’s Annual General Meeting, Gentoo is currently receiving more than it spends in donations, so I think making a little more use of this approach will allow our supporters to benefit Gentoo. Seeing donations in action probably will help encourage an increase in donation as well. However, I think we also need to tread carefully here, as the community matters far more than squashing a few bugs.

Finally, while I’d like to see policy around bounties formalized, I think doing so right away would be a mistake. I think we should try to consciously apply principles like these but wait until we see how they work in practice before trying to codify them.

Written by rich0

August 19, 2012 at 10:55 pm

7 Responses

Subscribe to comments with RSS.

  1. I got a number of comments via IRC, but I figure I’ll respond to them here to capture them.

    Comment: dberkholz – i think the critical point is the effort to get volunteers involved first, before resorting to a bounty, i’ve interacted with a number of other floss foundations about bounties in the past, and this has been the biggest issue. same applies to hiring part-time effort for things like legal, accounting, fundraising, sysadmining, etc.

    Couldn’t agree more. Beyond the expense of it spending money can make volunteers feel like second-class citizens and that can kill a community effort quickly. I’ve seen this in other non-profits – slowly the important jobs become paid, and you’re not really part of the “community” unless you are on some office 9-5. Then they wonder why everybody stops donating.


    August 20, 2012 at 9:17 am

  2. Comment: ioflow – are we no longer incorporated in germany as an e.V, or whatever it was?

    As dberkholz mentioned on IRC, the e.V. in Germany is actually a distinct organization with no formal relationship to the Gentoo Foundation. That organization hasn’t been quite as active (well, at least in recent time – it wasn’t that long ago that the Foundation was nearly dead). The US Foundation is the one that does most of the paying of the bills and which owns all the infrastructure (well, the parts that Gentoo actually owns at all – quite a bit of the non-core stuff is owned by others and donated to Gentoo use – THANKS!). I’m sure the e.V. has been up to things more locally – I don’t want to create the impression they don’t do anything, and I’d be interested in comments about their activities.

    I’d actually like to try to cement the relationship more formally. The Foundation has a legal obligation to defend its trademarks and such, and formalizing the relationship would avoid creating an argument that we just don’t care about the EU. Of course, nothing could be further than the truth, and in recent years I think we’ve spent more on conference giveaways and such in the EU than in the US. Working together allows us to combine our legal footprint rather than letting others use the division (on paper) to weaken it.


    August 20, 2012 at 9:24 am

  3. Comment: robbat2 – can developer be bounty recipients per IRS?

    Yes. We can pay developers just as corporations can pay employees who happen to be shareholders (developers aren’t actually employees though).

    All of the rules that apply to corporations apply to the Foundation – we need to honor embargoes and such, and report payments above certain thresholds (which I’m hoping we’ll rarely if ever hit).

    Some additional rules apply as we’re seeking 501c3 status. Payments need to be reasonable for the effort – in line with market rates and such. I think the reality is that we’ll always be below-market for stuff like this, so no risk there. Payments to insiders or those with relationships to insiders (officers and trustees mostly) have to be handled carefully (they aren’t prohibited, but we need to be careful and the reality is that the community will become rightfully outraged far before we step over IRS rules). Donations can’t be associated with any kind of direct benefit either – you can’t tax deduct a donation that basically comes back to benefit yourself (such as if I were to donate money to Gentoo and in return Gentoo were to reimburse me for a piece of hardware I were to essentially have near-exclusive use over, or I were to travel on that money, and so on).

    As long as the expenditure is clearly mission-related then I think we’re fine. If paying a non-dev $500 to do some work would be fine, then paying a Gentoo dev the same amount to do the same work is also fine.

    However, all of my and dberkholz’s comments still apply – payments to devs cause lots of non-legal issues and I think these are actually the more important ones to manage. If a dev is willing to do work for money but not without being paid, we need to ask why. However, some jobs really are time-consuming or uninteresting but if they’re one-time things and they really need to get done, I don’t want to rule that out.

    The same applies to travel or buying machines for individual devs. While I oppose both in general, I’m fine with making exceptions. For example, if we had some 14-year-old who was really dedicated and spending lots of time on Gentoo but their position in life really was limiting their ability to spend real money, then that is where I think we can step in. In general for most of us the money we spend on computer hardware costs little compared to the time we spend on Gentoo, and we get a lot out of what we put into it all.

    My principle #1 above also applies – Gentoo should be “highly leveraged” when it comes to spending money. That is, we should spend money when a little goes a long way (I’m not using the term to refer to borrowing in any way – which we should absolutely avoid). For example, we rely on donated mirrors, but I believe we own the boxes they all replicate from. So, the most critical infrastructure is owned so that we can control and rely on it, but we don’t try to buy our own mirror network. The money we spend on our own hardware is therefore greatly multiplied by the donations of others, minimizing our overhead.

    Well, enough random comments on this topic – I’m sure there is another blog in here…


    August 20, 2012 at 9:45 am

  4. Rich,

    I am planning to migrate to Gentoo from another distribution once some fires die down. I say this as a preference as I only a cursory level of knowledge of Gentoo at this time. Until I discovered the link to this blog today, 20 August 2012, I was not aware of the bounty and all related matters for the git-svn issue Gentoo wanted to find some solution for.

    One aspect that struck my mind right away is the donations incoming greater than outgoing levels. I do not know if that also refers to future contingency inclusive or not. What I think one needs to consider is all the related costs and frequency of such things as bounty, hardware costs to someone clearly not in a position to afford, et al verses the recurring costs. My thoughts are NFPs such as Gentoo need to be careful as sometimes unexpected expenses of a large amount can arise. Ergo the contingency that was excess can be burned quickly as a result, be it hardware or the more open ended legal costs for something unforeseen that may have to be defended, even if innocent, and be able to do so. Even if innocent the legal system may not award costs incurred to defend the innocence of the complaints. I am sure there are more examples, but as we all know legal expenses can be very be hard to predict.

    Of course the points you raise as well as the comments of others you posted are very important as well.

    John L. Males
    Toronto, Ontario
    20 August 2012 14:03

    John L. Males

    August 20, 2012 at 2:03 pm

    • I agree with your points, but I think we have sufficient contingency for now. So far we’ve had one bug bounty in a decade or so – I think we can safely spend a bit more than that, without necessarily spending thousands of dollars a year.

      While having a rainy day fund makes sense, I don’t want to have donors giving us piles of money just so that somebody can eventually sue us for it. If there is a lawsuit and we really need money, we can ask for it. Worst case we just clone our limited selection of hardware and change our name. Gentoo is more about community than IP – so while I’m charged as a Trustee to protect the latter, I’m still far more concerned with the former.


      August 20, 2012 at 2:08 pm

  5. Re #7:

    I would post them to -dev and -user, at the least. This bug in particular didn’t even have anything to do with Gentoo — I’d be willing to bet there at least as many Perl developers on -dev as off. The forums could help, too.

    Michael Orlitzky

    August 20, 2012 at 7:10 pm

  6. […] results of Gentoo offering a bounty to fix a bug. This reminded me of Arch Bounty, which failed to even get a single project […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: