A full post-mortem is coming, but here's what happened from my perspective. I got mentioned on twitter about this thread while I was on the bus. I asked @evanphx to put the site into maintenance mode immediately.
Once I got on wifi, I downloaded the "exploit" gems and found that they used the Psych hack to post config/initializers/* and config/*.yml to Pastie. Luckily, none of our API keys were actually in those files - our config is a bit different since you can still run RubyGems.org on Heroku, which needs to put secret keys in the ENV.
I deleted the exploit gems permanently along with the throwaway account that posted them...that's why this post's URL is a 404.
I've reset our S3 keys just to be sure and any other keys we had on our production site. From what we know right now, no other changes on our S3 bucket have taken place, and we're going to check the logs to make sure. Once we get a real fix for this issue out, pushing gems will be enabled again.
Just a general PSA -
Please, if you find an issue like this, be nice. Tell the maintainers privately. Don't post to Reddit, HN, or a public Gist. RubyGems.org is completely volunteer run. No one gets paid to work on it. Thanks for your patience everyone.
If you want security notices to go to the Rubygems team, you have to set up a security page that tells people how to do that. Like everyone else, I appreciate your volunteer work, but no amount of goodwill creates the ability for people to read your mind.
Please post a security page. It literally doesn't need anything more than an email address and a PGP public key.
By the way: if Rubygems needs security help, it looks like there's quite a number of people who are willing to pitch in. When I published a FreeBSD crt0 bug back in 1996, I was given commit privs. I thought that was a pretty effective way to co-opt adversarial researchers.
I know dealing with this stuff is no fun. Try to keep in mind, though, that people like Ben Murphy and Hal (Postmodern) are on your side. Again, maybe consider formalizing that relationship a little!
The Rubygems team (or even just the person who writes the security page) can just generate a new one, for "[email protected]" or whatever, and it can be shared by the team.
The purpose of the key is to allow people to report security vulnerabilities without worrying that by doing so they're giving ammunition to people snooping emails.
Maybe I dont understand PGP and this is a stupid question, but if the site is compromised, would it not be possible for them to just put a different key up? One which they knew the private key for? How would you know the public key on the compromised site has not been changed unless you could compare it to the PGP key from before the site was compromised?
You can't, but having a security.html page gives attackers the possibility to contact you privately and securely; this reduces the chance that they'll decide to "contact" you by completely pwning your public site.
What if the key is endorsed by the individual people on the team? Their personal keys, which they keep, must sign the security team key. A replacement cannot have this endorsement.
You don't want to send an actual exploit in clear text to a possibly compromised email account. PGP encrypts, and provides some level of validation that the message can only be read by someone on the security team.
Respectfully, RubyGems has already been compromised. Not telling anyone has zero advantages to RubyGem users.
Telling people to avoid using RubyGems and to audit if they've used it recently helps protect users.
Responsible disclosure is about avoiding giving bad guys new attack vectors and ensuring that the maintainer has the best chance of fixing the issue before a blackhat discovers and exploits it. That doesn't apply in this case.
I agree, but how do people audit? I think it's time to start putting application dependencies, like rubygems and npm modules, under version control, not in the same repo, but in a different repo.
Mikeal Rogers has some good reasons why, but there gets to be too much commit noise if you have big dependencies, and put them in the same repo. A separate repo solves this.
Yes, and thanks for telling us. Really, we should have disabled gem pushes immediately. Hindsight is 20/20, and I'm not sure why I didn't think of doing so earlier. The pain of not being able to push gems would have forced us to fix it.
I'm sorry about this. I don't know what else to say. I wish we didn't have to deal with this kind of problem in the Ruby community.
> Please, if you find an issue like this, be nice. Tell the maintainers privately. Don't post to Reddit, HN, or a public Gist. RubyGems.org is completely volunteer run. No one gets paid to work on it. Thanks for your patience everyone.
It'd probably go a long way if you guys considered making an easily-discoverable security page with information on who to contact, how you prefer that contact to take place and how long they should expect to have to wait to hear back from you.
Is this maybe an opportunity to re-evaluate the volunteer only aspect of RubyGems? I love the service (as I'm sure thousands of others do) and I'd happily pay for a subscription model that would fund a full-time developer/maintainer for the service. Based on how critical it is to the Ruby community I can't imagine that I'm alone in thinking this.
Hmm. That's for private gems. Interesting, but a different use-case. I'd chip in to pay for someone to maintain "rubygems.org" on a professional basis. It's awesome that some people take their private time to support such an essential service, but I'm with the grandparent that it might be good if it were run on a paid-for basis. Maybe through donations or similar, so that access is still free.
CPAN is all volunteer. We happen to have a few really good SAs volunteer to keep core infrastructure up (which PAUSE, our upload server, just moved onto) ... but it's still volunteer.
Hi Nick, instead of being a dick, I want to first of all thank you for understanding the urgency (despite being on a public bus, you took the pains to get it temporarily fixed somehow). Like others, I do agree that posting on HN does more good than bad, but at the same time, I do understand that it is also polite to inform the maintainers too (which I think in your case they didn't.). Anyway, thank you for being a responsible maintainer!
Once again, Thank you for maintaining rubygems.org! You're doing a great job :)
A full post-mortem is coming, but here's what happened from my perspective. I got mentioned on twitter about this thread while I was on the bus. I asked @evanphx to put the site into maintenance mode immediately.
Once I got on wifi, I downloaded the "exploit" gems and found that they used the Psych hack to post config/initializers/* and config/*.yml to Pastie. Luckily, none of our API keys were actually in those files - our config is a bit different since you can still run RubyGems.org on Heroku, which needs to put secret keys in the ENV.
I deleted the exploit gems permanently along with the throwaway account that posted them...that's why this post's URL is a 404.
I've reset our S3 keys just to be sure and any other keys we had on our production site. From what we know right now, no other changes on our S3 bucket have taken place, and we're going to check the logs to make sure. Once we get a real fix for this issue out, pushing gems will be enabled again.
Just a general PSA -
Please, if you find an issue like this, be nice. Tell the maintainers privately. Don't post to Reddit, HN, or a public Gist. RubyGems.org is completely volunteer run. No one gets paid to work on it. Thanks for your patience everyone.