Compromising the integrity of the npm registry.

Recently it was disclosed that the NPM registry leaked the usernames, salts and sha1 hashes of registry users. Essentially this amounts to a breach of about 4k user accounts.

The issue has since been taken care of and users are being asked (not forced) to change their passwords. The leaked data has been available for a very long time, probably since the registry has been using couch. Everyone should be resetting their passwords. Now.

I first found out and notified Isaac about this on 3/1/2012. I only found out about this because I was looking for potential ways that &! could be compromised.

One of the ways we build our development and production environments is by using npm to install packages. I was curious just how hard it would be to compromise the integrity of packages published to the registry, turns out not very. It’s great to point out however that npm is meant to be a distribution channel. It’s a free and open service in which anybody can distribute packages. It’s not meant to provide any level of integrity and quality checking. As developers we are responsible for the code that executes in our environments. Maybe checking verified packages into your projects repository isn’t such a bad idea after all.

It took only 24 hours using an old spare machine to crack 25% of the passwords. Very little effort or CPU power.

Passwords cracked included prominent, well respected members of the node.js community that control publishing rights to widely used packages.

To be clear, this was not done for the sake of gaining access to those passwords and the data has been destroyed and I never tried to log in with any account. It was just a test of how hard it would be to abuse it and thus, what level of actual threat it represented.

There are some great comments on if this was a couch problem or a npm problem in this thread.

During that time I also discovered a number of persistent and reflected cross-site scripting vulnerabilities that were patched in this pull request.

Finally I would like to thank Isaac for taking the time to communicate with me over email about this, keep me updated as things progressed, and most importantly shipping a fix and being transparent with the node community.

I’m very interested in comments on this and appreciate feedback either via email (baldwin@andyet.net) or twitter @adam_baldwin

tl;dr

• Only 24 hours to crack 25% of the user passwords.
• It’s not required for you to reset your password, do so anyway.
• You are responsible for the code you run in production, not NPM.
• Serious thank you to @izs and @_jhs for shipping a fix & being transparent

filed under node.js, npm, and security View Comments

Devops and Security Vodcast: Code Quality & Helpful Tools

&yet’s ops and security guys hash it out in this latest vodcast.

Nathan Lafreniere talks about what’s in his devops toolkit, his code deployment process, how ops can help maintain code quality, and his new documentation library, ape.

Adam Baldwin discusses his new Node.js header security library for express, helmet, a few headers that most apps should be including by default now, and some random bits about realtime security.

Fortunately for you this particular cut doesn’t include Adam singing Russian Unicorn but it does feature a yeti and Adam doing what he would consider dancing.

Please let us know what you would like to hear about in the future regarding ops and security.

Credits:

“Talent”: Nathan (left) and @adam_baldwin (right).

Video filmed and produced by the awesome Ms. Mel.

filed under devops, node.js, process, qa, security, and vodcast View Comments

Adam Baldwin and Nathan LaFreniere are yetis.

Security expert and dev/ops badass join the &yet team January 1

Because we are huge fans of human namespace collisions and amazing people, we’re adding two new members to our team: Adam Baldwin and Nathan LaFreniere, both in transition from nGenuity, the security company Adam Baldwin co-founded and built into a well-respected consultancy that has advised the likes of GitHub, AirBNB, and LastPass on security.

We have relied on Adam and Nathan’s services through nGenuity to inform, improve, and check our development process, validating and invalidating our team’s work and process, providing education and correction along the way. We are thrilled to be able to bring these resources to bear with greater influence, while providing Adam Baldwin with the authority to improve areas in need of such.

Adam Baldwin

Adam Baldwin has served as &yet’s most essential advisor since our first year, providing me with confidence in venturing more into development as an addition to my initial web design freelance business, playing “panoptic debugger” when I struggled with it, helping us establish good policy and process as we built our team, improving our system operations, and always, always, bludgeoning us about the head regarding security.

It really can’t be expressed how much respect I and our team at &yet have for Adam and his work.

He’s uncovered Basecamp vulnerabilities that encouraged 37Signals to change their policies for handling reported vulnerabilities, found huge holes in Sprint/Verizon MiFi (that made for one of the most hilarious stories I’ve been a part of), published vulnerabilities *twice* to root Rackspace, shared research to uberhackers at DEFCON, and has provided security advice for a number of first-class web apps, including ones you’re using today and conceivably right now.

Adam Baldwin will be joining our team at &yet as CSO—it’s a double title: Chief of Software Operations and Chief Security Officer.

Adam will be adding his security consultancy, alongside &yet’s other consulting services, but will also be overseeing our team’s software processes, something he has informed, shaped, and helped externally verify since, I think, before most of our team was born.

On a personal note (a longer version of which is here), I must say it’s a real joy to be able to welcome one of my best friends into helping lead a business he helped build as much as anyone our team.

Nathan LaFreniere

As excited as I am personally to add Adam Baldwin, our dev team is even more thrilled about adding Nathan, whose services we have become well accustomed to relying on in our contract with nGenuity and in a large project where we’ve served a mutual customer.

Nathan is a multitalented dev/ops badass well-versed in automated deployment tools.

He solves operations problems with a combination of experience, innovation, and willingness to learn new tools and approaches.

He’s already gained a significant depth of experience building custom production systems for Node.js, including some tools we’ve come to rely on heavily for &bang.

Nathan’s passion for well-architected, smoothly running, and meticulously monitored servers has helped our developers sleep at night, very literally.

I know getting the luxury of having a huge amount of Nathan’s time at our developers disposal sounds to them like diving into a pool of soft kittens who don’t mind you diving on them and aren’t hurt at all by it either oh and they’re declawed and maybe wear dentures but took them out.

So that’s what we have for you today.

We think you’re gonna love it.

filed under new hires, ops, and security View Comments

Unsafe at Certain Speeds: Dangers Designed into Django

A good software development framework should make the common things easy and make the uncommon things possible.

Unfortunately, Django sometimes makes the simple things easy and the hard things possible — and security is hard!

What Django does well

The Django community does take security very seriously.

The ORM makes it really difficult to expose your app to SQL injection attacks. The template processing system makes it hard to enable cross-site scripting. It takes work to avoid Django’s CSRF protection, and it’d be rare to subvert its well-tested session handling.

Not only that, but Django’s documentation and release notes go the extra mile, discouraging many poor practices and even warning against problems outside of Django that could affect the security of a web app.

Django’s target market

So what’s the problem?

Django has its roots in the publishing industry and got its wings as a basis for sharing-oriented “Web 2.0” sites. When a majority of resources are publicly available, or shared among all logged-in users, it’s possible to focus on securing a few private corners.

What Django’s design considers uncommon is “multitenant” apps — imagine that instead of adding a blog to your company website, you are building a corporate blog–hosting service.

With a single-tenant app, there’s generally some level of trust among all users. Maybe an intern is only supposed to edit customer support documents, but discovers a bug in the custom CMS built on Django that lets him post a funny picture of his boss on the homepage. Sure it was technically the Django-using programmer who let it happen, but it was the intern who betrayed the tenant’s trust.

With multiple tenants, the responsibility of trust is upon the developers. When some computer-savvy ACME Corp employees find a hole that lets them access Wonder Inc’s draft blog posts, they’re just doing their job. If Wonder Inc imagined their exciting product announcements were safe inside your Django app, they won’t care how easy it was to make that security mistake.

What Django makes easy

Most days, most developers are struggling valiantly just to get their code to work. Getting it to “compile”, getting it to “run”, getting it to run “on the production database”. Fixing it to stay running even when a user clicks Y before they click X.

Security is hard because you still have to do all the “just getting it to work”, but you also have to make sure it doesn’t work even if a different user clicks X, fakes Y and then does Z with a little help from A.

Let me make this clear: security mistakes are too common to be a problem of “stupid developers”. Leave the PEBKAC mentality for the poor techs who have to support what they can’t fix — we are developers and designers, busy developing our designs. Engineering, Enforcement and Education are wonderful, but usability is cheaper.

Django’s ORM makes it easy — too easy — to expose database rows to users who shouldn’t have access. It provides a very user-friendly mapping from SQL to model objects. The catch is, the database doesn’t give a rat’s rooty-tooty about your app’s permission model, and neither does the ORM. Its job is to be the floppy disk for your spreadsheets, the ORM’s job is to pretend the spreadsheet rows are documents. Fair enough. But the tools Django provides for validating data access are too difficult to customize for an app where every table is shared among mutually untrusted tenants. Remember that developers are naturally inclined to code until it works for them — not to prove that the same code won’t work when an attacker calls it up.

The template processing and file handling infrastructure encourage developers to expose private user uploads via statically hosted media directories. This is fine for a blog, but when a user notices their private upload got renamed to “/media/user_images/image_______.jpg” they might start figuring out that Apache will gladly let them see “image___.jpg” (and “image.php”!) in that directory too.

Finally, while most of Django’s middleware does enhance web app security, the error debugging system can lead to inadvertent storage of sensitive user data if an exception catches it mid-flight. This issue is being addressed for Django 1.4, although the design is opt-in and may be a bit fragile in practice — but this particular problem is both hard and uncommon. In this last case I suspect the solution being built in is a good enough design.

How to make secure apps more common

That leaves us with Django’s ORM and file handling — which I’m convinced are not good enough designs for a multitenant web app framework.

In a multitenant app it is very common that model lookups and form validation must be contained to a stricter subset of data than Django encourages.

The very best solution to this problem is to partition your app. Give each tenant their own virtual system, their own database — in short, their own copy of the hosted app.Partitioning does take more work to configure up-front, but that’s the best place for investments like that. It also complicates cross-account administration features: which is exactly the point. Make the uncommon use cases the harder ones, so that the normal stuff is securer by default.

If you’re not ready or it’s too late to partition, do your whole team a favor and stop using Django’s ORM and ModelForms directly in a multitenant codebase. You need to write an API and force all your code to use it, instead of the ORM. Django’s views are too presentation-focused. Not the place to expect secure code. When coding up a working user interface, it’s too easy to say “My code needs this object!” when you mean “Some user would like to access this data?”. Give day-to-day development the freedom to wholeheartedly fight For the user. Build an internal Python data access API for the sole purpose of standing between the user request and the ORM or filesystem; a good gate on this border can keep a thousand welcome mats safe.

Whether you partition your app into single-tenant instances or use an API to isolate data access, you should develop tests primarily for security. If a commit breaks functionality, it’s an obvious bug. Someone will complain soon enough. If a code change only adds “functionality” that isn’t supposed to exist, it’s a zero-day. Will you notice the mistake in time?

Interestingly enough, our security tests do tend to catch functionality regressions too, since they really must check that Mallory can’t do something Alice and Bob can.That’s a nice benefit, especially since you’re still updating tests because the app is getting better and its security needs to as well. (Having to maintain tests that only lock in functionality as it continually changes, sucks.)

Focus your programmatic testing efforts on permissions enforcement. Your time is precious — don’t bother with automated tests for anything less valuable than earning trust!

Make boring mistakes hard

Django is a great traditional web framework that makes many customizations easy. It’s possible to build secure multitenant apps using the pieces Django provides, although certain built-in features and certain patterns encouraged by the documentation need to be avoided.

I suspect this is also the case with many other web frameworks. And security might not be the only area where developers’ toolkits make doing things “the wrong way” the easy way.

Pay attention to design decisions at the framework level that distract your team from delivering a great user experience at a higher level.

Avoid shooting yourselves in the foot (feets?) by only picking fights on fronts where the troops will stay engaged. Make solving interesting problems the only uphill battle for your developers. Then level the field for your customers. (That’s what usability is about.)

To see us work it’s not self-evident, but nerds invented computers to avoid tedious mistake-prone work. Like end-users, developers have lives and are busy and are experts only in their own passions. Assume security will be taken for granted by users, and developers alike!

If secure web apps should be common, vulnerable code must be made hard to write. It is a good workman’s responsibility to blame his tools every now and then — occasionally we get something as useful as Django as a result!

filed under django, and security View Comments