Security Through Obscurity

Random thoughts on security and anti-abuse from a bunch of nerds.

Adversarial Cycle


The majority of application security problems stem from software bugs that leave the existing security controls broken. However, even if the code is perfect it doesn’t mean an attacker can’t exploit it. The vulnerability can be hidden inside the business logic, not the code that powers it. This type of vulnerability is called business logic vulnerability. It’s when an attacker abuses a legitimate flow of an application so that it results in negative consequences.

An example could be a contact form on a website that is used for sending out emails to the service owners. This form can be abused to send out spam messages instead of genuine support requests.

When dealing with business logic vulnerabilities we enter a branch of security called anti-abuse. Fighting abuse often requires thinking about these problems more creatively, because unlike patching the bugs in the code the business logic often can’t be changed to mitigate the risks. We're stuck in a land where we can’t really fix the problem, we can only manage it. It requires active participation because an attacker will change the tactics as we put defence mechanisms into the system.

It’s a big and complex topic, and before diving into it let’s concentrate on some theory that will help us think about it more clearly.

One of my favorite papers on the topic of anti-abuse is Facebook Immune System. It proposes a model through which we can view abuse attacks on any system. We can think of it as a cycle where some stages are controlled by attackers and some by defenders.

Adversarial Cycle

The transitions are marked with numbers, let’s enumerate them:

  1. Attack detection
  2. Attack mitigation
  3. Defence detection
  4. Defence mitigation

So the defender directly controls stages 1 and 2, and the attacker directly controls 3 and 4. However both of the adversaries control the entire cycle indirectly.

The objective of the game for the attacker is to make sure that the attack doesn’t get detected. If it does, they need to be able to change the plans fast enough to try a different approach. The life of the defender is the same, but for different transitions. The defender wants to detect the attack and stage the defence as fast as possible.

Translating that to the diagram the attacker wants for transitions 1 and 2 to be as long as possible and transitions 3 and 4 as short as possible. On the other side, the defender wants transitions 3 and 4 take a very short time but stretch out transitions 1 and 2.

So, you may be wondering, how can it be applied in practice?

As a defender, it's easy to fall into the trap of spending too much time on intuitive solutions that don’t make the life of the attacker that much harder. I’ll give you an example.

Let’s say we need to protect the contact form mentioned earlier from sending spam. We see that the attackers are using a bash script and that is reflected in the User-Agent header. Naturally you might think that we should block all attempts that come from this particular User-Agent. Since we don’t want the spam to be sent out we block the attempt right there on the page. However, what just happened is we gave the adversaries a very tight feedback loop to figure out our defence. To circumvent it they just need to spoof the User-Agent to look like a browser. It’s a simple change on their side and they are back in business. Not good for us, now it got harder to detect them.

The attackers are antifragile, which means they get stronger as we fight them. Thinking about anti-abuse in terms of adversarial cycles helps us avoid first-order-thinking and be deliberate when we evaluate consequences of an intuitive solution.

Adversarial cycle is a great tool that enables us to be better at threat modeling. It forces us to think more methodically about costs and benefits of a particular mitigation strategy. Understanding it also makes it easier to grasp other concepts that we’ll be discussing in further posts.