Casting Caution to the Wind (Ch. 5, Sec. 9) [Securing Java]

Attack Applets: Exploiting Holes in the Security Model
CHAPTER SECTIONS: 1 / 2 / 3 / 4 / 5 / 6 / 7 / 8 / 9 / 10 / 11 / 12 / 13 / 14 / 15 / 16 / 17 / 18 / 19 / 20

Section 9 -- Casting Caution to the Wind

Software consultant Tom Cargill has discovered two security flaws related to the way in which Java handles interface types. Both flaws involve a rare case in which Java fails to check whether a method is private. Both also use type-casting operations on Java's interface types. By exploiting these flaws, an attacker can call private methods normally prohibited by Java's security rules. Since some of the security-critical values inside the Java system are protected by private methods, a complete security breach using this attack is possible.

Simple Interface Casting

The core of Cargill's first discovery is shown in the following code:

interface Inter {
  void f(); 
}
class Secure implements Inter {
  private void f(); 
}
class Dummy extends Secure implements Inter {
  public void f();
  Dummy() {
     Secure s = new Secure();
     Inter i = (Inter) s;
     i.f();          // should be illegal
  }
}

This code allows the private f method of class Secure to be called illegally. The Java interpreter fails to determine if f is private when i.f() is called.

The Princeton team figured out how to use this flaw to achieve full system penetration. This was done by exploiting the fix to the class loader bug. The class loader bug was fixed by splitting the critical defineClass method into a private method and a public method. The private method, defineClass0, did the work. The public method checked the initialized flag and called defineClass0 only if the flag was true. Since the private defineClass0 method couldn't be called directly by an applet, this was supposed to fix the class loader bug.

Unfortunately, a variant of the interface-casting trick shown here allows an applet to call the private defineClass0 method directly, bypassing the check. This meant that the attack could create a class loader by exploiting the Verifier bug. The initialized flag would be false, but that wouldn't matter. A programmer could bypass the flag-check by exploiting the interface-casting trick to call the private defineClass0 method directly.

By using this trick, an attacker could gain full system penetration under Netscape Navigator 2.02.

The Full Fix

Netscape fixed this problem in two ways. First, it fixed the flaw in its Java Virtual Machine that allowed the interface-casting trick to work. Second, Netscape began storing and checking the initialized flag inside the Java Virtual Machine, rather than in programmer-generated Java code. Netscape eliminated the dangerous defineClass0 operation by integrating everything into the VM's implementation of defineClass. This change took effect in Navigator 3.0beta3.

In reaction to the interface-casting bug, Netscape changed its Java implementation to protect itself more generally against an attacker who had the ability to call private methods. By going beyond a simple bug fix to improve the structure of the system, Netscape practiced good security engineering. Its decision paid off when the next bug was discovered.

Advanced Interface Casting

Here is the core of Cargill's second discovery:

  interface Inter {
     void f(); 
  }


  class Secure implements Inter {
     private void f(); 
  }


  class Dummy implements Inter {
     public void f();


     static void attack() {
        Inter inter[2] = {new Dummy(), new Secure() };
        for(int j=0; j<2; ++j)     inter[j].f();
     }
  }

The first call, inter[0].f(), is legal since Dummy's f method is public. The next time around the loop, inter[1].f() is illegal since Secure's f method is private.

In this case, Java was too smart for its own good. In order to improve performance, it only checked for legality the first time through the loop. Theoretically, what was legal the first time would be legal the next time. (See Chapter 2.) Though this is often a correct assumption, it broke down for the code just shown.

This trick allows an attacker to call private methods in violation of Java's security rules. Had Netscape not improved the structure of their system after the previous bug was reported, this bug would have once again allowed the class-loader attack to work. However, because Netscape had protected their system against private method attacks, this flaw was not easy to exploit.

Chapter... Preface -- 1 -- 2 -- 3 -- 4 -- 5 -- 6 -- 7 -- 8 -- 9 -- A -- B -- C -- Refs
Front -- Contents -- Help