Rumors of this bug began spreading on Slashdot and other sites, thanks
to Steve Wiseman of intelliadmin.com who serendipitously discovered it
while writing a VNC client. At first it was only a rumor, as Steve's
site gave scant details and he himself was surprised such a huge hole
could possibly exist in such a widely deployed product. Here are the
results of my research into this rumor.

In the interests of full disclosure, the following message details a
critical vulnerability in RealVNC's authentication protocol. Using the
following method, it is trivial to gain access to any RealVNC server
without knowing the password. This allows full control of the target
machine, with privilege levels equalling that of the user under which
the RealVNC server runs - often full Administrator access on Windows
desktops.

RealVNC is a widely used program which "makes it possible to view and
fully-interact with one computer from any other computer or mobile
device anywhere on the Internet." (www.realvnc.com) As documented in
rfbproto.pdf by Tristan Richardson, the RFB (remote frame buffer)
protocol performs an initial handshake which allows clients and
servers to negotiate appropriate authentication measures. There are
several methods of authentication, including the standard DES
Challenge-Response, as well as an option to disable authentication
completely. Due to an incorrect implementation, clients are able to
force the server to disable authentication, and allow login without a
password.

Technical details:

1) Server sends its version, "RFB 003.008\n"
2) Client replies with its version, "RFB 003.008\n"
3) Server sends 1 byte which is equal to the number of security types offered
3a) Server sends an array of bytes which indicate security types offered
4) Client replies with 1 byte, chosen from the array in 3a, to select
the security type
5) The handshake, if requested, is performed, followed by "0000" from the server

In RealVNC 4.1.1 and possibly prior versions which implement RFB
003.008 (though not RealVNC 4.0), the server does NOT perform a check
to determine if the byte sent by the client in step 4 has actually
been offered by the server in step 3a. In effect, authentication is
moved from the server side to the client side. It is possible to force
your client to simply request "Type 1 - None" as the security type,
and gain access to the server without having to go through the time
consuming and cumbersome password entry field.

Here is a typical packet dump:

Server -> Client: 52 46 42 20 30 30 33 2e 30 30 38 0a <- Server version
Client -> Server: 52 46 42 20 30 30 33 2e 30 30 38 0a <- Client version
Server -> Client: 01 02 <- One field follows... and that field is 02
(DES Challenge)
Client -> Server: 01 <- Ahh, the lovely 1 byte exploit! Beautiful, isn't it?
Server -> Client: 00 00 00 00 <-- Authenticated!

Modifying the RealVNC client to exploit this is simple, and other
clients can be modified as well. Such exercises, however, are best
left to the skilled reader. To all admins, you are reminded to run
services like these behind firewalls and through SSH tunnels.

And now a very important message...

RealVNC is distributed under the GNU General Public License. As such,
the complete source code of RealVNC *must* be freely distributed. When
RealVNC (the company) received notice of this flaw in their software,
they were quite prompt in patching it. Such action is normally worthy
of praise. Yet, in this case, RealVNC immediately took down the source
code to their software. While this was probably done out of fear
rather than malice, I believe it violates both the spirit and law of
the GNU GPL. As we can see from the above, it is also not beneficial
to security. I was able to rediscover this flaw using only binaries,
and a little thought. Allowing for the benefit of doubt, I posted to
the RealVNC mailing list, congratulating them on patching the bug so
quickly and asking when the source code would be released. I received
one reply from another user, agreeing that he would like to see the
source, as it is under GPL. Upon returning the next day to check if
there were any more replies, I was surprised to see the entire mailing
list was deleted along with its archives. This is unfortunate, and it
clearly neither prevents discussion nor promotes security.

Best,
James Evans