Tuesday, September 14, 2010

DES Encryption As Used in VNC Authentication

A few notes about how DES is used in the VNC Authentication security type (type 2)

  1. DES is used in ECB mode.
  2. The ECB Key is based upon an ASCII password. The key must be 8 bytes long. The password is either truncated to 8 bytes, or else zeros are added to the end to bring it up to 8 bytes. As an additional twist, each byte in flipped. So, if the ASCII password was "pword" [0x 70 77 6F 72 64], the resulting key would be [0x 0E EE F6 4E 26 00 00 00].
  3. The VNC Authentication scheme sends a 16 byte challenge. This challenge should be encrypted with the key that was just described, but DES in ECB mode can only encrypt an 8 byte message. So, the challenge is split into two messages, encrypted separately, and then jammed back together.
Here is some pseudocode (in Erlang) that should explain better than words can.
password_to_key(Password) ->
    Flipped = lists:map(fun flip_byte/1, Password),
    Truncated = truncate(Flipped, 8),
    pad(Truncated, 8, $\0).

encrypt_challenge(Password, Challenge) ->
    Key = password_to_key(Password),
    <<High:8/binary, Low:8/binary>> = Challenge,
    EncHigh = crypto:des_ecb_encrypt(Key, High),
    EncLow = crypto:des_ecb_encrypt(Key, Low),
    <<EncHigh/binary, EncLow/binary>>.