Tuesday, September 14, 2010

DES Encryption As Used in VNC Authentication

A few notes about how DES is used in the VNC Authentication security type (type 2)

  1. DES is used in ECB mode.
  2. The ECB Key is based upon an ASCII password. The key must be 8 bytes long. The password is either truncated to 8 bytes, or else zeros are added to the end to bring it up to 8 bytes. As an additional twist, each byte in flipped. So, if the ASCII password was "pword" [0x 70 77 6F 72 64], the resulting key would be [0x 0E EE F6 4E 26 00 00 00].
  3. The VNC Authentication scheme sends a 16 byte challenge. This challenge should be encrypted with the key that was just described, but DES in ECB mode can only encrypt an 8 byte message. So, the challenge is split into two messages, encrypted separately, and then jammed back together.
Here is some pseudocode (in Erlang) that should explain better than words can.
password_to_key(Password) ->
    Flipped = lists:map(fun flip_byte/1, Password),
    Truncated = truncate(Flipped, 8),
    pad(Truncated, 8, $\0).

encrypt_challenge(Password, Challenge) ->
    Key = password_to_key(Password),
    <<High:8/binary, Low:8/binary>> = Challenge,
    EncHigh = crypto:des_ecb_encrypt(Key, High),
    EncLow = crypto:des_ecb_encrypt(Key, Low),
    <<EncHigh/binary, EncLow/binary>>.


Anonymous said...

I can't speak for other VNC servers, but at least the UltraVNC server (see http://www.uvnc.com/) uses "CBC" and NOT "ECB" ( see http://en.wikipedia.org/wiki/Block_cipher_modes_of_operation#Electronic_codebook_.28ECB.29 for details) mode without padding.

Anonymous said...

Turns out I wasn't entirely right. The above holds true for UltraVNCs MS-Logon. The ordinary "VNC password" login however uses ECB without an IV (of course).

Pseudo code VNC Authetication:

Read 16byte => $challenge
$A = encrypt( first8bytes( $challenge) , bitflipped( $userpassword), DES/ECB/NoPadding)
$B = encrypt( second8bytes( $challenge), bitflipped( $userpassword), DES/ECB/NoPadding)

send concat( $A, $B)

Pseudo code MS-Logon Authetication:

$key = DiffieHellman( Server).pubKey
$user = encrypt( pad( $username, 256), bitflipped( $key), DES/CBC/NoPadding)
$pwd = encrypt( pad( $userpassword, 64), bitflipped( $key), DES/CBC/NoPadding)

send $user
senr $pwd

DoN said...

You can also mirrored hard secret key of VNC:
vnc_key = [23, 82, 107, 6, 35, 78, 88, 7] in code
vnc_key_for_classic_DES = [232, 74, 214, 96, 196, 114, 26, 224]

If you use common alghorithm of DES encryption (ECB), you should use vnc_key_for_classic_DES without password changing.