A few notes about how DES is used in the VNC Authentication security type (type 2)
- DES is used in ECB mode.
-
The ECB Key is based upon an ASCII password. The key must be 8 bytes long. The password is either truncated to 8 bytes, or else zeros are added to the end to bring it up to 8 bytes. As an additional twist, each byte in flipped. So, if the ASCII password was "pword"
[0x 70 77 6F 72 64]
, the resulting key would be[0x 0E EE F6 4E 26 00 00 00]
. - The VNC Authentication scheme sends a 16 byte challenge. This challenge should be encrypted with the key that was just described, but DES in ECB mode can only encrypt an 8 byte message. So, the challenge is split into two messages, encrypted separately, and then jammed back together.
password_to_key(Password) ->
Flipped = lists:map(fun flip_byte/1, Password),
Truncated = truncate(Flipped, 8),
pad(Truncated, 8, $\0).
encrypt_challenge(Password, Challenge) ->
Key = password_to_key(Password),
<<High:8/binary, Low:8/binary>> = Challenge,
EncHigh = crypto:des_ecb_encrypt(Key, High),
EncLow = crypto:des_ecb_encrypt(Key, Low),
<<EncHigh/binary, EncLow/binary>>.