A few notes about how DES is used in the VNC Authentication security type (type 2)
- DES is used in ECB mode.
The ECB Key is based upon an ASCII password. The key must be 8 bytes long. The password is either truncated to 8 bytes, or else zeros are added to the end to bring it up to 8 bytes. As an additional twist, each byte in flipped. So, if the ASCII password was "pword"
[0x 70 77 6F 72 64], the resulting key would be
[0x 0E EE F6 4E 26 00 00 00].
- The VNC Authentication scheme sends a 16 byte challenge. This challenge should be encrypted with the key that was just described, but DES in ECB mode can only encrypt an 8 byte message. So, the challenge is split into two messages, encrypted separately, and then jammed back together.
password_to_key(Password) -> Flipped = lists:map(fun flip_byte/1, Password), Truncated = truncate(Flipped, 8), pad(Truncated, 8, $\0). encrypt_challenge(Password, Challenge) -> Key = password_to_key(Password), <<High:8/binary, Low:8/binary>> = Challenge, EncHigh = crypto:des_ecb_encrypt(Key, High), EncLow = crypto:des_ecb_encrypt(Key, Low), <<EncHigh/binary, EncLow/binary>>.