regne || writeups

BHack 2022 - Ultimate Encryptor - Reverse Engineering a Custom Cipher

This challenge provided an encrypted flag file and the binary that produced it. I used Ghidra to decompile the program and inspect both the assembly and the reconstructed C code. After reading through it, I identified three important functions:

  • read_file(): reads a file and returns its contents as a string
  • UltimateEncryption(): encrypts the message and returns the result
  • main():
    1. calls read_file() with the first argument, which is the plaintext file;
    2. passes that content to UltimateEncryption();
    3. saves the encrypted output to the second file argument.

After some debugging, variable renaming, and cleanup, I reconstructed the core function into something like this:

char* UltimateEncryption(char* msg) {
  char tmp, zeroChar;
  int index4, index3, index2, index, keyLen;

  size_t msgLen = strlen(msg);
  keyLen = (int)msgLen;

  // Step 0
  if ((msgLen & 1) != 0) { // Check if is Odd or Even with Bitwise Operators
    zeroChar = '0';
    strncat(msg, (char*)&zeroChar, 2);
    keyLen = keyLen + 1;
  }
  printf("%s\n", msg); // print to debug

  // Step 1
  for (index = 0; index < keyLen; index = index + 1) {
    msg[index] = msg[(index + 1) % keyLen] ^ msg[index];
  }
  printf("%s\n", msg); // print to debug

  // Step 2
  for (index2 = 0; index2 < keyLen; index2 = index2 + 1) {
    msg[index2] = msg[index2] ^ 0x2b;
  }
  printf("%s\n", msg); // print to debug

  // Step 3
  for (index3 = 0; index3 < keyLen; index3 = index3 + 1) {
    msg[index3] = msg[index3] + '\x0f';
  }
  printf("%s\n", msg); // print to debug
  
  // Step 4
  for (index4 = 0; index4 < keyLen; index4 = index4 + 2) {
    tmp = msg[index4];
    msg[index4] = msg[(long)index4 + 1];
    msg[(long)index4 + 1] = tmp;
  }
  printf("%s\n", msg); // print to debug
  
  return msg;
}

To make the analysis easier, I copied the raw main() and read_file() functions from Ghidra into my local encrypt.c file along with the function above. Then I commented out the loops one by one so I could understand what each stage was doing. My conclusions were:

  • Step 0: if the string length was odd, append '0' at the end;
  • Step 1: XOR each byte with the next byte, wrapping the last one to the first;
  • Step 2: XOR each byte with 0x2b (+ in ASCII);
  • Step 3: add 0x0f to each byte;
  • Step 4: swap characters in pairs.

Once I understood the transformation order, I implemented the reverse process and tested each step independently while encrypting and decrypting. This was the final script:

key = list(open('out.txt', 'rb').read().strip())
plain_size = len(key)

# Step 4
i4 = 0
while i4 < plain_size:
    c = key[i4]
    key[i4] = key[i4 + 1]
    key[i4 + 1] = c
    i4 += 2

# Step 3
for i3 in range(plain_size):
    key[i3] = key[i3] - 0x0f

# Step 2
for i2 in range(plain_size):
    key[i2] = key[i2] ^ 0x2b

# Step 1
index = plain_size-1
while index >=  0:
    key[index] = key[(index + 1) % plain_size] ^ key[index]
    index -= 1

print(''.join(map(chr,key)))
BHackReverse EngineeringCryptography