XOR Encryption with Anti-Debug Protection

Medium

October 27, 2025

#ReverseEngineering#XOR#AntiDebug#Ghidra#BinaryAnalysis#Crackme

Problem

The Challenge

This challenge from Root Me presents a classic crackme binary with a twist: it implements XOR encryption combined with an anti-debugging mechanism. The challenge description warns "Don't be discouraged by this poor binary" and hints that "there may be traps!"

Like all crackmes, the goal is to find the key that passes the validation check. However, this binary employs runtime code modification that only activates when the program is NOT being debugged, making dynamic analysis potentially misleading.

OBJECTIVE

Primary Objective: Reverse engineer a Linux ELF64 binary to recover the password that successfully passes validation, despite the presence of anti-debugging protection and XOR encryption.

TARGETS

Password/Key Validation

Anti-Debug Protection Mechanism

Challenge Details

Platform: Root Me
Category: Cracking
Points: 25
Validations: 407 (1% of challengers)

Reconnaissance

My approach began with static analysis using Ghidra, a powerful open-source reverse engineering framework developed by the NSA. Static analysis allows us to examine the binary's code without executing it, which is crucial when dealing with anti-debugging protections.

Initial Binary Inspection

BINARY INFO

FORMAT

ELF64

ARCHITECTURE

x86-64

PROTECTIONS NONE

✗ None✗ No PIE✗ No Stack Canary

After loading the binary into Ghidra and running the auto-analysis, I identified several key functions in the symbol tree:

main() - Entry point that prompts for user input
check() - Core password validation function
init() - Buffer initialization routine
__do_global_ctors_aux() - Constructor function (suspicious!)

Analyzing the Validation Logic

The check() function contained the password validation logic. At address 0x00101282, I found the critical comparison:

Code (C (Decompiled))

if (*param_1 == -0x5c8852a8fb9a0207) {
    puts("C'est correct !");
}

This checks if the first 8 bytes of the encrypted input match a specific target value. The function also revealed references to an XOR key stored at address 0x00104048:

Encryption Key Location

BYTE VIEW

ADDR 0x00104048

VALUE 0x950A943E7F4F96A8

XOR Key

The XOR Encryption Routine

Between addresses 0x00101221 and 0x00101280, I discovered a classic repeating-key XOR cipher implementation:

Code (C (Decompiled))

while (true) {
    sVar1 = strlen((char *)param_1);
    if (sVar1 <= (ulong)(long)local_1c) break;

    *(byte *)((long)param_1 + (long)local_1c) =
        *(byte *)((long)param_1 + (long)local_1c) ^
        *(byte *)((long)&key + (long)(local_1c % 8));

    local_1c = local_1c + 1;
}

This code XORs each byte of the user input with the corresponding byte of the 8-byte key, cycling through the key repeatedly for longer inputs. XOR encryption is symmetric - the same operation both encrypts and decrypts data.

Discovery of Anti-Debug Mechanism

The real trap revealed itself in __do_global_ctors_aux(), a constructor function that executes before main(). At addresses 0x00101189 to 0x001011a1, I found this assembly code:

x86-64

ptrace Anti-Debug Check @ 0x00101189



            MOV    EAX, 0x65      ; syscall number for ptrace
                


            MOV    EDI, 0x0       ; PTRACE_TRACEME
                


            MOV    ESI, 0x0
                


            MOV    EDX, 0x0
                


            SYSCALL

The program uses the ptrace system call with PTRACE_TRACEME to detect debuggers. This syscall has a clever property:

• When executed normally: returns 0 (success)
• When executed under a debugger: returns -1 (failure)

The Password Modification Trap

Immediately following the ptrace check, at addresses 0x001011c2 to 0x001011f6, I discovered runtime memory modification:

Code (C (Decompiled))

if (*param_1 == -0x6af56bc180b06958) {
    *param_1 = *param_1 ^ 0x119011901190119;
}

CRITICAL FINDING: The target validation value stored at data_start is XORed with 0x119011901190119 BEFORE the validation check runs - but ONLY during normal execution (not when debugged)!

This means there are effectively two different target values depending on the execution context:

Target Value Transformation

BYTE VIEW

ADDR Static

VALUE -0x6af56bc180b06958

Original (in debugger)

ADDR Runtime

VALUE -0x5c8852a8fb9a0207

Modified (normal execution)

Key Findings

The reconnaissance phase revealed the complete attack surface:

• XOR Encryption: Repeating-key XOR with an 8-byte key
• Anti-Debug Detection: ptrace syscall detects debugger presence
• Dynamic Code Modification: Target value is modified at runtime only during normal execution
• Hardcoded Secrets: Both XOR key and target values visible in static analysis
• Reversible Protection: XOR is self-inverse, making decryption straightforward

Attack & Steps

Understanding the XOR Chain

To recover the original password, I needed to reverse two XOR operations in the correct order:

Code (Pseudo-code)

# Operation 1: Anti-debug XOR (runtime modification)
modified_value = original_value ^ 0x119011901190119

# Operation 2: Password encryption XOR
encrypted = password ^ key

# Final comparison in check()
if (encrypted == modified_value) { success }

Since XOR is self-inverse (A ^ B ^ B = A), I could reverse both operations by applying the same XOR operations again:

Code (Pseudo-code)

# Step 1: Reverse anti-debug XOR
original_memory = encrypted ^ 0x119011901190119

# Step 2: Reverse encryption XOR
password = original_memory ^ key

Building the Decryption Script

I created a Python script to automate the password recovery process. The script performs both XOR operations in reverse order to recover the plaintext password:

Code (Python)

#!/usr/bin/env python3

def recover_password():
    # Target encrypted value from check() function
    encrypted = -0x5c8852a8fb9a0207
    encrypted_unsigned = encrypted & 0xFFFFFFFFFFFFFFFF

    # XOR key used in check()
    key = 0x950A943E7F4F96A8

    # Anti-debug modification value
    anti_debug_xor = 0x119011901190119

    # Step 1: Reverse the anti-debug XOR
    original_memory = encrypted_unsigned ^ anti_debug_xor

    # Step 2: Reverse the encryption XOR
    password_int = original_memory ^ key

    # Convert to bytes (little-endian)
    password_bytes = password_int.to_bytes(8, byteorder='little')

    # Decode as ASCII
    password_str = password_bytes.decode('ascii').rstrip('\x00')

    return password_str

if __name__ == "__main__":
    password = recover_password()
    print(f"Recovered password: {password}")

The script handles several important details:

• Converts signed integer to unsigned with bit masking
• Applies XOR operations in reverse order
• Converts the result to bytes using little-endian byte order (x86-64 default)
• Decodes as ASCII and strips null bytes

Executing the Decryption

I ran the script to recover the password:

Execute Decryption Script

python3 decrypt.py

Code (Output)

Recovered password: [REDACTED_FLAG]

Success! The script revealed the password that would pass validation.

Validating the Solution

With the recovered password, I executed the binary and provided the correct input:

Execute Challenge Binary

./ch60

Code (Output)

Key: [REDACTED_FLAG]
C'est correct !

The program validated the password and displayed the success message, confirming that my reverse engineering and decryption process was correct!

Solution & Root Cause

HIGH

Root Cause Analysis

The binary's protection mechanism suffers from fundamental cryptographic and security design flaws that make it trivially breakable through static analysis alone.

Hardcoded Cryptographic Material: Both the XOR key (0x950A943E7F4F96A8) and target comparison value are embedded as plaintext constants in the binary
Weak Encryption Algorithm: XOR encryption with a static key is symmetric and trivially reversible
Ineffective Anti-Debug: Static analysis bypasses the ptrace protection entirely
No Obfuscation: All security-critical values are immediately visible in a decompiler
Predictable Logic: The validation and encryption routines follow standard patterns easily recognized by reverse engineers

Attack Chain Breakdown

The successful cracking followed this path:

Code (Attack Flow)

Static Analysis with Ghidra
    ↓
Identify check() Function
    ↓
Extract XOR Key (0x950A943E7F4F96A8)
    ↓
Discover Anti-Debug Mechanism in Constructor
    ↓
Extract Anti-Debug XOR Value (0x119011901190119)
    ↓
Extract Target Encrypted Value (-0x5c8852a8fb9a0207)
    ↓
Reverse Both XOR Operations
    ↓
Recover Plaintext Password
    ↓
Validate Solution

Why the Anti-Debug Protection Failed

While the anti-debug mechanism successfully detects debuggers and modifies the target value at runtime, it has critical weaknesses:

• Static Analysis Immunity: Tools like Ghidra analyze the binary without executing it, completely bypassing runtime checks
• Visible Transformation: Both the original and modified values are visible in the disassembly
• Reversible Operation: The XOR modification can be reversed just as easily as the original encryption
• No Code Obfuscation: The ptrace call and modification logic are clearly visible in the decompiled output

Anti-debugging techniques like ptrace primarily slow down dynamic analysis (debugging), but provide zero protection against static analysis. A skilled reverse engineer can extract all necessary values without ever executing the binary.

The Mathematics of XOR Reversal

XOR encryption's fundamental weakness lies in its symmetry. Given the operation:

Code (Formula)

encrypted = plaintext ^ key

We can recover the plaintext by XORing again with the same key:

Code (Mathematical Proof)

plaintext = encrypted ^ key

Proof:
encrypted ^ key = (plaintext ^ key) ^ key
                = plaintext ^ (key ^ key)
                = plaintext ^ 0
                = plaintext

This property makes XOR unsuitable for protecting secrets when the key is known or discoverable.

Remediation

How to Properly Protect Password Validation

For production software, password validation should follow cryptographic best practices, not rely on obfuscation or reversible encryption.

Fix 1: Use Cryptographic Hash Functions

1 Replace XOR encryption with one-way hash functions (SHA-256, SHA-512)
2 Store only the hash of the correct password, never the plaintext
3 Hash the user input and compare hashes, not plaintext values
4 Use constant-time comparison to prevent timing attacks

Code (C)

// WRONG (Current Implementation)
if (*param_1 == encrypted_value) { ... }

// CORRECT (Cryptographic Hash)
#include <openssl/sha.h>

unsigned char hash[SHA256_DIGEST_LENGTH];
unsigned char stored_hash[SHA256_DIGEST_LENGTH] = { /* precomputed */ };

SHA256((unsigned char*)user_input, strlen(user_input), hash);

if (memcmp(hash, stored_hash, SHA256_DIGEST_LENGTH) == 0) {
    puts("C'est correct !");
}

Fix 2: Add Salt and Use Key Derivation Functions

1 Add a random salt to prevent rainbow table attacks
2 Use modern KDFs like bcrypt, scrypt, or Argon2
3 Make brute-force attacks computationally expensive
4 Never use fast hashes (MD5, SHA-1) for password storage

Code (C++)

// Using bcrypt (recommended for password storage)
#include <bcrypt/BCrypt.hpp>

const char salt[16] = { /* random salt */ };
char hash_output[61];

// Hash the user input with bcrypt (work factor: 12)
bcrypt::generateHash(user_input, salt, 12, hash_output);

// Compare with stored hash
if (bcrypt::validatePassword(user_input, stored_bcrypt_hash)) {
    puts("C'est correct !");
}

Fix 3: Remove Hardcoded Secrets

1 Never embed sensitive values directly in executable code
2 Load secrets from secure external storage (encrypted files, HSM, key vault)
3 Use environment variables for configuration secrets
4 Implement proper key management practices

Code (C)

// Instead of hardcoded key:
// const uint64_t key = 0x950A943E7F4F96A8;  // BAD!

// Load from secure storage:
unsigned char stored_hash[32];
FILE* hash_file = fopen("/etc/app/password.hash", "r");
fread(stored_hash, 1, 32, hash_file);
fclose(hash_file);

Fix 4: Reconsider Anti-Debug Techniques

While anti-debugging can be part of a defense-in-depth strategy for sensitive applications, it should not be the primary security mechanism:

1 Understand that anti-debug does NOT prevent static analysis
2 Use code obfuscation to make reverse engineering harder (but not impossible)
3 Implement multiple layers of protection, not a single mechanism
4 Focus on cryptographically sound primitives as the foundation
5 Consider that skilled reverse engineers can bypass any anti-debug technique

For CTF challenges and learning exercises, anti-debug techniques are educational. For production security, they provide minimal value compared to proper cryptography.

Secure Implementation Example

Here's how the validation should be implemented properly:

Code (C)

#include <sodium.h>  // libsodium for modern crypto

#define PASSWORD_HASH_SIZE crypto_pwhash_STRBYTES

int check_password(const char* user_input) {
    // Stored hash (computed offline with crypto_pwhash_str)
    const char stored_hash[PASSWORD_HASH_SIZE] =
        "$argon2id$v=19$m=65536,t=2,p=1$...";

    // Verify password using Argon2id
    if (crypto_pwhash_str_verify(stored_hash,
                                  user_input,
                                  strlen(user_input)) == 0) {
        puts("C'est correct !");
        return 1;
    }

    puts("Incorrect password");
    return 0;
}

// To generate the hash (done offline):
// char hash[crypto_pwhash_STRBYTES];
// crypto_pwhash_str(hash, password, strlen(password),
//                   crypto_pwhash_OPSLIMIT_INTERACTIVE,
//                   crypto_pwhash_MEMLIMIT_INTERACTIVE);

This implementation uses Argon2id (via libsodium), which:

• Is a modern, memory-hard key derivation function
• Makes brute-force attacks computationally expensive
• Includes built-in salt generation and management
• Is resistant to GPU/ASIC acceleration attacks
• Won the Password Hashing Competition (2015)

Key Learnings

◆ KEY INSIGHT

XOR is not encryption for security purposes. When used with a static, discoverable key, XOR provides zero cryptographic security. It's suitable for data obfuscation or error detection, but never for protecting secrets. Real security requires proper cryptographic primitives like hash functions and key derivation functions.

Key Takeaways

Static Analysis Wins: Tools like Ghidra can extract all secrets from unobfuscated binaries without execution
XOR ≠ Encryption: Repeating-key XOR is trivially reversible and unsuitable for security
Anti-Debug ≠ Security: Runtime protections like ptrace don't prevent static analysis
Hardcoding Kills Security: Embedded cryptographic material is visible to any reverse engineer
Use Proper Crypto: Modern KDFs (bcrypt, Argon2) make password cracking computationally infeasible
Defense in Depth: Multiple weak protections don't equal one strong protection
Constructor Functions Matter: Always examine __init and constructor functions for hidden logic

Reverse Engineering Techniques Learned

• Ghidra Workflow: Load binary → Auto-analyze → Examine symbol tree → Decompile functions
• Finding Crypto Keys: Look for large constants in data sections and function parameters
• Identifying Anti-Debug: Search for ptrace, SIGTRAP, timing checks, and debugger detection patterns
• Understanding Constructors: Examine __do_global_ctors_aux() for pre-main execution
• XOR Pattern Recognition: Identify XOR loops by the ^ operator and byte-by-byte iteration
• Memory Value Tracking: Follow data references to understand value transformations

Tools & Resources

→ Ghidra - Free and open-source reverse engineering framework
→ IDA Free - Industry-standard disassembler (free version)
→ Radare2 - Open-source reverse engineering toolkit
→ libsodium - Modern cryptographic library (NaCl fork)
→ Crackmes.one - Collection of reverse engineering challenges
→ Root Me - Hacking and cybersecurity training platform
→ Compiler Explorer - See assembly output from C/C++ code

XOR Encryption with Anti-Debug Protection

Problem

The Challenge

Challenge Details

Reconnaissance

Initial Binary Inspection

Analyzing the Validation Logic

The XOR Encryption Routine

Discovery of Anti-Debug Mechanism

The Password Modification Trap

Key Findings

Attack & Steps

Understanding the XOR Chain

Building the Decryption Script

Executing the Decryption

Validating the Solution

Solution & Root Cause

Root Cause Analysis

Attack Chain Breakdown

Why the Anti-Debug Protection Failed

The Mathematics of XOR Reversal

Remediation

How to Properly Protect Password Validation

Fix 1: Use Cryptographic Hash Functions

Fix 2: Add Salt and Use Key Derivation Functions

Fix 3: Remove Hardcoded Secrets

Fix 4: Reconsider Anti-Debug Techniques

Secure Implementation Example

Key Learnings

Key Takeaways

Reverse Engineering Techniques Learned

Tools & Resources

Further Reading