Protocol/DOCS/EQPACKET.md

EQPacket System Documentation

Table of Contents

1. Overview
2. Architecture
3. Packet Class Hierarchy
4. Packet Types and Opcodes
5. Packet Processing Pipeline
6. Compression and Encryption
7. Packet Combining
8. CRC Validation
9. Serialization Formats
10. Memory Management
11. Debug and Diagnostic Tools

Overview

The EQPacket system is the core packet handling infrastructure for the EverQuest 2 network protocol. It provides a hierarchical class structure for managing different packet types, from low-level protocol packets to high-level application messages. The system handles packet construction, serialization, compression, encryption, validation, and debugging.

Key Features

• Hierarchical packet types - Protocol, Application, and EQ2-specific packets
• Automatic opcode management - Conversion between emulator and network opcodes
• Compression support - Zlib and simple encoding
• Packet combining - Efficient bundling of multiple packets
• CRC validation - Integrity checking with CRC16
• Chat encryption - XOR-based rolling key encryption
• Debug utilities - Comprehensive packet dumping and analysis

Architecture

┌─────────────────────────────────────────────────┐
│                    EQPacket                     │
│  Base class for all packet types                │
│  - Raw buffer management                        │
│  - Network information (IP, port)               │
│  - Timestamp tracking                           │
│  - Basic serialization                          │
└────────────┬───────────────────┬────────────────┘
             │                   │
             ▼                   ▼
┌─────────────────────┐ ┌────────────────────────┐
│ EQProtocolPacket    │ │ EQApplicationPacket    │
│ Low-level protocol  │ │ High-level application │
│ - Compression       │ │ - Game opcodes         │
│ - Sequencing        │ │ - Emulator opcodes     │
│ - Reliability       │ │ - Opcode conversion    │
└──────────┬──────────┘ └────────────────────────┘
           │
           ▼
┌─────────────────────┐
│     EQ2Packet       │
│ EQ2-specific packet │
│ - Login opcodes     │
│ - App combining     │
└─────────────────────┘

Packet Class Hierarchy

EQPacket (Base Class)

The foundation class for all packet types, managing raw packet data and metadata.

class EQPacket {
public:
    // Core data
    unsigned char* pBuffer;     // Raw packet data buffer
    uint32 size;               // Size of packet data
    
    // Network information
    uint32 src_ip, dst_ip;     // Source and destination IPs
    uint16 src_port, dst_port; // Source and destination ports
    uint32 priority;           // Processing priority
    timeval timestamp;         // Packet timestamp
    int16 version;            // Protocol version
    
    // Methods
    void DumpRaw(FILE* to = stdout) const;
    const char* GetOpcodeName();
    uint32 Size() const { return size + 2; }  // Total size with opcode
    uint16 GetRawOpcode() const { return opcode; }
    
protected:
    uint16 opcode;            // Packet operation code
};

Key Features:

• Buffer Management: Dynamic allocation and ownership of packet data
• Network Tracking: Complete IP:port addressing for routing
• Timestamp Support: Precise timing for latency and ordering
• Version Control: Protocol version tracking for compatibility

EQProtocolPacket

Handles low-level protocol operations including compression, encryption, and reliability.

class EQProtocolPacket : public EQPacket {
public:
    // State flags
    bool eq2_compressed;      // Packet is compressed
    bool packet_prepared;     // Ready for transmission
    bool packet_encrypted;    // Packet is encrypted
    bool acked;              // Acknowledgment received
    
    // Reliability tracking
    int32 sent_time;         // When packet was sent
    int8 attempt_count;      // Retransmission attempts
    int32 sequence;          // Sequence number
    
    // Core operations
    bool combine(const EQProtocolPacket* rhs);
    uint32 serialize(unsigned char* dest, int8 offset = 0) const;
    EQApplicationPacket* MakeApplicationPacket(uint8 opcode_size = 0) const;
    
    // Static utilities
    static bool ValidateCRC(const unsigned char* buffer, int length, uint32 Key);
    static uint32 Decompress(const unsigned char* buffer, uint32 length, 
                            unsigned char* newbuf, uint32 newbufsize);
    static uint32 Compress(const unsigned char* buffer, uint32 length,
                          unsigned char* newbuf, uint32 newbufsize);
    static void ChatDecode(unsigned char* buffer, int size, int DecodeKey);
    static void ChatEncode(unsigned char* buffer, int size, int EncodeKey);
};

Key Features:

• Compression: Zlib for large packets, simple encoding for small
• Reliability: Sequence numbers, timestamps, acknowledgments
• Packet Combining: Bundle multiple packets for efficiency
• CRC Validation: Integrity checking with configurable keys
• Chat Encryption: XOR-based rolling key encryption

EQApplicationPacket

Manages application-level game packets with emulator opcode abstraction.

class EQApplicationPacket : public EQPacket {
public:
    static uint8 default_opcode_size;  // Global default (usually 2)
    
    // Constructors for different use cases
    EQApplicationPacket();                                      // Empty packet
    EQApplicationPacket(EmuOpcode op);                         // Opcode only
    EQApplicationPacket(EmuOpcode op, uint32 len);            // Pre-allocated
    EQApplicationPacket(EmuOpcode op, const unsigned char* buf, uint32 len);
    
    // Operations
    bool combine(const EQApplicationPacket* rhs);
    uint32 serialize(unsigned char* dest) const;
    EQApplicationPacket* Copy() const;
    
    // Opcode management
    void SetOpcode(EmuOpcode op);
    EmuOpcode GetOpcode();              // Caching version
    const EmuOpcode GetOpcode() const;  // Const version
    
protected:
    EmuOpcode emu_opcode;  // Cached emulator opcode
    
private:
    uint8 app_opcode_size; // Opcode size (1 or 2 bytes)
};

Key Features:

• Opcode Abstraction: Automatic conversion between emulator and network opcodes
• Flexible Construction: Multiple constructors for different scenarios
• Deep Copy Support: Safe packet duplication with memory management
• Variable Opcode Size: Support for 1-byte and 2-byte opcodes

EQ2Packet

EverQuest 2 specific packet handling with login opcodes and advanced combining.

class EQ2Packet : public EQProtocolPacket {
public:
    EmuOpcode login_op;  // EQ2 login/application opcode
    
    // EQ2-specific operations
    bool AppCombine(EQ2Packet* rhs);
    int8 PreparePacket(int16 MaxLen);
    const char* GetOpcodeName();
    EQ2Packet* Copy();
};

Key Features:

• Login Opcodes: Special handling for login server packets
• Application Combining: EQ2-specific packet bundling protocol
• Packet Preparation: Convert emulator format to network format

Packet Types and Opcodes

Protocol Opcodes

Low-level protocol control packets:

enum ProtocolOpcodes {
    OP_SessionRequest      = 0x0001,  // Initialize new session
    OP_SessionResponse     = 0x0002,  // Accept session
    OP_Combined           = 0x0003,  // Multiple packets bundled
    OP_SessionDisconnect  = 0x0005,  // Terminate session
    OP_KeepAlive         = 0x0006,  // Heartbeat/ping
    OP_SessionStatRequest = 0x0007,  // Request statistics
    OP_SessionStatResponse = 0x0008,  // Statistics reply
    OP_Packet            = 0x0009,  // Data packet
    OP_Fragment          = 0x000D,  // Fragmented packet piece
    OP_OutOfOrderAck     = 0x0011,  // Out-of-order acknowledgment
    OP_Ack               = 0x0015,  // Standard acknowledgment
    OP_AppCombined       = 0x0019,  // Application-level combining
    OP_OutOfSession      = 0x001D   // Not in session error
};

Emulator Opcodes

High-level game opcodes (partial list):

enum EmuOpcodes {
    OP_Unknown           = 0x0000,  // Unknown/invalid opcode
    OP_LoginRequestMsg   = 0x0001,  // Client login request
    OP_LoginByNumRequestMsg = 0x0002, // Login by number
    OP_WSLoginRequestMsg = 0x0003,  // World server login
    OP_ESLoginRequestMsg = 0x0004,  // Entity server login
    OP_LoginReplyMsg     = 0x0005,  // Login response
    OP_WSStatusReplyMsg  = 0x0006,  // World status
    // ... hundreds more game-specific opcodes
};

Opcode Conversion

The system maintains bidirectional mappings between emulator and network opcodes:

// Conversion flow:
// Game Logic → EmuOpcode → Network Opcode → Wire Format
// Wire Format → Network Opcode → EmuOpcode → Game Logic

// Example:
SetOpcode(OP_LoginRequestMsg);  // Set emulator opcode
// Internally converts to network opcode 0x00B3 for version 1193

Packet Processing Pipeline

1. Packet Reception

// Network → Protocol Packet
unsigned char buffer[MAX_PACKET_SIZE];
int bytes_received = recv(socket, buffer, MAX_PACKET_SIZE, 0);

// Create protocol packet from raw data
EQProtocolPacket* proto = new EQProtocolPacket(buffer, bytes_received);

2. Decompression

// Check for compression flags
if (buffer[offset] == 0x5a) {  // Zlib compressed
    unsigned char decompressed[MAX_PACKET_SIZE];
    uint32 new_len = EQProtocolPacket::Decompress(
        buffer, bytes_received, 
        decompressed, MAX_PACKET_SIZE
    );
    // Process decompressed data
} else if (buffer[offset] == 0xa5) {  // Simple encoding
    // Remove encoding flag
    memmove(buffer + offset, buffer + offset + 1, length - offset - 1);
}

3. CRC Validation

// Validate packet integrity
bool valid = EQProtocolPacket::ValidateCRC(buffer, length, crc_key);
if (!valid) {
    // Handle corruption - request retransmission
    SendAck(sequence, false);  // NAK
    return;
}

4. Protocol to Application Conversion

// Convert protocol packet to application packet
EQApplicationPacket* app = proto->MakeApplicationPacket();
if (app) {
    // Extract emulator opcode
    EmuOpcode op = app->GetOpcode();
    
    // Process based on opcode
    switch (op) {
        case OP_LoginRequestMsg:
            HandleLoginRequest(app);
            break;
        // ... handle other opcodes
    }
}

5. Packet Transmission

// Create application packet
EQApplicationPacket* response = new EQApplicationPacket(
    OP_LoginReplyMsg, 
    response_data, 
    response_size
);

// Convert to protocol packet for transmission
EQProtocolPacket* proto_out = CreateProtocolPacket(response);

// Compress if beneficial
if (proto_out->size > 30) {
    unsigned char compressed[MAX_PACKET_SIZE];
    uint32 comp_len = EQProtocolPacket::Compress(
        proto_out->pBuffer, proto_out->size,
        compressed, MAX_PACKET_SIZE
    );
    // Update packet with compressed data
}

// Add CRC
uint16 crc = CRC16(proto_out->pBuffer, proto_out->size, crc_key);
*(uint16*)(proto_out->pBuffer + proto_out->size) = htons(crc);

// Send to network
send(socket, proto_out->pBuffer, proto_out->size + 2, 0);

Compression and Encryption

Compression Methods

The system uses two compression strategies based on packet size:

Zlib Compression (packets > 30 bytes)

uint32 EQProtocolPacket::Compress(...) {
    if (length > 30) {
        // Use zlib deflate for larger packets
        newlength = Deflate(
            buffer + flag_offset,      // Skip opcode
            length - flag_offset,       // Data size
            newbuf + flag_offset + 1,   // Leave room for flag
            newbufsize
        );
        *(newbuf + flag_offset) = 0x5a;  // Zlib flag
        return newlength + flag_offset + 1;
    }
}

Compression Process:

1. Skip opcode bytes (1 or 2 bytes)
2. Insert compression flag (0x5a)
3. Deflate packet data using zlib
4. Append CRC to compressed data

Simple Encoding (packets ≤ 30 bytes)

// For small packets, just add encoding flag
memmove(newbuf + flag_offset + 1, buffer + flag_offset, length - flag_offset);
*(newbuf + flag_offset) = 0xa5;  // Simple encoding flag
return length + 1;  // Original size + flag byte

Benefits:

• Avoids zlib overhead for small packets
• Still marks packet as "encoded" for consistency
• Minimal CPU overhead

Chat Encryption

XOR-based rolling key encryption for chat messages:

void EQProtocolPacket::ChatEncode(unsigned char* buffer, int size, int EncodeKey) {
    int Key = EncodeKey;
    buffer += 2;  // Skip opcode
    size -= 2;
    
    // Process 4-byte blocks with rolling key
    for (int i = 0; i + 4 <= size; i += 4) {
        int pt = (*(int*)&buffer[i]) ^ Key;
        Key = pt;  // Update key with encrypted data
        *(int*)&test[i] = pt;
    }
    
    // Handle remaining bytes
    unsigned char KC = Key & 0xFF;
    for (; i < size; i++) {
        test[i] = buffer[i] ^ KC;
    }
}

Encryption Features:

• Rolling Key: Key updates with each 4-byte block
• Opcode Exemption: First 2 bytes (opcode) remain unencrypted
• Partial Block Handling: Remaining bytes use last key byte
• Bidirectional: Same algorithm for encode/decode (XOR property)

Packet Combining

Protocol-Level Combining

Bundles multiple protocol packets into a single transmission:

bool EQProtocolPacket::combine(const EQProtocolPacket* rhs) {
    // Check size constraints (max 256 bytes total)
    if (opcode == OP_Combined && size + rhs->size + 5 < 256) {
        // Already combined - append new packet
        auto tmpbuffer = new unsigned char[size + rhs->size + 3];
        memcpy(tmpbuffer, pBuffer, size);
        
        // Add size prefix and packet data
        tmpbuffer[size] = rhs->Size();
        rhs->serialize(tmpbuffer + size + 1);
        
        // Update packet
        delete[] pBuffer;
        pBuffer = tmpbuffer;
        size = size + rhs->size + 3;
        return true;
    }
}

Combined Packet Format:

[OP_Combined(2)] [Size1(1)] [Packet1] [Size2(1)] [Packet2] ... [CRC(2)]

Application-Level Combining (EQ2)

More sophisticated combining for application packets:

bool EQ2Packet::AppCombine(EQ2Packet* rhs) {
    // Handle oversized packets (> 255 bytes)
    if (tmp_size >= 255) {
        *ptr++ = 255;  // Oversized marker
        tmp_size = htons(tmp_size);
        memcpy(ptr, &tmp_size, sizeof(int16));
        ptr += sizeof(int16);
    } else {
        *ptr++ = static_cast<uint8>(tmp_size);
    }
}

EQ2 Combined Format:

[OP_AppCombined(2)] [0x00] [0x19] [Size1] [Data1] [Size2] [Data2] ...

Where Size can be:
- 1 byte: size 0-254
- 3 bytes: [0xFF] [Size_High] [Size_Low] for size >= 255

CRC Validation

CRC16 Implementation

The system uses CRC16 for packet integrity:

bool EQProtocolPacket::ValidateCRC(const unsigned char* buffer, int length, uint32 Key) {
    // Exempt packets (no CRC required)
    if (buffer[0] == 0x00 && 
        (buffer[1] == OP_SessionRequest || 
         buffer[1] == OP_SessionResponse || 
         buffer[1] == OP_OutOfSession)) {
        return true;  // Session packets exempt
    }
    
    // Calculate and compare CRC
    uint16 comp_crc = CRC16(buffer, length - 2, Key);
    uint16 packet_crc = ntohs(*(uint16*)(buffer + length - 2));
    
    return (!packet_crc || comp_crc == packet_crc);
}

CRC Properties:

• Algorithm: CRC16-CCITT polynomial
• Key-based: Uses session-specific key for security
• Position: Last 2 bytes of packet
• Exemptions: Session control packets don't require CRC

Serialization Formats

Protocol Packet Serialization

uint32 EQProtocolPacket::serialize(unsigned char* dest, int8 offset) const {
    // Handle variable opcode sizes
    if (opcode > 0xff) {
        *(uint16*)dest = opcode;  // 2-byte opcode
    } else {
        *(dest) = 0;              // High byte = 0
        *(dest + 1) = opcode;     // Low byte = opcode
    }
    
    // Copy packet data
    memcpy(dest + 2, pBuffer + offset, size - offset);
    return size + 2;
}

Application Packet Serialization

uint32 EQApplicationPacket::serialize(unsigned char* dest) const {
    uint8 OpCodeBytes = app_opcode_size;
    
    if (app_opcode_size == 1) {
        *(unsigned char*)dest = opcode;
    } else {
        // Special encoding for opcodes with low byte = 0x00
        if ((opcode & 0x00ff) == 0) {
            *(uint8*)dest = 0;
            *(uint16*)(dest + 1) = opcode;
            ++OpCodeBytes;  // Extra byte needed
        } else {
            *(uint16*)dest = opcode;
        }
    }
    
    memcpy(dest + app_opcode_size, pBuffer, size);
    return size + OpCodeBytes;
}

Wire Format Examples

Simple Data Packet:

[Opcode(2)] [Data(N)] [CRC(2)]
Example: [0x00 0x09] [Hello World] [0x3A 0x5C]

Compressed Packet:

[Opcode(2)] [Flag(1)] [Compressed Data(N)] [CRC(2)]
Example: [0x00 0x09] [0x5A] [Zlib data...] [0x3A 0x5C]

Combined Packet:

[OP_Combined(2)] [Size1(1)] [Packet1] [Size2(1)] [Packet2] [CRC(2)]
Example: [0x00 0x03] [0x08] [0x00 0x09 Data1] [0x0A] [0x00 0x06 Data2] [CRC]

Memory Management

Buffer Ownership

The packet classes follow strict ownership rules:

// Constructor takes ownership of buffer
EQPacket::EQPacket(uint16 op, const unsigned char* buf, uint32 len) {
    if (len > 0) {
        pBuffer = new unsigned char[len];
        if (buf) {
            memcpy(pBuffer, buf, len);  // Copy data
        } else {
            memset(pBuffer, 0, len);     // Zero-initialize
        }
    }
}

// Destructor releases buffer
EQPacket::~EQPacket() {
    safe_delete_array(pBuffer);  // delete[] with null check
    pBuffer = nullptr;
}

Deep Copy Operations

EQApplicationPacket* EQApplicationPacket::Copy() const {
    auto it = std::make_unique<EQApplicationPacket>();
    try {
        if (size > 0) {
            it->pBuffer = new unsigned char[size];
            memcpy(it->pBuffer, pBuffer, size);
        }
        it->size = size;
        it->opcode = opcode;
        it->emu_opcode = emu_opcode;
        it->version = version;
        return it.release();
    }
    catch (const std::bad_alloc& ba) {
        return nullptr;  // Allocation failed
    }
}

Buffer Reallocation

When packets are modified (compressed, combined, etc.):

// Safe buffer replacement pattern
unsigned char* new_buffer = new unsigned char[new_size];
// ... populate new_buffer ...
delete[] pBuffer;  // Release old buffer
pBuffer = new_buffer;  // Adopt new buffer
size = new_size;

Debug and Diagnostic Tools

Packet Dumping Functions

// Raw hex dump with network info
void EQPacket::DumpRaw(FILE* to = stdout) const {
    // Print header: [IP:port->IP:port] [Seq=N] [OpCode 0xXXXX (Name) Size=N]
    DumpRawHeader();
    
    // Dump packet contents in hex columns
    if (pBuffer && size) {
        dump_message_column(pBuffer, size, " ", to);
    }
}

// Application packet with info
void DumpPacket(const EQApplicationPacket* app, bool iShowInfo) {
    if (iShowInfo) {
        cout << "Dumping Applayer: 0x" << hex << setfill('0') << setw(4) 
             << app->GetOpcode() << dec;
        cout << " size:" << app->size << endl;
    }
    DumpPacketHex(app->pBuffer, app->size);
}

Output Formats

Hex Dump:

[192.168.1.100:5555->192.168.1.1:9000] [Seq=42] [OpCode 0x0009 (OP_Packet) Size=20]
00000000: 48 65 6C 6C 6F 20 57 6F  72 6C 64 21 00 00 00 00  |Hello World!....|
00000010: 01 02 03 04                                       |....|

ASCII Dump:

Hello World!....

Binary Dump:

01001000 01100101 01101100 01101100 01101111 ...

Diagnostic Information

The system tracks extensive metadata for debugging:

• Timestamp: Precise packet timing (microsecond resolution)
• Network Info: Complete IP:port source and destination
• Sequence Numbers: For tracking packet order and loss
• Attempt Count: Retransmission attempts for reliability analysis
• Version Info: Protocol version for compatibility debugging
• Compression State: Whether packet is compressed/encrypted
• CRC Validation: Integrity check results

Performance Considerations

Optimization Strategies

1. Selective Compression: Only compress packets > 30 bytes
2. Packet Combining: Reduce network overhead by bundling
3. Opcode Caching: Cache emulator opcodes to avoid repeated lookups
4. Memory Pooling: Reuse buffers where possible
5. Lazy Evaluation: Defer expensive operations until needed

Memory Footprint

Typical packet sizes:

• Control Packets: 4-20 bytes
• Chat Messages: 20-200 bytes
• Movement Updates: 30-60 bytes
• Item Data: 100-500 bytes
• Zone Data: 1000-8000 bytes (fragmented)

CPU Impact

Operation costs (relative):

• CRC Calculation: Low (lookup table)
• XOR Encryption: Very Low
• Zlib Compression: Medium-High
• Packet Combining: Low
• Opcode Conversion: Low (hash lookup)

Common Usage Patterns

Creating and Sending a Packet

// Create application packet
EQApplicationPacket* packet = new EQApplicationPacket(
    OP_ChatMessage,
    message_data,
    message_length
);

// Set version for opcode conversion
packet->setVersion(client_version);

// Queue for sending (stream handles conversion/compression)
stream->QueuePacket(packet);

Receiving and Processing

// Stream delivers application packets
EQApplicationPacket* packet = stream->PopPacket();
if (packet) {
    EmuOpcode op = packet->GetOpcode();
    
    switch (op) {
        case OP_LoginRequestMsg:
            ProcessLogin(packet);
            break;
        // ... handle other opcodes
    }
    
    delete packet;  // Clean up after processing
}

Error Handling

// Check for packet validity
if (!EQProtocolPacket::ValidateCRC(buffer, length, key)) {
    LogWrite(PACKET__ERROR, 0, "Packet", "CRC validation failed");
    return false;
}

// Handle compression errors
uint32 decompressed_size = EQProtocolPacket::Decompress(
    compressed, comp_len, decompressed, MAX_SIZE
);
if (decompressed_size == (uint32)-1) {
    LogWrite(PACKET__ERROR, 0, "Packet", "Decompression failed");
    // Fall back to uncompressed processing
}

Thread Safety

The packet classes themselves are not thread-safe. Thread safety must be handled at a higher level:

// Example thread-safe packet queue
class PacketQueue {
    std::mutex mutex_;
    std::queue<EQApplicationPacket*> packets_;
    
public:
    void Push(EQApplicationPacket* packet) {
        std::lock_guard<std::mutex> lock(mutex_);
        packets_.push(packet);
    }
    
    EQApplicationPacket* Pop() {
        std::lock_guard<std::mutex> lock(mutex_);
        if (packets_.empty()) return nullptr;
        auto packet = packets_.front();
        packets_.pop();
        return packet;
    }
};

Integration with EQStream

The packet system integrates tightly with EQStream for network operations:

1. Packet Creation: EQStream creates packets from network data
2. Sequencing: EQStream manages packet sequence numbers
3. Reliability: EQStream handles acknowledgments and retransmission
4. Fragmentation: EQStream splits/reassembles large packets
5. Flow Control: EQStream manages packet rate and congestion

Summary

The EQPacket system provides a robust, efficient framework for handling EverQuest 2 network packets. Its hierarchical design separates protocol concerns from application logic, while features like compression, combining, and encryption optimize network usage. The extensive debugging capabilities and clear separation of concerns make it maintainable and extensible for ongoing development.

Key strengths:

• Flexible Architecture: Clean separation between protocol and application layers
• Performance Optimized: Selective compression, packet combining, caching
• Robust Error Handling: CRC validation, safe memory management
• Comprehensive Debugging: Multiple dump formats, extensive metadata
• Protocol Agnostic: Opcode abstraction allows version independence

The system successfully handles the complex requirements of an MMORPG protocol while maintaining code clarity and performance.