eq2go/eq2_protocol_report.md

EverQuest 2 Network Protocol Documentation

Overview

The EverQuest 2 protocol is a custom UDP-based protocol that provides reliable delivery, encryption, compression, and session management. This document describes the protocol structure for reimplementation.

1. Protocol Architecture

1.1 Protocol Layers

Application Layer    - Game logic packets (EQApplicationPacket)
Protocol Layer       - Session management, reliability (EQProtocolPacket) 
Transport Layer      - UDP with custom reliability
Network Layer        - Standard IP

1.2 Packet Types

• EQProtocolPacket: Low-level protocol control packets
• EQApplicationPacket: High-level game data packets
• EQ2Packet: EQ2-specific application packets with login opcodes

2. Session Management

2.1 Session Establishment

Client -> Server: OP_SessionRequest
Server -> Client: OP_SessionResponse

SessionRequest Structure

struct SessionRequest {
    uint32 UnknownA;      // Usually 0
    uint32 Session;       // Proposed session ID
    uint32 MaxLength;     // Maximum packet length
};

SessionResponse Structure

struct SessionResponse {
    uint32 Session;       // Confirmed session ID
    uint32 Key;           // Encryption key
    uint8  UnknownA;      // Usually 2
    uint8  Format;        // Flags: 0x01=compressed, 0x04=encoded
    uint8  UnknownB;      // Usually 0
    uint32 MaxLength;     // Maximum packet length
    uint32 UnknownD;      // Usually 0
};

2.2 Session Termination

Either -> Other: OP_SessionDisconnect

3. Protocol Opcodes

3.1 Core Protocol Opcodes

#define OP_SessionRequest      0x01
#define OP_SessionResponse     0x02  
#define OP_Combined            0x03
#define OP_SessionDisconnect   0x05
#define OP_KeepAlive          0x06
#define OP_ServerKeyRequest   0x07
#define OP_SessionStatResponse 0x08
#define OP_Packet             0x09
#define OP_Fragment           0x0D
#define OP_OutOfOrderAck      0x11
#define OP_Ack                0x15
#define OP_AppCombined        0x19
#define OP_OutOfSession       0x1D

4. Reliable Delivery System

4.1 Sequence Numbers

• 16-bit sequence numbers for ordered delivery
• Wrap-around handling at 65536
• Window-based flow control (default window size: 2048)

4.2 Acknowledgments

• OP_Ack: Acknowledges packets up to sequence number
• OP_OutOfOrderAck: Acknowledges specific out-of-order packet
• Retransmission on timeout (default: 500ms * 3.0 multiplier, max 5000ms)

4.3 Packet Structure for Sequenced Data

[2 bytes: Sequence Number][Payload Data]

5. Encryption System

5.1 Key Exchange

1. RSA key exchange during initial handshake
2. 8-byte encrypted key transmitted in packet
3. RC4 encryption initialized with exchanged key

5.2 RC4 Encryption

• Applied to packet payload after headers
• Separate encryption state per connection
• Encryption offset varies by packet type and compression

5.3 CRC Validation

• 16-bit CRC appended to most packets
• CRC calculated using session key
• Some packets (SessionRequest, SessionResponse, OutOfSession) not CRC'd

6. Compression System

6.1 zlib Compression

• Individual packets compressed using zlib deflate
• Compression applied when packet size > 128 bytes
• Compression markers:
  • 0x5A: zlib compressed data follows
  • 0xA5: uncompressed data (small packets)

6.2 Compression Process

1. Check if packet size > compression threshold
2. Apply zlib deflate compression  
3. Prepend compression marker
4. If compressed size >= original, use uncompressed with 0xA5 marker

7. Packet Combination

7.1 Protocol-Level Combination (OP_Combined)

Multiple protocol packets combined into single UDP datagram:

[1 byte: Packet1 Size][Packet1 Data]
[1 byte: Packet2 Size][Packet2 Data]
...

If size >= 255:

[1 byte: 0xFF][2 bytes: Actual Size][Packet Data]

7.2 Application-Level Combination (OP_AppCombined)

Multiple application packets combined:

[1 byte: Packet1 Size][Packet1 Data without opcode header]
[1 byte: Packet2 Size][Packet2 Data without opcode header]
...

8. Fragmentation

8.1 Large Packet Handling

Packets larger than MaxLength are fragmented using OP_Fragment:

First Fragment:

[2 bytes: Sequence][4 bytes: Total Length][Payload Chunk]

Subsequent Fragments:

[2 bytes: Sequence][Payload Chunk]

8.2 Reassembly

1. Allocate buffer based on total length from first fragment
2. Collect fragments in sequence order
3. Reconstruct original packet when all fragments received

9. Data Structure System

9.1 Data Types

#define DATA_STRUCT_INT8               1
#define DATA_STRUCT_INT16              2  
#define DATA_STRUCT_INT32              3
#define DATA_STRUCT_INT64              4
#define DATA_STRUCT_FLOAT              5
#define DATA_STRUCT_DOUBLE             6
#define DATA_STRUCT_COLOR              7
#define DATA_STRUCT_SINT8              8
#define DATA_STRUCT_SINT16             9
#define DATA_STRUCT_SINT32             10
#define DATA_STRUCT_CHAR               11
#define DATA_STRUCT_EQ2_8BIT_STRING    12
#define DATA_STRUCT_EQ2_16BIT_STRING   13
#define DATA_STRUCT_EQ2_32BIT_STRING   14
#define DATA_STRUCT_EQUIPMENT          15
#define DATA_STRUCT_ARRAY              16
#define DATA_STRUCT_ITEM               17
#define DATA_STRUCT_SINT64             18

9.2 String Types

• EQ2_8BitString: [1 byte length][string data]
• EQ2_16BitString: [2 bytes length][string data]
• EQ2_32BitString: [4 bytes length][string data]

9.3 Color Structure

struct EQ2_Color {
    uint8 red;
    uint8 green; 
    uint8 blue;
};

9.4 Equipment Structure

struct EQ2_EquipmentItem {
    uint16 type;
    EQ2_Color color;
    EQ2_Color highlight;
};

10. Application Opcodes

10.1 Opcode System

• Two-byte opcodes for game servers (WorldServer, ZoneServer)
• One-byte opcodes for login servers
• Version-specific opcode mappings stored in database
• Translation between internal EmuOpcodes and client opcodes

10.2 Key Application Opcodes

// Login Operations
OP_LoginRequestMsg
OP_LoginReplyMsg
OP_AllCharactersDescRequestMsg
OP_AllCharactersDescReplyMsg
OP_CreateCharacterRequestMsg
OP_CreateCharacterReplyMsg

// World Operations  
OP_ZoneInfoMsg
OP_UpdateCharacterSheetMsg
OP_UpdateInventoryMsg
OP_ClientCmdMsg

// Chat Operations
OP_ChatTellUserMsg
OP_ChatJoinChannelMsg

11. Implementation Guidelines

11.1 Connection State Machine

CLOSED -> SessionRequest -> ESTABLISHED
ESTABLISHED -> SessionDisconnect -> CLOSING -> CLOSED

11.2 Buffer Management

• Maintain separate inbound/outbound queues
• Implement sliding window for flow control
• Handle out-of-order packet storage
• Implement packet combining logic

11.3 Threading Considerations

• Separate reader/writer threads recommended
• Reader processes incoming UDP packets
• Writer sends outbound packets and handles retransmission
• Combine packet processor for optimization

11.4 Error Handling

• Validate CRC on all received packets
• Handle malformed packets gracefully
• Implement connection timeout detection
• Retry logic for failed transmissions

11.5 Performance Optimizations

• Packet combination to reduce UDP overhead
• Compression for large packets
• Rate limiting and congestion control
• Efficient data structure serialization

12. Stream Types

Different stream types have different characteristics:

enum EQStreamType {
    LoginStream,     // 1-byte opcodes, no compression/encryption
    WorldStream,     // 2-byte opcodes, compression, no encryption
    ZoneStream,      // 2-byte opcodes, compression, no encryption  
    ChatStream,      // 1-byte opcodes, no compression, encoding
    EQ2Stream        // 2-byte opcodes, no compression/encryption
};

13. Sample Packet Flow

13.1 Login Sequence

1. Client -> Server: OP_SessionRequest
2. Server -> Client: OP_SessionResponse (with key, compression flags)
3. Client -> Server: OP_Packet[OP_LoginRequestMsg] (with credentials)
4. Server -> Client: OP_Packet[OP_LoginReplyMsg] (success/failure)
5. Client -> Server: OP_Packet[OP_AllCharactersDescRequestMsg]
6. Server -> Client: OP_Packet[OP_AllCharactersDescReplyMsg] (character list)

13.2 Reliable Data Transfer

1. Sender: Assign sequence number, add to retransmit queue
2. Sender: Transmit OP_Packet[seq][data]
3. Receiver: Process packet, send OP_Ack[seq]
4. Sender: Receive ack, remove from retransmit queue
5. On timeout: Retransmit packet up to max attempts

This documentation provides the foundation for implementing the EQ2 protocol in any programming language while maintaining compatibility with the existing server and client implementations.