eq2go/eq2_protocol_report.md

# EverQuest 2 Network Protocol Documentation

## Overview

The EverQuest 2 protocol is a custom UDP-based protocol that provides reliable delivery, encryption, compression, and session management. This document describes the protocol structure for reimplementation.

## 1. Protocol Architecture

### 1.1 Protocol Layers
```
Application Layer    - Game logic packets (EQApplicationPacket)
Protocol Layer       - Session management, reliability (EQProtocolPacket) 
Transport Layer      - UDP with custom reliability
Network Layer        - Standard IP
```

### 1.2 Packet Types
- **EQProtocolPacket**: Low-level protocol control packets
- **EQApplicationPacket**: High-level game data packets  
- **EQ2Packet**: EQ2-specific application packets with login opcodes

## 2. Session Management

### 2.1 Session Establishment
```
Client -> Server: OP_SessionRequest
Server -> Client: OP_SessionResponse
```

#### SessionRequest Structure
```c
struct SessionRequest {
    uint32 UnknownA;      // Usually 0
    uint32 Session;       // Proposed session ID
    uint32 MaxLength;     // Maximum packet length
};
```

#### SessionResponse Structure  
```c
struct SessionResponse {
    uint32 Session;       // Confirmed session ID
    uint32 Key;           // Encryption key
    uint8  UnknownA;      // Usually 2
    uint8  Format;        // Flags: 0x01=compressed, 0x04=encoded
    uint8  UnknownB;      // Usually 0
    uint32 MaxLength;     // Maximum packet length
    uint32 UnknownD;      // Usually 0
};
```

### 2.2 Session Termination
```
Either -> Other: OP_SessionDisconnect
```

## 3. Protocol Opcodes

### 3.1 Core Protocol Opcodes
```c
#define OP_SessionRequest      0x01
#define OP_SessionResponse     0x02  
#define OP_Combined            0x03
#define OP_SessionDisconnect   0x05
#define OP_KeepAlive          0x06
#define OP_ServerKeyRequest   0x07
#define OP_SessionStatResponse 0x08
#define OP_Packet             0x09
#define OP_Fragment           0x0D
#define OP_OutOfOrderAck      0x11
#define OP_Ack                0x15
#define OP_AppCombined        0x19
#define OP_OutOfSession       0x1D
```

## 4. Reliable Delivery System

### 4.1 Sequence Numbers
- 16-bit sequence numbers for ordered delivery
- Wrap-around handling at 65536
- Window-based flow control (default window size: 2048)

### 4.2 Acknowledgments
- **OP_Ack**: Acknowledges packets up to sequence number
- **OP_OutOfOrderAck**: Acknowledges specific out-of-order packet
- Retransmission on timeout (default: 500ms * 3.0 multiplier, max 5000ms)

### 4.3 Packet Structure for Sequenced Data
```
[2 bytes: Sequence Number][Payload Data]
```

## 5. Encryption System

### 5.1 Key Exchange
1. RSA key exchange during initial handshake
2. 8-byte encrypted key transmitted in packet
3. RC4 encryption initialized with exchanged key

### 5.2 RC4 Encryption
- Applied to packet payload after headers
- Separate encryption state per connection
- Encryption offset varies by packet type and compression

### 5.3 CRC Validation
- 16-bit CRC appended to most packets
- CRC calculated using session key
- Some packets (SessionRequest, SessionResponse, OutOfSession) not CRC'd

## 6. Compression System

### 6.1 zlib Compression
- Individual packets compressed using zlib deflate
- Compression applied when packet size > 128 bytes
- Compression markers:
  - `0x5A`: zlib compressed data follows
  - `0xA5`: uncompressed data (small packets)

### 6.2 Compression Process
```
1. Check if packet size > compression threshold
2. Apply zlib deflate compression  
3. Prepend compression marker
4. If compressed size >= original, use uncompressed with 0xA5 marker
```

## 7. Packet Combination

### 7.1 Protocol-Level Combination (OP_Combined)
Multiple protocol packets combined into single UDP datagram:
```
[1 byte: Packet1 Size][Packet1 Data]
[1 byte: Packet2 Size][Packet2 Data]
...
```
If size >= 255:
```
[1 byte: 0xFF][2 bytes: Actual Size][Packet Data]
```

### 7.2 Application-Level Combination (OP_AppCombined)  
Multiple application packets combined:
```
[1 byte: Packet1 Size][Packet1 Data without opcode header]
[1 byte: Packet2 Size][Packet2 Data without opcode header]
...
```

## 8. Fragmentation

### 8.1 Large Packet Handling
Packets larger than MaxLength are fragmented using OP_Fragment:

**First Fragment:**
```
[2 bytes: Sequence][4 bytes: Total Length][Payload Chunk]
```

**Subsequent Fragments:**
```
[2 bytes: Sequence][Payload Chunk]
```

### 8.2 Reassembly
1. Allocate buffer based on total length from first fragment
2. Collect fragments in sequence order
3. Reconstruct original packet when all fragments received

## 9. Data Structure System

### 9.1 Data Types
```c
#define DATA_STRUCT_INT8               1
#define DATA_STRUCT_INT16              2  
#define DATA_STRUCT_INT32              3
#define DATA_STRUCT_INT64              4
#define DATA_STRUCT_FLOAT              5
#define DATA_STRUCT_DOUBLE             6
#define DATA_STRUCT_COLOR              7
#define DATA_STRUCT_SINT8              8
#define DATA_STRUCT_SINT16             9
#define DATA_STRUCT_SINT32             10
#define DATA_STRUCT_CHAR               11
#define DATA_STRUCT_EQ2_8BIT_STRING    12
#define DATA_STRUCT_EQ2_16BIT_STRING   13
#define DATA_STRUCT_EQ2_32BIT_STRING   14
#define DATA_STRUCT_EQUIPMENT          15
#define DATA_STRUCT_ARRAY              16
#define DATA_STRUCT_ITEM               17
#define DATA_STRUCT_SINT64             18
```

### 9.2 String Types
- **EQ2_8BitString**: [1 byte length][string data]
- **EQ2_16BitString**: [2 bytes length][string data]  
- **EQ2_32BitString**: [4 bytes length][string data]

### 9.3 Color Structure
```c
struct EQ2_Color {
    uint8 red;
    uint8 green; 
    uint8 blue;
};
```

### 9.4 Equipment Structure
```c
struct EQ2_EquipmentItem {
    uint16 type;
    EQ2_Color color;
    EQ2_Color highlight;
};
```

## 10. Application Opcodes

### 10.1 Opcode System
- Two-byte opcodes for game servers (WorldServer, ZoneServer)
- One-byte opcodes for login servers
- Version-specific opcode mappings stored in database
- Translation between internal EmuOpcodes and client opcodes

### 10.2 Key Application Opcodes
```c
// Login Operations
OP_LoginRequestMsg
OP_LoginReplyMsg
OP_AllCharactersDescRequestMsg
OP_AllCharactersDescReplyMsg
OP_CreateCharacterRequestMsg
OP_CreateCharacterReplyMsg

// World Operations  
OP_ZoneInfoMsg
OP_UpdateCharacterSheetMsg
OP_UpdateInventoryMsg
OP_ClientCmdMsg

// Chat Operations
OP_ChatTellUserMsg
OP_ChatJoinChannelMsg
```

## 11. Implementation Guidelines

### 11.1 Connection State Machine
```
CLOSED -> SessionRequest -> ESTABLISHED
ESTABLISHED -> SessionDisconnect -> CLOSING -> CLOSED
```

### 11.2 Buffer Management
- Maintain separate inbound/outbound queues
- Implement sliding window for flow control
- Handle out-of-order packet storage
- Implement packet combining logic

### 11.3 Threading Considerations
- Separate reader/writer threads recommended
- Reader processes incoming UDP packets
- Writer sends outbound packets and handles retransmission
- Combine packet processor for optimization

### 11.4 Error Handling
- Validate CRC on all received packets
- Handle malformed packets gracefully
- Implement connection timeout detection
- Retry logic for failed transmissions

### 11.5 Performance Optimizations
- Packet combination to reduce UDP overhead
- Compression for large packets
- Rate limiting and congestion control
- Efficient data structure serialization

## 12. Stream Types

Different stream types have different characteristics:

```c
enum EQStreamType {
    LoginStream,     // 1-byte opcodes, no compression/encryption
    WorldStream,     // 2-byte opcodes, compression, no encryption
    ZoneStream,      // 2-byte opcodes, compression, no encryption  
    ChatStream,      // 1-byte opcodes, no compression, encoding
    EQ2Stream        // 2-byte opcodes, no compression/encryption
};
```

## 13. Sample Packet Flow

### 13.1 Login Sequence
```
1. Client -> Server: OP_SessionRequest
2. Server -> Client: OP_SessionResponse (with key, compression flags)
3. Client -> Server: OP_Packet[OP_LoginRequestMsg] (with credentials)
4. Server -> Client: OP_Packet[OP_LoginReplyMsg] (success/failure)
5. Client -> Server: OP_Packet[OP_AllCharactersDescRequestMsg]
6. Server -> Client: OP_Packet[OP_AllCharactersDescReplyMsg] (character list)
```

### 13.2 Reliable Data Transfer
```
1. Sender: Assign sequence number, add to retransmit queue
2. Sender: Transmit OP_Packet[seq][data]
3. Receiver: Process packet, send OP_Ack[seq]
4. Sender: Receive ack, remove from retransmit queue
5. On timeout: Retransmit packet up to max attempts
```

This documentation provides the foundation for implementing the EQ2 protocol in any programming language while maintaining compatibility with the existing server and client implementations.