42 KiB
42 KiB
EQStream - Complete Technical Documentation
Table of Contents
- Overview
- Core Components
- Protocol Architecture
- Packet Types and Structures
- Encryption System
- Compression
- Packet Sequencing and Reliability
- Fragmentation and Reassembly
- Connection Management
- Flow Control
- Implementation Details
- Utilities and Helper Functions
Overview
EQStream is a custom network protocol implementation that sits on top of TCP/UDP, providing:
- Reliable delivery with acknowledgments and retransmission
- Ordered delivery with sequence numbers
- Encryption using RC4 stream cipher
- Compression using zlib
- Fragmentation for large packets
- Flow control to prevent congestion
- Session management with keepalives
Design Goals
- Low latency for real-time gameplay
- Reliable delivery of critical packets
- Efficient bandwidth usage
- Security through encryption
- Compatibility with EverQuest II client
Core Components
EQStream Class
The main class that manages a single connection:
class EQStream {
private:
// Connection state
enum State {
CLOSED, // No connection
WAIT_CONNECT, // Awaiting session response
CONNECTED, // Active connection
DISCONNECTING, // Graceful shutdown
DISCONNECT // Connection terminated
};
// Core components
RC4_KEY* encrypt_key; // Encryption key
RC4_KEY* decrypt_key; // Decryption key
z_stream* stream_compress; // Compression stream
z_stream* stream_decompress; // Decompression stream
// Sequence management
uint16_t next_in_sequence; // Expected incoming sequence
uint16_t next_out_sequence; // Next outgoing sequence
uint32_t session_id; // Unique session identifier
uint32_t key; // RC4 key seed
// Buffers and queues
std::queue<EQProtocolPacket*> send_queue; // Outgoing packets
std::queue<EQProtocolPacket*> receive_queue; // Incoming packets
std::map<uint16_t, EQProtocolPacket*> future_packets; // Out-of-order
std::map<uint16_t, Fragment*> fragment_buffer; // Fragmented packets
// Configuration
uint32_t max_packet_size; // Maximum packet size
uint8_t crc_length; // CRC bytes (usually 2)
bool compressed_mode; // Compression enabled
bool encoded_mode; // Encryption enabled
};
EQStreamFactory Class
Factory pattern for creating and managing multiple streams:
class EQStreamFactory {
private:
std::map<std::string, EQStream*> stream_map; // Active streams
std::queue<EQStream*> new_streams; // Pending connections
int listen_socket; // Listening socket
uint16_t port; // Server port
public:
bool Open(uint16_t port); // Start listening
EQStream* Pop(); // Get new connection
void CheckTimeout(); // Timeout inactive streams
void Close(); // Shutdown factory
};
Protocol Architecture
Layer Model
┌─────────────────────────────────────┐
│ Application Data │
├─────────────────────────────────────┤
│ EQApplicationPacket │ Game packets
├─────────────────────────────────────┤
│ EQProtocolPacket │ Protocol control
├─────────────────────────────────────┤
│ Compression (optional zlib) │ Data compression
├─────────────────────────────────────┤
│ Encryption (RC4 cipher) │ Stream encryption
├─────────────────────────────────────┤
│ CRC32 Checksum │ Integrity check
├─────────────────────────────────────┤
│ UDP Transport │ Unreliable datagrams
└─────────────────────────────────────┘
Protocol Flow
Client Server
│ │
├──────── SessionRequest ──────────►│
│ (unencrypted) │
│ │
│◄──────── SessionResponse ─────────┤
│ (contains RC4 key) │
│ │
├─ [All subsequent packets encrypted]─┤
│ │
├──────── OP_Packet(data) ─────────►│
│ │
│◄────────── OP_Ack ───────────────┤
│ │
Packet Types and Structures
Protocol Control Packets
enum EQStreamOp : uint16_t {
// Session Management
OP_SessionRequest = 0x0001, // Client initiates connection
OP_SessionResponse = 0x0002, // Server accepts connection
OP_SessionDisconnect = 0x0005, // Graceful disconnect
// Connection Maintenance
OP_KeepAlive = 0x0006, // Heartbeat packet
OP_SessionStatRequest = 0x0007, // Request statistics
OP_SessionStatResponse = 0x0008, // Statistics response
// Data Transfer
OP_Packet = 0x0009, // Application data
OP_Combined = 0x0003, // Multiple packets bundled
OP_Fragment = 0x000d, // Part of fragmented packet
OP_OutOfOrderAck = 0x0011, // Acknowledge out-of-order
// Acknowledgments
OP_Ack = 0x0015, // Standard acknowledgment
OP_AckFuture = 0x0016, // Acknowledge future packet
OP_AckPast = 0x0017, // Acknowledge past packet
// Error Handling
OP_Error = 0x0019, // Error notification
OP_Reset = 0x001d // Reset connection
};
Packet Structure Definitions
Base Protocol Packet
struct EQProtocolPacket {
uint16_t opcode; // Operation code
uint8_t data[0]; // Variable length data
// Methods
uint32_t Size() const; // Total packet size
void SetOpcode(uint16_t op);
uint16_t GetOpcode() const;
};
Session Request
struct SessionRequest {
uint16_t opcode; // OP_SessionRequest (0x0001)
uint32_t unknown; // Protocol version (usually 0x0003)
uint32_t session_id; // Client's session ID
uint32_t max_length; // Maximum packet size client can handle
};
Session Response
struct SessionResponse {
uint16_t opcode; // OP_SessionResponse (0x0002)
uint32_t session_id; // Server's session ID
uint32_t key; // RC4 encryption key seed
uint8_t crc_length; // CRC bytes (2 for CRC16, 4 for CRC32)
uint8_t compressed_mode; // 0x01 if compression enabled
uint8_t encoded_mode; // 0x01 if encryption enabled
uint32_t unknown; // Reserved/padding
uint32_t max_length; // Maximum packet size server can handle
};
Standard Data Packet
struct DataPacket {
uint16_t opcode; // OP_Packet (0x0009)
uint16_t sequence; // Sequence number
uint8_t app_data[0]; // Application layer data
};
Combined Packet
struct CombinedPacket {
uint16_t opcode; // OP_Combined (0x0003)
uint8_t count; // Number of sub-packets
struct SubPacket {
uint8_t size; // Size of this sub-packet
uint8_t data[0]; // Sub-packet data
} packets[0];
};
Fragment Packet
struct FragmentPacket {
uint16_t opcode; // OP_Fragment (0x000d)
uint16_t sequence; // Sequence number
uint32_t total_size; // Total size when reassembled
uint16_t fragment_number; // Fragment index (0-based)
uint16_t fragment_count; // Total number of fragments
uint8_t data[0]; // Fragment data
};
Acknowledgment Packet
struct AckPacket {
uint16_t opcode; // OP_Ack (0x0015)
uint16_t sequence; // Sequence being acknowledged
};
Encryption System
RC4 Implementation
Key Initialization
void EQStream::SetupEncryption(uint32_t key_seed) {
// Generate key bytes from seed
uint8_t key_bytes[4];
memcpy(key_bytes, &key_seed, 4);
// Initialize encryption key
encrypt_key = new RC4_KEY;
RC4_set_key(encrypt_key, 4, key_bytes);
// Initialize decryption key (separate state)
decrypt_key = new RC4_KEY;
RC4_set_key(decrypt_key, 4, key_bytes);
encoded_mode = true;
}
Packet Encryption
void EQStream::EncryptPacket(EQProtocolPacket* packet) {
if (!encoded_mode || !encrypt_key) return;
// Get packet data (skip opcode)
uint8_t* data = packet->data;
uint32_t length = packet->Size() - 2; // Exclude opcode
// Apply RC4 encryption
RC4(encrypt_key, length, data, data);
}
Packet Decryption
void EQStream::DecryptPacket(uint8_t* data, uint32_t length) {
if (!encoded_mode || !decrypt_key) return;
// Skip first 2 bytes (opcode is not encrypted)
if (length > 2) {
RC4(decrypt_key, length - 2, data + 2, data + 2);
}
}
Security Considerations
- Key Exchange: Key is transmitted in clear during session setup
- Stream Cipher: RC4 maintains state, packets must be processed in order
- No IV: Each session uses the same initial RC4 state
- CRC Protection: CRC is calculated on encrypted data
Compression
zlib Integration
Compression Setup
void EQStream::SetupCompression() {
// Initialize compression stream
stream_compress = new z_stream;
memset(stream_compress, 0, sizeof(z_stream));
deflateInit(stream_compress, Z_DEFAULT_COMPRESSION);
// Initialize decompression stream
stream_decompress = new z_stream;
memset(stream_decompress, 0, sizeof(z_stream));
inflateInit(stream_decompress);
compressed_mode = true;
}
Packet Compression
EQProtocolPacket* EQStream::CompressPacket(EQProtocolPacket* packet) {
if (!compressed_mode || packet->Size() < COMPRESS_THRESHOLD) {
return packet;
}
// Prepare compression buffer
uint32_t source_size = packet->Size();
uint32_t dest_size = source_size + 128; // Extra space
uint8_t* dest_buffer = new uint8_t[dest_size];
// Setup zlib stream
stream_compress->next_in = (Bytef*)packet->data;
stream_compress->avail_in = source_size;
stream_compress->next_out = dest_buffer;
stream_compress->avail_out = dest_size;
// Compress
int result = deflate(stream_compress, Z_SYNC_FLUSH);
if (result == Z_OK) {
uint32_t compressed_size = dest_size - stream_compress->avail_out;
// Create compressed packet
EQProtocolPacket* compressed = new EQProtocolPacket(
OP_Compressed, dest_buffer, compressed_size);
compressed->SetFlag(FLAG_COMPRESSED);
delete[] dest_buffer;
delete packet;
return compressed;
}
// Compression failed, return original
delete[] dest_buffer;
return packet;
}
Packet Decompression
bool EQStream::DecompressPacket(uint8_t* data, uint32_t length,
uint8_t* dest, uint32_t dest_length) {
if (!compressed_mode) return false;
// Setup decompression
stream_decompress->next_in = data;
stream_decompress->avail_in = length;
stream_decompress->next_out = dest;
stream_decompress->avail_out = dest_length;
// Decompress
int result = inflate(stream_decompress, Z_SYNC_FLUSH);
return (result == Z_OK || result == Z_STREAM_END);
}
Packet Sequencing and Reliability
Sequence Number Management
class SequenceManager {
private:
uint16_t next_out_sequence; // Next sequence to send
uint16_t next_in_sequence; // Expected incoming sequence
uint16_t last_acked_sequence; // Last acknowledged by peer
// Retransmission queue
struct RetransmitEntry {
uint16_t sequence;
EQProtocolPacket* packet;
uint32_t timestamp;
uint32_t retransmit_count;
};
std::map<uint16_t, RetransmitEntry> retransmit_queue;
public:
uint16_t GetNextSequence() {
return next_out_sequence++;
}
bool IsValidSequence(uint16_t seq) {
// Handle sequence wrap-around
int16_t diff = seq - next_in_sequence;
return (diff >= 0 && diff < MAX_SEQUENCE_DIFF);
}
void AcknowledgeSequence(uint16_t seq) {
// Remove from retransmit queue
auto it = retransmit_queue.find(seq);
if (it != retransmit_queue.end()) {
delete it->second.packet;
retransmit_queue.erase(it);
}
// Update last acknowledged
if (CompareSequence(seq, last_acked_sequence) > 0) {
last_acked_sequence = seq;
}
}
};
Retransmission Logic
void EQStream::CheckRetransmit() {
uint32_t current_time = Timer::GetCurrentTime();
for (auto& [seq, entry] : retransmit_queue) {
uint32_t elapsed = current_time - entry.timestamp;
// Retransmit if timeout exceeded
if (elapsed > RETRANSMIT_TIMEOUT) {
if (entry.retransmit_count < MAX_RETRANSMIT) {
// Resend packet
SendPacketInternal(entry.packet->Copy());
entry.timestamp = current_time;
entry.retransmit_count++;
LogWrite(NET__DEBUG, "Retransmitting seq %d (attempt %d)",
seq, entry.retransmit_count);
} else {
// Max retransmits exceeded, connection lost
LogWrite(NET__ERROR, "Max retransmits for seq %d", seq);
Disconnect();
}
}
}
}
Out-of-Order Handling
void EQStream::ProcessOutOfOrder(uint16_t sequence, EQProtocolPacket* packet) {
// Check if this is a future packet
if (CompareSequence(sequence, next_in_sequence) > 0) {
// Store for later processing
future_packets[sequence] = packet;
// Send future acknowledgment
SendAckFuture(sequence);
LogWrite(NET__DEBUG, "Stored future packet seq %d", sequence);
}
// Check if this is a duplicate
else if (CompareSequence(sequence, next_in_sequence) < 0) {
// Already processed, send past acknowledgment
SendAckPast(sequence);
delete packet;
LogWrite(NET__DEBUG, "Duplicate packet seq %d", sequence);
}
}
void EQStream::ProcessFuturePackets() {
// Check if any future packets can now be processed
while (!future_packets.empty()) {
auto it = future_packets.find(next_in_sequence);
if (it != future_packets.end()) {
// Process this packet
ProcessPacket(it->second);
future_packets.erase(it);
next_in_sequence++;
} else {
break; // Gap in sequence
}
}
}
Fragmentation and Reassembly
Fragment Management
class FragmentManager {
private:
struct FragmentSet {
uint32_t total_size;
uint16_t fragment_count;
uint16_t received_count;
std::map<uint16_t, uint8_t*> fragments;
uint32_t timestamp;
};
std::map<uint16_t, FragmentSet> fragment_sets;
public:
void AddFragment(uint16_t sequence, uint16_t fragment_num,
uint16_t fragment_count, uint32_t total_size,
uint8_t* data, uint32_t data_length);
EQProtocolPacket* CheckComplete(uint16_t sequence);
void CleanupOldFragments(uint32_t timeout);
};
Fragmentation Process
std::vector<EQProtocolPacket*> EQStream::FragmentPacket(
EQProtocolPacket* packet, uint16_t max_fragment_size) {
std::vector<EQProtocolPacket*> fragments;
uint32_t total_size = packet->Size();
uint16_t fragment_count = (total_size + max_fragment_size - 1)
/ max_fragment_size;
uint8_t* data_ptr = packet->data;
uint32_t remaining = total_size;
for (uint16_t i = 0; i < fragment_count; i++) {
uint32_t fragment_size = std::min(remaining,
(uint32_t)max_fragment_size);
// Create fragment header
FragmentHeader header;
header.opcode = OP_Fragment;
header.sequence = GetNextSequence();
header.total_size = total_size;
header.fragment_number = i;
header.fragment_count = fragment_count;
// Create fragment packet
uint32_t packet_size = sizeof(FragmentHeader) + fragment_size;
EQProtocolPacket* fragment = new EQProtocolPacket(
OP_Fragment, nullptr, packet_size);
// Copy header and data
memcpy(fragment->data, &header, sizeof(FragmentHeader));
memcpy(fragment->data + sizeof(FragmentHeader),
data_ptr, fragment_size);
fragments.push_back(fragment);
data_ptr += fragment_size;
remaining -= fragment_size;
}
delete packet; // Original no longer needed
return fragments;
}
Reassembly Process
EQProtocolPacket* FragmentManager::Reassemble(uint16_t sequence) {
auto it = fragment_sets.find(sequence);
if (it == fragment_sets.end()) {
return nullptr;
}
FragmentSet& set = it->second;
// Check if all fragments received
if (set.received_count != set.fragment_count) {
return nullptr;
}
// Allocate buffer for complete packet
uint8_t* complete_data = new uint8_t[set.total_size];
uint32_t offset = 0;
// Assemble fragments in order
for (uint16_t i = 0; i < set.fragment_count; i++) {
auto frag_it = set.fragments.find(i);
if (frag_it == set.fragments.end()) {
// Missing fragment - shouldn't happen
delete[] complete_data;
return nullptr;
}
// Copy fragment data
uint32_t frag_size = (i == set.fragment_count - 1) ?
(set.total_size - offset) : MAX_FRAGMENT_SIZE;
memcpy(complete_data + offset, frag_it->second, frag_size);
offset += frag_size;
// Clean up fragment
delete[] frag_it->second;
}
// Create reassembled packet
EQProtocolPacket* packet = new EQProtocolPacket(
OP_Packet, complete_data, set.total_size);
// Clean up fragment set
fragment_sets.erase(it);
delete[] complete_data;
return packet;
}
Connection Management
State Machine
class ConnectionStateMachine {
public:
enum State {
CLOSED, // No connection
CONNECTING, // Session request sent
CONNECTED, // Active connection
DISCONNECTING, // Graceful shutdown initiated
ERROR // Error state
};
private:
State current_state;
uint32_t state_timestamp;
// State transition table
struct Transition {
State from;
Event event;
State to;
std::function<void()> action;
};
std::vector<Transition> transitions = {
{CLOSED, EVENT_CONNECT, CONNECTING, [this]() { SendSessionRequest(); }},
{CONNECTING, EVENT_SESSION_RESPONSE, CONNECTED, [this]() { OnConnected(); }},
{CONNECTED, EVENT_DISCONNECT, DISCONNECTING, [this]() { SendDisconnect(); }},
{DISCONNECTING, EVENT_DISCONNECT_ACK, CLOSED, [this]() { OnClosed(); }},
// ... more transitions
};
public:
void ProcessEvent(Event event) {
for (const auto& trans : transitions) {
if (trans.from == current_state && trans.event == event) {
current_state = trans.to;
state_timestamp = Timer::GetCurrentTime();
if (trans.action) trans.action();
break;
}
}
}
};
Keep-Alive Mechanism
class KeepAliveManager {
private:
uint32_t keepalive_interval; // Milliseconds between keepalives
uint32_t last_sent; // Last keepalive sent
uint32_t last_received; // Last packet received
uint32_t timeout_threshold; // Connection timeout
public:
void Process() {
uint32_t current_time = Timer::GetCurrentTime();
// Check if we need to send keepalive
if (current_time - last_sent > keepalive_interval) {
SendKeepAlive();
last_sent = current_time;
}
// Check for connection timeout
if (current_time - last_received > timeout_threshold) {
LogWrite(NET__ERROR, "Connection timeout - no response");
Disconnect();
}
}
void OnPacketReceived() {
last_received = Timer::GetCurrentTime();
}
};
Flow Control
Congestion Window
class CongestionControl {
private:
uint32_t cwnd; // Congestion window size
uint32_t ssthresh; // Slow start threshold
uint32_t outstanding_bytes; // Bytes in flight
uint32_t rtt_estimate; // Round trip time estimate
enum State {
SLOW_START,
CONGESTION_AVOIDANCE,
FAST_RECOVERY
} state;
public:
bool CanSend(uint32_t packet_size) {
return outstanding_bytes + packet_size <= cwnd;
}
void OnAck(uint32_t acked_bytes) {
outstanding_bytes -= acked_bytes;
switch (state) {
case SLOW_START:
cwnd += acked_bytes;
if (cwnd >= ssthresh) {
state = CONGESTION_AVOIDANCE;
}
break;
case CONGESTION_AVOIDANCE:
cwnd += (acked_bytes * acked_bytes) / cwnd;
break;
case FAST_RECOVERY:
cwnd = ssthresh;
state = CONGESTION_AVOIDANCE;
break;
}
}
void OnLoss() {
ssthresh = cwnd / 2;
cwnd = 1;
state = SLOW_START;
}
};
Rate Limiting
class RateLimiter {
private:
uint32_t max_bytes_per_second;
uint32_t bytes_sent_this_second;
uint32_t current_second;
// Token bucket algorithm
double tokens;
double max_tokens;
double tokens_per_ms;
uint32_t last_update;
public:
bool AllowSend(uint32_t bytes) {
UpdateTokens();
if (tokens >= bytes) {
tokens -= bytes;
return true;
}
return false;
}
void UpdateTokens() {
uint32_t current_time = Timer::GetCurrentTime();
uint32_t elapsed = current_time - last_update;
tokens = std::min(max_tokens,
tokens + (elapsed * tokens_per_ms));
last_update = current_time;
}
};
Implementation Details
Packet Processing Pipeline
void EQStream::ProcessIncomingData(uint8_t* data, uint32_t length) {
// Stage 1: CRC Verification
if (!VerifyCRC(data, length)) {
LogWrite(NET__ERROR, "CRC check failed");
return;
}
// Stage 2: Decryption
if (encoded_mode) {
DecryptPacket(data, length);
}
// Stage 3: Decompression
if (IsCompressed(data)) {
data = DecompressData(data, length);
}
// Stage 4: Protocol Processing
uint16_t opcode = *(uint16_t*)data;
switch (opcode) {
case OP_SessionRequest:
HandleSessionRequest(data, length);
break;
case OP_SessionResponse:
HandleSessionResponse(data, length);
break;
case OP_Packet:
HandleDataPacket(data, length);
break;
case OP_Fragment:
HandleFragment(data, length);
break;
case OP_Ack:
HandleAck(data, length);
break;
case OP_Combined:
HandleCombined(data, length);
break;
case OP_KeepAlive:
HandleKeepAlive();
break;
default:
LogWrite(NET__ERROR, "Unknown opcode 0x%04x", opcode);
}
}
Outgoing Packet Pipeline
void EQStream::SendPacket(EQApplicationPacket* app) {
// Stage 1: Convert to protocol packet
EQProtocolPacket* packet = WrapPacket(app);
// Stage 2: Check fragmentation
if (packet->Size() > max_packet_size) {
auto fragments = FragmentPacket(packet, max_packet_size);
for (auto frag : fragments) {
SendPacketInternal(frag);
}
return;
}
// Stage 3: Compression
if (compressed_mode && packet->Size() > COMPRESS_THRESHOLD) {
packet = CompressPacket(packet);
}
// Stage 4: Add to send queue
SendPacketInternal(packet);
}
void EQStream::SendPacketInternal(EQProtocolPacket* packet) {
// Add sequence number
packet->sequence = GetNextSequence();
// Store for retransmission
StoreForRetransmit(packet);
// Apply encryption
if (encoded_mode) {
EncryptPacket(packet);
}
// Add CRC
AddCRC(packet);
// Send over network
SendRaw(packet->GetBuffer(), packet->GetSize());
}
Utilities and Helper Functions
CRC Calculation
class CRCUtil {
private:
static uint32_t crc_table[256];
static bool table_initialized;
static void InitTable() {
for (uint32_t i = 0; i < 256; i++) {
uint32_t c = i;
for (int j = 0; j < 8; j++) {
c = (c & 1) ? (0xEDB88320 ^ (c >> 1)) : (c >> 1);
}
crc_table[i] = c;
}
table_initialized = true;
}
public:
static uint32_t Calculate(const uint8_t* data, uint32_t length) {
if (!table_initialized) InitTable();
uint32_t crc = 0xFFFFFFFF;
for (uint32_t i = 0; i < length; i++) {
crc = crc_table[(crc ^ data[i]) & 0xFF] ^ (crc >> 8);
}
return crc ^ 0xFFFFFFFF;
}
static bool Verify(const uint8_t* data, uint32_t length,
uint32_t expected_crc) {
return Calculate(data, length - 4) == expected_crc;
}
};
Packet Combining
class PacketCombiner {
private:
std::vector<EQProtocolPacket*> pending_packets;
uint32_t combined_size;
uint32_t max_combined_size;
public:
void AddPacket(EQProtocolPacket* packet) {
if (combined_size + packet->Size() > max_combined_size) {
Flush();
}
pending_packets.push_back(packet);
combined_size += packet->Size();
}
EQProtocolPacket* Flush() {
if (pending_packets.empty()) return nullptr;
if (pending_packets.size() == 1) {
auto packet = pending_packets[0];
pending_packets.clear();
combined_size = 0;
return packet;
}
// Create combined packet
uint32_t total_size = sizeof(uint16_t) + sizeof(uint8_t);
for (auto p : pending_packets) {
total_size += sizeof(uint8_t) + p->Size();
}
EQProtocolPacket* combined = new EQProtocolPacket(
OP_Combined, nullptr, total_size);
uint8_t* ptr = combined->data;
*ptr++ = pending_packets.size(); // Packet count
for (auto p : pending_packets) {
*ptr++ = p->Size(); // Sub-packet size
memcpy(ptr, p->GetBuffer(), p->Size());
ptr += p->Size();
delete p;
}
pending_packets.clear();
combined_size = 0;
return combined;
}
};
Sequence Number Utilities
class SequenceUtil {
public:
// Compare sequence numbers with wrap-around
static int CompareSequence(uint16_t a, uint16_t b) {
int16_t diff = a - b;
if (diff > 0x7FFF) {
return -1; // a is before b (wrapped)
} else if (diff < -0x7FFF) {
return 1; // a is after b (wrapped)
} else {
return diff; // Normal comparison
}
}
// Check if sequence is in valid window
static bool InWindow(uint16_t seq, uint16_t expected,
uint16_t window_size) {
int diff = CompareSequence(seq, expected);
return diff >= 0 && diff < window_size;
}
// Calculate distance between sequences
static uint16_t Distance(uint16_t from, uint16_t to) {
if (to >= from) {
return to - from;
} else {
return (0xFFFF - from) + to + 1;
}
}
};
Buffer Management
class BufferPool {
private:
struct Buffer {
uint8_t* data;
uint32_t size;
bool in_use;
};
std::vector<Buffer> buffers;
std::mutex mutex;
public:
uint8_t* Acquire(uint32_t size) {
std::lock_guard<std::mutex> lock(mutex);
// Find existing buffer
for (auto& buf : buffers) {
if (!buf.in_use && buf.size >= size) {
buf.in_use = true;
return buf.data;
}
}
// Allocate new buffer
Buffer new_buf;
new_buf.data = new uint8_t[size];
new_buf.size = size;
new_buf.in_use = true;
buffers.push_back(new_buf);
return new_buf.data;
}
void Release(uint8_t* data) {
std::lock_guard<std::mutex> lock(mutex);
for (auto& buf : buffers) {
if (buf.data == data) {
buf.in_use = false;
break;
}
}
}
};
Statistics Tracking
class StreamStatistics {
public:
// Packet counters
std::atomic<uint64_t> packets_sent{0};
std::atomic<uint64_t> packets_received{0};
std::atomic<uint64_t> bytes_sent{0};
std::atomic<uint64_t> bytes_received{0};
// Error counters
std::atomic<uint64_t> crc_errors{0};
std::atomic<uint64_t> sequence_errors{0};
std::atomic<uint64_t> timeout_errors{0};
std::atomic<uint64_t> retransmissions{0};
// Performance metrics
std::atomic<uint32_t> avg_rtt{0};
std::atomic<uint32_t> min_rtt{UINT32_MAX};
std::atomic<uint32_t> max_rtt{0};
std::atomic<uint32_t> packet_loss_rate{0};
// Compression statistics
std::atomic<uint64_t> bytes_before_compression{0};
std::atomic<uint64_t> bytes_after_compression{0};
void UpdateRTT(uint32_t rtt) {
// Exponential moving average
avg_rtt = (avg_rtt * 7 + rtt) / 8;
// Update min/max
uint32_t current_min = min_rtt.load();
while (rtt < current_min &&
!min_rtt.compare_exchange_weak(current_min, rtt));
uint32_t current_max = max_rtt.load();
while (rtt > current_max &&
!max_rtt.compare_exchange_weak(current_max, rtt));
}
double GetCompressionRatio() const {
if (bytes_before_compression == 0) return 1.0;
return (double)bytes_after_compression / bytes_before_compression;
}
void DumpStats() const {
LogWrite(NET__INFO, "=== Stream Statistics ===");
LogWrite(NET__INFO, "Packets: sent=%llu, received=%llu",
packets_sent.load(), packets_received.load());
LogWrite(NET__INFO, "Bytes: sent=%llu, received=%llu",
bytes_sent.load(), bytes_received.load());
LogWrite(NET__INFO, "RTT: avg=%u, min=%u, max=%u",
avg_rtt.load(), min_rtt.load(), max_rtt.load());
LogWrite(NET__INFO, "Errors: crc=%llu, seq=%llu, timeout=%llu",
crc_errors.load(), sequence_errors.load(),
timeout_errors.load());
LogWrite(NET__INFO, "Compression ratio: %.2f%%",
GetCompressionRatio() * 100);
}
};
Debug Utilities
class PacketDebugger {
public:
static void DumpPacket(const EQProtocolPacket* packet) {
LogWrite(NET__DEBUG, "Packet: opcode=0x%04x, size=%u, seq=%u",
packet->GetOpcode(), packet->Size(), packet->sequence);
HexDump(packet->GetBuffer(), packet->Size());
}
static void HexDump(const uint8_t* data, uint32_t length) {
char line[80];
char ascii[17];
for (uint32_t i = 0; i < length; i++) {
if (i % 16 == 0) {
if (i > 0) {
LogWrite(NET__DEBUG, "%s %s", line, ascii);
}
sprintf(line, "%04x: ", i);
memset(ascii, 0, 17);
}
sprintf(line + strlen(line), "%02x ", data[i]);
ascii[i % 16] = isprint(data[i]) ? data[i] : '.';
}
// Print last line
if (length % 16 != 0) {
for (uint32_t i = length % 16; i < 16; i++) {
strcat(line, " ");
}
}
LogWrite(NET__DEBUG, "%s %s", line, ascii);
}
static void TracePacketFlow(const char* stage,
const EQProtocolPacket* packet) {
LogWrite(NET__TRACE, "[%s] opcode=0x%04x seq=%u size=%u",
stage, packet->GetOpcode(), packet->sequence,
packet->Size());
}
};
Error Recovery
Connection Reset
void EQStream::Reset() {
LogWrite(NET__INFO, "Resetting connection");
// Clear all queues
ClearSendQueue();
ClearReceiveQueue();
ClearRetransmitQueue();
ClearFragmentBuffer();
// Reset sequence numbers
next_in_sequence = 0;
next_out_sequence = 0;
last_acked_sequence = 0;
// Reset encryption
if (encrypt_key) {
delete encrypt_key;
encrypt_key = nullptr;
}
if (decrypt_key) {
delete decrypt_key;
decrypt_key = nullptr;
}
// Reset compression
if (stream_compress) {
deflateEnd(stream_compress);
delete stream_compress;
stream_compress = nullptr;
}
if (stream_decompress) {
inflateEnd(stream_decompress);
delete stream_decompress;
stream_decompress = nullptr;
}
// Send reset packet
SendResetPacket();
// Reinitialize connection
state = WAIT_CONNECT;
SendSessionRequest();
}
Performance Considerations
Optimization Strategies
- Packet Batching: Combine multiple small packets
- Compression Threshold: Only compress packets > 100 bytes
- Buffer Pooling: Reuse buffers to reduce allocations
- Lock-Free Queues: Use atomic operations where possible
- Zero-Copy: Minimize data copying in pipeline
Benchmarking
class PerformanceBenchmark {
struct Metric {
std::string name;
uint64_t total_time;
uint64_t call_count;
double Average() const {
return call_count ? (double)total_time / call_count : 0;
}
};
std::map<std::string, Metric> metrics;
public:
class Timer {
PerformanceBenchmark* benchmark;
std::string metric_name;
uint64_t start_time;
public:
Timer(PerformanceBenchmark* b, const std::string& name)
: benchmark(b), metric_name(name) {
start_time = GetHighResTime();
}
~Timer() {
uint64_t elapsed = GetHighResTime() - start_time;
benchmark->AddSample(metric_name, elapsed);
}
};
void AddSample(const std::string& name, uint64_t time) {
metrics[name].total_time += time;
metrics[name].call_count++;
metrics[name].name = name;
}
void Report() {
LogWrite(NET__INFO, "=== Performance Report ===");
for (const auto& [name, metric] : metrics) {
LogWrite(NET__INFO, "%s: avg=%.2fus, calls=%llu",
name.c_str(), metric.Average() / 1000.0,
metric.call_count);
}
}
};
UDP Implementation Details
Why UDP Instead of TCP?
EverQuest II uses UDP for game client connections for several critical reasons:
- Lower Latency: No TCP handshake or connection establishment overhead
- No Head-of-Line Blocking: Lost packets don't block subsequent packets
- Custom Reliability: Only retransmit what's necessary for gameplay
- Better for Real-Time: Position updates don't need 100% reliability
- Multicast Support: Can efficiently broadcast to multiple clients
UDP Socket Implementation
class UDPSocket {
private:
int sock_fd;
struct sockaddr_in bind_addr;
public:
bool Bind(uint16_t port) {
// Create UDP socket
sock_fd = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);
if (sock_fd < 0) return false;
// Set socket options
int reuse = 1;
setsockopt(sock_fd, SOL_SOCKET, SO_REUSEADDR,
&reuse, sizeof(reuse));
// Increase buffer sizes for high throughput
int rcvbuf = 256 * 1024; // 256KB receive buffer
int sndbuf = 256 * 1024; // 256KB send buffer
setsockopt(sock_fd, SOL_SOCKET, SO_RCVBUF,
&rcvbuf, sizeof(rcvbuf));
setsockopt(sock_fd, SOL_SOCKET, SO_SNDBUF,
&sndbuf, sizeof(sndbuf));
// Bind to port
bind_addr.sin_family = AF_INET;
bind_addr.sin_port = htons(port);
bind_addr.sin_addr.s_addr = INADDR_ANY;
return bind(sock_fd, (struct sockaddr*)&bind_addr,
sizeof(bind_addr)) == 0;
}
ssize_t SendTo(const void* data, size_t len,
const sockaddr_in& dest) {
return sendto(sock_fd, data, len, 0,
(struct sockaddr*)&dest, sizeof(dest));
}
ssize_t RecvFrom(void* buffer, size_t len,
sockaddr_in& source) {
socklen_t addr_len = sizeof(source);
return recvfrom(sock_fd, buffer, len, 0,
(struct sockaddr*)&source, &addr_len);
}
};
Client Identification
Since UDP is connectionless, the server must track clients by their endpoint:
struct ClientEndpoint {
uint32_t ip;
uint16_t port;
bool operator<(const ClientEndpoint& other) const {
return (ip < other.ip) ||
(ip == other.ip && port < other.port);
}
};
std::map<ClientEndpoint, EQStream*> client_streams;
EQStream* GetStreamForPacket(const sockaddr_in& source) {
ClientEndpoint endpoint{source.sin_addr.s_addr,
ntohs(source.sin_port)};
auto it = client_streams.find(endpoint);
if (it != client_streams.end()) {
return it->second;
}
// New client - check if this is a session request
// If so, create new stream
return nullptr;
}
Reliability Layer Over UDP
EQStream implements TCP-like features over UDP:
class ReliabilityLayer {
// Sliding window for flow control
uint16_t send_window_start;
uint16_t send_window_size;
uint16_t recv_window_start;
uint16_t recv_window_size;
// Retransmission timers
struct RetransmitEntry {
EQProtocolPacket* packet;
uint32_t send_time;
uint32_t retransmit_count;
uint32_t rto; // Retransmission timeout
};
// Round-trip time estimation (Jacobson/Karels algorithm)
uint32_t srtt; // Smoothed RTT
uint32_t rttvar; // RTT variance
uint32_t rto; // Retransmission timeout
void UpdateRTT(uint32_t measured_rtt) {
if (srtt == 0) {
// First measurement
srtt = measured_rtt;
rttvar = measured_rtt / 2;
} else {
// Update estimates
uint32_t delta = abs((int)measured_rtt - (int)srtt);
rttvar = (3 * rttvar + delta) / 4;
srtt = (7 * srtt + measured_rtt) / 8;
}
// Calculate RTO (with minimum and maximum bounds)
rto = srtt + 4 * rttvar;
rto = std::max(rto, 200u); // Min 200ms
rto = std::min(rto, 3000u); // Max 3 seconds
}
};
Conclusion
EQStream provides a robust, feature-rich networking layer that handles the complexities of real-time game communication over UDP. Key features include:
- Reliability through sequence numbers and retransmission (TCP-like over UDP)
- Security through RC4 encryption
- Efficiency through compression and packet combining
- Low Latency by using UDP instead of TCP
- Custom Flow Control optimized for game traffic
- Scalability through proper resource management
- Maintainability through clear separation of concerns
The protocol is specifically designed to handle the demands of MMO gameplay over UDP while maintaining compatibility with the EverQuest II client protocol specifications. By implementing reliability at the application layer rather than using TCP, the protocol achieves better performance for real-time gaming scenarios.