eq2go/MODERNIZE.md

# Package Modernization Instructions

## Goal
Transform legacy packages to use focused, performance-optimized bespoke master lists with simplified database operations. REMEMBER THAT THIS IS FOR A TOTAL REWRITE, NOT A VARIATION. If any step does not apply to a given package, it is okay to skip it and to focus on modernizing other areas of the package.

## Steps

### 1. Create Bespoke MasterList Implementation
Replace generic implementations with specialized, high-performance master lists:
```go
// Create domain-specific master list optimized for the use case
type MasterList struct {
    // Core storage
    items map[K]*Type // ID -> Type
    mutex sync.RWMutex

    // Specialized indices for O(1) lookups
    byCategory map[string][]*Type   // Category -> items
    byProperty map[string][]*Type   // Property -> items

    // Cached metadata with invalidation
    categories []string
    metaStale  bool
}
```

### 2. Consolidate Package Structure

**Remove:**
- Legacy wrapper functions (LoadAll, SaveAll, etc.)
- Duplicate database files (database.go, database_legacy.go)
- Generic MasterList dependencies
- README.md (use doc.go instead)
- Separate "active_record.go" files

**Consolidate into:**
- `{type}.go` - Main type with embedded database operations
- `types.go` - Supporting types only (no main type)
- `master.go` - Bespoke MasterList optimized for domain
- `player.go` - Player-specific logic (if applicable)
- `doc.go` - Primary documentation
- `{type}_test.go` - Focused tests
- `benchmark_test.go` - Comprehensive performance tests

### 3. Refactor Main Type

Transform the main type to include database operations:

```go
// Before: Separate type and database operations
type Achievement struct {
    ID    uint32
    Title string
}

func LoadAchievement(db *database.Database, id uint32) (*Achievement, error)
func SaveAchievement(db *database.Database, a *Achievement) error

// After: Embedded database operations
type Achievement struct {
    ID    uint32
    Title string

    db    *database.Database
    isNew bool
}

func New(db *database.Database) *Achievement
func Load(db *database.Database, id uint32) (*Achievement, error)
func (a *Achievement) Save() error
func (a *Achievement) Delete() error
func (a *Achievement) Reload() error
```

### 4. Create Bespoke MasterList

Replace generic implementations with specialized, optimized master lists:

```go
// Before: Generic base with limited optimization
type MasterList struct {
    *common.MasterList[uint32, *Achievement]
}

// After: Bespoke implementation optimized for domain
type MasterList struct {
    // Core storage
    achievements map[uint32]*Achievement
    mutex        sync.RWMutex

    // Domain-specific indices for O(1) lookups
    byCategory  map[string][]*Achievement
    byExpansion map[string][]*Achievement

    // Cached metadata with lazy loading
    categories  []string
    expansions  []string
    metaStale   bool
}

func (m *MasterList) AddAchievement(a *Achievement) bool {
    m.mutex.Lock()
    defer m.mutex.Unlock()

    // Check existence
    if _, exists := m.achievements[a.AchievementID]; exists {
        return false
    }

    // Add to core storage
    m.achievements[a.AchievementID] = a

    // Update specialized indices
    m.byCategory[a.Category] = append(m.byCategory[a.Category], a)
    m.byExpansion[a.Expansion] = append(m.byExpansion[a.Expansion], a)

    // Invalidate metadata cache
    m.metaStale = true
    return true
}

// O(1) category lookup
func (m *MasterList) GetByCategory(category string) []*Achievement {
    m.mutex.RLock()
    defer m.mutex.RUnlock()
    return m.byCategory[category]
}
```

### 5. Optimize for Domain Use Cases

Focus on the specific operations your domain needs:

```go
// Implement GetID() for consistency
func (a *Achievement) GetID() uint32 {
    return a.AchievementID
}

// Add domain-specific optimized methods
func (m *MasterList) GetByCategoryAndExpansion(category, expansion string) []*Achievement {
    // Use set intersection for efficient combined queries
    categoryItems := m.byCategory[category]
    expansionItems := m.byExpansion[expansion]

    // Use smaller set for iteration efficiency
    if len(categoryItems) > len(expansionItems) {
        categoryItems, expansionItems = expansionItems, categoryItems
    }

    // Set intersection using map lookup
    expansionSet := make(map[*Achievement]struct{}, len(expansionItems))
    for _, item := range expansionItems {
        expansionSet[item] = struct{}{}
    }

    var result []*Achievement
    for _, item := range categoryItems {
        if _, exists := expansionSet[item]; exists {
            result = append(result, item)
        }
    }
    return result
}
```

### 6. Simplify API

**Remove:**
- Generic MasterList dependencies
- Legacy type variants (Achievement vs AchievementRecord)
- Conversion methods (ToLegacy, FromLegacy)
- Duplicate CRUD operations
- Complex wrapper functions
- Slow O(n) filter operations

**Keep:**
- Single type definition
- Direct database methods on type
- Domain-specific optimized operations
- O(1) indexed lookups where possible
- Lazy caching for expensive operations

### 7. Update Documentation

Create concise `doc.go`:

```go
// Package achievements provides [brief description].
//
// Basic Usage:
//
//	achievement := achievements.New(db)
//	achievement.Title = "Dragon Slayer"
//	achievement.Save()
//
//	loaded, _ := achievements.Load(db, 1001)
//	loaded.Delete()
//
// Bespoke Master List (optimized for performance):
//
//	masterList := achievements.NewMasterList()
//	masterList.AddAchievement(achievement)
//
//	// O(1) category lookup
//	combatAchievements := masterList.GetByCategory("Combat")
//
//	// O(1) expansion lookup
//	classicAchievements := masterList.GetByExpansion("Classic")
//
//	// Optimized set intersection
//	combined := masterList.GetByCategoryAndExpansion("Combat", "Classic")
package achievements
```

### 8. Testing

Create focused tests:
- Test core type operations (New, Save, Load, Delete)
- Test MasterList basic operations
- Remove legacy compatibility tests
- Keep tests simple and direct

## Key Principles

1. **Single Source of Truth**: One type definition, not multiple variants
2. **Embedded Operations**: Database methods on the type itself
3. **Bespoke Master Lists**: Domain-specific optimized implementations
4. **Performance First**: O(1) lookups, lazy caching, efficient algorithms
5. **Thread Safety**: Proper RWMutex usage with minimal lock contention
6. **No Legacy Baggage**: Remove all "Legacy" types and converters
7. **Documentation in Code**: Use doc.go, not README.md
8. **Comprehensive Benchmarking**: Measure and optimize all operations

## Migration Checklist

- [ ] Identify main type and supporting types
- [ ] Consolidate database operations into main type
- [ ] Add db field and methods to main type
- [ ] **Create bespoke MasterList optimized for domain**
- [ ] **Add specialized indices for O(1) lookups**
- [ ] **Implement lazy caching for expensive operations**
- [ ] **Add set intersection algorithms for combined queries**
- [ ] Implement GetID() for consistency
- [ ] Remove all generic MasterList dependencies
- [ ] Remove all legacy types and converters
- [ ] Update doc.go with performance-focused examples
- [ ] Simplify tests to cover core functionality
- [ ] **Add concurrency tests for thread safety**
- [ ] **Create comprehensive benchmarks (benchmark_test.go)**
- [ ] **Verify performance meets targets (sub-microsecond operations)**
- [ ] Run `go test -race` to verify thread safety
- [ ] Run `go fmt`, `go test`, and `go test -bench=.`

## Expected Results

- **80% less code** in most packages
- **Single type** instead of multiple variants
- **Thread-safe** operations with optimized locking
- **O(1) performance** for common lookup operations
- **Sub-microsecond** response times for most operations
- **Specialized algorithms** for domain-specific queries
- **Consistent API** across all packages
- **Better maintainability** with focused, understandable code
- **Performance predictability** through comprehensive benchmarking

## Benchmarking

After modernization, create comprehensive benchmarks to measure performance improvements and ensure no regressions:

### Required Benchmarks

**Core Operations:**
```go
func BenchmarkTypeCreation(b *testing.B)         // New type creation
func BenchmarkTypeOperations(b *testing.B)       // CRUD operations
func BenchmarkMasterListOperations(b *testing.B) // Collection operations
```

**Performance Critical Paths:**
```go
func BenchmarkCoreAlgorithm(b *testing.B)      // Main business logic
func BenchmarkConcurrentAccess(b *testing.B)   // Thread safety
func BenchmarkMemoryAllocation(b *testing.B)   // Memory patterns
```

**Comparison Benchmarks:**
```go
func BenchmarkComparisonWithOldSystem(b *testing.B) // Before/after metrics
```

### Benchmark Structure

Use sub-benchmarks for detailed measurements:
```go
func BenchmarkMasterListOperations(b *testing.B) {
    ml := NewMasterList()
    // Setup data...

    b.Run("Add", func(b *testing.B) {
        for i := 0; i < b.N; i++ {
            ml.Add(createTestItem(i))
        }
    })

    b.Run("Get", func(b *testing.B) {
        b.RunParallel(func(pb *testing.PB) {
            for pb.Next() {
                _ = ml.Get(randomID())
            }
        })
    })
}
```

### Mock Implementations

Create lightweight mocks for interfaces:
```go
type mockPlayer struct {
    level    int16
    location int32
}

func (p *mockPlayer) GetLevel() int16 { return p.level }
func (p *mockPlayer) GetLocation() int32 { return p.location }
```

### Performance Expectations

Target performance after bespoke modernization:
- **Creation operations**: <100ns per operation
- **ID lookup operations**: <50ns per operation (O(1) map access)
- **Indexed lookup operations**: <100ns per operation (O(1) specialized indices)
- **Combined queries**: <5µs per operation (optimized set intersection)
- **Cached metadata**: <100ns per operation (lazy loading)
- **Memory allocations**: Zero allocations for read operations
- **Concurrent access**: Linear scaling with cores, minimal lock contention
- **Specialized algorithms**: Domain-optimized performance characteristics

### Running Benchmarks

```bash
# Run all benchmarks
go test -bench=. ./internal/package_name

# Detailed benchmarks with memory stats
go test -bench=. -benchmem ./internal/package_name

# Compare performance over time
go test -bench=. -count=5 ./internal/package_name

# CPU profiling for optimization
go test -bench=BenchmarkCoreAlgorithm -cpuprofile=cpu.prof ./internal/package_name
```

## Example Commands

```bash
# Remove legacy files and generic dependencies
rm README.md database_legacy.go active_record.go

# Remove generic MasterList imports
grep -r "eq2emu/internal/common" . --include="*.go" | cut -d: -f1 | sort -u | xargs sed -i '/eq2emu\/internal\/common/d'

# Rename if needed
mv active_record.go achievement.go

# Test the changes
go fmt ./...
go test ./...
go test -race ./...  # Verify thread safety
go test -bench=. ./...
go build ./...

# Performance verification
go test -bench=BenchmarkMasterListOperations -benchtime=5s ./internal/package_name
go test -bench=. -benchmem ./internal/package_name
```