wc3lib: MPQ support

A long time ago I’ve posted the last web log entry, so now it’s time for a new one.
In this article I want to describe Blizzard’s MPQ format for those of you who don’t know already and I want to write how I’m going to add read and write support to the wc3lib.

The MPQ format a format created by Blizzard Entertainment to create file archives for their games.
Like most (or all) Blizzard’s formats its documentation has never been made public so people had to analyse it themselves.
As some sources do state ([1], [2]) Tom Amigo was the first person who created a “stormless” MPQ editor (in the year 2000).
Stormless means without the Storm library. The Storm library is Blizzard’s own MPQ library which is dispatched with all of their games.
It doesn’t only provide MPQ read- and writing functionality, it does also provide many other cross-platform functions which are required by Blizzard’s games.
The easiest way to use MPQ archives is to use this library. Since it has been checked out and documented well enough to read and write MPQ archives and there are already some closed and open source tools which do use its API, people should be able to easily create their own library/tool or whatever.
So maybe at this point you’re asking yourself why we should discuss any longer since there’s already a nice possibility to use the format.
The answer is easy: Blizzard’s Storm library is closed source and maybe not optimized for your usages.
Surely it’s not really necessary to work any further on this but (for me) it’s very interesting to work on such a format, although. Additionally there comes the fact that I nearly always use Linux and want to write some scripts for my project (“Die Macht des Feuers”) which create/release my custom MPQ archives automatically.
Using Windows code is very annoying since most times it’s very unclear and almost unreadable. Besides I’ve to use wine every time I want to do something with it.
As I did not want to create or write everything myself when I started the wc3lib project I searched for some MPQ libraries/tools which do implement their own functions.
My searching results were: SFmpqapi, StormLib, MpqLib and libmpq.
The last one does only support reading MPQ archives and is written in C. The MpqLib is written in C++ but it’s .NET and therefore not cross-platform. Additionally it’s only a wrapper over the StormLib which makes it (in my opinion) completely useless for most people. The StormLib in my opinion is the best choice if you want to create a cross-platform application which can read AND write MPQ archives.
It was ported to Linux and Mac OS X (Marko Friedemann and Sam Wilkins) in 2006 and provides nearly the same API as SFmpqapi does.
The main disadvantages of the SFmpqapi library are that it doesn’t seem to be maintained anymore (last update 2002, StormLib 2009) and it does only provide Linux support which didn’t work on my system (tried to compile it with g++, it also doesn’t provide any makefiles).
The StormLib and the libmpq libraries were the only ones which I was able to compile on my platform.
libmpq seems to work as well but since it does only provide read functionality it’s useless. However, its code is written very well (ISO C) and it provides some nice tools (fuse MPQ plug in) which makes it particularly interesting for Linux users/programmers.
However, SFmpqapi is used by the WEHelper project and additionally by the Jass NewGen Pack project since it probably was the first real usable MPQ library when people started modifying Warcraft 3 .
When I began writing and designing the wc3lib I intended to use the StormLib for providing MPQ support. I wanted to write some wrapper classes (as MpqLib creator Magos did).
My first contact to the StormLib goes back to June in 2009 when started creating a simple MPQ editor. In fact I was able to use the StormLib as programming interface but creating and reading archives didn’t work correctly.
However, I still wanted to use the library and to improve it as needed.
Later (after writing first implementations of BLP and MDLX modules) I started reading Justin Olbrantz’ MPQ format specification (version 1.0, year 2006) and became more interested in it. First I wrote some classes for providing a nice programming interface to MPQ archives but then I started implementing my own functions. For this reason wc3lib’s MPQ support is a custom one now and unfortunately still in beta stage.
Since wc3lib is created by using the programming language C++ it provides some classes to read and modify MPQ archives.
At the moment there are four different classes which provide access to all MPQ archive’s elements:

  • Mpq: Main class which represents single archive.
  • Hash: Should be read-only class and provides access to single hash table entry.
  • Block: Also read-only class which provides access to single block table entry.
  • MpqFile: Represents single file contained by an archive.
  • Sector: Provides access to an MPQ file data sector. Usually this class should not be used by users.

Normally you just need to know the classes Mpq and MpqFile which provide all functions you need to add, rename, move or remove files from archives or to modify files and archives attributes.
MPQ archives do always contain hash and block table which provide entry lists for files.
Each file has exactly one hash table entry which allows user to search for files using their hash values and the hash table of the archive.
Additionally the file consists of blocks which again do consist of sectors.
Each sector can be compressed differently by using various compression methods (Huffman, Bzip2 etc.).
Some compression and decompression functions have been imported from the StormLib (e. g. PKWare and Wave) since it would be much and unnecessary work to write custom ones.
Some others (like inflate and deflate – Zlib) were implemented by myself using some documentation stuff.
All MPQ related compressing, decompression and hashing functions can be reached by including header file “” but should usually not be required by user.
There are various surrounding conditions to clarify when creating such a library. First of all I had design the classes and their methods based on the MPQ specification (which incorrectly uses int8 instead of int16 by the way).
At the moment you can only get constant access to MPQ files and writing file functionality is not implemented yet.
The variety of compression algorithm made me creating some kind of dependencies since I have to use external libraries for some of the compression algorithm.
These dependencies did also occur when I created the BLP module since it needs to compress and decompress JPEG images.
At the moment I work on changes from compile time linking support (to shared objects) to run time linking by using functions like dlopen.
This feature would prevent unnecessary recompiling when using other compile flags but also means much more work.
There’s a simple class now called LibraryLoader which I’ve added to solve this.
Another problem I’ve to consider is code licensing of externally and internally used code.
For instance Zlib and bzip2 do use open source licenses and has to be installed additionally to the wc3lib when using MPQ module but as I mentioned there’s also some code which is directly implemented in the wc3lib itself (took from StormLib and MPQ specification) which I’ll have to license separately.
The author of the MPQ specification, Justin Olbrantz, mentions that you’re allowed to distribute everything as long as stating the author which should not be a big problem for me.
By contrast, Ladislav Zezula author of StormLib does not specify any license and therefore owns copyright to it.
It would be nice to see StormLib with GPL or any similar license.
mpqlib uses GPLv2 license, MpqLib uses GPLv3 (but consider that it needs StormLib) and SFmpqapi is licensed in a similar way to the MPQ specification.
C++ does provide nice features (operator overloading, friends) to develop such API. For example you can use stream operators to add files to archives:

#include // Include MPQ module of wc3lib.

using namespace wc3lib::mpq; // Use MPQ module namespace.

void testFunction()
{
class Mpq mpq1;

// Using exception handling to catch errors.
try
{
mpq1.open("War3x.mpq"); // Opens an existing MPQ archive.
}
catch (class Exception &exception)
{
std::cerr << "Unable to open archive: " << exception << std::endl;

return;
}

class MpqFile *listfileFile = mpq1.findFile("(listfile)");

if (listfileFile == 0) // Did not find file.
return;

std::cout << "(listfile) file data:\n" << *listfileFile << std::endl; // Writes file content into output stream.

class Mpq mpq2;

try
{
mpq2.open("War3.mpq");
}
catch (class Exception &exception)
{
std::cerr << "Unable to open archive: " << exception << std::endl;

return;
}

try
{
mpq2 << *listfileFile; // Copies file content data and adds file to second archive.
}
catch (class Exception &exception) // Should an exception if file does already exist.
{
std::cerr << "Unable to add file: " << exception << std::endl;

return;
}

// Like fstreams, MPQ archives are being closed automatically when being deleted (at end of scope).
}

This little (completely useless) example shows you how to use stream operators and some functions (open and findFile). Of course there is already more functionality than this but I just wanted to show you that it’s not hard to use the library.
Exception handling improves (in my opinion) error handling and makes it easier for developers to check if everything was fine.
Internally Mpq and MpqFile classes do only read meta data (like hash, block and file sector table). Primary when user wants to read some file data (e. g. file data is being written into output stream as in the example above) it’s allocated. For this reason it would be hard to work with two instances of the same MPQ archive and adding or deleting some files (multithreading). I hope I will have left some time to add multithreading support one day but usually it should be enough without for most applications. Consider that MPQ archives do not save file paths in header, hash or block table. They’re always stored separately in the “(listfile)” file which is not required necessarily.
Further plans I’ve made are for the editor module of wc3lib which allow users some GUI support for Blizzards formats. Maybe I’m going to implement some formats like the MPQ one as KIO slaves.
This would allow KDE users to open and modify MPQ archives e. g. in Konqueror directly.
Contribution would be appreciated, so if you know Python or C++ (I would like to add Python bindings or to write some parts of wc3lib in Python since it’s easier to maintain) and want to help just contact me.

Advertisements

5 Responses to “wc3lib: MPQ support”

  1. Hello,

    I have started working on project SMPQ [1] which will be cross platform command line tool (like unix tar or zip) for manipulating (create/append/read/extract files) mpq archives using StormLib library [2]. Part of this project will be (read/write) KIO slave KDE4 plugin [3] too.

    1 – https://launchpad.net/smpq
    2 – http://zezula.net/en/mpq/stormlib.html
    3 – http://en.wikipedia.org/wiki/KIO

    • Baradé Says:

      Nice to hear that something more is going on in Warcraft 3 formats related programming. Maybe I will check out some of your KIO code which might be very interesting for my implementation.
      Unfortunately, I have to restrict my programming work to MDLX implementation at the moment since I’ll probably use some parts of it as project work for my current school.
      As I can remember, MPQ functionality is relatively far in wc3lib (considering that I am still the only developer). You can already read some data but there are some serious problems about getting correct file sectors numbers and compressing/decompressing files with those various policies (using C implementations etc.).

      • My project SMPQ is complete. smpq command line program is very powerfull (it implement all options from stormlib library) and kio-smpq kio slave is done too.

        Source code, binaries for Windows and PPA for Ubuntu on project site: https://launchpad.net/smpq

  2. Baradé Says:

    That’s a really great thing! When I tried to add a similar KDE IO slave for the MPQ format and used class KArchive to support it as usual archive format I had to create a new wishlist entry (https://bugs.kde.org/show_bug.cgi?id=277615). As far as I understand your code you did it yourself and implemented a completely new slave. Hopefully they will make class ArchiveProtocol extensible for custom formats in later KDE versions that I do not have to rewrite everything. Otherwise class KArchive would be useless for external people who do not develop kdebase runtime stuff. Unfortunately it will still take a long time until wc3lib’s MPQ support has been finished but I’ve increased performance massively and I am using Boost bzip2 and zlib filters now which makes code much more maintainable (sorry for that but I hate StormLib’s C-like API).
    Probably I will create a new wc3lib related blog article in some weeks when I’ve finished my current big revision (new repository is on Gitorious now: https://gitorious.org/wc3lib).

    • You are right that StormLib has not good looking API… (It is not C, but ugly WINAPI C++ style) But last year StormLib was only one free and open source library which support creating/extracting MPQ archives version 1 – version 4.

      I did not know that something like KArchive API exists. But kio slave API was not new for me. 2 years ago I wrote gammu kio slave (for nokia S40 internal filesystem throw USB) which only run external program gammu. I needed it to backup and cleanup my old phone… With my plugin deleting all files, applications, sms, … was really easy: One delete key on top folder in Dolphin 🙂 (and wait 15 minutes).

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: