wc3lib: MDLX view implementation

Preface

School is annoying and therefore we have to do some project work this year, so I got the idea to combine one of my current projects with this.
Initially, I wanted to do some compiler things with Bison and Flex to create first JASS++ or vJass compiler support but since I haven’t been very familiar with these things (not enough to make sure that I had got finished with everything I would have planned) I dropped the idea. Later I dropped the MPQ support as possible project work as well because I thought it would be too boring only presenting some archive format and tools, so finally I came to MDLX support.
The really big problem about this is the huge amount of things which could be supported by an implementation. Of course I limited my project work goals for that reason and made sure I would be able to use OGRE, Qt, KDE and everything else I needed to finish my program.
Besides, most of MDLX loading had already been supported at the end of my recent summer holidays (2010) when I made that decision.

Let’s come to the point. Here’s a list of things I originally wanted to support with my project work:

• loading MDX files
• loading texture files (as well BLP)
• setup a light configurable model view widget in KDE
• displaying Geosets of a model file with material layers and their corresponding textures
• displaying light sources
• using pivot points
• displaying attachments
• using cameras
• displaying and using collision forms for “hit tests”

In the meantime, some things have changed. Fortunately, I’ve been able to add BLP reading support for JFIF compression. Hence I’m able to add all kinds of “Warcraft: III” BLP textures now. On the other hand unfortunately OGRE does not support image object pixel modification like Qt does. For this reason, I have to read BLP files into a Qt image object and write it as raw data (e. g. PNG/BMP) into another data stream which OGRE has to read from. Recently I’ve come a bit more familiar with OGRE’s class ImageCodec so maybe I’m going to implement more direct read and write functionality for BLP in OGRE in future. Furthermore, I’ve added some team color and glow selection actions and dialogs for displayed models.

Back in 2009 I was already starting to create C++ classes for Blizzard Entertainment’s MDX format by using the unofficial specification. Later I did some more abstraction to it and added classes for “blocks”, “nodes”, “objects”, “scalings”, “translations”, “rotations”, “alphas”, “animated properties” etc.

The whole reading process of MDX files has never been a big problem since it is a simple binary format. Reading MDL files is much more complicated because you have to defined correct, abstract parsing rules.

Actually, the main question was “How should it be displayed?”. There are several ways of doing this:

• converting MDLX files into another format and reading this format – slow but useful for editing it with a format supporting tool like Blender
• reading MDLX files into heap and displaying it by using state functions of OpenGL or Direct3D – fast but complicated and dependent to one renderer implementation
• reading MDLX files into heap and displaying it by using an abstract render library – faster than converting it, more abstract (you could choose your native renderer implementation)

In my opinion, first solution is the best one in most cases. It’s always very useful to have an importing/exporting tool for a common format such as Blend because it allows you to edit models of MDLX format with some great tool and additionally allows you to convert it into many other formats by using Blender’s importing/exporting tools for instance.

The problem about that solution mainly is its waste of performance. Imagine you want to display models like in its corresponding game, as fast as possible. The only possibility to do this would be to use Blender’s renderer which probably is not optimized for games/clearly displaying loaded scenes.

Of course there is a game engine by now but there still would be loaded too much data for being really fast since Blender has always been a tool for editing not only displaying scenes and supports much more rendering features than actually required by MDLX format (besides some physics and game logic features etc.).

Additionally Blend format is very special and not abstract enough to display MDLX files easily. It would be a long and taxing process to create a really good import/export add-on such as 3ds Max one’s which still has a massive lack of supported features as well.

If you want to have the fastest displaying tool you would have to choose the second solution. Calling renderer routines directly probably is the most efficient way to render scenes by using hardware acceleration. The main problem about that solution is that you have to rely on one specific render API (e. g. OpenGL). Besides you can’t expand it easily. It’s like a comparison between C and C++ code. The second one is much more abstract than the first one and therefore can be expanded and maintained much easier.

Mago’s War3 Model Editor uses the second solution by calling Direct3d routines to display MDL and MDx files. Unfortunately that is one of the reasons why it is only available on Windows systems (except you’re using some kind of emulation like wine). The lack of abstraction brings much more further issues if you think of implementing advanced 3d editor and scene rendering functionality to your tool.

That’s the main reason why I’ve chosen the last solution for my project work/the wc3lib.

Implementation

Finally, I handed in my project work on 6th May 2011. At that moment I had finished common loading display of MDX files in an OGRE based scene. By reading and copying some stuff about OGRE and QtOgre on the Internet I was able to create the C++ class “ModelView” deriving from class QWidget. That class probably is the most basic one of the whole project work since it connects the two big frameworks Qt and OGRE and allows you to create 3d graphics scene displaying Qt widgets.

OGRE has some really useful classes which allow you to modularize and split up views and scene data. For example you can add multiple viewports to one single render window. At the moment my class “ModelView” has one primary viewport and render window member but theoretically one view could be expanded to multiple views. Due to this well designed abstraction it’s very easy to add new features once you got used to OGRE’s API.

The actual implementation of my school project work’s goals differs a little bit to its original list. I dropped light sources, attachments and pivot points. The latter was dropped because pivot points are only used by nodes which aren’t a part of the project work at all (originally I thought they would be used by Geosets, too). Light sources and attachments were dropped because they are some kind of animated properties (which aren’t part of my project work as well).

As already mentioned above I have been able to add BLP support which became a new item on my goals list.

Let’s take a look at some screenshots I made for the project work’s documentation:

That’s the famous “dark portal” you may know from the “Warcraft III: The Frozen Throne” alliance campaign. I took it as example since it most people who know the game will know the portal as well.

It’s not animated as you might see when you look at the ray texture.

In wc3lib we have different modules for Blizzard’s file formats which allow us to read and write them actually. There’s the abstract class “Format” which is derived from all format classes such as “Mdlx” and “Blp”.

For all categories of formats there are different modules in wc3lib which are logically separated by sub directories. “mdlx” is the MDLX format’s sub directory, for instance. Indeed, there’s only one module which relies on all formats. It’s called “editor” and combines all formats for graphical usage using Qt, KDE and OGRE libraries. “ModelView” class which was mentioned above is part of it as well since it uses Qt and OGRE for MDLX display.

Actually, “editor” module is optional and has not to be compiled together with the other modules. Initially it had been created as playground module to test the other modules correctness. Now, among class “ModelView” there are some World Editor and World Editor modules emulation classes which allow simple global resource and MPQ archive handling (for data sources, consider MPQ support of wc3lib has been not finished yet). For my school project work the class “OgreMdlx” is one of the most important ones among “ModelView”. It allows you to create ManualObject instances in its corresponding OGRE scene/model view by using its corresponding MDLX data. Simply imagine it as OGRE/MDLX pair.

The initial creation process of the OGRE object can be very time expansive so it wouldn’t be a good idea to use “OgreMdlx” instances for every unit, doodad etc. if you want to display your “Warcraft: III” map rather than creating Mesh instances for all MDLX geosets (via convertToMesh) and using them as entities. Probably, this has to be done anyway since only meshes can use animations. Therefore “OgreMdlx” only represents a bunch of static geometry data which can be used for dynamic animated entities once it has been loaded.

As animations are not part of my school project work the second “secret” class which uses those meshes and can be animated and instanced via OGRE’s entity system hasn’t been created or designed yet.

OGRE allows you to change user’s view easily for example to wireframe mode as you can see in the picture above. For simple user control I’ve added some mouse rotation and translation control to class “ModelView” via Qt’s events. There’s still a massive lack on “when the scene should be rendered”. At the moment member function called “render” of class “ModelView” is called every time something is changed via control/view actions or Qt’s widget paint/resize etc. events.

In my opinion a multithreaded solution would be much nicer and more efficient than calling “render” all the time which can be very annoying if you think of the nearly unlimited range of model editing scenarios which could be added in the future.

As OGRE’s API reference states using Ogre::Root::startRendering() which does exactly what I want to have (automatic rendering cycle) should not be used in model editing programs for example. Probably, because it is much faster to change only things which were changed actually. The best compromise could be optional rendering loop for model views which show you map terrain with placed objects for example where objects are constantly animated.

Some days before I had to hand in my project work I was able to fix an annoying display bug which actually occurred because of one line in my code. There you can read about my problem which fortunately could be solved since it was that simple. I just thought you would have to call the index function not only for creating faces. Thank you Kojack for helping a noob like me finishing his school work on time :-).

Here’s a picture of the nasty model faces distortion:

As mentioned in my goals list I’ve added collision shapes and team color/glow support as well. Collision shapes are some kind of simplification for Geoset selection which can either be spheres or boxes. Team colors are used in “Warcraft: III” to indicate a unit’s or building’s owner/player. Team glows are nearly the same but used for heroes only.

Team color and glow textures are loaded globally in my project work and can be used by member functions of class “Editor”.  Collision shape “hit test” (as it is called by Blizzard in their Art Tools’ manual) is implemented in class “ModelEditorView” (deriving from class “ModelView”) using an OGRE-based ray query.

Unfortunately, collision shapes seem to be too big at the moment as you can see on the following screenshot (the little thing it the center of the view is an Arachnathid):

Future

At the beginning of the creation of the “editor” module of “wc3lib” I defined the abstract class “Resource” from which class “OgreMdlx” derives. Meanwhile, I’ve expanded it and added some multithreading support via virtual member functions and by the fact that “Resources” derives from QThread. It should be possible to load as many resources as possible at the same time to take advantage of multiprocessor systems. For this reason I will have to take a look on Qt’s classes and their threading support since I’ve got already some issues when creating objects in separated threads etc.. When I begun writing this article I had already taken a look on ImageCodec of the OGRE framework. By the meantime, I was able to implement some read functionality for the BLP format which should heavily improve the speed of loading BLP textures in OGRE (without Qt’s import/export step).

Furthermore, I want to improve the mouse and key control and make some model view settings more customizable. Actually the most important issues I will have to fix are the ones which are related to my school project work’s goals. As I have to present my project work in school on 28th or 29th June 2011 it would be annoying to have some unfinished stuff, so I will try to fix the collision shapes size, the corresponding “hit test” implementation and to improve the team color and glow dialog selection until that date.

Hopefully, the new C++ ISO will be released this year. The new language standard would allow you to create more maintainable code and use some completely new language features.

Publication

Of course I’ve uploaded my school project work which is almost completely written in German (except its required English summary). Some parts are nearly equal to this article but it’s larger, I think and written more detailed for people who don’t know much about the MDLX format (as my teacher(s): one of my teachers thought MDLX must be a Roman number …):

Project Work (German)

All corresponding source code of “wc3lib” can be found in my SVN respository on SourceForge.net.