RGB Classic Games Forum

UPDATE (Dec 25th 2015): This is a generic notice stating that this post does not have to be edited after any update to the repository or this topic; And so, possibly mentioning in this post for the last time, the update for today is the addition of versions 1.0 and 2.0 (1 and 6 episodes releases) of Blake Stone: Aliens of Gold.

UPDATE (August 28th 2015): The "wolf3d" subdirectory has been renamed "w3d_plus", now that the same codebase can be used to recreate (ignoring debugging information differences) the EXE for the DOS port of Super 3-D Noah's Ark, v1.0. The same applies to the .diff file. Other than this edit, I haven't touched the original post.

Hi,

There's been a bit weird project of mine hosted here: https://bitbucket.org/NY00123/gamesrc-ver-recreation/

I'm going to quote-and-edit some of the texts originally written by me here: http://wolf3d.darkbb.com/t3302-wolf3d-s ... estoration
There's also a relevant topic in this place: http://diehardwolfers.areyep.com/viewtopic.php?t=7098

Let's begin with telling how did I get to all of that.

Back in September of 2014, following a campaign to acquire the rights to publish the "lost" episode of Commander Keen known as "Keen Dreams", original sources for this episode were made available. These include sources not just for one, but apparently all versions of Keen Dreams originally made in the 90s for DOS, with the possible exception of 1.93 (which isn't that different from 1.92 anyway and could also be reconstructed).

It looked to me like the sources were in a relatively good state, and that the released forms were not greatly different from the originals. It's also stated that Borland C++ 2.0 was originally used to build the EXEs, and it also looks clear that a lot of the EXEs were originally packed using LZEXE 0.91.

Therefore, I decided to have a little attempt, using Borland C++ 2.0 and LZEXE 0.91 to build and pack version 1.13 (Shareware release). Turns out, I got the exact same EXE as created in 1992, byte-by-byte. I also got such results with at least a few other versions.

This looks like the main motivation behind starting the "gamesrc-ver-recreation" repository above (although originally under a different location). I think that initially, I wanted to try and recreate the Catacomb Abyss v1.13 EXE. Eventually, this followed with the Catacomb 3-D v1.0 EXE and the Catacomb Abyss v1.24 EXE. To compare, the Catacombs sources as released on June of 2014 can be used to build the Catacomb 3-D v1.22 EXE, as well as an EXE *very* close to Catacomb Abyss v1.24. However, for some reason, in order to get exactly the original v1.24 EXE, STARTTILE8 has to be edited in GFXE_ABS.EQU, but not in GFXE_ABS.H (there's some function in ID_VW_AE.ASM which shouldn't be called but is compiled into the EXE, for which this has an effect). The exact same edit is also required for recreation of the v1.13 EXE.

Eventually, I decided to fiddle with the Wolf3D/SOD sources for similar purposes, gradually adding projects for recreation of certain EXEs over time. With a few exceptions, the right tools can now be used to build original EXEs from the sources above, byte-by-byte. The EXEs I refer to are these:
- Wolfenstein 3D Shareware v1.0+1.1+1.2+1.4, Apogee releases (with cheats).
- Wolfenstein 3D Registered v1.1+1.2+1.4, Apogee releases (with cheats).
- Wolfenstein 3D Registered v1.4, early GT Interactive release.
- Wolfenstein 3D Registered v1.4, ID Software release (same EXE as in the early
GT Interactive release, except for a different logo in the signon screen).
- Wolfenstein 3D Registered v1.4, late GT Interactive release
(no three-letter code is shown after completing an episode).
- Wolfenstein 3D Registered v1.4, Activision release.
- Spear of Destiny demo v1.0, FormGen release.
- Spear of Destiny versions 1.0 (possibly with a different BSS section size) and 1.4, FormGen releases (copy protected).
- Spear of Destiny v1.4, Activision release (no copy protection).

It should be noted that the original SOD v1.0 EXE, as well as the originally released Activision EXEs (as currently available from Steam), appear to be a bit different. The former has the "LZ91" string from its first 32 bytes replaced with NUL bytes, while the latter two have "shifted" LZEXE signatures. In addition, the very beginning of the SOD v1.0 EXE has a couple of bytes differing from what you may currently get from the sources, implying an increased so-called "BSS section". The STRIPBSS tool included in the same repository wasn't made with this in mind, and I'm not sure about this. However, apparently it was very common to use modified EXEs in which the copy protection is skipped or removed, and this may further add some confusion.

There are also a few EXEs which are *not* reproduced by any project above, and there are great chances they won't be (but at least we have all of the above!):
- A release of SOD v1.4 made for FormGen, mistakenly labelled as v1.0 in the signon screen. (If it's just the v1.4 FormGen EXE, but with another signon screen, then it should be relatively easy to recreate it now.)
- Any translated release.

Later, I also got to recreate EXEs originally distributed as parts of some versions of Blake Stone, including Aliens of Gold. The currently available versions are these:
- Blake Stone: Aliens of Gold v2.1+3.0, Shareware and Registered releases.
- Blake Stone: Planet Strike v1.00+1.01.

Not remembering a lot from the two Blake Stone games, I thought that I may encounter quite large differences and a lot of code pieces removed. Turns out this wasn't really the case. 3D_MAIN.C has a couple of AOG-specific functions related to player alignment, and the AOG implementations of ShowOverhead and InputFloor have great differences from the PS implementations, but a lot is really the same. There are also a few commented out AOG-specific code pieces in the released Planet Strike sources, like Vital sprite definitions. Interestingly, there are at least a few occasion in which AOG-specific line codes apparently has different indentation (when compared to similar codebases). Furthermore, this is probably known, but it looks like the vital was "transformed" into the rotating cube during the development of Planet Strike.

As expected, though, I think that Planet Strike was started as a separate branch even before Aliens of Gold v2.1 was released. As a hint of that, the implementation of CA_LoadAllSounds in Planet Strike is the same as in the Catacomb 3-D sources, while there are minor edits in Aliens of Gold (at least in versions 2.1 and 3.0).



About all games:

I grabbed and modified GFXV_APO.H from Wolf4SDL by Moritz "Ripper" Kroll, for the restoration of Wolfenstein 3D Apogee EXEs. Similarly, I used GFXV_BS1.H definitions from the bstone port by Boris Bendovsky. I also "borrowed" a few song definitions in the very beginning from the same port. Regarding AOG restoration in general, most of the source code recreation was done using earlier research work by Braden Obrzut.

Unfortunately, there aren't exactly tools I'm aware of that can automatically compare two EXEs from different revisions of very similar source codes, given the EXEs and one source code revision as an input. Hence, the differences had to be manually located. There are still enough ways to ease the work, though. Obviously disassemblers can be used so we don't have to look just at machine code. Furthermore, Borland C++'s linker can generate a MAP file that shows the locations of most functions and variables in the EXE (as segment:offset pairs).

It's not just a matter of modifying the C/H/ASM/EQU files themselves, though. Project settings (like optimizations) had to be guessed for at least 2-3 projects, the ordering of a few source files to had to change in certain projects, there are files that should be removed from some projects (possibly most), and the Wolf3D GAMEPAL data has to be linked in different ways for different projects (either into its own segment, or as a part of the data segment).

Finally, these are still not exactly the original revisions of the sources, since we don't have access to them. However, what can be done now, is telling differences between versions as expressed in the compiled EXE files (e.g., the differences in secret elevator handling between Shareware versions 1.1 and 1.2 of Wolf3D, although these were basically known for a while).

That's a very interesting project, thanks for sharing!

[Edit] Since your work theoretically paves the way for unofficial patches for said games, are there any particular bugs that should be fixed? I remember that The Catacomb Abyss shareware v1.2 froze once in mid-level 2 for no apparent reason, whereas shareware v1.3 does not seem to have any such issues.

Jesus Christ!hehe

I believe I did at one point looked at a DEMO from Wolf3D maker.

I am still impress with their 3D shadings and such.

MrFlibble wrote:That's a very interesting project, thanks for sharing!

You're welcome!

[Edit] Since your work theoretically paves the way for unofficial patches for said games, are there any particular bugs that should be fixed? I remember that The Catacomb Abyss shareware v1.2 froze once in mid-level 2 for no apparent reason, whereas shareware v1.3 does not seem to have any such issues.

Are you talking about patches similar to the official ones, upgrading EXE and data of one version to another one? I haven't really investigated the differences between versions, except for making sure I get the original EXEs' layouts (or at least get very close).

What I know is, as mentioned in the topic above at DHW (http://diehardwolfers.areyep.com/viewtopic.php?t=7098), it turns new bugs were introduced to Wolfenstein 3-D v1.4 and Spear of Destiny (compared to Wolf3D v1.1/1.2), as mentioned by the topic's creator who clearly appears to prefer version 1.1.

In addition, I was told that this could make it much less difficult to adapt older Wolfenstein 3D mods made for e.g., version 1.1 to other codebases. While it's possible to manually change/fix behaviors without the restoration, I think this is still a technical way to take advantage of that.

NY00123 wrote:Are you talking about patches similar to the official ones, upgrading EXE and data of one version to another one? I haven't really investigated the differences between versions, except for making sure I get the original EXEs' layouts (or at least get very close).

I meant any kind of bug fixes (if necessary) that could be accomplished by editing the code (as opposed to say, hacking the official executables) and then compiling a "fixed" EXE.

NY00123 wrote:What I know is, as mentioned in the topic above at DHW (http://diehardwolfers.areyep.com/viewtopic.php?t=7098), it turns new bugs were introduced to Wolfenstein 3-D v1.4 and Spear of Destiny (compared to Wolf3D v1.1/1.2), as mentioned by the topic's creator who clearly appears to prefer version 1.1.

That's some very interesting information, both on its own, and also because I'm somewhat interested in the topic of older versions being better/more advantageous/more interesting in certain ways

[Edit] BTW, is it true that Blake Stone v2.1 and v3.00 are identical code-wise, with only the ordering and price info having been changed (plus some corrected typos), as described here?

LATE EDIT (MAY 7 2017): Corrected a minor typo regarding the first demo played in AOG shareware v3.0.

MrFlibble wrote:I meant any kind of bug fixes (if necessary) that could be accomplished by editing the code (as opposed to say, hacking the official executables) and then compiling a "fixed" EXE.

The DHW topic above has a post by Chris with a link to archived EXEs made out of modified Wolf3D v1.1/1.2 sources containing a few fixes. Note that at the moment, the downloaded file may end with .rar, but it is actually a ZIP.

MrFlibble wrote:[Edit] BTW, is it true that Blake Stone v2.1 and v3.00 are identical code-wise, with only the ordering and price info having been changed (plus some corrected typos), as described here?

There are a few more changes between v2.1 and v3.0 code-wise, but not a lot. I've just added descriptions of the changes in code between the recreated versions of AOG, and similarly between versions of PS, in the same DHW topic. These are repeated here.

As expected, I'm ignoring what's trivial (meaning the definition of the __VERSION__ macro, plus original compilation time and date).

There are only two code changes between versions 1.00 and 1.01 of Planet Strike, which can be found by looking for mentions of the macro GAMEVER_RESTORATION_ANY_LATE:
- In version 1.00, any of the shift keys can be used to double movement speed while using the mouse. The keyboard's key configured for running is totally ignored here. In version 1.01, any keyboard/mouse/joystick button defined for running can be used to do the same, in *addition* to the shift keys.
- No mouse button can be configured for running in v1.00. This was changed for v1.01.

Regarding AOG v2.1 and v3.0, the two changes above apply between these, too, but there are a few more changes in the code. You can find these by looking for mentions of GAMEVER_RESTORATION_BS1_300, GAMEVER_RESTORATION_BS6_300 and GAMEVER_RESTORATION_AOG_300:
- The amount of demos was reduced by one in-between versions 2.1 and 3.0.
- DemoLoop begins with demo no. 3 in registered v3.0, and with demo no. 0 otherwise. (Technically, LastDemo is also initialized to 3 in shareware v3.0, but the demo number is taken mod 3.)
- A different TERM_BACK_COLOR value is used in the menus as of v3.0.
- Planet Strike promotion pictures were added to v3.0.
- The registration phone number shown in the default high scores table was edited for v3.0.

NY00123 wrote:There are a few more changes between v2.1 and v3.0 code-wise, but not a lot. I've just added descriptions of the changes in code between the recreated versions of AOG, and similarly between versions of PS, in the same DHW topic. These are repeated here.

As expected, I'm ignoring what's trivial (meaning the definition of the __VERSION__ macro, plus original compilation time and date).

There are only two code changes between versions 1.00 and 1.01 of Planet Strike, which can be found by looking for mentions of the macro GAMEVER_RESTORATION_ANY_LATE:
- In version 1.00, any of the shift keys can be used to double movement speed while using the mouse. The keyboard's key configured for running is totally ignored here. In version 1.01, any keyboard/mouse/joystick button defined for running can be used to do the same, in *addition* to the shift keys.
- No mouse button can be configured for running in v1.00. This was changed for v1.01.

Regarding AOG v2.1 and v3.0, the two changes above apply between these, too, but there are a few more changes in the code. You can find these by looking for mentions of GAMEVER_RESTORATION_BS1_300, GAMEVER_RESTORATION_BS6_300 and GAMEVER_RESTORATION_AOG_300:
- The amount of demos was reduced by one in-between versions 2.1 and 3.0.
- DemoLoop begins with demo no. 3 in v3.0, and demo no. 0 otherwise.
- A different TERM_BACK_COLOR value is used in the menus as of v3.0.
- Planet Strike promotion pictures were added to v3.0.
- The registration phone number shown in the default high scores table was edited for v3.0.

Thanks for the detailed explanation!

Is there anywhere some exhaustive description all the bugs fixed an introduced in the different revisions of Wolfenstein 3-D (and/or other open sourced games). I remember some overview of fixes in all the versions in some FAQ but I'm not sure if this is accurate (IIRC it was not based on code analysis). Does Wolfenstein 3-D v1.2 have the same advantages over v1.4, or does it also introduce something undesired compared to v1.1? Does all of said by Chris apply to v1.1 shareware?

Now do Blood with the aid of the Blood alpha source code

Calvero wrote:Now do Blood with the aid of the Blood alpha source code

Hehe, I think that I know the feeling here. Unfortunately, I think that it's going to be enormously more difficult and time-consuming, for the following reasons at the least:
- There's much more Blood code which isn't available.
- It's a much larger codebase.
- While I don't know a lot about it, I understand it can be way more difficult to work with Build-engine based codebases.
- At the least, BLOOD.EXE (as coming with Blood: One Unit Whole Blood) has an external dependency on DOS4GW.EXE, unlike e.g., Duke Nukem 3D for which the extender is embedded in the EXE. There may be any other external dependency, though, adding some possible complications.
- I may just be less experienced with non-16-bit DOS code made with Borland C++, Turbo Assembler and/or similar.

MrFlibble wrote:Thanks for the detailed explanation!

Is there anywhere some exhaustive description all the bugs fixed an introduced in the different revisions of Wolfenstein 3-D (and/or other open sourced games). I remember some overview of fixes in all the versions in some FAQ but I'm not sure if this is accurate (IIRC it was not based on code analysis). Does Wolfenstein 3-D v1.2 have the same advantages over v1.4, or does it also introduce something undesired compared to v1.1? Does all of said by Chris apply to v1.1 shareware?

While lists of changes can be found, I believe they don't show all little (and somewhat larger) edits found in the restoration repository. You may currently find a wolf3d.diff file in the "patches" with all changes from the sources as originally released by id Software: https://bitbucket.org/NY00123/gamesrc-v ... at=default

As expected, even when you limit yourself to a specific game (say Wolfenstein 3D but not Spear of Destiny), there are way more changes than in the Blake Stone cases above. Of course, the fact that more versions of Wolf3D and SOD are covered has an impact, but I think it's not just that. Truly, the Activision release of Wolf3D doesn't seem very different from the other "GOODTIMES" releases in terms of source code (all of which originally made available on 1993 or later). However, there are a lot of changes between shareware versions 1.0 and 1.1 of Wolfenstein 3D (you can look for mentions of GAMEVER_RESTORATION_WL1_APO10 and GAMEVER_RESTORATION_ANY_APO10). For one example, a lot of the code for the bosses found in episodes 4-6 is missing from v1.0, while it is present in shareware version 1.1 and later (including all registered versions).

It's also a good time to clear up a possible confusion regarding releases known as versions 1.1 and 1.2. From what I learnt, the following releases were made available in the given order:
- Shareware v1.0 (initial release).
- Shareware v1.1.
- Registered v1.1: Based on a newer revision of the codebase than shareware v1.1, including a change to the way secret elevators are handled (look for mentions of ex_secretlevel in WL_GAME.C WL_AGENT.C). Originally, the player had reside on a tile of the form (10,y), but this was changed in the registered release so the tile should rather be filled with a special floor code. This was presumably done during the addition of the maps for episodes 2-6. However, the corresponding secret elevator tile from episode 1 was *not* edited to reflect this change, thus a bug was introduced.
- Registered v1.2: The EXE is exactly the same as in registered v1.1, it's just that a few data files were edited. This includes a fix for the secret elevator bug. Note that the game's version is still recognized as 1.1 in the signon screen (which is embedded in the EXE).
- Shareware v1.2: This appears to be based on the same revision of the codebase used for registered v1.1/1.2, just with the definition of the UPLOAD macro. Changes in data relevant to what's present in the shareware release are not necessarily the same. As in the registered release, though, this EXE is identified as v1.1 in the signon screen.

Another random fact: Before registered v1.1/1.2 (and shareware v1.2), all bosses except for Hitler were coded to spawn a gold key after being defeated. Since registered v1.1 was the first such release, this was never observed in practice.

Thanks for yet another extensive explanation!

I remembered that I have read some overview of the version differences in this FAQ but it is indeed very brief and incomplete.

There are also some details on Litude's website (currently inaccessible for some reason), such as the maps not having been updated between v1.1 and v1.2.

NY00123 wrote:However, there are a lot of changes between shareware versions 1.0 and 1.1 of Wolfenstein 3D (you can look for mentions of GAMEVER_RESTORATION_WL1_APO10 and GAMEVER_RESTORATION_ANY_APO10). For one example, a lot of the code for the bosses found in episodes 4-6 is missing from v1.0, while it is present in shareware version 1.1 and later (including all registered versions).

I assume this is because v1.0 was released before the registered version was completed (I'm actually trying to compile a list of such shareware releases here).

Somewhat off-topic, I've just remembered that some people reported a bug in Blood which was apparently introduced in the latest version. I'm thinking that maybe it's a good idea to start a separate thread for broken updates and other cases when the newest known version is for any reason not always the best one to play.

This is an interesting Project. Would love to see this done to Doom also!

Wouldn't Doom be problematic because of the proprietary sound code?

Yes, that may be so, didn't Think about that.

The point about the lack of the DMX library code is correct. It may theoretically be possible to do a partial source code restoration, though, by extracting the binary DMX code from the original EXE and then including it in an OBJ or LIB file that can be used by the Watcom Linker.

Also, it may be easier to at least begin with the Heretic or Hexen sources, since these are the codes for the original DOS versions and have references to the DMX API. They can't be built as-is, which is where the additional OBJ/LIB file may help. On the contrary, the released Doom sources are for the Linux port, lacking DMX API calls and having other modifications.

Finally, there may still be more challenges I haven't thought about, possibly related to the way Watcom works and/or any combination of project settings (e.g., optimizations) in use.

POST EDIT: Just made a minor correction spotted by Blzut3 (had a misleading mention of SNES texture/sprite size, which may really apply to the Jaguar and Mac versions of Wolf3D).

All right, I think it's time for another update:

- To begin with, please note that the "wolf3d" directory has been renamed "w3d_plus", for a reason that should be quite obvious very soon. Similarly, the wolf3d.diff file has been replaced with w3d_plus.diff.
- For each project you previously had to use STRIPBSS, there's no need to anymore. While working on Blake Stone, a way to strip the BSS section right from Borland C++ was found: Disabling the addition of debugging information to the EXE.
- As in the Blake Stone tree, PREP.BAT can now be used for making a clean up, before a project is built in w3d_plus.
- The most significant update, though, is the recreation of code for the DOS port of Super 3-D Noah's Ark, a title which many may consider a Wolfenstein 3D mod with changes.

Now, there's enough to tell about Super 3-D Noah's Ark (i.e., S3DNA) which differs from previous restoration efforts. I should begin with a little introduction, being that S3DNA was originally released as a SNES title, using the SNES port of Wolfenstein 3D as a base. The latter port, known to belong to the so-called Mac Family of Wolf3D ports, has some clear differences from the original DOS title. These include variable-length episodes, enemies appearing to face the player for all time, two additional weapons (flamethrower and rocket launcher) and generally a different storyline and somewhat different maps. The SNES version also had some censoring.

As said before, S3DNA was started as a SNES game based on the port above. The original DOS codebase was then used as a base for a DOS version of S3DNA. It is more similar to the SNES version than the DOS and SNES versions of Wolf3D are, though. For instance, the two versions of S3DNA let you use an auto-map, and there's an arsenal of up to 6 feeders (weapons), including hand feeding. The most technically significant addition to the DOS version, though, may be MIDI music playback (still via an AdLib or Sound Blaster), using what appears to be the MIDI data from the SNES version.

Now, according to Braden Obrzut (aka Blzut3), developer of the ECWolf port of Wolfenstein 3D, which is also used as a base for a new commercial port of S3DNA, unfortunately the original source code for S3DNA is assumed to be lost. However, I figured out that S3DNA isn't that different from Wolf3D in many ways. In fact, it feels much more like Wolf3D than the Blake Stone games may be. For instance, it's now clear to me that each animal in S3DNA is basically just one of the Wolf3D enemies with a few changes. There are still some modifications, the largest possibly being the MIDI code, but otherwise it's more-or-less the same. And so, the recreation of the EXE has started.

It may be a good chance to tell that a disassembler known as IDA Pro has been in use (there are a few versions which can be used for free for non-commercial purposes). While it does not translate the code in the EXE back to readable C code, it can still be useful for many purposes, like an initial automatic analyzing of the code, locating functions, variables, string literals and more.

One thing it can't seem to do (at least the version in use), though, is reading debugging information appended to the original S3DNA EXE. I've decided that I initially skip this and make sure the actual code and data match with the original, which is most of the work (and further the practically interesting part) anyway, and also what I've been familiar with. Some earlier research work done by Blzut3 has also been very useful.

Let's take one step back. Debugging information? That's right, not only the original S3DNA EXE is not an EXE packed with anything like LZEXE, it still has intact debugging symbols. This includes names of functions, almost all variables and even some enum definitions (probably as long as there's a variable of the given enum's type mentioned somewhere). As said above, though, I've initially skipped these. Only towards the end, having a little problem in the compiled MissileTryMove function, I've decided to start the Turbo Debugger and inspect the function. Turns out, while the original sources may be lost, the EXE still has line numbers information. Surprisingly enough, splitting an "if" expression with an "&&" operator into 2 lines has solved the issue, and I've finally gotten the exact same EXE image as the original (ignoring the debugging stuff).

Afterwards, I've figured out that Turbo Dump can be used in order to dump the debugging information in a somewhat more readable way. The -v argument may be used control the way the output appears to be. With the help of this, and by comparing the output of the original EXE with a recreated one's output, I've assigned names used in the original codebase to all functions and almost all of the variables, as well as some enum values (in case of differences in the TDUMP outputs). There are very few (local) variable names which may be missing, but it isn't a great deal.

There are still a few differences in the Turbo Dump outputs, mainly with some type symbols. Missing function prototypes may be one cause, while there can be more. However, given that the actual EXE image has been successfully recreated, and original symbol names are now in use where found, I think this can be sufficient for now.