Prehistorik music

[IllidanS4]

Hello, I was trying to extract Prehistorik music (MDI format). I successfully opened the game archives (FILESA.CUR and FILESB.CUR) and found lots of MDI files inside. However, the format appears to be different than one described on vgmpf.com. The files share a similar header (no ASCII signature), but nothing that I could recognize. Could you please help me? The files are available here.

[Malvineous]

It's not a format I recognise, and the data looks too random to be music. If it's music there is usually a lot of repetition. I suspect the data is compressed. Are there other game files you can extract to see whether they are also compressed or not? If other files are compressed then it's certain these are too.

If they are compressed, you might be able to experiment with something like STUNS to see if you can work out what sort of compression the game uses.

[IllidanS4]

There are some uncompressed Creative Voice Files and also some PC1 files (graphics?), which also look random. Unfortunately, STUNS doesn't recognize any compression (I hope I use it correctly). However, I was able to dump DOSBox memory and extract the following MDI files:

INTRO.MDI
RIK1.MDI
(the names are by guess)

They are played correctly with MIDIPLAY and match how it sounds in the game. How can I work out the compression? I've noticed the files share a similar header, first 0000, then two bytes (uncompressed size?) and then always 26950D06400400035030080549034E46.

Edit:
Despite not having worked out the compression, I have at least recorded all music by replacing the intro.mdi file in the archive with other ones, running it in the game, dumping the memory, extracting the MDI file, playing it in MIDIPLAY, and capturing OPL in DOSBox. You can find the songs here.

[Malvineous]

Nice idea to get the songs extracted! Your .mdi rips are cut off however - they are missing the last few bytes (you can see that the 'decompressed size' field in the header is a few bytes larger than your rips.) You can use something like Ripper6 to rip the .mdi files from a DOSBox memory dump to correctly extract the full size, if that's easier.

I'm guessing the first few bytes always being the same are because the MDI files all start off the same too, with the MThd header. I haven't used STUNS but I'm guessing you need to trim off the four bytes at the start of the compressed file that store the uncompressed size.

If you do cut off the first four bytes, then reading the file in 9-bits-per-byte big-endian mode shows the text "MThd", so I'm guessing it's LZW compression. Looks like there's some variation though as it quickly diverges from the usual type so might be some custom tweak to the LZW algorithm.

[Malvineous]

After looking at this a bit more closely, I'm pretty sure the compression algorithm is LZSS, or at least a variant (maybe LZ78) based on looking back into the data that has just been written. I think this algorithm might work to decompress it, but I haven't tested it yet:

• First four bytes are UINT32BE storing decompressed size in bytes, followed by the LZSS data:
• Read one bit as Flag
• If Flag is 0, read another eight bits and output those eight bits as a byte
• If Flag is 1, read two bits as Length then eight bits as Offset
• Add 2 to Length, add 1 to Offset
• Go back Offset bytes in the output data file, and copy Length bytes from there onto the end of the output data
• Loop (read next Flag)

All bit arrangements appear to be big-endian - I'm guessing the game was ported from a non-Intel platform as this seems to be the only time big-endian data shows up in a DOS game.

[Malvineous]

Sorry for the triple post but I'm just confirming this is indeed LZSS. I've added support for compressing and decompressing these files to Camoto (and when I extract and decompress the .mdi files I get them in full, the last couple of bytes aren't cut off). I've also created a Prehistorik page on the ModdingWiki to document it all.

It looks like a number of other files are compressed with the same algorithm - pretty much everything except the .voc files by the looks of it. The .pc1 files are standard .lbm images you can open in an image editor after decompression. I think this file format originated on the Amiga which IIRC was big-endian so it looks more and more like all this data was ported straight from another platform this game was on.

[pablolupo]

Hello there!
@Malvineous, I downloaded the last version of Camoto Studio but I couldn't find the Prehistorik game in the list.
How should I use Camoto in order to decode those MID files ?
Thanks to the information you shared with us in the wiki about the CUR files format, I have written my own C/C++ application and succesfully extracted those CUR and VGA files.
But I cannot decode none of those files extracted from the CUR.
I tried using an LZSS implementation library written by Michael Dipperstein in C++ but without success.

Hope you can help me, Thanks!
Lupo

[Malvineous]

Sorry I should have been clearer! I have only added support for the files to the Camoto component called "libgamearchive". There is nothing in Camoto Studio yet as there is nothing to show (no graphics formats or maps or anything supported yet.)

If you can compile libgamearchive from source, then you can use the "gamearch" command-line utility to extract and decompress the .cur files (e.g. "gamearch -X blah.cur" will decompress and extract all the files into the current directory.)

[pablolupo]

Malvineous, thank you very much for your quick response.
Well, I have tried to compile the libgamearchive project in order to use the gamearchive process, to extract CUR files, but it gave me a lot of errors during the compilation process...

I followed the building steps from the guide at http://www.shikadi.net/camoto/help/build/win32, I have downloaded all the projects from the GitHub on its version 1. After a successful build of the complete solution, I downloaded the master branch for the libgamearchive and unziped over the previous libgamearchive folder...
There is where I have a lot of errors, even if I just try to compile the master branch version without previously extracting the version 1, It even fails.

Please, can you give me a clue ?
Best regards,
Lupo

[developertn]

I don't know which compiler you're using. However Borland tends to get a little complicated until you get used to it.

For instance, if you compile Time1973.c you'd get a bunch of errors:

http://www.mediafire.com/download/fntki ... me1973.zip

However all the errors are not anything exception little warnings that you can usually ignore. If the program doesn't run then you need to fix it. What is peculiar about Borland is that the compiler allows forward reference catching through multiple passes. So you'll have to do a little bit of wacky way out there command-line manual coding.

A case in point would be to compile the above Time1973.c program, the sequence of commandline would be:

tcc -S t
tasm /m2 t
tlink c0s t, time1973, , fp87 maths cs

As you can see, it's a little convoluted. However it's needed.

The reason you need it is that the normal way would not resolve the forward reference. So you had to pass it through twice.

Anyways through diligent work these so called secrets are unearthed - THANK YOU JESUS CHRIST!

Bless God then my real mom Huong Thi Vu for buying for me the officially handbook so I know what the heck the commandlines.

Honours to my real mom Huong Thi Vu for being the socialitest.

Honours to my real dad Thuy Binh Nguyen for being the elitest.

Love to my real two sis for being examples of how a wonderful family is: Nguyen Khoa Thi, and Nguyen Khoa Thuyen

[Malvineous]

Malvineous, thank you very much for your quick response.
I followed the building steps from the guide at http://www.shikadi.net/camoto/help/build/win32, I have downloaded all the projects from the GitHub on its version 1. After a successful build of the complete solution, I downloaded the master branch for the libgamearchive and unziped over the previous libgamearchive folder...
There is where I have a lot of errors, even if I just try to compile the master branch version without previously extracting the version 1, It even fails.

Sorry I didn't realise you were using Windows. I haven't tested the latest release under Windows. It should work, but there might be some compiler tweaks you need to do to get it to work. If you use the latest git version of libgamearchive, then it won't compile unless you also use the latest git version of libgamecommon. The other libraries are not needed if you are only interested in extracting the .cur files.

Do you still have problems compiling the git version of libgamecommon? You will need to enable C++11 mode - I am not sure if that is on by default.

If you can't get these to compile, feel free to e-mail me your .cur and .vga files to malvineous@shikadi.net and I will extract them for you and e-mail you back the contents.

[pablolupo]

Hello guys,

Thank you all for your support!

@Malvineous, I actually couldn't compile it by my self, and I sent you an email with those CUR and VGA files. So if you kindly may extract and send me them back I will be very greatful.

Anyway, when I have tried to compile libgamecommon using visual studio, I could resolve some errors that were related to filenames that had to change (for instance lzw.cpp to filter-lzw.cpp), also files that no longer exists and new ones that I had to add.

Besides that, there were some other errors that I couldn't find a solution, for instance:

git\include\camoto/util.hpp(65): error C2144: syntax error : 'unsigned long' should be preceded by ';' (git\src\iff.cpp)

git\include\camoto/util.hpp(65): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int (git\src\iff.cpp)

...\VC\include\xrefwrap(283): error C2064: term does not evaluate to a function taking 1 arguments (git\src\bitstream.cpp)

Then I tried using MinGW, but with errors too:

$ make
make all-recursive
make[1]: Entering directory '/cygdrive/c/Users/pbacigalupo/Documents/GitHub/camoto-win32-master/libgamecommon/git'
Making all in src
make[2]: Entering directory '/cygdrive/c/Users/pbacigalupo/Documents/GitHub/camoto-win32-master/libgamecommon/git/src'
CXX iostream_helpers.lo
<command-line>:0:12: error: expected identifier before numeric constant
../include/camoto/error.hpp:34:7: note: in expansion of macro ‘DLL_EXPORT’
class DLL_EXPORT error: public std::exception
^

Best regards,
Lupo

[developertn]

Hello guys,

Thank you all for your support!

@Malvineous, I actually couldn't compile it by my self, and I sent you an email with those CUR and VGA files. So if you kindly may extract and send me them back I will be very greatful.

Anyway, when I have tried to compile libgamecommon using visual studio, I could resolve some errors that were related to filenames that had to change (for instance lzw.cpp to filter-lzw.cpp), also files that no longer exists and new ones that I had to add.

Besides that, there were some other errors that I couldn't find a solution, for instance:

git\include\camoto/util.hpp(65): error C2144: syntax error : 'unsigned long' should be preceded by ';' (git\src\iff.cpp)

git\include\camoto/util.hpp(65): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int (git\src\iff.cpp)

...\VC\include\xrefwrap(283): error C2064: term does not evaluate to a function taking 1 arguments (git\src\bitstream.cpp)

Then I tried using MinGW, but with errors too:

$ make
make all-recursive
make[1]: Entering directory '/cygdrive/c/Users/pbacigalupo/Documents/GitHub/camoto-win32-master/libgamecommon/git'
Making all in src
make[2]: Entering directory '/cygdrive/c/Users/pbacigalupo/Documents/GitHub/camoto-win32-master/libgamecommon/git/src'
CXX iostream_helpers.lo
<command-line>:0:12: error: expected identifier before numeric constant
../include/camoto/error.hpp:34:7: note: in expansion of macro ‘DLL_EXPORT’
class DLL_EXPORT error: public std::exception
^

Best regards,
Lupo

Hey Lupo,

I've only been programming for a serious year on a fulltime basis. However I can tell you the three errors you are encountering is not that hard to fix. I get them quite often enough that it is recognized immediately. So off the top of my head, if it says unsigned long should be preceded by a semicolon then please check to make sure the previous statements all have a semicolon after the commands. This is because if there are no semicolon it does not know that the next commandline is starting.

For missing type specifier please go to your place where you define your variation. For instance you define "int varA;" however you forgot the "int". This is what that line teaches you.

Last please check the function actually receives an input of 1 variation. For instance:

void funcA(int varB)
{
}

If it is simply:

void funcA()
{
}

So you can see it is not receiving any input yet you may accidentally give it the call funcA(12);

Jesus Christ!
Well, let me know what else your compiler messages come up with. I've delt with literally thousands of errors over my lifetime.

So I might know a little about them.

I'm sorry if my English is misunderstood.
Thank you my The LORD of all hosts#!!

[developertn]

God, Jesus Christ, is number one!
Jesus!

Bless Jesus Christ, then my real mom Huong Thi Vu for always being rich in luxury for the remainder of our family's lives.

Honour to my real mom Huong Thi Vu for giving me the power to program professionally from this day forth until after I pass.

Honour to my real dad Nguyen Binh Thuy for protecting the family and giving us only good karma for eternity.

Love to my two real sisters Nguyen Khoa Thi and Nguyen Khoa Thuyen for obeying our two real parents.

[Malvineous]

@Lupo: Thanks for the info. I've replied to your e-mail with the extracted contents. The first one bounced so hopefully the second one made it through. (One of the data files is called "jumbo.exe" and even though it's not an .exe your mail provider's spam filter doesn't want to take any chances! I put it in a .tar.bz2 which hasn't bounced yet.)

Anyway, when I have tried to compile libgamecommon using visual studio, I could resolve some errors that were related to filenames that had to change (for instance lzw.cpp to filter-lzw.cpp), also files that no longer exists and new ones that I had to add.

This would be the new files that have changed. I've only updated the standard Makefiles so far, not yet the Visual Studio project files.

Besides that, there were some other errors that I couldn't find a solution, for instance:

git\include\camoto/util.hpp(65): error C2144: syntax error : 'unsigned long' should be preceded by ';' (git\src\iff.cpp)

git\include\camoto/util.hpp(65): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int (git\src\iff.cpp)

...\VC\include\xrefwrap(283): error C2064: term does not evaluate to a function taking 1 arguments (git\src\bitstream.cpp)

The first two errors are complaining about the unknown keyword "constexpr" which is a new C++11 keyword. So that would indicate that Visual Studio is compiling in C++03 mode, and needs to be changed to C++11 mode.

Not sure about the third error as I don't have "xrefwrap" handy, but I'm guessing it's likely to be a C++11 issue again. The new code uses C++11 quite heavily so it definitely needs compiler support now.

../include/camoto/error.hpp:34:7: note: in expansion of macro ‘DLL_EXPORT’
class DLL_EXPORT error: public std::exception
^

That one is just assuming that when compiling under Windows, DLL_EXPORT will be defined already. I guess Cygwin confuses things because it defines DLL_EXPORT as something different (apparently a number?) I guess you'd have to override that definition to proceed.

Thanks again for the details. It looks like there are still a few issues to resolve but luckily they don't seem as bad as I was expecting! Perhaps it's time to update the win32 build instructions...