Author Topic: Smashing .IDX for fun and profit (or how hglzap got started)  (Read 2656 times)

NiteHawk

  • Administrator
  • Dark Will
  • *****
  • Posts: 529
  • Karma: 8
  • Lurking in the shadows
:arrow: The information here was duplicated from previous posts for future reference. While the 'old' forum is still available, you can read the original thread here: http://www.hellgateaus.info/forum/viewto ... ?f=9&t=862.
___

Smashing .IDX for fun and profit

Hi everybody!

Have you ever peeked inside Hellgate's logfiles? Usually (under XP) these are located in C:Documents and Settings<Your username>My DocumentsMy GamesHellgateLogs. You might notice that the NODIRECT*.log files frequently contain "FILE NOT FOUND ON DISK" messages referring to game data files. It seems the game (under certain circumstances) actually looks for files directly on your hard disk.

This has been (ab)used for "modding" the game by taking away certain 'container' files (.IDX and .DAT) and replacing them with "unhelled" content. The original UnHell utility by Benjamin Haisch allows extracting the .IDX/.DAT content, and in turn makes it possible to change/replace files. Since the game no longer finds the original versions, it will instead revert to using the files directly from disk. Examples for this technique are Tessera's "Nude Patch" and also the Revival Patch.

Now if you are like me, you probably don't like the idea of having a truckload of unhelled files lying around in your Hellgate installation. I prefer my patches 'lean and mean' :). I also got the impression that some people had difficulties fiddling with all those files and where they need to go. Which kept a "there must be a better way" nagging at me from time to time... I've been experimenting with Hellgate's data files a bit before. Basically the .IDX ("index") files are only 'pointers' into the corresponding .DATs, telling the game what files are 'there' and how to access them.

Let's summarize: A) the game will load data files from disk if it can't find them otherwise, and B) the .IDX files tell Hellgate about the 'available' files. Combining these two quickly lead me to a different idea: Why not make use of the .IDX mechanism to 'hide' data from the game, so it will look for the corresponding files directly on disk? (So instead of unpacking all the .DAT content and overwriting only a few files, I start 'the other way round' and selectively create 'blind spots' within the original game data, and thus force Hellgate to explicitly look for the replacement files.) I have nicknamed this "zapping" the IDX. 8-)

This has a number of advantages:
- We only need to back up the original .IDX files, all the .DAT files stay where they are and remain fully intact. All the modding/patching can be easily reverted by simply restoring the original .IDX.
- The game's (optimised) loading mechanism largely remains unchanged. Direct file access is only needed/used for the modified content. Only a minimum of additional files and directory structures is added/required.
- Patching can easily be automated. Just drop the mod files into their appropriate places and have a utility parse for anything that is an 'extra' to the original Hellgate installation. After that simply make sure that there are no 'visible' references to these files left inside the .IDX hierarchy (i.e. let the tool remove them, if necessary).
"'Chocolate Park' sounds... suspiciously cute."

NiteHawk

  • Administrator
  • Dark Will
  • *****
  • Posts: 529
  • Karma: 8
  • Lurking in the shadows
Considering checksums and general criticism
« Reply #1 on: Jul 28, 2009; 08:53 PM »
Quote from: "batman"
Not to rain on anyone's parade. This functionality has already been implemented in hellmix. Yes, the idx files are the index for the data path structures in the dat files. Yes you can add files/update files into the idx AND the dat. You are about to run into the same problem the original coder ran into. There is a checksum check on load, if this deviates too much, it will not load your idx/dat regardless. This means, that while changing the idx, you also have to recalculate the checksums of the files and update either datchecksum.xml or localizeddatchecksum.xml.

While the general concept of what you are doing will save load times by having the files located properly in the index and data files. Recalculating and updating the checksums to the proper xml will be a little more challenging.
Hello batman!

Can you name additional sources of information on this? Are you sure these checksums actually matter in SP gameplay? I have been thinking about this too, but maybe the checksums were meant to be used for doing the online update check (e.g. making sure you have the correct version of MP patch in place) or to prevent tampering with those files which are shared by the MP client...

What is the SP executable supposed to do when it 'refuses' an .IDX? Simply not load it at all? If this happened to the SP patch files, you'd end up with a black "console" starting screen - I did not come across that yet. Still my tool has already (heavily?) modified the corresponding .IDX - and changing a single byte should result in the checksum being invalid. So how much of a change is needed to "deviate too much"? :)

I have the hellmix.cpp source (v1.0) here. My impression was that properly creating/adding to an .IDX never worked with that tool. I have also stepped back from the idea of appending the actual mod files to Hellgate's .DATs - since I cannot recreate .IDX entries either with my utility. There's a certain 'magic' 32bit value associated with the strings in the .IDX (a hash?) that I can't reproduce (at least for now).

Be aware that hglzap 'by design' never interferes with the .DAT files (as this is not necessary). It simply 'punches holes' into the .IDX, in turn forcing Hellgate to look for / load the mod files directly from the disk / directory structure...

I'd be interested in any specific problems you ran into when using hglzap, as well as any suggestions you have for circumventing these problems and/or solving the modding issues in a better / more elegant way...

Regards, NiteHawk
"'Chocolate Park' sounds... suspiciously cute."

NiteHawk

  • Administrator
  • Dark Will
  • *****
  • Posts: 529
  • Karma: 8
  • Lurking in the shadows
FAILURE to replace certain files / ingame 'assets'
« Reply #2 on: Jul 28, 2009; 08:59 PM »
Quote from: "Norak"
UGH!
This is annoying....
It won't take over movies...
I placed my movie in the correct folder, it said that it "zapped" it. Though in game it doesn't show. It should overwrite the starting movie, where that book is floating etc.
:(
Quote from: "Norak"
Hey,
I have a movie, in which I want to replace the starting movie, I have given it the same name and placed it in the correct place. (I think.. :? )
The scene I want to replace is called "scene 1-5_low" and "scene 1-5_high", I have placed them in a "cinematic" folder in Data.
I can confirm this. :(

After "zapping" the related entries from the .IDX file the game simply acts as if the intro movie wasn't there at all. The effect I get is that it jumps straight to the title screen after playing the 'logo' movies on startup.

I also notice that when taking away the 'replacement' files, Hellgate won't complain (i.e. log an "FILE NOT FOUND ON DISK" entry to NODIRECT*.log). Therefore I believe that movies get treated/loaded differently from 'ingame data'. If they're not found within the .IDX, no attempt is made to load them 'direct'.

Were you able to get your replacement video to work with the "unhelled" variant?

(The only other way I could think of would be to actually modify the .DAT files to include the new movies. This is currently not possible, but - at least in theory - it could be done.)

Regards, NiteHawk

Quote from: "Norak"
I have been mucking around with other cinematics and music.
I tried getting Promy's main menu music to work, I also tried replacing the Hellgate  logo with my own creation...Nothing, I don't think...
Were you able to achieve this with the "unhelled" data before? If so I'd have to rethink my understanding of how Hellgate looks for / loads those files. The only other thing I can imagine right now would be to actually modify the .DAT files instead...

Regards, NiteHawk
"'Chocolate Park' sounds... suspiciously cute."

NiteHawk

  • Administrator
  • Dark Will
  • *****
  • Posts: 529
  • Karma: 8
  • Lurking in the shadows
FAILURE to replace certain files / ingame 'assets'
« Reply #3 on: Jul 28, 2009; 09:01 PM »
Quote from: "Norak"
Sadly, I wasn't able to do it the other way either..
Which is why I was like. :o
When you released this..
Well, I guess it's "no luck" then. :( But at least this reassures me that both 'unhelling' and hglzap more or less achieve exactly the same thing. And as they both rely on the game 'falling back' to direct loading, they're equally helpless if it doesn't do this...

So this sort of mod would probably have to go straight into the .DAT (which is currently not possible with my tool).

[...]

I noticed two things during my experiments:

- The game didn't accept a renamed "scene 6_*.bik" as a replacement for "scene 1-5_*.bik". I would have expected HG:L's own files to have the 'correct' format...

- When 'taking away' (deleting) the files from disk after zapping the IDX, no "FILE NOT FOUND ON DISK" showed up in the NODIRECT-*.log. To me this indicates that the game doesn't try this load method at all? (It can be demonstrated with the posters [...] mentioned: temporarily rename databackground_advertisementsad02.dds, say to no-ad02.dds - start HG:L, go to Covent Garden - and not only will the poster be visibly missing, but you'll also find a "FILE NOT FOUND ON DISK: databackground_advertisementsad02.dds" in your logfiles afterwards!)

Regards, NiteHawk
"'Chocolate Park' sounds... suspiciously cute."

NiteHawk

  • Administrator
  • Dark Will
  • *****
  • Posts: 529
  • Karma: 8
  • Lurking in the shadows
Re: Smashing .IDX for fun and profit (or how hglzap got started)
« Reply #4 on: Jul 28, 2009; 09:07 PM »
hglzap is intended to be run against a "clean" installation of Hellgate SP Patch version 1.2.

IMPORTANT NOTE:
Make sure you actually run the game once after updating to the original 1.2 patch! Not doing this may result in failure when trying to installl the Revival "fun patch" afterwards...

Quote from: "maeyan"
Quote from: "Malachor"
Actually, you might not need to start the game before running hglzapper.
You do need to do it if you want to unhell files though.
And I've just kept that going, maybe wrongly.
The game must be executed because it modifies the .idx first time the game is run after installation/patch. I'm not sure why
"'Chocolate Park' sounds... suspiciously cute."