Sacred 2 Downloads - Sacred 2 modding tool: .txt Comparison extractor

[Lindor 359]

Sacred 2 modding tool: .txt Comparison extractor

View File

---

This is a tool coded by myself to compare two different mods and extract all differences between each supported .txt file. Which is nearly all of them.
Use together with S2CE_Insert.exe so you can pull one mod out and insert it somewhere else. Especially useful for mods based on other mods if the first mod gets an update.
The tool will only extract actual changes and not the whole .txt file, that's what makes it so powerful!
The tools latest version will only work on windows OS, latest non-OS-dependend version is 2.0.0!

It works like that: One mod is the basic and one is the modded. It will only compare and extract things that the modded one contains. After each file comparison, if it found any changes, it will create an "EXTRACTED_...".txt file and only list the changes in that extracted file. If there exists something in the basic file that the modded one deleted in that file, it won't list it. If there exists something that's same in basic and modded, it won't list it. If there exists something in modded that's different to basic, it will list it. It works define-based, so if e.g. there's one entry changed in a spell in spells.txt, it will list the whole spell define in EXTRACTED_spells.txt.

 

Usage:

1. unzip/unrar this tool and your two mods you want to compare.
2. put first mod folder into same folder as S2C_Extract.exe and lua54.dll
3. [If the first mod are the CM-Patch scripts, replace them like I've described further down]
4. rename the first mod folder to "basic" (without quotation marks)
5. put the second mod folder into same folder as above
6. rename second mod folder to "modded" (without quotation marks)
7. douple-click S2C_Extract.exe
8. Q'apla!

If you want to compare a mod to the CM-Patch-scripts, then you must replace

• scripts/server/creatures.txt
• scripts/server/weaponpool.txt
• scripts/shared/spells.txt

with the three three files in the "CM 1.60 files debugged" folder first, as there are some problems and inconsistencies with the CM-patch.

• Line-by-line based comparison .txt files:
  • scripts
    • autoexec.txt
    • genMipMapInfo.tx
    • options_readme.txt
    • startPos.txt
  • scripts/client
  • scripts/server
    • questscripts.txt
    • respawn.txt
    • spawnpos.txt
    • worldobjecthints.txt
  • scripts/shared
    • defines.txt
    • staticinfo.txt
• Partially supported define based comparison .txt files:
  • scripts/server
    • creatures.txt
      • The "mapPosition"s at the end of the file are not supported due to two different safe patterns. Everything else works.
      • Unlike in earlier versions, Creature-Blueprint-Relations work again
    • quest.txt
      • Task Creatures work
      • Task Items work
      • Quests work
      • nothing else works
• Fully supported define-based comparison .txt files:
  • scripts
    • behaviour.txt
    • heightmap.txt
    • landscape.txt
    • optionsDefault.txt
    • roadmap.txt
  • scripts/client
    • animation.txt
    • collision.txt
    • Eliza.txt
    • environment.txt
    • keycodes.txt
    • minitypes.txt
    • patches.txt
    • poidata.txt
    • relations.txt
    • soundcluster.txt
    • soundprofile.txt
    • soundresources.txt
    • surface.txt
  • scripts/server
    • balance.txt
    • blueprint.txt
    • creatures.txt
    • drop.txt
    • equipsets.txt
    • faction.txt
    • pathObjects.txt
    • portals.txt
    • region.txt
    • spawn.txt
    • treasure.txt
    • triggerarea.txt
    • triggervolumes.txt
    • waypoints.txt
    • weaponpool.txt
    • worldobjects.txt
  • scripts/server/heroes
    • centurio.txt
    • dragonmage.txt
    • dryadin.txt
    • highelve.txt
    • inquisitor.txt
    • seraphim.txt
    • templeguard.txt
  • scripts/shared
    • books.txt
    • creatureinfo.txt
    • iteminfo.txt
    • itemtype.txt
    • material.txt
    • spells.txt
    • typification.txt

 

 

 

The tool is completely coded in lua and fully open source. The Script is included in the upload since version 1.1.0.

Credits:

• Luiz Henrique de Figueiredo, I used his srlua to compile the script into a standalone .exe.

---

• Submitter
  Lindor
• Submitted
  01/31/2022
• Category
  Fallen Angel Tools

 

Edited February 11, 2022 by Lindor
Reason: It had to be done.

[Lindor 359]

There is a major bug I discovered in the adddollars() and killdollars() functions, resulting in some comparisons failing (or better: not even happening), esp. in spells.txt. There will be a very important update coming soon which fixes the bug.

[Lindor 359]

The bug has been fixed, the fix has been successfully tested and the download file has been updated.

[Lindor 359]

I've created line-based comparison for all previously unsupported files and fixed a few file-specific bugs.

I wish to upload it but man, creatures.txt is giving me some headache right now. There are two defineclasses with different safe patterns: creature-blueprint-relations (depending on mod) and mappositions (even in the same mod).

@Hooyaah, @Flix, @dimitrius154, @Charon117, @Azrin14, @NathyielNathyiel, @Schot basically all mod creators (sry if I forgot one), may I ask you to do two things:

1. For creature-blueprint-relations, I would prefer the 1-line safe pattern. Problem is that CM-patch uses a 4-line safe pattern. Can we please change all 4-line safe patterns to the 1-line safe pattern in all mods including CM-patch?

2. For mappositions, can we please change this pattern wheneverver it occurs:

mgr.addMapPos {
	merchantType = 2,
    position = { 31,7,0 , 1821.700,2051.750,58.460, 349.000 },
}

into this pattern which is preferred by me:

mgr.addMapPos {
	merchantType = 5,
    position = {
      x = 100500.00,
      y = 24450.00,
      z = 0.00,
      layer = 0,
      orientation = 20.000,
    },
}

?

 

The alternative would be to add creatures.txt to line-by-line comparisons as well, but I really don't want to do it unless absolutely necessary and I have a good reason for that:

I want to create a define-based insertion tool next. It should be easy since I already wrote and tested all of the recognition functions. It is basically a modmerging system for all define based text files, quick, easy to use and keeping all the comments outside of defines intact. And it is highly needed for mods based on other mods if an update of the first mod occurs. But first I need to make sure that everything works 100%. If I have to make creatures.txt into line-based comparison, it would exclude it from the define-based modmerging system which I really don't want to happen for such an important file.

A line-based insertion tool would be completely useless, since it's not doing anything else than replacing one file with another. We already have JSGME for that.

Step three on this journey will be automated debugging and comment adding.

[Lindor 359]

I don't know what happens, every time I try to post in the download section it becomes a double post. Poor moderator who needs to check my comments

[gogoblender 2,777]

2 minutes ago, Lindor said:

I don't know what happens, every time I try to post in the download section it becomes a double post. Poor moderator who needs to check my comments

hmmm, is this actually a SAcred Underworld file Lindor ? I saw the root and edited the title to match.. if its an underworld post I or you can re edit the title, to demarcate tween sacred underworld or Sacred 2 

Thanks LIndor for getting your work up for sharing with the Gang!!

gogo

[Lindor 359]

5 minutes ago, gogoblender said:

hmmm, is this actually a SAcred Underworld file Lindor ? I saw the root and edited the title to match.. if its an underworld post I or you can re edit the title, to demarcate tween sacred underworld or Sacred 2 

Nah it was the modding tool in the Sacred 2 Fallen Angel tools section. Just scroll up and click "View File".

Btw the tool is not really for Fallen Angel nor for any other game version specifically, it's more of a modding tool for all versions of the game but the Fallen Angel section was the only one with a "tools" sub.

8 minutes ago, gogoblender said:

Thanks LIndor for getting your work up for sharing with the Gang!!

gogo

I love doing it! Thx

[dimitrius154 501]

@LindorNo problem, though I'm postponing further development, until I complete the current Season in Inquisitor: Martyr. Got people to burn, you see.

[gogoblender 2,777]

2 hours ago, dimitrius154 said:

@LindorNo problem, though I'm postponing further development, until I complete the current Season in Inquisitor: Martyr. Got people to burn, you see.

 

 

 

gogo

[Lindor 359]

15 hours ago, dimitrius154 said:

@LindorNo problem, though I'm postponing further development, until I complete the current Season in Inquisitor: Martyr. Got people to burn, you see.

Let them heretics pay!

-----

For the inserter, I have to use os.execute() command which is highly system dependend which would mean the program would only work on windows. And in order to work together with the insertion program, I'd have to make the extraction windows-dependend as well. Which is going to happen. 2.0.0 will be the latest version running on non-windows systems.

[Lindor 359]

On 2/8/2022 at 3:35 PM, Lindor said:

including CM-patch

Hmm I wonder who's in charge of CM-patch nowadays? czevak and markuswob are both not active anymore. Frank Rentmeister's website isn't online anymore either. Is it even possible to bring out a new version or is it a completely closed project? Because I thought about alternatives. I could upload an updated version of CM-patches creatures.txt together with the tool, I wouldn't change anything but the safe pattern, the contents of the file would completely stay intact.

[dimitrius154 501]

1 hour ago, Lindor said:

Hmm I wonder who's in charge of CM-patch nowadays?

My guess - no one. Ever since CMP 1.60 release there's been a split between EE and the Addendum. 

[Lindor 359]

Thx! The alternative it is, then.

[Flix 4,924]

Last time that I talked to Marcus and Czevak they talked of moving on to other priorities, family, career etc.  Development continues in EE and Addendum, as D said.  We just have different visions.

[Lindor 359]

While testing my streamlined version of CM-Patches creatures.txt, I saw another issue and that issue is spaces.

If two defines have same content but different spaceage, the comparison function will recognize them as different.

I can make it so that it ignores spaces, but that would:

• at least double memory usage (but not necessarily at the same time, just the total amount of memory usage during the process)
• Significantly increase runtime (also I think around double it, but not sure)
• Ignore ALL spaces

What do you think? Should I do it? The amount of false positives (or negatives, depending on what you define as a positive comparison) can be pretty hefty on my streamlined CM-patch file, but the main theme of the program is "quick and easy to use".

[Lindor 359]

14 hours ago, Lindor said:

Significantly increase runtime (also I think around double it, but not sure)

It's a hefty amount of runtime increase.

From average of 17.365 sec to average of 30.1635 sec on comparing EE 3.0 to CM-Patch. I've tried to speed up by cutting support of some unnecessary files which need additional checks every define, barely reduced to 30.1455 sec average, not worth it.

The best solution would be uniform formatting over all mods including spaces, which is too much to ask for.
The best I can come up with is releasing two versions, one ignoring spaces at the cost of runtime and one with half runtime at the cost of not ignoring spaces.

[Lindor 359]

OH. MY. GOD.

 

I've spent the entire sunday trying to hunt down a seamingly IMPOSSIBLE bug with my inserter in creatures.txt. Yeah. turns out that the CM-Patch creatures.txt entry 

mgr.createCreature {
    id = 750,
    itemtype_id = 7778,
    name = "HU_cult_fk_elite",
    behaviour = "Enemy_hunter",
    dangerclass = 6,
    groupmaxcount = 1,
    elite_creature_id = 750,
    probabilityforelite = 0.000000,
    rank = 60,
    tenergy_creature_id = 441,
    template_creature_id = 1382,
    livesremaining = 0,
    unconscioustime = 20,
    palettebits = "1111111111111111",
    monstertype = 1,
    speakertype_id = 13,
    faction_id = 86,
    equipset_id = 179,
    modelscale = 1.150000,
    rise_from_ground = 0,
    has_corpse = 1,
    has_soul = 1,
    can_strafe = 0,
} 

has ONE EXTRA LITTLE SPACE at the last line, "} " instead of "}".

That goddamn ONE LITTLE SPACE messed up my WHOLE program! WRRRAAR sometimes....

 

[gogoblender 2,777]

41 minutes ago, Lindor said:

OH. MY. GOD.

 

I've spent the entire sunday trying to hunt down a seamingly IMPOSSIBLE bug with my inserter in creatures.txt. Yeah. turns out that the CM-Patch creatures.txt entry 

mgr.createCreature {
    id = 750,
    itemtype_id = 7778,
    name = "HU_cult_fk_elite",
    behaviour = "Enemy_hunter",
    dangerclass = 6,
    groupmaxcount = 1,
    elite_creature_id = 750,
    probabilityforelite = 0.000000,
    rank = 60,
    tenergy_creature_id = 441,
    template_creature_id = 1382,
    livesremaining = 0,
    unconscioustime = 20,
    palettebits = "1111111111111111",
    monstertype = 1,
    speakertype_id = 13,
    faction_id = 86,
    equipset_id = 179,
    modelscale = 1.150000,
    rise_from_ground = 0,
    has_corpse = 1,
    has_soul = 1,
    can_strafe = 0,
} 

has ONE EXTRA LITTLE SPACE at the last line, "} " instead of "}".

That goddamn ONE LITTLE SPACE messed up my WHOLE program! WRRRAAR sometimes....

 

dude, netting that bug on a Sunday... !

your coffee is WAY better than mine

gogo

 

[Lindor 359]

16 minutes ago, gogoblender said:

obviously your coffee is WAY better than mine

TEA!

-----

I have manually checked all relevant CM-patch files now for an additional space at the beginning/end of the last line of a define and removed them if existent..

@dimitrius154, @Flix, @NathyielNathyiel

Can you please check the following three entries and remove the additional spaces at the end of a define in your mods in case they're there?

creatures.txt:

mgr.createCreature {
    id = 750,
    itemtype_id = 7778,
    name = "HU_cult_fk_elite",
    behaviour = "Enemy_hunter",
    dangerclass = 6,
    groupmaxcount = 1,
    elite_creature_id = 750,
    probabilityforelite = 0.000000,
    rank = 60,
    tenergy_creature_id = 441,
    template_creature_id = 1382,
    livesremaining = 0,
    unconscioustime = 20,
    palettebits = "1111111111111111",
    monstertype = 1,
    speakertype_id = 13,
    faction_id = 86,
    equipset_id = 179,
    modelscale = 1.150000,
    rise_from_ground = 0,
    has_corpse = 1,
    has_soul = 1,
    can_strafe = 0,
} 

weaponpool.txt:

mgr.addWeaponPool { -- dm_dm_chest
    dbid = 1126,
    prefDmg = 1,
    prefDmgProb = 0,
    content = {3223,},
    contentProb = {1,},
} 

spells.txt:

mgr.defineSpell( "trap_nova_onyx", {
    eiStateName = "cSpellCast",
    fxTypeCast = "FX_SK_STANDARTE_C",
    fxTypeSpell = "FX_BOSSGOLEM_SHOCKWAVE",
	fxTypeCastSpecial = "",
    duration = 0.000000,
    animType = "ANIM_TYPE_IDLE",
    animTypeApproach = "",
    animTypeRide = "",
    animTypeSpecial = "",
    causesSpellDamage = 1,
    tokens = {
        entry0 = {"et_duration_sec", 100, 0, 0, 8 },
        entry1 = {"et_range_area", 300, 0, 0, 4 },
        entry2 = {"et_spelldamage_magic", 220, 310, 0, 133 },
        entry3 = {"et_spelldamage_physical", 320, 310, 0, 133 },
	},
    fightDistance = 500.000000,
    aspect = "EA_ENEMY_ANY",
    cooldown = 5.000000,
    soundProfile = 0,
    cost_level = 10,
    cost_base = 10,
    focus_skill_name = "skill__enemy_focus",
    lore_skill_name = "skill__enemy_lore",
    spellClass = "cSpellSeSchockwelle",
    spellcontroltype = "eCAtype_a_effect_self",
    sorting_rank = 0,
}) 

The new and I think final version of the extractor will be released in a couple of minutes.

The inserter will be released right after that as well.

With the help of these tools, I will update my graphics mod for EE 3.1 as the final and ultimate proof of the modmerging system working. It's just a couple of mouseclicks.

[Lindor 359]

There is a problem with surface.txt for the extractor. Sometimes, the define start has a two-space-offset and sometimes it doesn't. I knew this could happen so I've bult a possible configuration for the program to search for the define start from the end of the line and not from the beginning, so the fix is super easy. I'm actually kinda proud of this one. But not so proud that I've overseen the need for it in surface.txt. The fix is gonna be included in 2.1.1.

[Lindor 359]

11 minutes ago, Lindor said:

I don't know what happens with behaviour.txt in detail, but I found out that only the first 5 entries are affected:

• "JOB_DRYADSTATIST", "JOB_COMMUNICATOR", "JOB_WALKPATH", "JOB_SENATOR" and "JOB_DOCKWORKER".

The program assigns those the numbers 0-4 in the order I listed them. Everything else is sorted alphabetically by name, as it should. If someone recognizes a pattern for those numbers appearing in the entries of those defines in behaviour.txt, pls tell me.

 

 

Another file not sorted correctly is soundprofile.txt:

• "Weather Set 1", "Weapon 2H Sword", "Item Stone3", "Item Stone4", "Item Crystal Standard5 S", ...

Notice the pattern? The problem is that the program by default searches for numbers first if not otherly stated, and that the strings contain numbers.

• Enabling the config.prefer_name_over_Id configuration for soundprofile.txt should fix the bug.
• Another, more general but also more calculation heavy solution would be to alter the searchfor_number() function to return nil or false if the number is inside two quotation marks. And that has again the potential to bring its own bugs, like I explained above I don't want to be caught in an infinite fix of the fix of the fix of the... loop.

The decision on how I want to fix it has not been made yet, but I'm leaning towards the config atm. After the mysterious issue of behaviour.txt has been narrowed down, this will be fixed in the next iteration of the S2CE.

 

 

Last but not least, I want to pull this conversation over to the S2CE discussion where it belongs more imho, so I will be quoting myself there.

 

Edited March 26, 2022 by Lindor