Code-like syntax for XmlTools patch files

[Iritscen]

Let's reiterate what 'dox just stated in another ticket and then add an even worse scenario to the mix, to understand where XmlTools currently stops being useful. I've been procrastinating making this ticket because it's a doozy, but really, it has to be discussed and dealt with. Keep in mind that this is not intended to minimize the work that s10k has already done, which has been very helpful. Also, s10k, please don't start hyperventilating while reading this because there may be a "simple" solution that doesn't require much work from you.

Paradox outlined a sample use case where we would need to be able to modify one part of a tag based on another part of the same tag. So, to alter the length of time (in levels) for which an autoprompt appears, we have to look for an autoprompt like:

<ONGSAutoPrompt>
   <Notes>hypo</Notes>
   <FirstAutopromptLevel>1</FirstAutopromptLevel>
   <LastAutopromptLevel>2</LastAutopromptLevel>
   <SubtitleName>autoprompt_hypo</SubtitleName>
</ONGSAutoPrompt>

according to the value of the SubtitleName tag, then modify the LastAutopromptLevel tag if we find what we're looking for. How would we normally want this to look, in a computing environment? Probably something like this:

if ONGSAutoPrompt.SubtitleName is "autoprompt_hypo" then
   set LastAutopromptLevel to 2
end if

See how simple that can look with standard procedural syntax? Now technically we're missing a 'for' loop there, but we could easily just ask for XmlTools to treat 'if' statements as a request to iterate over all the named elements.

But let's take this a step further with a much-less hypothetical scenario. Let's say that I want to add glass-breaking particles to all TRAMs (because I do). Gumby actually had to write his own tool to do this for the 2009 AE because it requires a conditional syntax and variables. This mod consists of four parts:

1. The glass-breaking particle. This is the easiest part, we just include the file BINA3RAPglass_break.oni in the mod, no patching needed.
2. A list of attacks to affect. This could be based on a pattern like "*punch*", "*kick*", etc., but there will have to be a list of names or patterns. This is currently able to be handled by the AEI's file name-matching of the .oni-patch files, so we can leave that alone, although it does limit the kind of pattern-matching we can do.
3. Now we need to let Oni know that a character can create this particle. We do this by adding this to their ONCC's ONCP array this element:

   <ONCPParticle>
      <Name>glass_break</Name>
      <Type>glass_break</Type>
      <BodyPart>-1</BodyPart>
   </ONCPParticle>
   

Just a standard @ADDTO command, here. Preferably, we only do this if this ONCPParticle has not been added already. Not sure if duplicating the listing does any harm, though, so let's move on.

4. We add the actual emission of the particle to attacks. Here's where it gets tricky. Here's the relevant part of a TRAM (Barabas' Earthquaker):

   <Attack>
      <Start>51</Start>
      <End>55</End>
      <Bones>LeftThigh LeftCalf LeftFoot RightThigh RightCalf RightFoot</Bones>
      [...]
   </Attack>
   

Notice how Barabas does damage with all his lower bones for the duration of frames 51-55. To add glass-breaking (according to how Gumby did it), we have to go to the Particles array in the TRAM, where it currently lists only the barab_wave for one foot (since it only needs to spawn that effect once), and add this:

<Particle>
   <Start>51</Start>
   <End>55</End>
   <Bone>LeftThigh</Bone>
   <Name>glass_break</Name>
</Particle>
<Particle>
   <Start>51</Start>
   <End>55</End>
   <Bone>LeftCalf</Bone>
   <Name>glass_break</Name>
</Particle>
[etc. etc.]

So let's count the ways that this isn't possible with XmlTools:

1. We need to read and save into memory the Start and End frames from the Attack, otherwise we have no idea when the glass-breaking takes place. We could just say frames 1-100, but then the character will break glass as far away as the distance of his attack's Extents before his fist/whatever gets anywhere near the glass.
2. We need to read, parse and save the list of Bones from the Attack array. Otherwise we'd have to guess which bones to add the particle to, and if we guessed wrong, a character standing near glass would break it with the wrong parts of his body. Also, we'd be generating particles unnecessarily if we added glass_break to all bones.

(Yes, we don't need to do A and B programmatically; the use of separate files for each patch allows us to specify which bones and frame ranges to use for each TRAM, but who wants to set up over 400 patches by hand?)

3. Finally, many existing characters had the glass-breaking added to them already. Therefore we don't want to add anything to the ONCCs/TRAMs that are already modded or we double the number of glass_break particles in the system.

In traditional computer terms, just performing step 4, with A, B and C, would require us to be able to say:

for Attack in Attacks
   set start_frame to Attack.Start
   set end_frame to Attack.End
   set array Bones{} to Attack.Bones
   for Bone in Bones{}
      set bone_name to Bone
      set add_this_bone to true
      for Particle in ONCP
         if Particle.Bone is bone_name and Particle.Name is "glass_break" then
            set add_this_bone to false
         end if
      end for
      if add_this_bone is true then
         add to ONCP "Particle" with properties:
            Start:start_frame
            End:end_frame
            Bone:bone_name
            Name:"glass_break"
      end if
   end for
end for

I haven't carefully reviewed that pseudo-code, as it's just an example, so there might be a logical error in there, but you get the idea. This is totally beyond the current scope of Vago's syntax.

Now, we could help s10k in adding this kind of capability to XmlTools manually, but perhaps that's reinventing the wheel when there's already a ton of scripting languages out there. It may be a better idea to allow an existing scripting program to be called from within XmlTools and to pass it the instructions to run. Or, it may make more sense to integrate an open-source interpreter into XmlTools' code.

Thoughts? Questions? Headaches? Angry recriminations? I'm almost sorry to bring this subject up, but I simply don't know how else we can make a glass-breaking patch mod. The only alternative is to write yet another private tool like the one Gumby wrote and didn't release, so why not make something that the community can benefit from?

[Iritscen]

Re: "three" -> "four", I can count, really. But I originally wrote the glass-breaking to have three steps :-)

[Samer]

Hi Iritscen glad to see this written down finally, I have one note though, I mentioned this in the email in which we talked about the glass mod before
if a character does attack with left thigh, left calf, left foot .. It's enough to add the glass particle to the left foot without the left calf and thigh ... This minimizes the amount of particles you have to add ... (if the attack happens with say wrist and fist you can add it to the fist only, since the fist will make contact first) .. But i think that might be harder to autogenerate .. Adding it to wrist and fist or calf and foot doesn't hurt either but it's a unnecessary as many particles are created at same time, and in my experimentation there also seems to be a limit to how many particles a tram can hold .. If it has a contrial and glow for example on 3 bones and then you add glass particles to extra 4 bones there's a high chance it might crash.
Also for example barabas's heavy kick .. This doesn't need any glass breaking particle on the bones since the shockwave will already break the glass. (i know maybe you just chose that tram because it has many bones as an example, but it's something to keep in mind)

[Iritscen]

Thanks for the input, those are good points. If we are going to be writing a script that adds glass-breaking, then it's not much additional trouble to add it to only the feet and hands. One could modify my above script, where it says "for Particle in ONCP", to say something like:

for Particle in ONCP
   if Particle.Bone is bone_name and Particle.Name is "glass_break" then
      set add_this_bone to false
end for
for Bone in Bones{}
   if Bone is "*Foot" and bone_name is ("*Calf" or "*Thigh") then
      // no need to add particle to calf or thigh, attack is being done with foot
      set add_this_bone to false
   end if
   // do same check for fists vs. arms here
end for

You raise a good point about the Earthquaker already breaking glass. I think we might just end up hardcoding into the script some particle names that we know already break glass, like barabwave, and leave the TRAM alone if we detect them. In the original glass mod, the Earthquaker does indeed have glass_break added to all those bones because (1) Gumby didn't take barabwave and similar particles into account, or (2) we just weren't thinking about particle limitations as a community at the time.

[Iritscen]

To add to my earlier thoughts at the end of the ticket, it could be that this isn't a good fit for XmlTools. I know it's a lot bigger of a scope than you originally planned, s10k. It might be that, if we don't incorporate this functionality into XmlTools, it doesn't even make sense to use XmlTools as a middleman to call another program or interpreter, because then we have the AEI calling XmlTools calling something else. So in that case, it would make more sense to add a scripting language to the AEI itself (there are many available for Java), and all the patching can just take place within that single environment without having to call another program. So just let us know how you honestly feel about the subject, s10k.

P.S.: This is filed under "Vago" because I didn't feel like making an "XmlTools" component for a single request :)

[s10k]

I have two solutions for this:

use python (.net implementation)

• possible pros: easier to learn for beginners, less lines of code
• possible cons: slower

use c#

• possible pros: faster, C like
• possible cons: harder to learn, more lines of code

The code would be read from a file (maybe even a patch file).

This would work for every complex xml editing.

The code could be something like:

@CUSTOMCODE File<quoted file name (wildcards accepted)>
<code>
if ONGSAutoPrompt.SubtitleName is "autoprompt_hypo" then
   set LastAutopromptLevel to 2
end if
</code>

[Iritscen]

It's hard to say whether "python.net" or straight C# would be better. I assume by "C#" you mean creating this kind of syntax yourself. I don't know if there would be much of a performance hit with Python, can we find out in advance through any simple tests?

Certainly it seems easier to add in support for Python, but you'll have to limit the Python support to what the modder should be allowed to do (no file I/O, for instance). But it should still be easier than creating a new language from scratch.

You do make a good point about how a user can use XmlTools to "bake" his patches into .oni files to put in his mod. When possible, that will always be preferable to "live-patching" during installation for speed reasons, so that's one obvious advantage of XmlTools doing this instead of the AEI.

[s10k]

I mean really C# not thinking in creating the syntax myself. (.net will compile that code at runtime)

Limit what the user may do with the code may be trickier and I'm not seeing any safe way to do it.

[Iritscen]

Oh! I didn't realize we could supply C# code and have it compiled at runtime, that's interesting. But as you indicated, it could lead to security issues.

[Samer]

I've got to ask .. Is it worth the effort ? If it's meant for a new technique for modding in the future then never mind but if this just for the glass mod i think it's better to just use oni files for the TRAMS .. Use the vanilla TRAMs and manually add the glass particle to them without any other changes in xml and have them as oni files. The gumby ones can be used but compare to vanilla and remove any other changes besides the glass particle .. And modify as to not include unnecessary glass particle as i discussed before (only have it at the foot or fist when the extremeties are involved and remove from moves which break glass already like earthquaker) then any other TRAM fixes are going to be made as xml patches anyway so these will easily add to the TRAM with the glass if it's installed since the xml patch will take the TRAMS in the highest package. All other TRAM modifications can be easily and should be transformed to xml patches (i'm working on that) so there's no incompatability risk that the TRAMs with glass will be overwritten by other TRAM. Then the ONCC modification can be made as xml patch to add the particle to the oncc and avoid incompatabilities with other oncc modifiying mods ... So in the end there will be an optional glass pack with xml patch for onccc and oni files for trams built from vanilla with only exception adding the particle and an oni file of the particle itself .. Will be compatible with all other xml tram patches .. If not that's why we have the incompatability field.

The approach to add the glass particle to the TRAM with coding or the xml patch sounds good in theory but do the advantages outweigh the disadvantages (effort, delay, security issues) vs just using the TRAMS as oni ? I don't see any added benefits for this particular mod.

[Iritscen]

Hmm, well I suppose if every other mod that affects TRAMs was a patch mod -- Domino Knockdowns, Fallen Aiming, VDG throws, the TCTF back throw, hit_jewels, etc., then perhaps we can get away with just using .onis for the TRAMs in Glass-Breaking. Certainly it would apply faster than a patch, which is good.

But what about the Andrashi TRAMs? Are you willing to make all those into patches? There's anywhere from 200-400 TRAMs that he's modded (depending on which set of files you look at), and unlike the glass mod, those changes cannot be described programmatically -- there's no simple pattern to those changes, like the script I provided above for adding glass_break.

Also, do we really want to force modders who want to mod TRAMs to choose between learning and using XML patching, and being incompatible with glass-breaking? And what about the modders who add new TRAMs? We'll have to update the glass mod with modded versions of their .onis every time they release something (and update the glass mod if they alter their new TRAMs afterward in a later version of the mod). Whereas a patch could capture all TRAMs with certain file names like *punch*, new to Oni or not, and modify them all.

Finally, as s10k pointed out, if this feature is added to XmlTools it can be used to apply patches at install-time OR to bake those patches into .oni files before the mod is released (which, after all, was the original purpose of XmlTools). So even as a tool for mass-modification of .oni files, this feature can be helpful in ways that we probably can't anticipate with future mods.

[Samer]

"But what about the Andrashi TRAMs? Are you willing to make all those into patches? There's anywhere from 200-400 TRAMs that he's modded (depending on which set of files you look at).."
his modifications are sometimes unnecessary .. he himself didn't want them to be released ... it's a mess whether glass mod is applied or not .. and yes I do believe they're best be broken down into either bug fixes or just gameplay challenging changes and into xml patches, whether I will be able to turn all of it into xml mod patches is questionable because it's time consuming but it can be done, also they have been missing for years (AE2010 didn't have them) and no one really missed those changes so how necessary are they ? they can be released much later as a challenging pack while the bug fixes which are limited can be isolated to xml patches.

"Also, do we really want to force modders who want to mod TRAMs to choose between learning and using XML patching, and being incompatible with glass-breaking?"
no we don't .. they can just include the glass particle in their new TRAMS as we've been doing for the last few years ... and if they don't learn xml patching for trams they'll risk incompatibility one way or the other if several modders release TRAM modifications.

"Whereas a patch could capture all TRAMs with certain file names like *punch*, new to Oni or not, and modify them all."
I mentioned this before in an email .. this takes the power away from modders .. they may not want their attack to break glass .. example: hammer's heavy kick is an attack where he uses konoko powerup animation and then release muro like electricity it's called heavy_kick .. but it shouldn't break glass, it's a static energy charging animation and it's pushing particle limit for a tram if the patch is added that applies to all *kicks .. it will force t to break glass and also exceed particle limit for that TRAM.

"Finally, as s10k pointed out, if this feature is added to XmlTools it can be used to apply patches at install-time OR to bake those patches into .oni files before the mod is released"
yeah i can see it being useful for other mods, but I still think it's unnecessary for the glass mod. your call though ...

[Iritscen]

"his modifications are sometimes unnecessary"
That's a really subjective statement ^_^ That's why I think you need to (once you have the time) put up a table or a list of changes that can be sorted, grouped and discussed. But since you expressed an interest in this mod (or set of mods), I'm trying to be hands-off in order to let you show us your project organization skills ;)

"he himself didn't want them to be released"
Loser has probably said that about every single mod he's ever made. He also consistently deletes his own work, sometimes before we can obtain a backup copy. Loser is the last person to trust when it comes to an opinion of his own work :)

"whether I will be able to turn all of it into xml mod patches is questionable because it's time consuming but it can be done"
The suggestion implicit in my question about "glass patch vs. Andrashi TRAMs" was that the Andrashi TRAMs might be a better candidate for a whole-.oni mod instead of a patch because of how painful it will be to make a patch, whereas glass-breaking will be easy to describe in patch-script form. Of course the downside is that Andrashi will be incompatible with lots of stuff in that case. Perhaps a whole-.oni release could come first, and an improved, patch version can come Later^TM.

"also they have been missing for years (AE2010 didn't have them) and no one really missed those changes so how necessary are they ?"
I assume you're referring to the ~200 files love_Oni wrote about. Those were released in 2008. Since we did offer another ~200 files with the 2009 and 2010 AEs under the name "Andrashi TRAMs", I think players were less likely to notice what was missing since fewer people played the original 2008 AE or mod.

But I partially agree with you. I definitely don't consider this a project that needs to be done urgently; nor do I consider glass-breaking moves to be an urgent need of the community either. It's better to take our time and do them right.

"they can just include the glass particle in their new TRAMS as we've been doing for the last few years... and if they don't learn xml patching for trams they'll risk incompatibility one way or the other if several modders release TRAM modifications."
That's a good point. Perhaps we shouldn't attempt to patch new TRAMs at all. After all, many (most?) of them will be getting skipped anyway by the patch script if they have glass_break already.

"it shouldn't break glass, it's a static energy charging animation and it's pushing particle limit for a tram if the patch is added that applies to all *kicks .. it will force it to break glass and also exceed particle limit for that TRAM."
It's true that a pattern-based patch might have minor drawbacks like that, but honestly, nobody's gonna care if an electrical attack breaks glass ^_^ That's really splitting hairs, but ​here's a reason why that could still be considered an appropriate effect. As for particle limits, the script could easily count the existing particles to make sure it doesn't exceed a certain number.

"yeah i can see it being useful for other mods, but I still think it's unnecessary for the glass mod."
Just to be clear, this ticket isn't for the glass-breaking mod. Glass-breaking was just one of my two examples of the usefulness of having a real scripting language available for patching. I picked it because it's the most complex example I'm aware of, and so I thought it was a good example of the uppermost limit of what we will need. If you acknowledge that this functionality will be useful for some mods, then you are agreeing with the ticket's request after all :)

[Iritscen]

Edited just the paragraph that begins "That's a good point".

[Samer]

"his modifications are sometimes unnecessary."
what i meant was they aren't all bug fixes as in necessary modifications. it's not necessary to make super moves have more vulnerability frames or make directional punches of other characters cause knockdown. but it's necessary to fix bugs. these aren't bugs these are challenging modifications (not being subjective) .. will get to more details later on.

"Perhaps a whole-.oni release could come first, and an improved, patch version can come LaterTM" maybe but i'm interested in the bug fixes these should be xml patches while the Andrashi package aspects that are to increase challenge can be oni optional package that's why it's important to isolate the bug fixes from the challenging modifications.
requires more investigation on my part.

"It's true that a pattern-based patch might have minor drawbacks like that, but honestly, nobody's gonna care if an electrical attack breaks glass .."
I gave that as an example there are others though where the animation isn't meant to be damaging with the bones ... but since you say it can keep track of particle limit then yes i guess there's no harm in that.

"If you acknowledge that this functionality will be useful for some mods" yes sure, I did say this upfront "If it's meant for a new technique for modding in the future then never mind but if this just for the glass mod..." I'm sure if the technique is available it will be put into good use later.

[Iritscen]

Replying to Samer:

"his modifications are sometimes unnecessary."
what i meant was they aren't all bug fixes as in necessary modifications. [...] these aren't bugs these are challenging modifications

Sure, okay, I agree with that.

the bug fixes should be xml patches while the Andrashi package aspects that are to increase challenge can be oni optional package

Okay, that might be a good way to proceed.

Sorry to anyone getting these off-topic emails ^_^;

[s10k]

Ok after some research I got two possible options:

C# -> allows run code in a secure sandbox (already tried and seems to work great). As downside it doesn't seem to create the sandbox correctly in the MacOS (mono doesn't seem to support fully CAS correctly ​http://www.mono-project.com/CAS) so insecure may be able to run in this OS.

Lua -> scripting engine which have also support for sandbox environment seems that are some .NET implementations which also support mono. I will need to integrate a XML library. Still need some research but looks a nice approach.

Right now I'm more inclined to the Lua implementation due the fact of the C# mono security issues. Lua may not be an easy language to start with, but I think this advanced feature may be mainly used by advanced users too.

[Paradox-01]

Gedankenexperiment:

Case 1: Evil person uploads an package with an evil script. XmlTools runs it in a sand box, nothing happens.

Case 2: Evil person learn from his mistake and replaces XmlTools on ModDepot with an evil version. Now evil script works.

Is case 2 not possible?

[Christian]

Replying to Paradox-01:

Is case 2 not possible?

Shouldn't be as not just anyone can upload stuff to the Depot ;)
(We'd probably even notice if someone else than s10k would upload something to the xmlTools node)

[s10k]

Case 2 is possible but improbable because it will be probably fast detected plus we will know who did it. (the upload)

Still is a security breach but that may happen also with OniSplit or any other AEI executable.

[Paradox-01]

I would feel more save if you create whitelist* of uploaders.

Only persons involved in the development of executables should be listed there.

Alloc
EdT (if he continues AETool)
Iritscen
Neo
Script10k

So then.. sandbox + whitelist.

[Iritscen]

I've been thinking about the issue of executables myself. I could easily create a new node type called Modding Tool that is limited to certain users and which accepts .exes, and change the Mod node type to not allow .exes.

The problem, as you have probably already realized by this sentence, is that we're not uploading .exes, we're uploading .zips that contain all sorts of files including .exes. It seems that Drupal doesn't have support for examining and rejecting .zips based on content, so we would have to write a script that would run on the server that would list the files in the .zip and reject it if it contained a .exe and the node type was "Mod". I haven't done anything like this before, but maybe with Alloc's help it would be possible.

Unfortunately, there's always the chance that someone's password might be guessed or hacked. I'm also thinking of installing an extension that can email me and Alloc when a node is changed. That way, nothing can get by us for very long.

[Samer]

uhm is it that dangerous ?
I mean besides the people paradox mentioned, only paradox himself, gumby, geyser, demos kratos and I can upload to the depot ...
I know nothing about coding .. so that's one person you don't have to worry about :P and maybe disallow uploading for those who have become inactive ... who exactly are we worried about having ill intentions from our current uploaders ?
and in a chance someone's depot account is hacked by a random person would the person who hacked it be interested in hacking\damaging an old game fan base :\ ? they'll have to know about Oni, and about which executable to change and about modding .. I find the chances of that slim to null ... and can't antiviruses detect that ?
just seems to me like that effort is better spent on something else ... like making the mod.cfg up-datable from the mod's depot info like discussed before or fixing the particle limit :P..

[Paradox-01]

can't antiviruses detect that

Antivirus programs know only that what has been found in the anitvirus company labs. New viruses, new dangers. Also there are ways to trick such software: encrypted code waiting for a distraction - looped file generation (e.g. zipbomb) or buffer overflows.

would the person who hacked it be interested in hacking\damaging an old game fan base :\ ?

Fan or not fan, they are people with computers and as such I guess a hacker would be interested in passwords for facebook, twitter or online banking, or turning the system into a zombie (-> botnet for spam / DDoS attacks).

I think it would be good to have some measurements to discourage hackers to even try it. "Better save than sorry."

[Samer]

but how are they even going to get access to that ? if only a few of us can upload ..
they'll have to hack one of the uploaders, then change the executable then upload it to the depot knowing how to package it and fill the right info etc.. then wait for users to run AEinstaller and update their onisplit or whatever all this while no one of us detected it. I'm just not getting how some random hackers will pay attention to Oni AE in particular as vast as the internet is and know that it's running onisplit or xml-tools and know that the AEinstaller automatically updates these even though our uploads are zipped and know who the up-loaders are, to then hack them and all this while no one detects that they've been hacked .. seems very farfetched unless they are targeting Oni fan base in particular and have internal insights of how our modding and uploading and how AEinstaller works .. in that case those who have that knowledge are our few fellow modders whom I trust .. anyway ... ok better safe than sorry :\

[Iritscen]

Sorry, Samer, but this is just one of those things that responsible coders have to think about, no ifs, ands or buts. We've had at least one unstable individual in the community who would hack us if he was smart enough, so we can't say that there will never be a disgruntled fan who has just enough time on his hands to guess a password and to learn some simple scripting.

Besides that, you have to consider the possibility of a trusted modder who simply makes a mistake. If we've allowed a patch script to alter files "directly", then a simple typo could have devastating consequences on users' computers. I wrote fairly extensive code along with the AE 1.1 to prevent an update installation (this was back with the offline update system which never actually got used, as it turns out) from affecting the wrong files on the computer. I went to the lengths of writing up a list of every possible security hole I could think of and then testing them to make sure they were closed. All this despite the fact that only Gumby and I were ever going to be releasing the file that drove this update. The goal was simply to prevent costly mistakes from occurring.

Moral of the story: even if you're only writing code that will be called by a single scripter, a developer wants to have the peace of mind of knowing that his code shouldn't be able to possibly do any harm to the computer of an innocent user.

[s10k]

Seems I will forget Lua. I implemented it with success in .NET however I couldn't find a complete and easy to use xml library.

However good news. I was able to find a really nice native javascript library to parse XML. Already tried it in .NET and works great even in MONO.

I will probably choose Javascript as script language for xmlTools since it is really easy to use and you can find plenty of information in the web. (plus it's sandbox language)

I will work on this in the weekend. News soon.

[s10k]

Implementation of the javascript custom code is now complete. It will be released in XmlTools 0.9. It may need some optimizations (working on it).

[Iritscen]

Just an update, s10k has moved XmlTools to C++ for a new implementation of JavaScript with better performance and Mac compatibility.