AEI2 bsl core packages override installed mods

[s10k]

Currently AEI2 install the BSL core mods after the user BSL mods have been installed.

This is causing serious problems for BSL mods that use the original levels.

For example the package that I have attached to this issue doesn't work correctly with AEI2 because the core package "Widescreen Fixes" always replaces the "Manplant - My friends (by AWP)" "manplant_cutscene.bsl" and "manplant_level_logic.bsl" by its own.

This is incorrect because it simply kills the "Manplant - My friends (by AWP)" mod.

I suggest that the BSL core packages (not sure about .ONI packages) to be installed before any other mod. This way the packages that use their own versions of the existing scripts aren't affected.

[Iritscen]

I'm not sure why we never noticed this before, but the installation order for BSL files, according to the end of Installation.log, is:

1. Mods (e.g. Manplant - My friends)
2. Vanilla BSL (excluding levels covered by mods in step 1)
3. Addons (including Widescreen Fixes in Core)

Thus, the BSL files in Widescreen Fixes unfortunately replace any files by the same name in any mods that the user has chosen to install. It seems to me that the correct order would be:

1. Core Mods (including Vanilla BSL)
2. Core BSL Addons (currently just Widescreen Fixes)
3. Mods (including BSL->Yes mods)
4. BSL Addons

There are other more complex solutions to this problem, but this is a simple approach that should guarantee that third-party mods are not overwritten by core mods.

[s10k]

I think it is also necessary to clear the vanilla bsl folder if the user's bsl mod uses it.

For instance if there is a core mod that use Airport folder and a user's mod that also uses this folder it may result in problems, even if the files are different. Because the user's mod may have files that contain variables or functions present in vanilla or core bsl files (this usually results in a bsl runtime error).

So as a safer measure I suggest that a user's mod always clear the vanilla and core bsl files (and only those, other user's mod bsl packages are ok, even if they share the same folder, since they can always be marked as incompatible).

[Alloc]

Ok, looking at the code it should currently be like this:
Take all non-addon BSL mods in package number *descending* order, then apply addons in *ascending* order.
As vanilla and widescreen are just normal mods too this means that it's normally (as Iritscen already pointed out :) ):

1. Non-vanilla mods (non-addons)
2. Vanilla
3. Widescreen
4. Other addons

Non-addons are looked at file by file, copying the file if it does not exist yet in the target folder: That means different mods (including vanilla) can add up files to one single mission folder.
Addons are looked at file by file too but always overwrite existing BSL files too.

As I wouldn't have made it descending order without a good reason I suppose we should first look at why it's in that way. Could be I simply didn't want to code an exception and by that order it will first add mod files and then not overwrite those mod files with the vanilla files.

If I got the posts by you two right what you'd like to see in the end is:

1. Vanilla
2. Clear out mission folders that are later on touched by non-addon mods in step 4
3. Add addon-files from core
4. Add files from non-addon mods
5. Add addon-files from mods

?

What if multiple non-addon mods add to the same mission? What about addon files from core? If we could switch steps 3 and 4 it would make things cleaner again on code (i.e. not handling core addons differently than mod addons).

[Iritscen]

I looked at the code too, and was puzzled why the non-addons were being installed in descending order, but the reason must be exactly as you said: so the vanilla BSL would be installed for any levels that didn't already get a BSL mod installed in them. However, installing optional mods in descending order is counter to what we have taught the user to expect. It also seems like, if we are installing .onis in ascending order and BSL in descending order, then mods that use both components will get jumbled together. So we need to keep everything in ascending order.

Anyway, your proposed steps are almost correct. However, if the level folders are cleared out before core addons (i.e., Widescreen Fixes) are installed, then there is still a conflict between the BSL files in Fixes and in any non-addons that might come after it in the installation. We have to give optional mods higher priority than Fixes. Thus the desired order would be:

1. Install all Vanilla BSL (I guess you will have to hardcode the package number like you did for OniSplit and XmlTools)
2. Install Core addons (deleting any individual pre-existing vanilla BSL files that are in the way)
3. Install BSL mods (deleting all pre-existing BSL for the affected level)
4. Install BSL addons (deleting any individual pre-existing BSL files that are in the way)

To answer your questions:

"What if multiple non-addon mods add to the same mission?"

I don't think we ever intended to support multiple BSL scenario mods in one level. That was the whole reason we introduced the "BSL->Addon" flag in AE 2010-05 (even though it wasn't properly implemented at the time) -- so that users could deliberately add onto a level or mod instead of replacing all its BSL.

"What about addon files from core?"

We don't really have a choice but to remove Widescreen Fixes' BSL files whenever a non-core mod (addon or not) is about to be installed. The BSL in Fixes corrects some cutscene logic to prevent issues in widescreen resolutions, however if a modder wrote his own cutscene logic then we have to consider that to be more important. The only way to make Fixes a true addon would be to introduce a BSL patching feature to go along with our existing .oni patching. Otherwise we have no choice but to erase the Fixes that conflict with user-selected mods.

[Alloc]

Hm, your explanations seem to contradict themselves a bit ;)

deleting any individual pre-existing vanilla BSL files that are in the way

Suppose that is simply meant as "overwriting existing files"?

Step 3 says remove all if already anything in there, step 4 does not. Later on:

We don't really have a choice but to remove Widescreen Fixes' BSL files whenever a non-core mod (addon or not) is about to be installed

So what about level1 receiving *only* an addon. Do we clear out anything or just overwrite what's in this addon itself?

[Iritscen]

"Suppose that is simply meant as "overwriting existing files"?

Yes, that's what I meant, I was just trying to be precise, and to draw a distinction between deleting individual files in steps 2 and 4 vs. deleting all the existing files in that level folder in step 3.

"Step 3 says remove all if already anything in there, step 4 does not."

That's right; that's the difference between a full scenario and an addon :-) As far as Widescreen Fixes, if my steps are followed, then the fixes will get automatically replaced by any BSL files that are installed in steps 3 and 4.

"So what about level1 receiving *only* an addon."

Step 1 will install the vanilla BSL, nothing will happen in steps 2 and 3, and then step 4 will install the addon BSL in place of any conflicting vanilla BSL.

[Alloc]

Why nothing in step 2? Wouldn't it get the widescreen (and possibly others from core) addon?

[Iritscen]

Sorry, I was taking you too literally. The only core addon is Widescreen Fixes, which doesn't affect level 1.

But if we use level 2 as an example, then step 1 will install the vanilla BSL, step 2 will install the Fix for that level, and step 4 will install an addon for that level. If the addon specifically contains a manplant_cutscene.bsl or a manplant_level_logic.bsl, then it will replace the manplant_cutscene.bsl or manplant_level_logic.bsl that Fixes installed, which is the behavior we want.

[s10k]

"I looked at the code too, and was puzzled why the non-addons were being installed in descending order..."

I agree with this Iritscen's solution. I see this as the correct way to fix the BSL issues. I also agree with him in the next comments.

I think Iriscen's steps are the best to fix this situation:

"

1. Install all Vanilla BSL (I guess you will have to hardcode the package number like you did for OniSplit and XmlTools)
2. Install Core addons (deleting any individual pre-existing vanilla BSL files that are in the way)
3. Install BSL mods (deleting all pre-existing BSL for the affected level) -> the clear of the pre-existing BSL folder is too avoid files / functions / variables BSL conflits in runtime.
4. Install BSL addons (deleting any individual pre-existing BSL files that are in the way)

"

"What if multiple non-addon mods add to the same mission?"

In this case the owners of the optional mods should add an incompability flag with each other. (you can decide what is the best way to handle it in AEI2 if this happens and user's haven't added the incompability flag, maybe clear the folder, maybe replace the files, maybe give a warning etc, in the end it's user's fault for not marking them as incompatible).

Core mods should use the addon option as Iritscen mentioned.

[Alloc]

Ok, just to reiterate for confirmation on the new implementation I added:

1. Iterating all core non-addon mods: Copying everything over (without overwriting) to the target
2. Iterating all core addon mods: Copying everything over (with overwriting)
3. Iterating all non-core non-addon mods: If a level is affected for the first time in this step that already exists in the target it's deleted. Also copying everything over (without overwriting)
4. Iterating all non-core addon mods: Just as step 2, overwrites existing files

For steps 1. and 3.: If multiple mods in either step (or a single mod with platform-dependent folders with e.g. both "win" and "common" folder affecting the same file) would place the same file in a target folder the first mod will win, the second file will be ignored.

Is that correct?

[alloc]

In 1082:

Error: Processor CommitTicketReference failed

Unsupported version control system "svn": No module named svn

[s10k]

@Alloc

"For steps 1. and 3.: If multiple mods in either step (or a single mod with platform-dependent folders with e.g. both "win" and "common" folder affecting the same file) would place the same file in a target folder the first mod will win, the second file will be ignored.
Is that correct?"

That looks ok for me (including all your steps). As I said this should be user's mods responsibility to update the packages as incompatible (AEI2 could only at most just report a warning that an existing bsl has being replaced, but it is ok for now).

Also if someone writes a mod that is dependent on another, let say as example to add additional difficulty to an existing level, they can use the non-core addon mod to override the desired BSL files of the dependent mod.

[Iritscen]

Hmm, not sure why you rephrased everything. The way I wrote it above was very precisely worded. The main issue I see with your rephrased steps is that non-addon mods are, by definition, supposed to overwrite each other in ascending order. So if mod 10000 is installed and then mod 10001, 10001 is intended to overwrite whatever came in 10000. The rule should be the same for BSL as it is for .oni files, otherwise we end up with the .oni files from one mod (10001) mixed with the BSL from another mod (10000). Also, my request in step 3 was for the entire level folder to be cleared any time the next mod in line modifies the same BSL folder.

Finally, your use of the term "without overwriting" makes me nervous. First, the idea behind the steps as I wrote them is supposed to be that there cannot be any name conflicts, because step 1 was only installing vanilla BSL, and step 3 could only proceed after any existing BSL files were systematically deleted. Now, if by some chance there *is* a name conflict between files, it seems better to me to overwrite than not to overwrite. That's how it works for .oni files.

[Alloc]

Hmm, not sure why you rephrased everything. The way I wrote it above was very precisely worded.

Maybe it's just not been as "perfectly" phrased as you might think and rephrasing with my own words was to make sure we're talking about the same.
So basically what you want to see changed still is in step 3: every time a new mod affects a level clear out that folder, not just when the first mod affects a level in that step? So only the highest number mod for each level is used at all (excluding addons of course)?

[Iritscen]

I think that's the only change needed, yes. Can we do some testing of this version before you release it into the wild, though?

[alloc]

In 1086:

Error: Processor CommitTicketReference failed

Unsupported version control system "svn": No module named svn

[Alloc]

Doesn't really make a lot of sense with the previous change out in 2.28 anyway? Also, these are minor changes, shouldn't break anything (besides changing the order in which BSL stuff is applied).
It's in 2.29 now.

[Iritscen]

Oh, I didn't even realize you had already released what you had done so far :-/ So you asked if it was correct and then immediately shipped it? All right, well I'm out of time right now, but will try to test 2.29 later today.