Read Incompatibility\Dependency from Depot

[Samer]

the mods read the incompatibility and dependency from the mod.cfg while almost all other info is read from the depot, would it be possible to add to the depot 2 new fields for incompatibility and dependency and have the AEinstaller read them from there ? like it does for description, package number etc ..
reason is when new incompatabilites are discovered or package changes that require new dependencies added or removed, modder is forced to re-upload the package and player to re-download when all was changed is the mod.cfg info .. if it's on the depot, we can change it anytime we need without having to reupoad\ redownload the whole package and we can "supervise" mods whose modders might forget to add them correctly.
they can also be kept in the mod.cfg to be read in offline mod but letting those on the depot take precedence like description or the other info.
I can only imagine that this would require some heavy rework of code ... so I don't know if it's worth the effort.

[Christian]

As you said yourself: If it's not in the .cfg it won't help for mods which are not from the depot (e.g. pre-uploading) and also won't work if someone doesn't have a Depot cache (e.g. because of continuously being in offline mode).

And if we want to have it at both places it doesn't change anything about having to reupload a package anyway so that doesn't make any sense ;)

Let's wait what Irit's saying on this though.

[Iritscen]

I'd like to tackle this subject at an even higher level. I've been thinking about this general situation for a while -- I've never liked having to duplicate the mod name, version, creator, and description between the .cfg file in the .zip and the Depot node fields. It seems even more unnecessary now that the AEI reads the Depot fields and displays that info automatically. That makes the .cfg totally redundant except when the package is not on the Depot.

We could remove the .cfg altogether and have the info only entered on the Depot, but that would mean that mods couldn't be distributed without the Depot. The AEI could probably build a .cfg from the mod node info upon downloading, but then you have a system whereby, if someone's mod is on the Depot, it needs to be downloaded through the AEI or it won't work locally.

We could leave the .cfg in the .zip, but then we have the probability that someone will update the mod node fields but not re-upload the mod with the updated .cfg (which is already possible to have happen, just that it becomes even more likely if we add the incompat. and depend. fields). Not that we have a bunch of careless or lazy people updating the Depot, but any time you have duplicated info, it can lead to mistakes. I'm sure we already have a number of minor discrepancies between mods' node fields and their .cfgs.

So if it doesn't make much sense to have duplicated info, and it's not a good idea to have the info only on the Depot as node fields, and we don't like having to re-upload a mod that might be dozens of megs (in the case of textures or music) just to update a tiny amount of text, that leaves one option: have the .cfg file be read directly by the Depot and used to populate the node fields.

Right now the .cfg file is of course buried in the ZIP. I could have the user upload the .cfg separately from the ZIP, but then you have the same issue of separating the .cfg from the package so it can't stand alone outside of the Depot. The only perfect solution I can imagine is a two-way process:

1. User starts a new mod node and begins by uploading the ZIP of his package (in the same format we're currently using).
2. Depot unzips mod internally, reads .cfg, and fills in the node fields for the uploader (this means the form would probably consist of just an upload widget).
3. When the mod's info needs to be updated, someone edits the fields on the Depot.
4. The Depot builds a new .cfg from the node's fields, unzips the package it has stored, replaces the .cfg, and re-zips it.

The AEI could continue to work as usual, modders could continue to use packages locally with the .cfg they made or that was generated by Vago, and players could download packages from the Depot by HTTP or get them through some totally different means, and they would still have the .cfg file in them.

In theory the code required to do this is not hard. I've written PHP to do some of these things before. The trickiest part would probably be just integrating that code with Drupal. I don't really see a point in doing any less than this if we're still going to be stuck with duplicated data and having to re-upload entire mods.

[Iritscen]

To round out my previous comment, I am also thinking of a couple other factors when I suggest this. For one, the request to add mod preview images to the Depot. If the Depot has the ability to look inside a ZIP, then it can also extract preview images that are found in, say, a previews/ folder that the modder has included.

Also, to expand on step 2 above, what I mean is that the upload form could be reduced to one editable field -- the file attachment -- and then once the ZIP is uploaded and processed, the Creator and other fields would be filled in as non-editable fields. Then the user would be able to hit Save if everything looked good. Going back to the mod in the future, all the fields would be editable and the changes would feed back into the ZIP's .cfg file.

A second reason for my suggestion is that if you look at how Overgrowth fans upload their mods, it's a bit simpler than our process ^_^: ​http://ogmods.antonriehl.com/. You'll note that the site is being phased out; that's because in the future you'll be able to upload mods from within one of their modding tools. And if all the needed info for an Oni mod is stored in the ZIP, that's the direction we could move in too: the Depot could be a mostly-unseen backend and anyone could write a modding tool that could send submissions to it (probably the tool will require your account's login info to pass to the Depot before it can upload, since I don't plan on allowing uploading without an account).

I realize this is all outside the scope of Samer's original suggestion. I also don't feel like doing it just yet :-) I'd rather finish my AE work, then make some more progress on my other project, and then come back to this "Depot 2.0" idea. Since Alloc pointed out that we would still need to re-upload a mod if we needed to update an incompatibility, or dependency, I'm not sure that adding those fields to the Depot will help.

For now I'll wait for comments on this idea, but unless I see a good argument for that info being on the Depot, I'll probably close this as wontfix. No matter what, this is probably a Depot request, not an AEI request, so I'll change the component.

[Samer]

Replying to Alloc:

And if we want to have it at both places it doesn't change anything about having to reupload a package anyway so that doesn't make any sense ;)

I didn't get that. right now if the mod.cfg has a description, an old description, and on the depot we have another description, the description that's on the depot is the one that's shown, if you go offline then the description in mod.cfg is shown .. that's what i was suggesting for the incompatibility .. say a mod is incompatible with 34000 and 23500 in its mod.cfg, then on the depot we update the incompatibility field to have 34000, 23500, 23600 or whatever, it will read now that it's incompatible with these 3 mods, s long as it's online, if it's offline it will still think it's incompatible with 34000, 23500 only, true, but here we didn't need to update the whole package to update this info, you're saying they'll have to reuplaod the package for the offline use, if the package is updated then they will also need to be online to be notified about the update anyway, so both require to be online, but one necessities re-uploading whole package and redwonlaoding whole thing, and the other only updates the new info which is more practical. and anytime user is using offline mode they risk outdated info\packages that's inevitable. but the downfall here will be if it's online it's incompatible with 3 mods, if it's offline only incompatible with 2,
so i like to amend my first suggestion combining it with some of Iritscen's: the AEinstaller should be able to save the new info into the mod.cfg so we then can update the mod.cfg by simply changing the info on the depot and that info is saved once updated, (like it generates the aei file in the pack it should be able to generate a new mod.cfg from the depot info and update the mod.cfg included (after unzipping on hard drive) ... modders will still include mod.cfg when they upload their packs but now they will be able to be updated by changing the info on the depot .. same as package updates that will require online connection, but once updated they stay updated offline too. this will also solve the difference in the info between the depot and the mod.cfg .. which many mods currently have especially in the description field.
this will only need to add 2 fields to the depot, and for the AEinstaller to be able to pull the info from the depot and update the mod.cfg, I don't know much about coding but I think that it can be relatively easily done(?) because the AEinstaller already does that to display the info in the AEinstaller now it will only need to save that info in a new mod.cfg.
some packs are 40 mb or more it doesn't make sense to have to re-download\re-upload for only a change in mod.cfg ... example once the exploding barrel is made an xml patch it will not be incompatible with my mods anymore, so i'll have to reuplaod and ask users to redownlaod 8 packages for only a change in mod.cfg .. this could happen anytime as incompatibilities are resolved or new ones arise.
@Iritscen I think like i said easiest way is to be able to update the mod.cfg from the depot info like you also suggested, but I find the local zipping, unzipping and the filling info from mod.cfg first time a bit unnecessary .. we can fill the info on the depot manually and it will update the mod.cfg one automatically.

[Iritscen]

That's an interesting idea, but the main issue that comes to mind at the moment is that if someone downloads a mod through HTTP while browsing the Depot, and installs it with the AEI while offline, they'll get the out-of-date package. Yes, someone working offline has to expect that they are at risk of out-of-date packages, but the risk is higher when we intentionally are not updating the mod's .cfg file and expecting the AEI to do it based on the latest Depot info. But let's see how Alloc feels about this.

[s10k]

I also thought about this before. I think this half read from depot, half read from mod.cfg will produce inconsistencies with the time.

I think the simplest solution would be to just display in AEI the information contained in mod.cfg, ignoring the one in depot.

(The main disadvantage will be that slower internet users will need to upload an entire zip to update the configurations which may take a bit due to their connection speed)

[Christian]

Replying to s10k:

I think the simplest solution would be to just display in AEI the information contained in mod.cfg, ignoring the one in depot.

Seems like you forget the main idea behind AEI: Displaying information before actually downloading hundreds of MiB of data ;)
(If the info was in the cfg only we couldn't show anything before downloading the packages)

Basically I think the idea to only update Depot fields and replace the mod_info.cfg online is the best idea. Though before doing anything we'll have to think about all the possible workflows.
E.g. what about modders uploading actually new versions of their mods? Take the old information that was on the Depot and put that in the new Zip? Always take the information from the newly uploaded Zip?
How do we handle new additions? Upload the Zip already containing the cfg and take the information from there or type the data on a web form and replace the cfg from the Zip? (I assume Modders will always create their cfg locally first to test their mods prior to uploading to the Depot so there'll be some file anyway)

Excluding the config completely from the package makes no sense as iritscen pointed out as that would make a matching Depot cache mandatory to use the mods. Only having it in the package is also not possible as I stated as reply to s10k. So we will have that redundancy anyway. It's just up to us to make it less likely to become inconsistent =)

[Samer]

"E.g. what about modders uploading actually new versions of their mods? Take the old information that was on the Depot and put that in the new Zip? Always take the information from the newly uploaded Zip?"
mmm I think we're over complicating things a bit .. when a modder is updating his package with new zipped mod.cfg it's very easy to just copy paste the new info from the text file and paste it on the depot fields (or vice versa) ... they'll be editing the nod anyway.
So it all really needs to do is read the info from the depot and update the mod.cfg on hard disk with it .. so when someone updates their pack and the info in mod.cfg they can easily manually edit the depot fields when they're uploading the package (just copy paste them manually, that's what we've always done, takes a few seconds) ... however after it's uploaded editing mod.cfg becomes hard without reuplaoding it, hence here the feature of updating the downloaded mod.cfg by updating the depot info later. many packs currently have had their description updated on the depot but still have old description in their mod.cfg... allowing the depot info to overwrite that of the mod.cfg and save it to local disk will fix that quickly. and I personally can quickly edit the depot nodes and add the incompatibilities and dependencies to them if these fields are made available before this is implemented so there's no risk the depot will overwrite the incompatabilities\dependecies with null ones or something while the fields are still empty.

"Basically I think the idea to only update Depot fields and replace the mod_info.cfg online is the best idea."
maybe I'm not fully understanding how the online zipping unzipping works but won't that take double the time ? unzip package somewhere online, rewrite the mod.cfg and then rezip and then download locally and then unzip on local disk and then install ?
why not download and unzip locally like it does now, then when AEinstaller is opened update the mod.cfg from the depot like it does when it pulls the info to display in the installer, it just needs to save it here locally as well. even people who download the mods from the depot directly still need the installer to install the mods and once it's connected once it will update the mod.cfg of the mods user has downloaded. The main feature of the new anniversary is the ability to update and download packages through the installer I don't think we should worry too much about exceptions when someone downloads the mods from the depot directly, if they do they'll still need installer to run, once it does it will update them, and also here's an issue if user downloads mod from depot directly and puts it in packages folders even if it's same version the installer will consider it outdated because no aei file was generated, so downloading from depot directly should be discouraged and already has its problems.

[Iritscen]

"So [all it] really needs to do is read the info from the depot and update the mod.cfg on hard disk with it"
Wouldn't that defeat the purpose of even having a .cfg file in the package, besides local testing by the modder during development?

Moreover, it once again puts the player at risk of having outdated mods if he doesn't have an Internet connection. What if someone's traveling, or took his computer to a friend's house for a LAN party and doesn't have Internet access so that the AEI can get the latest info from the Depot?

"maybe I'm not fully understanding how the online zipping unzipping works but won't that take double the time ?"
The server would be doing this behind the scenes after the node was edited, so to the user it wouldn't take any more time at all (except that the mod would be unavailable for download for a few seconds until processing was finished).

"I don't think we should worry too much about exceptions when someone downloads the mods from the depot directly, if they do they'll still need installer to run, once it does it will update them"
I think you're assuming the presence of an Internet connection, but Alloc specifically made sure that the AEI would operate offline when coding it.

"and also here's an issue if user downloads mod from depot directly and puts it in packages folders even if it's same version the installer will consider it outdated because no aei file was generated"
That's a good point. There's probably a good solution to that but we might want to consider that separately since it's an existing issue.

[Samer]

Moreover, it once again puts the player at risk of having outdated mods if he doesn't have an Internet connection.
If he doesn't have internet connection he won't know the package is updated either .. So whether the whole package is reuploaded, reupdated or only the info it won't make a difference right ? .. (If we're saying that it's better to update entire package so we won't risk outdated info) .. That also still needs internet connection to be notified and to download the update .. Only difference is whether it is redownloaded as whole or just updating the mod.cfg ...
Say a player downloads a pack with the AEinstaller the zipped pack mod.cfg is outdated ...after pack is downloaded immediately the AEI will read the info from depot and update the mod.cfg and save it .. Now even if he goes offline later he'll still have the newer info. (traveling or lan or whatever .. It is and remains updated)
rare exception is if he downloads from depot directly the zips on one computer then unzips and moves them on another computer without internet connection.
(because if he can download from depot means he has internet connection and once he uses AEinstaller to install it will update the mod.cfg)

The mod.cfg is useful for testing yes, and if a pack is local, it also specifies bsl tags and tools tag that the depot doesn't have, so it's still necessary.

... Anyway if the zipping unzipping thing happens only when node is edited then that's a great idea .. What i thought was that it will do this everytime someone downloads it ... Is there any risk for zip corruption while it's being rezipped or unzipped though ?

[Iritscen]

"Say a player downloads a pack with the AEinstaller the zipped pack mod.cfg is outdated ...after pack is downloaded immediately the AEI will read the info from depot and update the mod.cfg and save it .. Now even if he goes offline later he'll still have the newer info."
True, that part would work fine. Still not sure how I feel about the fact that HTTP-downloaded mods could be out-of-date, though, as long as the Depot is offering them for download directly like it does. Why, just today I downloaded a few mods from the Depot because I wanted to compare files -- this was greatly preferable to waiting for the AEI to launch and then selecting them for installation in order to get them to download (when I didn't even want to install them at all).

"Anyway if the zipping unzipping thing happens only when node is edited then that's a great idea"
Yep, now you've got it, it remakes the zip only when the node is edited.

"Is there any risk for zip corruption while it's being rezipped or unzipped though?"
A programmer has to assume that any file operation can fail, even if it seems impossible (recently part of our Oni2.net server ran out of space and broke Depot uploading, for instance -- stuff happens :-), so he always has to figure out a way to catch and handle that failure. Probably it would be set up to email me if a ZIP operation failed. I don't know if corruption (an uncaught failure) could happen because I'm not an expert in the ZIP format, but I doubt it is likely to happen (also, I see there is a "test" feature in the 'zip' command to double-check the newly-created archive).

[Samer]

this was greatly preferable to waiting for the AEI to launch and then selecting them for installation in order to get them to download (when I didn't even want to install them at all).

out of topic but that's another use for the "re-download - download" right context menu in the installer :) can download without installing :)

anyway now that I got how it will work, I will keep my fingers crossed that it gets implemented :) thanks for clarifying.