Keeping the various package repositories consistent

Developer
Dec 2, 2010 at 9:05 PM

Based on my understanding of the architecture, we have three different database tables that hold the package list:

  1. The Orchard gallery's Orchard_Gallery_PackagePartRecord table
  2. The GalleryServer's Package table
  3. The GalleryServer's PublishedPackage table

The distinction between 2 and 3 is well understood as part of the publish process, but I'm more concerned about consistency between 1 and 2.  It seems we can easily end up in a state where #2 will have a package that #1 doesn't have, or vice versa (e.g. if one side hits errors).

Do you agree this is a potential issue?

Why does the orchard gallery even need to keep it's own package table?  Could it instead get all the information from the server?

Coordinator
Dec 3, 2010 at 1:32 PM

Yes, that is how the architecture is setup. The two sets of tables used in the gallery server were done intentionally to separate the packages that have been published to the public feed, vs. the data that the website or other tools would interact with for creating/updating packages. Another reason for that was to allow the two schemas to be different if needed.

As for the table in the Orchard Gallery, that was another intentional decision in order to take advantage of Orchard itself. We wanted a package to be represented as a content type within Orchard so that we could use Orchard's theming, templating, etc. as well as compose it with other parts of Orchard. We could have created a gallery which just queried the server directly for it's data, but then there wouldn't have been much point in using Orchard. There were a couple of high level goals that were part of the original spec for the Orchard gallery. One was that we would have a reusable back-end system for other systems to use in building a gallery (this became Gallery Server). So somebody else building a gallery on Gallery Server may very well decide to do just that (i.e. just use the server directly instead of syncing the data to a third location). Another goal was to have the gallery website built in Orchard to serve as an example of an Orchard site. So this is why we decided to go this direction.

I do agree that it makes things a little more complicated and there are potential issues with keeping things in sync. We're definitely aware of the issue and will try to make the synchronization process as robust as possible.

Developer
Dec 3, 2010 at 5:27 PM

Thanks Kevin, I understand the rationale behind the choices.  I think what we'll probably end up needing is extra logic to help get things back to a good state if it gets messed up.  e.g. by default you only sync things that are newer than the last time you had synced. But maybe there needs to be a fuller admin-triggered action that would look at the entire feed and try to fix up whatever it can.

Obviously, being super robust in the first place is very important, but I'm concerned about things *somehow* getting out of whack and not having a way out to get back to health.

Thoughts?

Coordinator
Dec 3, 2010 at 8:56 PM

I agree, and we have considered the same idea, where we would have some way for an admin to manually deal with sync issues. We might (depending on time) include a feature in the Orchard admin panel to either sync the whole feed or individual packages. Right now, the synchronizer stores the last time it ran in a settings file, which is located in Orchard.Web/App_Data/Sites/Default/Packages. So to trigger a re-sync an admin could delete that file (which would cause the entire feed to re-sync) or could alter the DateTime value which would cause a re-sync of packages modified since that time. Ideally we'll have this ability surfaced in the admin UI at some point.

Developer
Dec 3, 2010 at 9:01 PM

That sounds good, thanks.