FS#2329 - Database syncing/coordination with main site database

Attached to Project: AUR web interface
Opened by Paul Mattal (paul) - Monday, 07 March 2005, 04:29 GMT
Last edited by Callan Barrett (wizzomafizzo) - Thursday, 27 March 2008, 08:47 GMT
Task Type Feature Request
Category Backend
Status Closed
Assigned To Paul Mattal (paul)
Architecture All
Severity Medium
Priority Normal
Reported Version
Due in Version 1.5.0
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

We need to import and periodically sync the main current/extra database package info into AUR for dependencies and quick searching for existing packages.
This task depends upon

Closed by  Callan Barrett (wizzomafizzo)
Thursday, 27 March 2008, 08:47 GMT
Reason for closing:  Won't fix
Additional comments about closing:  AUR is separate form the main site, no longer has the broken option to search the official repos.
Comment by Simo Leone (neotuli) - Friday, 15 April 2005, 02:32 GMT
I have no clue (tm) how the main Arch site works as far as those package lists, but if it's mysql based (which I bet it is), why don't we give whatever sql user the AUR uses read-only access to that database (they run on the same server don't they?). We could then put some new code into the AUR that looks up packages on the other database as well.

I think this approach might be more effective than periodic syncs.
Comment by Paul Mattal (paul) - Friday, 15 April 2005, 15:21 GMT
This sounds like a reasonable idea, but it would depend a lot, I'd think, on the structure of the other database. It might be less prone to bugs if we kept them separate -- for instance, when doing an insert of a new package into AUR, you have to be very careful not to insert one with the same name as one in current/extra. Right now, the table constraints guarantee that even if our code has a bug.. this would mean sacrificing some of that db protection.

But I'm still open to considering it. Let me dump the main DB for you and you can have a look at it.
Comment by Simo Leone (neotuli) - Sunday, 08 May 2005, 02:56 GMT
Ok I ran across the schema for the main page's sql db in the arch cvs, so I have have enough information to proceed.

It seems to have all the fields we might need, and they correspond pretty well with the AUR ones. The most notable difference is that it's got pkgver and pkgrel, while we only have Version, but that's not difficult to work around.

As a first step, we could set up something simple and non-cached where dependency links link to official repo pages if they exist (checked by name) in official repos. This is easy enough to do, just grant our sql user SELECT privilege on the packages table in database archweb, and add a couple lines of code to check if a dep exists in the official db, if it does, generate the link appropriatly.

Later on, we could add checking during submit and update that does a similar existence check, and blocks uploads if they are already in official repos.
Comment by Simo Leone (neotuli) - Sunday, 08 May 2005, 02:58 GMT
Errr... a cached solution would be easy as well, python script that runs once a day or something and syncs them into our db, translating fields accordingly (and resolving conflicts), and we could add a new location to PackageLocations , then do checking based on this.
Comment by Simo Leone (neotuli) - Wednesday, 28 December 2005, 04:13 GMT
Progress?

Loading...