FS#57137 - {archweb} Provide a JSON interface to query for multiple package versions

Attached to Project: Arch Linux
Opened by Felipe Contreras (felipec) - Thursday, 18 January 2018, 02:26 GMT
Last edited by Jelle van der Waa (jelly) - Monday, 24 January 2022, 08:11 GMT
Task Type Feature Request
Category Web Sites
Status Closed
Assigned To Angel Velasquez (angvp)
Jelle van der Waa (jelly)
Architecture All
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

It's very easy and efficient to query for the latest version of a bunch of packages with the AUR RPC interface:

https://aur.archlinux.org/rpc/?v=5&type=info&arg[]=spotify&arg[]=pacaur

It would be extremely convenient to provide a similar interface for official packages.

Currently if I want to check the latest version of 100 packages I'd have to do 100 requests.
This task depends upon

Closed by  Jelle van der Waa (jelly)
Monday, 24 January 2022, 08:11 GMT
Reason for closing:  Upstream
Additional comments about closing:  Please include your request in https://github.com/archlinux/archweb/iss ues/199
Comment by Eli Schwartz (eschwartz) - Thursday, 18 January 2018, 02:43 GMT
I don't recall us having a json search interface for official packages at all. Although yes, the package details page can be relayed as json, but that is per-package.
Is this a generic request for any sort of json search interface at all?

Either way, I'm not sure what the utility of this is. The repos will be present on your computer anyway, so you can just use expac to query the databases directly.
Comment by Felipe Contreras (felipec) - Friday, 19 January 2018, 00:02 GMT
Yes, currently it's per-packages, that's why I'm saying it's one HTTP request per package.

This is a specific request, although it would be nice to have a generic interface I guess, for other people.

The repos present on my system have the information since my last sync, not the up-to-date information.

The point is to check efficiently if there are updates available. Currently there's a script called `checkupdates` but that also fetches all the content of all the repositories.

I have a script that does exactly that for AUR: https://gist.github.com/felipec/94752ddd08e1adfb80ac57947982443c

I want to do the same for official packages.

Currently my AUR script checks 10 packages in 1.033 seconds, and 1 package in 1.025 seconds, so it's safe to say that the bottleneck is not the amount of packages.

My WIP script for official packages takes 3.527 seconds for 10 packages, and 1 package in 1.640 seconds, so the bottleneck is the amount of packages, because of the network.

I have not dared to check all the packages, but it probably would take minutes, and it's not a good use of the network. I believe a simple query to retrieve all the last versions of a list of packages wouldn't consume many resources, certainly less than fetching the whole databases of all the repositories, which takes 5.125 seconds on my machine.
Comment by loqs (loqs) - Friday, 19 January 2018, 00:48 GMT
When checkupdates downloads the databases it is fetching static files from a mirror then performing the search locally
I am not sure it is fair to compare the resources that uses to a server provide by arch performing the search and generating the result.
Also the base group has 50 members plus 83 required dependencies so the minimum expect query size would seem to be 100+ packages
larger installations can easily exceed 1000 packages so concluding there would be no bottleneck from a test of 10 packages seems optimistic.
Comment by Jelle van der Waa (jelly) - Friday, 19 January 2018, 22:01 GMT
There is a search JSON endpoint for Archweb https://www.archlinux.org/packages/search/json/?q=linux but that doesn't query multiple packages. Since it uses the same logic as the normal search functionality.
Comment by Felipe Contreras (felipec) - Saturday, 29 February 2020, 02:15 GMT
Is there any way I work on the code myself and test on my local machine to see if it's feasible to do?

Loading...