swh-cc: Create code to efficiently run scans

[pombredanne]

Create new ScanCode toolkit run mode for efficient running at scale.

• It could be client/server mode runs as a daemon and exposes an API to receive per file content to scan.

• It could also ensuring we can run efficiently to amortize startup costs, and/or reuse cached scan results across runs.

[JonoYang]

@pombredanne

It could also ensuring we can run efficiently to amortize startup costs, and/or reuse cached scan results across runs.

In #5289, we have an experimental cli option --use-cached-results that now stores scanner results for Resources such that they can be used in future scans. In the scan_resource function, we check to see if we have a cache file for the results of a scanner on a particular Resource. If we do, we return those results, if not, we run the scanner on that Resource and update the cache. The cache is organized using a hash key based on the content and filename of the Resource.

It could be client/server mode runs as a daemon and exposes an API to receive per file content to scan.

Since we have this results cache, it would be possible to share this amongst other scancode instances, where a scancode worker can share its results with another worker. This could be shared via a server with a rest api, a nfs store where there are a cluster of machines running scancode on the same network, a federated scancode cache. Was there a particular implementation you had in mind?