selinux: improve role transition hashing

The number of buckets is calculated by performing a binary AND against
the mask of the hash table, which is one less than its size (which is a
power of two).  This leads to all top bits being discarded, e.g. with
the Reference Policy on Debian there exists 376 entries, leading to a
size of 512, discarding the top 23 bits.

Use jhash to improve the hash table utilization:

    # current
    roletr:  376 entries and 124/512 buckets used,
             longest chain length 8, sum of chain length^2 1496

    # patch
    roletr:  376 entries and 266/512 buckets used,
             longest chain length 4, sum of chain length^2 646

Signed-off-by: Christian Göttsche <cgzones@googlemail.com>
Reviewed-by: Stephen Smalley <stephen.smalley.work@gmail.com>
[PM: line wrap in the commit description]
Signed-off-by: Paul Moore <paul@paul-moore.com>

Author: cgzones

security/selinux/ss/policydb.c

@@ -491,7 +491,7 @@ static u32 role_trans_hash(const void *k)
 {
 	const struct role_trans_key *key = k;
 
-	return key->role + (key->type << 3) + (key->tclass << 5);
+	return jhash_3words(key->role, key->type, (u32)key->tclass << 16 | key->tclass, 0);
 }
 
 static int role_trans_cmp(const void *k1, const void *k2)