Methodology
What Appraysal Claims
Appraysal separates static compliance checks into assertions we can support from repository evidence and items that still need user confirmation. The product should not claim that source code alone proves store-surface, backend-only, runtime, or subjective reviewer decisions.
1. Deterministic
Manifest, plist, entitlement, binary, and narrow static rules where the required evidence is directly present in the submitted project.
2. Semantic
Code-behavior rules judged from cited evidence. Promoted semantic findings require adjudicated evaluation and abstentions are routed to review.
3. Checklist
Store listing, server-side, runtime, and reviewer-judgment items that users must confirm because the repository is not a complete source of truth.
How Accuracy Is Measured
Deterministic rules are measured against canonical v2 holdout labels across 876 repositories.
The current deterministic holdout summary reports 93.6% precision, 75.9% recall, 83.8% F1, and 97.0% accuracy.
Semantic promotion uses decided-basis evaluation with explicit abstention routing, not forced binary claims.
Current Measured Numbers
| Rule | Precision | Recall | F1 |
|---|---|---|---|
| gp_firebase_app_check | 98.0% | 72.7% | 83.5% |
| 5.6_firebase_app_check | 95.7% | 87.0% | 91.2% |
| 3.2_hidden_code | 95.3% | 89.1% | 92.1% |
| 4.8_siwa_credential_state | 100.0% | 91.2% | 95.4% |
| gp_manage_external_storage | 100.0% | 90.2% | 94.8% |
| gp_query_all_packages | 99.1% | 99.1% | 99.1% |
| 2.3.2_iap_unlabeled_premium semantic | 75.0% | 100.0% | 85.7% |
| 5.1.1_privacy_policy semantic | 95.9% | 100.0% | 97.9% |
| 5.1.1_account_deletion semantic | 97.6% | 100.0% | 98.8% |
Semantic metrics are decided-basis: findings are only asserted when the judge reaches a verdict, and abstentions route to Needs Review. The 2.3.2 evaluation abstained on 25.6% of adjudicated labels; the 5.1.1 privacy-policy evaluation abstained on 12.5% and the 5.1.1 account-deletion evaluation on 13.5%, each across 400 adjudicated repositories.
Label provenance: evaluation labels are adjudicated by a cross-model reviewer (not the judge model) with zip-level evidence verification, and disagreements between the judge and labels are individually re-reviewed against repository evidence before metrics are finalized. This alignment step means the reported numbers measure agreement with an evidence-checked construct, not a fully independent blind test.
What Users Must Confirm
- Store listing URLs, deletion URLs, and metadata not supplied to the scanner.
- Backend-only deletion, billing, moderation, or data-handling behavior.
- Runtime UX such as paywall order, minimum functionality, and subjective reviewer presentation.
- Any Needs Review item caused by semantic abstention, evidence insufficiency, or budget overflow.