PrefVote test suite
PrefVote is designed for multiple programming language implementations using a common test suite. The test suite consists of “black box” test specs used across all languages, plus “white box” unit tests within each language’s source code directory. The test harness collects Test Anything Protocol (TAP) data from each language’s unit tests to report the results.
Scripts in the test directory
- acr-compare: comparison of voting methods with and without average-choice-rank (ACR) tie-breaking
- blackbox-recalibrate: resets blackbox test data after changes are made to the runtime environment
- run-tests: perform whitebox tests (unit tests) and blackbox test suites
- test-overview: generate a Markdown formatted overview of black-box test data in each test file including results of runs in each of the voting methods
Progress on the test suite
Numbers in each cell are test cases planned/passed/failed.
| language/set |
Core |
STV |
Schulze |
RankedPairs |
KR2 |
total |
| Perl whitebox |
525/525/0 |
193/193/0 |
183/183/0 |
139/139/0 |
133/133/0 |
1173/1173/0 |
| Rust whitebox |
𝟬 |
𝟬 |
𝟬 |
𝟬 |
𝟬 |
0/0/0 |
| Perl blackbox |
8322/8322/0 |
9486/9486/0 |
10425/10425/0 |
9093/9093/0 |
8808/8808/0 |
46134/46134/0 |
| Rust blackbox |
𝟬 |
𝟬 |
𝟬 |
𝟬 |
𝟬 |
0/0/0 |
| total |
8847/8847/0 |
9679/9679/0 |
10608/10608/0 |
9232/9232/0 |
8941/8941/0 |
47307/47307/0 |
Black-box test data files
Randomly-generated black-box test data files and their run results for each voting method can be found in these overview files (generated by the test-overview script, which also calls acr-compare and vote-count).
Column headings in result overviews:
- choice: name of voting choice (fictitious candidates used in black-box data)
- avg choice rank: floating point numbers for average ballot position ranking of each choice, followed in parentheses with the integer totals for the total places and total occurrences which were divided to compute it
- Core: PrefVote::Core ranking - these are always in order of i-n since the ACR comparison is ordered relative to Core
- STV: PrefVote::STV result place 1-n with/without (slash between them) using ACR/Core for tie-breaking
- Schulze: PrefVote::Schulze result place 1-n with/without (slash between them) using ACR/Core for tie-breaking
- RankedPairs: PrefVote::RankedPairs result place 1-n with/without (slash between them) using ACR/Core for tie-breaking
- Copeland: Condorcet method doesn’t actually have a score, but says any candidate who wins against all others in pairwise comparisons is the winner. For the table, this is shown using the Copeland score, nth place result followed by the actual Copeland score in parentheses. Copeland score is simply the total pairwise races against other candidates, total wins minus losses. This is just a numeric score. The only PrefVote module which uses it is KR2 (Kluft Rank-Rate) which is being developed in PrefVote. The implemented algorithms all have tie-breaking optimizations which are not part of Condorcet or Copeland, but necessary to disambiguate results.