[Comment 3 of 4] Overview of Live Auditing Procedures for Incorporation in the Voluntary Voting System Guidelines The proposed Voluntary Voting System Guidelines are missing guidelines for live auditing, and should include them. "Live auditing" is the process of independent testing of voting system functional units during an election, via methods that do not modify, create, or destroy votes on live ballots. Live auditing is similar to the 'parallel testing' process described in [1] and [2]; however, modern auditable voting system designs enable live auditing to take place without having to take voting machines out of service or to otherwise affect the voting process. Existing voting system test approaches are inadequate to verify voting system reliability. Currently, election officials perform "logic and accuracy tests" before, and in some cases after, the election. However, even if these tests are successful, they disclose nothing about the functionality of the system _during_ the election. Logic and accuracy tests also generally do not use live ballots, and are often run in a 'test' mode on the voting systems. Some examples of live audit applications: 1. Live auditing can be used to verify the accuracy of the 'ballot scanning and interpretation' component of some optical scan systems and VVPAT scanners. With modern auditable voting systems, very high accuracy tests can be implemented with a minimum of additional work. One such technique is described in [4]. 2. Live auditing can also be used to verify the accuracy of the vote tabulation component, when it is decoupled from the rest of the voting process (as it is in several current voting systems). One such technique is described in [5]. The components are tested as "black boxes." Inputs and outputs are defined, and an independent process must be able to verify to a high degree of confidence that the black boxes are producing valid outputs given particular inputs. In our first example, the inputs to the process are paper ballots or audit trail records, and the outputs are a set of cast vote records. Similarly, in the second example, the input is a set of cast vote records, and the output is the vote total across those records. One important prerequisite for live auditing is that the voting system components must themselves be auditable to independent observers. DREs without VVPATs generally do not possess this property, as there is no independent, non-computer-corruptible record of voter intent that can be compared to the machine-stored record. The Voluntary Voting System Guidelines should discourage the development and use of voting systems which cannot be live audited. Similarly, some voting systems are difficult to live audit efficiently, even though they may technically be live auditable. One example would be a paper-based optical scan system without unique identifiers. Live auditing the ballot scanning and interpretation component of such a system to a reasonable degree of confidence would involve the independent interpretation of a large number of ballots, and therefore make live auditing practically infeasible for this system. The Voluntary Voting System Guidelines should discourage the use of voting systems with components that are practically infeasible to live audit. Another requirement for live auditing is that the systems under audit must not have any means of detecting that they are under audit. The system must be auditable without putting it into any "test modes" or other modes of operation that are not used during a live election. Live auditing is not a new concept for election administrators. Many jurisdictions use teams of election judges, composed of citizens with different party affiliations, at various points in the voting process, including ballot box unsealing and questionable vote interpretation. In these teams, each judge is live auditing the decisions of the other judges. This basic philosophy - that decisions made by one observer must be verified by at least one other independent observer - should also be used to test the voting system equipment used during an election. The Voluntary Voting System Guidelines should promote the incorporation of peer-reviewed live audit techniques in voting system design and election official best practices. [1] Jones, Douglas W. _Testing Voting Systems._ [2] Jones, Douglas W. _Parallel Testing: a menu of options._ [3] Jones, Douglas W. _Auditing Elections._ Communications of the ACM, 47, 10 (October 2004) 46-50. [4] Walmsley, Paul. _Requirements for Statistical Live Auditing of Optical-Scan and VVPAT Records._ September 30, 2005. [5] Walmsley, Paul. _Requirements for Live Auditing for Vote Tabulation._ September 30, 2005.