Last transcript - Calif. ITA hearings. (Plus link to the FULL transcript!)

[Steve A Play]

------------------------------------------------------------
Black Box Voting : Latest Consumer Reports from Black Box Voting: 4-13-06: Last transcript - Calif. ITA hearings.
------------------------------------------------------------

Posted by Bev Harris on Thursday, April 13, 2006 - 12:45 pm:

(A link to a more formal transcript for the entire hearing is included
at the bottom of this portion of the transcript.)

Systest (Brian Phillips): Well if the testing organization should have
found it yes, we're responsible, we should have found that. I mean,
that's the case in all of the work that my company does, whether it's
IV&V work or commercial testing or this.

What is the impact if that happens? If this were the commercial world,
we would be working with the customer for them to fix it, because it's
still a defect they have to fix. And we would then go into regression
test mode and then maybe reissue new reports to them and various other
things. In this case here I would have expected the product to be --
that particular memory card device and software to be brought back to,
let's say if it were Systest Labs doing it, we would get it back, we'd
retest it, and then amend our report to NASED. Then NASED would make a
determination based on that. That's what I would expect to have
happened.

They didn't do that here and I don't have any insights as to why they
felt they should take this course of action. The one that I suggested
is the way that I would expect it.

Senator Debra Bowen: That's what I would have expected too, but that's
not what happened. It never came back, and so the result of it is
ultimately that you've got a testing lab -- or more than one testing
lab has got their name on this thing, and with a qualification
recommendation. And with the memory card in particular, I don't know
any way to fix it, because if you take the interpreted code off the
memory card it's a different system, it won't work. It's an essential
part of the system. You can't record votes without that code on that
memory card. So you no longer have that system.

Systest (Brian Phillips): I do know that we've done source code review
for a vendor and I won't name who it is, but we've done source code
review for a vendor, and they had thousands and thousands of lines of
code that all basically didn't meet a very important requirement of
the VSS. And in the end they rewrote every line of code. They had to.
They knew they were not going to get qualified. And by the way, I'm
not sure if you're aware of this. Once a vendor starts a qualification
process with an ITA, they cannot switch in the middle.

Senator Debra Bowen: I didn't know that.

Systest (Brian Phillips): You cannot go shopping, say "Well, Systest
Labs says that our stuff is no good so now we're going to go to Ciber
and maybe they'll pass us." And NASED put that on there specifically
for that purpose.

Senator Debra Bowen: That's exactly what I was getting at earlier when
I asked about the professor who gave the easy Bs. How do you keep that
from happening.

Systest (Brian Phillips): Oh yeah, they shop around and they find out
you're the hardest professor of all, then you're stuck with that
professor, you can't drop the course. Now they can stop and then come
back with a so-called newer version, different version. There are some
things about that. But we talk amongst each other. We know what's
going on for the most part.

Senator Debra Bowen: The relationships here are complex. And part of
the problem we get into is you've got the ITAs, you have NASED, you
have the state officials, you have the EAC, now you're going to have
NIST, it's a real accountability question.

What happens if you discover – first, I don't know why the memory card
didn't get tested. I don't know whether it's because somebody said
"Well, it's commercial off the shelf software," or if you get specs as
to what you're going to test, that's not one of the modules you think
you need to test. Later on it's determined that it should have been
tested. You look at it, it's got interpreted code.

Now we have a situation where, that's not your responsibility but we
have a situation where the certification of this equipment that has
interpreted code, that doesn't comply with the 2002 standards, is
about to be purchased. We're going to have taxpayers spending a whole
bunch of money on systems that don't meet the 2002 requirement, and I
believe that that's carried over into the 2005 requirement. So the
situation's not going to get better in 2007 when we go to NIST and we
have all of the current requirements plus the new mostly ADA-related
requirements. And the taxpayers have bought it.

Wyle (Joe Hazeltine): Well is this hearing specifically about one card
or is it about a system?

Senator Debra Bowen: Well, the system doesn't work without the card.

Wyle (Joe Hazeltine): No, that system DOES work without the card.

There is an issue on this card which has been addressed four or five
times now in our discussion here. What happened, and then the feedback
loop to correct the situation, or at least to have a group of experts
come up with a determination of is it acceptable or not, and that's
done.

Senator Debra Bowen: Well, how would the system work without the card?
If it works without the card I'd love to--

Wyle (Joe Hazeltine): --I'm saying the issue of the card has been
addressed four or five times in our talk here.

Senator Debra Bowen: But there's more--

Wyle (Joe Hazeltine): --Is this whole discussion about the memory
card, or is it about our process?

Senator Debra Bowen: Ah, no, I think—

Wyle (Joe Hazeltine): --The process -- certainly something happened in
our process that people had to interpret. They didn't expect it to
occur. But it was identified, it was corrected, it's been addressed.

Senator Debra Bowen: How can you say it was corrected if the result is
that equipment that doesn't meet the standard is now going to be--

Wyle (Joe Hazeltine): --A decision has been made corrective action,
put it that way.

Senator Debra Bowen: But this, you said, is not a system that you
would have passed in the first instance if you'd gotten it that way--

Wyle (Jim Neu): --Wouldn't have passed it because there is a
prohibition against interpreted code. And IF we had reviewed it and IF
we had found interpreted code we would not have passed it.

Wyle (Joe Hazeltine): It was not in our scope.

Senator Debra Bowen: So why are you defending it?

Wyle (Joe Hazeltine): No I'm saying that you've got a card, and you've
got an issue. We've talked about that. Is that the intent of the
hearing that card, or is the intent of the hearing the process around
the entire certification of all voting systems, not just this one
card.

Senator Debra Bowen: Well you've been here for the whole system, I
think it's been about a lot more things than the card. But the card is
a pretty big indicator of where the breakdowns in the certification
system and the testing can occur. And I use it, and I probably would
if we were doing this two or three weeks, be referencing what happened
in Chicago and Texas. It just hasn't, we haven't had as rigorous a
failure analysis there so I'm as familiar with exactly what happened,
aside from 4,000 people in Chicago got assigned to use equipment that
we'd never physically laid eyes on and that's a problem. But we use,
when we're trying to evaluate policy, we try to use concrete examples
to learn more generally about how something works, so failure analysis
is an important part of what we do.

Wyle (Jim Neu): Unfortunately I don't think that we have all of the
data here. Again, what the information appears to indicate is it was
characterized as COTS, that it was part of the review done by the
software ITA, and that it was not reviewed, apparently because of
that. But I've just said "apparently" about three times. And without
knowing that I don't think we can get any further into it.

Systest (Brian Phillips): But let me, if I may, I think, there's a
process for doing the testing here and human error's going to occur.
We've got that. So something occurred here, it did get tested. It did
get found. The decision is not up to the ITA in whether we can pull it
back or anything else.

So what I encourage you is to bring this up to NASED and the EAC. "Why
would you go and violate your own standards and allow this product to
continue to be qualified. Why don't you revoke qualification on this
particular product and either have it corrected or redesigned or
whatever, but you know, whatever, something done."

Now, had we found it or had Wyle found it as they said, it never would
have even reached this point, because it wouldn't have gotten to
NASED, as we said before, we would make them change it.

Senator Debra Bowen: And the point of this is just, yes, we do need to
go to NASED and to the EAC. But it is important to understand how the
ITAs deal with a problem that's discovered later.

Systest (Brian Phillips): But no, we do exactly what I think you're
wanting. If we discover it – by the way there's -- after we've issued
reports, we've done additional analysis, and we've found some other
things. We will amend our reports to NASED or we'll do some additional
testing, all during the process of reviewing the reports and so forth.
We're very cognizant that we have to have the best test results that
have to go back there and show that we've done everything that we're
supposed to do and that the results are indicative of how well the
system operates.

If NASED or someone else comes back with a problem against one of the
systems we've tested, yes we want to go to every extent to correct
whatever happened. I mean we can't correct the system, but to retest
it, to force the vendor to make the changes, to make it correct. But
we can't go back to NASED and say "you've got to remove qualification
for that system."

Senator Debra Bowen: I understand that. My question just is, is there
a process for amending your report?

Systest (Brian Phillips): Oh absolutely. Yes, there are. And we've
done that. We've had to amend reports, most of the time, the only time
I remember is when there's been an enhancement requested by a
particular state during that process.

Senator Debra Bowen: But your report -- if you learn of something that
didn't come from the vendor you won't amend that report, that's a
vendor-driven process?

Systest (Brian Phillips): We've not had a situation, I'll be very
honest, that I know of, where we've learned of a failure in the field
of a system that we've qualified, than other from the vendor. But I
believe that if we had learned of a failure and we had access to
confirm that, then we would amend our report.

Wyle (Jim Neu): As I said before, we would amend the report if we for
some reason reviewed the raw data and found that our report was wrong.
But our report is essentially a compilation of the raw data. And so if
in fact if something outside of our raw data was subsequently found to
be a problem and was known or became known, I think we would believe
that that would become known by NASED and that they would take the
appropriate steps.

Senator Debra Bowen: And again, my discussion about the memory card
and the testing and how this gets reported really comes from an early
concern that I had in looking at these things the hardware and
firmware was split from the evaluation of the operating system and
what the result of that can be. That you get something that, you know,
your report's finished then something else comes up and it goes
somewhere else. And then there's nobody who's got the ultimate bottom
line.

Systest (Brian Phillips): Well, I did mention this before, but going
forward, is something I just reminded myself, going forward, and
gentlemen correct me if I'm wrong on this as well, but in the VSTL
there won't be a separation. The VSTL is responsible for qualification
testing of the product. There will be hardware testing companies that
will be able to test environmental hardware, but all the software,
whether it's firmware on the polling place device, election management
systems, central count and so forth, that's all going to be software.
At least that's the latest I have heard from the EAC. Which is
something that we've been pushing for from the very beginning.

When we became a software ITA we assumed if it was all software. And
that any firmware, any memory chip firmware that was burned on, yes
that's still part of the hardware. But if it was an operating system
running on a hard drive, software on a polling place device, that fell
under our jurisdiction. It wasn't until after we got into it was it
clarified. It's my understanding that that's supposed to go away.
Software does all software and hardware will do all hardware.

Senator Debra Bowen: All right, well that will help. But even within
that, and this is a question about variations on a system, does each
version of the same voting system have a separate test?

Systest (Brian Phillips): Yes. I mean if you have election management
system 1.0 and then election management system 1.1, they will each
have the same test. Now, having said that, we may not rerun certain
tests if that part of the software or the hardware has not changed.
But from 1.1 to 1.2 for instance, if certain parts of the software
have changed, we may run any or all of the tests, depending on our
engineering evaluation of the changes.

Senator Debra Bowen: So if you've got a system that's certified and
the manufacturer shifts to a Crusoe chip from an Intel chip, is that a
different system that requires a different test?

Wyle (Joe Hazeltine): Yeah, it would be a hardware change, yes.

Systest (Brian Phillips): Yes.

Senator Debra Bowen: Okay. Would the same be true if a different
motherboard is used on the system?

Systest (Brian Phillips): Absolutely.

Wyle: (Joe Hazeltine): Yeah.

Systest (Brian Phillips): Absolutely.

Senator Debra Bowen: So do you routinely get from vendors multiple
versions if they're going to buy from more than one hardware
manufacturer, they'll submit to you--

Systest (Brian Phillips): Now there are times when a vendor will
submit what's called an equipment change request – I believe you all
have that equipment change order request, where there's a process –

Wyle: Mm-hm.

Systest (Brian Phillips): Where they're changing a component within a
piece of hardware, for instance, but there's no form, fit or function
change, it's just that the manufacturer – a capacitor, for instance,
they used to use ECO5 capacitor, now they're going to go to ECO6, no
form fit or function change. We may decide not to run any tests on
that because there's really no impact to the system. So those types of
things will happen. We could engineer equipment change requests rather
frequently, but it has to pass that test of no form, fit or function
change.

Senator Debra Bowen: Is that an industry standard test so that it's
the same, whether you're dealing with satellites or whatever?

Systest (Brian Phillips): Yes, even FDA medical equipment.

But if any part of the software changes between version 1.1 and 1.2,
we look at exactly what that change is, we do source code differences.
We look at exactly the changes and the impact on that code. Even if
it's a few lines change it could have an enormous impact on the
software. We may require a complete retest. Or, it may have zero
impact on the software other than maybe correcting a few things,
clarifying some stuff. So we do have to look at that and we do have to
look at that and we do, we do an extensive comparison version between
version 1.1 and 1.2 to determine exactly what has been changed and to
what extent we need to test.

Senator Debra Bowen: So what happens, and I'll give you two scenarios
and you can tell me what would happen on the lab's end of it. One is,
you've certified software that's version 1.1.1.1 and you subsequently
are informed not by the vendor but, you know, the secretary of state
in Pennsylvania says "Gee we just discovered that this vendor's
running 1.1.1.2" -- what if anything does the lab do, and the second
is, the system is certified it's got motherboard A on it and you
subsequently learn that it was shipped and billed as being the
certified system, with Motherboard B in it, but nobody ever said
anything to you, you didn't go through the – what was the terminology
– form function and use?

Systest (Brian Phillips): Form fit function

Senator Debra Bowen: Form fit function test. What happens at that
point? Anything?

Wyle (Joe Hazeltine): If we were to find that they shipped either
software or hardware that was different that what we certified, the
certification's null and void. But we have no way that we would find
that out unless somebody else notified us. Because we're not going to
see other than the system that went through the first test.

Wyle (Jim Neu): It would probably be the customer who would find that
out.

Wyle (Joe Hazeltine): Right. We wouldn't find it out. But if they've
intentionally changed out things which would affect that, then the
report that we have is no longer valid. It wouldn't be on the system
that they've actually shipped. And that's common for, you know, safety
work. Oftentimes a customer will come to us and say, "I'm going to
have to use two or three different microprocessors because of vendors"
and we'll approve all three of them at the same time. So oftentimes
the bill of materials listing will more than one component on it. And
in a case like that everything's okay.

Senator Debra Bowen: But the only way that you have to find that out
is if the vendor tells you?

Wyle (Jim Neu): Or if the customer finds out.

What we certify is to a specific configuration – and, I'm sorry, we
don't certify, what we report on is to the configuration that we
tested. And we specify in the report the configuration that we tested.
Beyond that NASED would not issue a certification to a configuration
different from what we provided in the test results on.

Systest (Brian Phillips): For instance, if the customer found the
hardware device had Motherboard A when it was qualified with
Motherboard B. The device they currently have is not a qualified
system. Only the motherboard, in this case I said Motherboard A
was the qualified one, only the system with Motherboard A is a
qualified system. What they have is NOT a qualified system. Same thing
with software. If they have a software version 2.3.1 but only 2.3.0
was qualified, they don't have qualified software it's not a question
of whether we've done anything, it's not qualified.

Wyle (Jim Neu): We've simply submitted a report and again if the
vendor chooses to represent that as being something that is certified
by NASED and it's not, then the customer has a problem with the
vendor. But what we have done is specified, clearly stated in the
report what we tested, provided to NASED.

Systest (Brian Phillips): And that includes the hardware
configurations of the components as well as the software. And that's
something that NIST is trying to implement, where counties can go and
do a hash code check on software, for instance, and find out "Do I
have qualified software? Oh, I found that my hash code is different.
My software's not qualified, where did it get changed." Now they no
longer trust their relationship with the vendor because somebody's
implemented a patch without their knowing. But in that case they don't
have a qualified product.

Senator Debra Bowen: Okay. So there's no further action for the ITAs
to take at that point because you've already done your work and you
passed on motherboard A or software version whatever it is, and
anything else is just plain old, you know, not the subject of your
report so, and presumably not the--

Wyle (Jim Neu): --I would assume at this point since this new piece of
equipment is not certified by NASED, that the vendor would contact one
of the ITAs and endeavor to get that piece --

Senator Debra Bowen: Well that's not what's actually happened. In this
instance that I'm talking about, the vendor has just shipped with
different hardware and, you know, again--

Systest (Brian Phillips): We would have absolutely no way of knowing
that. We've got no control over what the vendor does.

Wyle (Jim Neu): That would be for us to take a police function, which
we clearly can't do.

Senator Debra Bowen: I'm not advocating it, I'm just trying to
determine what your view of your report is if somebody shipped
something other that's got something in it other than what you tested.

Systest (Brian Phillips): If they do than that's completely not a
qualified system.

Senator Debra Bowen: But the difficulty for me is trying to reconcile
that, which makes sense – you didn't qualify it -- but if something
should have been tested, but didn’t get tested and there the report
stands, even if there was a problem.

Wyle (Jim Neu): Well if something should have been tested and wasn't
tested then it's an error. And all of us strive to reduce errors. And
we've gone over what happened in that particular case.

Senator Debra Bowen: I think what's important here is, first, that we
strive to eliminate errors, but also that there's a clear path for
what happens next. Because we're going to have errors. Every human
endeavor. And the question is at first, how do you reduce error, how
do you test, how do you audit, and that's a subject that we've dealt
with here too. Because to some extent you can catch in an audit things
that you won't catch in testing. In all human endeavors. And then the
question is when you do find a mistake how do you handle it, what do
you do about it, who has the ultimate responsibility, how you figure
it out. It's actually why we went with IV&V vendors in our big
software projects because we didn't have anybody who had the clear
responsibility to pull the plug on something that was just not working
and not likely to, and so we basically said we're going to have a
parallel--

Systest (Brian Phillips): But in a situation with an IV&V vendor the
project is completed, they've implemented your new tax system, and
then you start to find problems in the field after it's done. The same
thing applies. Your IV&V vendor -- or are you saying that the IV&V
vendor would come back and say "Now we're going to reject the product
that's in the field." They really don't have anything that they can do
at that point. And that's kind of what you're saying about the ITAs.
Once it's gone past our process, if we've made a mistake, we will try
to correct that and do everything we can. But in our case, NASED has
made a decision they're going to let it stay qualified. In the IV&V
case the state has accepted the product and now they have to deal with
the vendor getting its problems fixed and production fixes and so
forth. So the process has to, who has the ultimate authority on
whether a product stays qualified or isn't qualified. And as far as
qualification is concerned, that now rests with the EAC. Then it's up
to the state to determine whether or not they want to accept that
system in their certification process. I mean if I were the state of
California, and this is a significant issue in the state of
California, they may be qualified, so what? You don't pass state
certification. That's kind of the process on down the ladder, the way
I looked at it.

Senator Debra Bowen: What we're dealing with here is a system that
shouldn't have been qualified and passed, both and got a NASED number
and then passed state certification. So it just has a lot of people
out there in the public very concerned.

And in part it's an issue because they don't get to look under the
hood to see what happened during the process. They just know the bits
and pieces of what they've read or something that has been discovered
by somebody else's efforts. I think to some extent we've created this
problem of our own making by having all of this stuff protected.
That's not your doing. But we have a system in which it becomes very
difficult for anybody to look at what's actually happened. And you can
understand the kind of concerns that people have. Particularly about
voting.

If it were about cell phone and it was one cell phone customer's
conversation might be interrupted -- but I think the other thing
that's fair to say is that, just had a very good columnist in the LA
Times just a few weeks ago, write that all of us when it comes to our
computers are security slobs. We write down our password on our desk,
we program stuff in to places where we shouldn't, and so we really
look for engineering to help us, to save us from our own failings when
it comes to the fact that, you know, every Web site you go to you're
supposed to have a different name and a different password, and
sometimes you can use an asterisk and some sites you can't, and some
it's case sensitive and some it's not. We've made things so
complicated that it's difficult. It's just difficult for everybody to
figure out what's happening.

I don't have any further questions. I do want to thank you and I think
we've learned a lot about how the testing system works, what happens
when there are issues, what gets passed, what role various parties
have in this, and I think it will be grounds for a further discussion
about how California wants to handle some of this. So it will be a
useful discussion and I want to thank you again, very much, for
participating. I learned a lot.

Our hearing is adjourned.

FULL TRANSCRIPT:
http://www.bbvforums.org/forums/messages/2197/27280.html

PERMISSION GRANTED TO EXCERPT OR REPRINT, WITH LINK TO
http://www.blackboxvoting.org



------------------------------------------------------------


Use this link to go directly to full article:
http://www.bbvforums.org/cgi-bin/forums/show.cgi?1954/27282

[nicknameless]

1. Kick for Sen. Bowen

:patriot: