When a software or an application is created, it is vital to make several types of tests, to make sure the product is complete, secure and efficient. To make these tests, several methods are available: “black box”, “white box” and “grey box”. Each of these methods offers different possibilities, which we will explain in this article.
“Black box” tests consist in reviewing only the functionnalities of an application, ie if it does what it is supposed to, no matter how it does it. Its internal structure and functionning are not studied. The tester thus needs to know what the role of the system is, and its functionnalities, but does not know its internal mechanisms. He has a “user” profile.
So this method aims at checking, after the final stages of a project, if a software or an application works well, and serves its users efficiently. Usually, testers are looking for missing or incorrect functions; interface, performance, program initialization and exit errors; data structure or external database access errors.
For this, they prepare scenarii based on the different user paths that can be followed on the tested system. All functionnalities have to be taken into account, for all of them to have been tested at the end of the operation. Each test follows a scenario, and checks, for every functionnality, if the valid inputs are accepted, the invalid ones refused, and of course, every time, if the output or result expected is indeed obtained. That is the “trial and error” method.
The benefits of this method are as follows:
- Simplicity: these tests are easy to make, because one focuses on the inputs and outputs. The tester does not need to know how the system works internally, or its source code, which is not accessible. Thus, this method is also non intrusive.
- Rapidity: the preparation time of these tests is very short, since little knowledge of the system is needed. Creating and testing the scenarii takes little time, since it follows the user paths, which are relatively few, depending on the size of the system.
- Impartiality: we are here following a “user” point of view rather than a “developper” point of view. The test results are neutral: the system works, or its does not. There is no possible contestation, like the use of one process rather than another one, depending on the developper’s opinion.
But “black box” tests also have drawbacks:
- Superficiality: given the fact that the code is not reviewed, the tests do not allow to see which parts of the code are the cause to a potential problem. Moreover, testers can miss underlying issues or vulnerabilities. Some issues are also hard to spot with this method, like the ones related to cryptography, or caused by bad quality randomness. It is one of the less complete tests.
- Redundancy: if other tests are made, this one can lose its benefits, since its range tends to be included in other tests’ one.
In the case of penetration testing
In the particular case of penetration testing, the principle remains the same: testers know nothing about how the application works, and they do not have access to its source code. They find themselves in the shoes of a normal Internet user: that usually is the same state hackers are in, which makes these tests really realistic. However, some knowledge in programming can be useful or even necessary for the tester (or for the hacker), because it will help him or her identify the relevant tests to be made.
However, “black box” penetration tests can be very long (the testers have to try many different attack methods to make sure none of them are working). Moreover, as mentioned earlier, some parts of the system’s infrastructure won’t be tested, since they have no influence on the user interface on which the test happens.
“White box” tests consist in reviewing the functionning of an application and its internal structure, its processes, rather than its functionnalities. Here, all of the internal components of the software or application are tested through the source code, main work base of the tester.
To make a “white box” test, the tester thus has to have competences in programming, in order to understand the source code he or she studies. He or she must also have a global view of the functionning of the application, of the elements it is made of, and of course, of its source code. Unlike in “black box” testing, the tester has a “developper” profile, not a “user” profile.
By making a “white box” test, the tester can see which code line is called for each functionnality. It allows to test the data flow, and the handling of exceptions and errors. The resource dependancy, as well as the code’s internal logic and righteousness are also studied. That is why these tests are mainly useful during the developpment of an application, even if they can be made during several phases of a project’s life. The “white box” method can be applied for (mainly) unit tests, integration tests, and system tests.
“White box” testing uses test scenarii, created by the tester depending on what he or she found out about the environment’s source code. Testing all these scenarii aims ar checking all the lines of code. What is looked at is the process made by the application after an input, to get a certain result. The tester only checks if the code produces the expected results.
This method has several advantages:
- Anticipation: doing these tests during a program’s development allows to spot blocking points that could turn into errors or issues in the future (for instance during a version upgrade, or even during the integration of the tested component into the main system).
- Optimization: given the fact that he or she works on the code, the tester can also take advantage of his or her access to optimize the code, to bring better performances to the studied system (not to mention security…).
- Exhaustiveness: the tester works on the code, he or she thus has the opportunity to fully review it. That is the kind of test that allows, if it is well done, to test all of the system, without missing anything. It enables to spot bugs and vulnerabilities that might be hidden on purpose (like backdoors for instance). That is why the ARJEL demands “white box” code audits to online games operators, to make sure that no hidden mechanism can penalize the players.
Since one cannot have everything, these tests also have drawbacks.
- Complexity: these tests demand competences in programming, and a strong knowledge of the studied system.
- Duration: the length of the source code can make these tests last a long time.
- Automation: to make “white box” tests, one needs tools such as code analyzers, debuggers… They can have a negative impact on the system’s performances, or even impact the results of the tests.
- Scalability: scaling the project can be very complicated. The source code of a program is often very long, it can thus be hard to find out what is tested, what can be put aside… Indeed, one cannot always test everything, it would be too long. It is also possible for the tester not to realize that a functionnality, planned in the program, was forgotten and not integrated. The tester, thus, does not have to check if everything is there: he or she just tests what is already in the code.
- Intrusion: this method is very intrusive. It can, indeed, be risky to make one’s code available to another person, external to one’s company: there are risks of breakage, thief, or even integration of backdoors… Always choose professional testers!
In the case of penetration testing
If this method is used in the case of a penetration test, it means that testers know how the attacked system works: they have access to the source code, the application’s architecture design… Testers create tests based on this knowledge to check the security of the application.
It especially allows, unlike “black box” testing, to check the quality of the code, and to work on a much larger range. However, this method does not allow to test the application under “real conditions”, since no hacker has access to that much information. It still enables to efficiently secure one’s application, especially against internal threats!
“Grey box” testing compiles the two previous approaches: they test both the functionnalities and functionning of a website. That means that a tester gives an input to the system, checks that if result is what is expected, and checks through which process this result was obtained.
In this type of tests, the tester knows the role of the system and of its functionnalities, and also knows (though not extensively) its internal mechanisms (especially the internal data structure and the algorithms used). However, he or she does not have access to the source code!
These tests can hardly be done during the developping phase of a project, since it implies tests on the programm’s functionnalities: it thus has to be in a state close to the final one for these tests to be relecant. Indeed, during “grey box” testing, testers mainly use “black box” methods, since the source code is not accessible. However, the scenarii are oriented to impact the underlying processes, and thus test them too.
Of course, the “grey box” method mainly combines advantages from the “white box” and “black box” methods. We can still mention two big benefits of this method:
- Impartiality: “grey box” tests keep a demarcation between the developpers and the tester, since the latter does not study the source code and can lean on the results obtained by testing the user interface.
- Intelligence: by knowing the internal structure of the program, a tester can create more varied and smart scenarii, in order to be certain to test all the functionnalities, but also all the corresponding processes of the program.
At the same time, one of the main drawbacks of these tests is the following:
- Non exhaustiveness: given the fact that the source code is not accessible, “grey box” tests cannot hope to provide a complete coverage of the program.
In the case of penetration tests
In the case of penetration tests, the “grey box” method consists in doing the tests while being logged on to the application with a user account. It is an intermediary level, where testers know the internal mechanisms of the application, which guides them in their choice of tests. It allows to create relatively exhaustive tests, while maintaining them in a relatively realistic view, close to what a real hacker could do.
To better understand…
An analogy is often used to differenciate these methods, by comparing the tested system to a car.
- With the “black box” method, one checks that the car works by turning on the lights, honking and turning the key for the motor to switch on. If everything goes as planned, the car works.
- With the “white box” method, one takes the car to a garage. The technician looks at the motor and at all the other parts (both mechanical and electrical) of the car. If they are in a good state, the car works.
- With the “grey box” method, one takes the car to a garage. While someone turns the key, one checks that the motor switches on, and the technician looks at the motor at the same time, to make sure it comes on the way it should.