blob: 94dbc07d95546969f7becea7b6c8b8b062fefd1f (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
|
<html>
<head>
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta content="text/html; charset=ISO-8859-2" http-equiv="content-type">
<title>µCsim: Regression Testing</title>
</head>
<body bgcolor="white">
<h1>Regression Testing</h1>
<h2>Running Tests</h2>
<p>If you are making changes to µCsim you should both run tests
to ensure nothing breaks and write new tests to ensure your changes
work and do not get broken in the future!</p>
<p>In all cases the tests are run using the locally built simulators.
i.e. you need to have completed a successful build before you can
test it! Tests produce output when run which is captured to a file
and compared with baseline output held in the source tree. If there
are any differences they are output and the test is considered
failed. In that case either the code needs to be changed to correct
the output or the baseline needs to be updated to reflect the new
expected output. Only when there are no differences to the baseline
can tests be said to have passed.</p>
<h3>All Tests</h3>
<p>To run all tests run <code>make test</code> in the top-level
<code>ucsim</code> directory.<br />
E.g.:
<pre>.../ucsim$ <font color="#118811">make test</font></pre>
</p>
<h3>For a Single MCU</h3>
<p>To run tests for a single MCU type run <code>make test</code>
in the directory for that simulator.<br />
E.g.:
<pre>.../ucsim/s51.src$ <font color="#118811">make test</font></pre>
or run <code>make</code> in the <code>test</code> subdirectory.<br />
E.g.:
<pre>.../ucsim/s51.src/test$ <font color="#118811">make</font></pre>
</p>
<p>Note that tests for functionality generic to all simulators and that
does not depend on a particular MCU type lives in
<code>sim.src/test</code> and should be run there. These tests require
at least one simulator to have been built but unless there is only one
it is indeterminate which simulator will be used.</p>
<h3>Running a Single Test</h3>
<p>To run a single test run <code>make <i>test_name</i></code>
in the <code>test</code> subdirectory of the relevant simulator. E.g.:
<pre>.../ucsim/s51.src/test$ <font color="#118811">make config</font></pre>
In this case the test is run verbosely and the commands executed by make
are displayed. In all other cases only the final differences to baseline
(if any) are displayed.</p>
<h3>Updating the Baselines</h3>
<p>If changes to the simulator code base lead to changes in test outputs
<em>and these changes are as expected</em> then the baselines should be
updated to match the new output by copying the contents of the <code>out</code>
directories to the corresponding <code>baseline</code> directories.</p>
<p>To update all baselines run <code>make baseline</code> in the top-level
<code>ucsim</code> directory.<br />
<p>To update baselines for a single MCU type run <code>make baseline</code>
in the directory for that simulator.<br />
<p>If you need to update the baseline for a single test you should copy the
relevant files manually.</p>
<p><em>Remember that the updated baselines form part of your changes and
need to be committed or submitted as part of your patch!</em></p>
<h2>Writing Tests</h2>
<ol>
<li>Create a directory named after your test under the relevant simulator's
<code>test</code> directory:
</li>
<li>Place the data needed for your test, <code>.cmd</code> files, <code>.asm</code>
or <code>C</code> files etc, in your test directory:
</li>
<li>Create a <code>baseline</code> directory in your test directory:
</li>
<li>Create an empty file in the <code>baseline</code> directory for each
output that your test generates:
</li>
<li>Create a <code>Makefile</code> that lists the output(s) to be generated,
the recipes to create them and which includes <code>test-lib.mk</code>:
</li>
<li>Run your test, verify the contents of the <code>out</code> directory
and then copy it to the <code>baseline</code> directory:
</li>
</ol>
<pre>
.../ucsim/s51.src/test$ <font color="#118811">mkdir <i>name</i></font>
.../ucsim/s51.src/test$ <font color="#118811">cd <i>name</i></font>
.../ucsim/s51.src/test/<i>name</i>$ <font color="#118811">vim test.{cmd,asm}</font>
.../ucsim/s51.src/test/<i>name</i>$ <font color="#118811">mkdir baseline</font>
.../ucsim/s51.src/test/<i>name</i>$ <font color="#118811">touch baseline/<i>stdout</i></font>
.../ucsim/s51.src/test/<i>name</i>$ <font color="#118811">cat > Makefile <<EOF</font>
OUTPUTS = <i>stdout</i>
<i>stdout</i>: <i>mytest</i>.ihx <i>mytest</i>.cmd
$(call run-sim)
include test-lib.mk
EOF
.../ucsim/s51.src/test/<i>name</i>$ <font color="#118811">cd ..</font>
.../ucsim/s51.src/test$ <font color="#118811">make <i>name</i></font>
.../ucsim/s51.src/test$ <font color="#118811">cp -r <i>name</i>/out <i>name</i>/baseline/.</font></pre>
<p><code>test-lib.mk</code> takes care of building any <code>.ihx</code> files
needed from corresponding <code>.asm</code> or <code>.c</code> files.
The call to <code>run-sim</code> will invoke the simulator with the
<code>.ihx</code> and <code>.cmd</code> prerequisites, capturing the
output.</p>
<p>If there are no <code>.cmd</code> prerequisites and there is no
<code>-e</code> in the arguments to <code>run-sim</code> it
will add a <code>-g</code> argument to start the simulator running.
It is assumed there is at least one <code>.ihx</code> file in the
prerequisites in this case and that it places code at the reset
address.</p>
<p>If you wish to pass arguments of your own to the simulator these
can be given as arguments to run-sim. This can be used to pass commands
with the <code>-e</code> option rather than using a <code>.cmd</code> file
for instance, as is done with the config tests.</p>
<p>If you create any other output from within the simulator you should
specify a pathname starting with <code>out/</code>. Anything written
to the <code>out</code> directory will be compared to the baseline
when the test is complete.</p>
</p>
<hr>
</body>
</html>
|
