1 # Anna Agents-Oriented Testing Setup
3 ## Agents configuration
6 $> python3 loader.py -h
7 usage: loader.py [-h] -f FILE
9 Anna Agents-Oriented Testing Setup Loader
12 -h, --help show this help message and exit
13 -f FILE, --file FILE Agents yaml configuration file
16 The python script 'loader.py' will preload and configure all the needed agents.
17 Also, ADML ones will start with the configured endpoints, ready for work.
20 python3 loader.py --file <tests directory>/agents.yml
23 See example at './tests.example/agents.yml'.
25 It is recommended to have that file within the tests directory compatible with such layout.
26 Anyway, you could have a common agents definition for multiple directories wherever you want to store them.
27 Next procedure (execution) will get a working directory which will be the parent for all the tests processed.
29 Also, you could load several agents definitions.
31 The format for agents file specification is:
37 description: <description>
46 description: <description>
48 application_id: <value>
49 dictionary: <xml dictionary path (absolute or relative to ./agents/ADML)>
51 origin_host: <origin host>
52 origin_realm: <origin realm>
53 address: <ip/server:port[,ip2/server2:port,...]>
60 description: <description>
62 address: <ip/server:port>
69 description: AF Diameter Rx Node
71 application_id: 16777236
72 dictionary: stacks/DictionaryRx.16777236.xml
74 origin_host: machine.source.server.realm.com
75 origin_realm: source.server.realm.com
76 address: 127.0.0.1:3868
79 description: SMPC N7 Node
93 $> python3 launcher.py -h
94 usage: launcher.py [-h] -t TESTS_DIR [-k] [-s] [-i] [-d] [1/708]
96 Anna Agents-Oriented Testing Setup Launcher
99 -h, --help show this help message and exit
100 -t TESTS_DIR, --tests-dir TESTS_DIR
101 Tests parent directory where to find .yml files (from
102 the next directories level)
103 -k, --keep-list-if-exists
104 Keeps intact the list of test cases (<test-
105 dir>/launcher.list), creates it if missing
106 -s, --stop-adml-at-the-end
107 At the end, ADML keeps running to ease debugging. You
108 could force stop with this option
109 -i, --interactive Interactive execution to ease debugging of test cases
110 -d, --dry-run Used to test and debug provision, no execution is
112 -p IP_LIMIT, --ip-limit IP_LIMIT
113 In-Progress limit is the number of coexisting In-
114 Progress State test cases. Defaults to 1 (sequential),
115 -1 would be 'no limit').
116 -r TTPS, --ttps TTPS Rate of test cases launched (test ticks per second).
117 By default 50 (recommended for monothread version).
120 The python script 'launcher.py' will execute the test cases under the provided directory.
121 Test cases at the directory provided itself will be ignored (searchs are performed at second level in order to avoid catching agents definitions).
122 Then we recommend this structure:
125 <tests directory>/agents.yml
126 <tests directory>/<test suite>/.../<test case>.yml
127 <tests directory>/<test suite(i)>/.../<dir(j)>/.../<test case(k)>.yml
130 The organization of test cases, chapters, test suites, or whatever desired concept is completely free for user preferences. Anyway, we recommend the use of numeric prefixes to control the default order of execution (alphabetical order).
132 To execute the tests, just provide the global parent directory.
133 NO TEST CASES SHALL BE LOCATED THERE BY CONVENTION (indeed, you could have them, but they would be ignored).
136 python3 launcher.py --tests-dir <tests directory>
139 The procedure will look for the file ```'<tests directory>'/launcher.list'``` in order to plan the execution.
140 That list will be created/refreshed always, except if you pass the option '-k|--keep-list-if-exists' (in case you want to play with edited lists).
141 The algorithm to create the list is based in a recursive search of yaml files in alphabetical order under the tests directory provided, in a similar way than the output of:
144 find <tests directory> -mindepth 2 -name "*.yml" | sort -t'/'
149 One interesting feature from AOTS is the possibility to interact test cases even to deep level (step by step).
150 You could enable the interactive mode by mean:
153 python3 launcher.py --tests-dir <tests directory> --interactive
156 A menu will be shown after test cases programming to give you a complete control of the ADML operations:
159 MAIN INTERACTIVE MENU
160 =====================
161 (prefix option with 'h' to get detailed help)
170 Test cases execution:
171 3. Start with test rate
172 4. Start next N test cases
175 Low level execution: test case steps
176 6. Execute next N steps
183 11. Reports configuration
186 ### User-defined lists
188 You could edit such list and remove or comment (#) entries (each line is a test case) in order to "disable" such test cases.
189 You could also mess the order with something like:
192 sort -R <tests directory>/launcher.list
195 Don't forget to pass the option '-k|--keep-list-if-exists' or the list will be overwritten during launcher procedure execution.
199 Files used by parsers within the test cases (xml diameter messages, json kafka ones, etc.), are relative to the test case file path location, anyway you could set absolute paths for the stuff used.
203 * Alternative 1: create a directory called ```'$(basename <test case file> .yml)'```
204 in order to keep order:
207 <dir>/tc1-initial-with-ipv4.yml
208 <dir>/tc1-initial-with-ipv4/AAA.xml
209 <dir>/tc1-initial-with-ipv4/smpc-consumed.json
210 <dir>/tc2-initial-with-ipv6.yml
211 <dir>/tc2-initial-with-ipv6/AAA.xml
212 <dir>/tc2-initial-with-ipv6/smpc-consumed.json
216 * Alternative 2: create a unique directory for the definition and all the test case stuff:
219 <dir>/tc1/initial-with-ipv4.yml
221 <dir>/tc1/smpc-consumed.json
222 <dir>/tc2/initial-with-ipv6.yml
224 <dir>/tc2/smpc-consumed.json
228 The last one seems to be better and cleaner alternative.
230 ### Test case definition
232 A test case is a yaml list of steps. Each step is a dictionary object with mandatory keys called 'action' and 'arguments'.
233 An action could be generic (no agent id specified) or agent-related.
234 Arguments depends on the type of action and are documented at ```'agents/<template>/ACTIONS.md'``` file.
235 Generic actions are documented below ([Generic actions](#generic-actions)).
238 - action: [<agent id>/]<action id>
240 <argument 1>: <value 1>
242 <argument m>: <value m>
249 # Step 1: Establish a timeout for the whole test case
254 # Step 2: Clean kafka message bus for the SMPC topic
257 operation: clean_topic
259 # Step 3: Send the AA-Request to the server
260 - action: AF/send_xml_to_entity
264 # Step 4: Wait for AA-Answer for the previous request sent
265 - action: AF/wait_xml_from_entity
270 # Step 4: Check the message bus for a kafka notify reception
271 - action: SMPC/consume_json
274 auto_offset_reset: earliest
278 It is good to comment each step with the position number (1..N). Take into account that lists are processed in order. Some steps could reference to other steps and this position could be useful for that purpose (see step 4 in the example above).
284 Sets a shell command script execution.
287 * value: command and arguments
298 Sets timeout (milliseconds) to complete the whole test case, or get a Failed state if not completed.
299 This could be configured at any step point, where will be started as time measuring reference.
300 It is recommended in order to ensure that test will not be in-progress eternally.
301 The launcher executor will calculate 15 seconds per test case by default just in case this
302 timeout feature is not used.
305 * value: numeric value for the lapse of time in milliseconds.
316 Sets delay (milliseconds) to sleep the test case at the corresponding delay step.
319 * value: numeric value for the lapse of time in milliseconds.
330 Controller In-Progress limit. A value of 1 means that only 1 test case with
331 status "in progress" is allowed. That is to say, ip_limit=1 is the same as
332 sequential execution of test cases.
334 This value can be set for the whole launcher execution by mean '-p|--ip-limit'
335 argument, but anyway, the value set in a test case definition has priority
336 because it will remain for the following test cases executed if the action
337 is not used to update the value again. Summing up, the command line for
338 launcher only sets the initial ip-limit value (1 by default).
340 Also we can restore the command-line value in any moment just configuring
341 "launcher" for this action value.
343 A typical use case for this is when a chapter or a whole test suite need a
344 initial (or intermediate) administration step to do global actions which
345 could impact in the real test cases if everything is executed in parallel.
346 For example, remove kafka bus or create first dummy message, could impact
347 in the real test cases which are consuming data from the kafka message bus.
348 Then, we will define a first test-suite case with the proper administrative
349 operations setting ip-limit to 1, doing needed operations and updating this
350 limit again (to -1, no limit, or any other). The reserved word "launcher"
351 helps to restore the value which the user provided through the launcher
352 procedure (in this way, if user wanted sequential, the tests will be
353 sequentially executed).
355 To ensure that an intermediate test case is the only executed we should
356 first set the ip-limit to 1, then perform a delay step with the maximum
357 timeout of the suite test cases, and then, perform the administrative
358 operations. Finally, as it is unpredictable to know which test case is the
359 following, in the same administrative test case, we end defining another
360 ip_limit action to restore the parallel value (ip limit > 1 or "launcher").
361 Also, those administrative operations should be blocking until completed.
364 * value: in progress limit amount (-1: no limit, 0: stops the ttps clock). Reserved 'launcher' sets the command-line launcher received value.
373 #### modify_xml_avp_data
375 Replace operations over ANNA diameter message xml representation.
376 Replaces data (or hex-data) fields for specific avps before processing.
377 XPath is used to select the node(s) to be processed. The value provided
378 will replace the data or hex-data field. A list is used to allow one or
379 more replace operations over the same xml file.
381 Arguments: xml, new_xml, xpath_value_list
382 * xml: the diameter message in xml format. Could be a symlink to common template.
383 * new_xml: new diameter message in xml format. Should be the one referred at test case.
384 * xpath_value_list: list of items with fields: xpath, value
388 - action: modify_xml_avp_data
390 xml: AAR.xml.template
393 - xpath: "avp[@name='Session-Id']"
394 value: "mytest;afNodeHostname.nodeHostRealm.com;1;005"
395 - xpath: "avp[@name='Framed-IP-Address']"
399 Important note: better don't use for diameter messages used in wait conditions.
400 The reason is that wait conditions acts using the xml content as regular
401 expression to be matched with serialized message received at the endpoint.
402 The thing is that diameter xml representation compared with python parser
403 representation will be probably different and regular expression won't be
404 valid to match. Summing up, this kind of messages must be statically
405 created with the expected content. Anyway, fields like hop-by-hop, end-to-end
406 and Origin-State-Id AVP values, will be automatically matched regardless its
407 value (regular expression is internally modified to replace them by '[0-9]+').
409 #### modify_json_key_value
411 Replaces value fields for specific keys before processing.
413 Arguments: json, new_json, kpath_value_list
414 * json: the json file. Could be a symlink to common template.
415 * new_json: new json file. Must be the one referred at test case.
416 * kpath_value_list: list of items with fields: kpath, value
418 The 'key path' concept (kpath field) consist in json node location
419 by mean a dot-separated string: <node1>[.<node2>]..[.<nodeN>].
423 - action: modify_json_key_value
425 json: notify.json.template
426 new_json: notify.json
428 - kpath: "session.smSessionId"
429 value: "the_sm_session_id"
434 Launchs an inline shell command or a user script file.
436 Arguments: [shell, file, file_parameters]
437 * shell: inline shell command and parameters.
438 * file: script or procedure defined in a test case file
439 * file_parameters: script file parameters (empty by default)
445 shell: echo "this is a test"
449 shell: rdate remote_host
453 file: check_consumer_step.py
457 ## Environment variables
459 Agents directory, where templates ADML and KAFKA are located, could be accessible through environment variable 'AGENTS_DIR'.
460 For example, you could use it for an system_cmd action:
466 value: check_consumption.sh $AGENTS_DIR/KAFKA/SMPC-consumer.sh.output
471 After launcher execution, a '<tests directory>.logs' directory is created within test directory processed.
472 This is an example of logs created by AOTS:
475 drwxrwxr-x 2 vagrant vagrant 4096 Aug 18 21:24 traffic
476 -rw-rw-r-- 1 vagrant vagrant 16065 Aug 18 21:24 junit.xml
477 -rw-rw-r-- 1 vagrant vagrant 15166 Aug 18 21:24 tests.summary
478 -rw-rw-r-- 1 vagrant vagrant 10747 Aug 18 21:24 tests.oam
479 -rw-rw-r-- 1 vagrant vagrant 697 Aug 18 21:24 tests.stats
480 -rw-rw-r-- 1 vagrant vagrant 83839 Aug 18 21:24 adml.context
481 drwxrwxr-x 2 vagrant vagrant 4096 Aug 18 21:24 counters
482 drwxrwxr-x 2 vagrant vagrant 4096 Aug 18 21:24 test-reports
483 drwxrwxr-x 2 vagrant vagrant 4096 Aug 18 21:24 debug
486 * traffic: folder containing traffic logs, for example diameter interfaces flow information.
487 * junit.xml: JUnit report for the testsuite execution.
488 * tests.summary: ADML director tests summary in xml format.
489 * tests.oam: ADML director Operation & Maintenance information (counters)
490 * tests.stats: ADML director statistic module information (average, deviation, max & min values for any of the statistic concepts registered, for example processing time in every interface, messages size, etc.).
491 * adml.context: context information in xml format for the ADML director process.
492 * counters: counters information, for example diameter events.
493 * test-reports: ADML tests reports in xml format.
494 * debug: debug information (ADML traces, yml test case files list, relation between ids and descriptions, ADML operations dumps, etc.).