TermAppISONFT: Orkhestra Cross Test Performance Summary

## Loading required package: lattice
## 
## Attaching package: 'BSDA'
## The following object is masked from 'package:datasets':
## 
##     Orange
##  : starts:  Mon Nov  1 19:52:23 2021

Introduction

There are three elements to this cross-test performance summary report. The first shows a summary of the percentage success of each function/operation/call. The second element compares the performance of the latest test(s) to the pooled performance of previous tests for each of the functions/operations/calls and outcomes. And the third element of this report compares the performance by function/operation/call by the outcome across multiple NFT result sets.

The percentage successful outcomes are presented as a summary for the latest test(s). This summary is ranked in increasing order of the percentage of good outcomes of that scenario against all attempts of that scenario in the test.

As a summary and for ranking the performance of the last tests results against previous test results, for each function/operation/call and outcome, the tests in the last test session are compared against the tests in previous sessions. This is accomplished by pooling the sample mean of the response times and pooling the sample standard deviations of response times across all prior tests, and then comparing the latest test(s) with the previous tests using tsum.test. The results are by ranked by the corresponding \(p\)-values in increasing order and tabulated. For each function/operation/call request, three comparison tests are made: The first determines a measure of the difference between the respective response time distributions; the second determines a measure of those response times that could be considered worse in the latest test(s) as compared to the pooled previous test; and the third determines a measure of those response times that could be considered better in the latest test(s) as compared to the pooled previous tests.

In addition to tabulating the response time means and standard deviations against function/operation/call and outcomes across the tests, box-plots are produced to visually compare the performance/outcomes over the various tests. In each case, the box-plots show up the 15 most extreme functions/operations/calls that are most different to the historic response time distributions, and then a box-plot each showing those that have response times greatest increase and decrease in their response times when compared to their respective historic counterparts.

The last section of the report compares the performance by function/operation/call by the outcome across multiple NFT result sets. The summary results have been taken from the application performance sections of the individual NFT sessions. The Resp value is the sample mean of the response times in seconds and the StdDevis the corresponding sample standard deviation. In each case only those values where the customer or business function arrival rate did not materially exceed the peak observed/production target are included in the calculation.

Summary of successful outcomes for latest testing

Test 1 - TermAppISONFT - TermAppISO

The following table is a summary of the outcomes of test 1 (TermAppISONFT - TermAppISO), showing the percentage of functions/operations/calls considered successful. The scenarios are shown from worst percentage good outcomes to best:

StartTime TestNumber Label Description Basename Outcome Count Percent Resp StdDev
2021-11-01 17:46:01 1 TermAppISONFT TermAppISO transaction_advice_response_1230 TRANSACTION_ADVICE_RESPONSE_1230_OK 15609 99.872 0.109 0.039
2021-11-01 17:46:01 1 TermAppISONFT TermAppISO authorisation_request_1100 AUTHORISATION_RESPONSE_1110_OK 15686 100.000 0.213 0.039

Comparison of latest tests to pooled previous tests

The last test date in the summary data is used to delimit the prior tests from the tests in the last test session. This section compares the tests performed on testdate to the tests that ran in sessions prior to this date. Comparisons are made only for the successful outcomes, and only the performance data where the rate in each of the tests included in the comparison did not exceed the target rate is included in the comparison.

Differences in response time distributions

The following show the comparisons of the good outcomes of the tests performed on 2021-11-01 as compared to the tests performed before this date. The table is ranked in increasing order of the \(p\)-values from the corresponding Welch Modified Two-Sample t-Test (two.sided), starting from the function/operation/call where the response time distribution differences are the greatest. Results are only shown for which the \(p\)-value is less than or equal to the cutoff value (\(\alpha\) = 0.05).

Test 1 - TermAppISONFT - TermAppISO

The following compare the responses time differences from the test started at 2021-11-01 17:46:01 to the tests from previous test sessions.

Basename Outcome Count Resp StdDev PrevCount PrevMean PrevStdDev pvalue.d
authorisation_request_1100 AUTHORISATION_RESPONSE_1110_OK 15686 0.213 0.039 294238 1.594 4.109 0
transaction_advice_response_1230 TRANSACTION_ADVICE_RESPONSE_1230_OK 15609 0.109 0.039 292624 0.260 0.354 0
## Loading required package: grid

Key Basename
A authorisation_request_1100
B transaction_advice_response_1230

Increases in the response times

There were no significant response time increases when comparing the test(s) in the last test session to tests from earlier test sessions for any of the items.

Decreases in the response times

The following show the comparisons of the good outcomes of the tests performed on 2021-11-01 as compared to the tests performed before this date. The table is ranked in increasing order of the \(p\)-values from the corresponding Welch Modified Two-Sample t-Test (less), starting from the function/operation/call where the response time decreases are the greatest. Results are only shown for which the \(p\)-value is less than or equal to the cutoff value (\(\alpha\) = 0.05).

Test 1 - TermAppISONFT - TermAppISO

The following compare the responses time decreases from the test started at 2021-11-01 17:46:01 to the tests from previous test sessions.

Basename Outcome Count Resp StdDev PrevCount PrevMean PrevStdDev pvalue.l
authorisation_request_1100 AUTHORISATION_RESPONSE_1110_OK 15686 0.213 0.039 294238 1.594 4.109 0
transaction_advice_response_1230 TRANSACTION_ADVICE_RESPONSE_1230_OK 15609 0.109 0.039 292624 0.260 0.354 0

Key Basename
A authorisation_request_1100
B transaction_advice_response_1230

Comparison across all tests individually

This section compares the performance between the NFT tests to date for each of the functions/operations/calls included in the corresponding test.

In the box-plots that follow, in each case, the centre is the sample mean response time value in seconds for that particular function/operation/call qualified by the outcome of that function/operation/call. The lower edge of the box is the corresponding sample mean response time value less the sample standard deviation, and the upper edge of the box is the corresponding sample mean response time value plus the standard deviation. The minimum and maximum values are calculated by taking two times the standard deviation in a similar manner.

Performance of authorisation_request_1100 with outcome: AUTHORISATION_RESPONSE_1110_OK

The following table shows the performance descriptive statistics for authorisation_request_1100 when the outcomes are AUTHORISATION_RESPONSE_1110_OK.

TestDate Description Basename Outcome Count Percent Resp StdDev
2020-12-07 TermAppISO Integrator Routing Regression authorisation_request_1100 AUTHORISATION_RESPONSE_1110_OK 15652 100.000 0.287 0.123
2020-12-08 TermAppISO Integrator Routing Regression authorisation_request_1100 AUTHORISATION_RESPONSE_1110_OK 15873 100.000 0.281 0.124
2020-12-09 TermAppISO Int Routing Regression authorisation_request_1100 AUTHORISATION_RESPONSE_1110_OK 18635 98.182 6.813 8.841
2020-12-09 TermAppISO authorisation_request_1100 AUTHORISATION_RESPONSE_1110_OK 16455 100.000 0.283 0.122
2020-12-16 TermAppISO Integrator Routing Changes authorisation_request_1100 AUTHORISATION_RESPONSE_1110_OK 19400 99.954 4.531 7.550
2020-12-16 TermAppISO with 40 provider threads authorisation_request_1100 AUTHORISATION_RESPONSE_1110_OK 19591 99.974 4.483 7.560
2020-12-16 TermAppISO with 80 provider threads authorisation_request_1100 AUTHORISATION_RESPONSE_1110_OK 19369 99.943 5.214 8.069
2021-01-15 TermAppISO authorisation_request_1100 AUTHORISATION_RESPONSE_1110_OK 16427 100.000 0.213 0.049
2021-05-10 TermAppISO authorisation_request_1100 AUTHORISATION_RESPONSE_1110_OK 7173 50.000 0.229 0.074
2021-05-20 TermAppISO authorisation_request_1100 AUTHORISATION_RESPONSE_1110_OK 7095 50.021 0.213 0.028
2021-06-28 TermAppISO authorisation_request_1100 AUTHORISATION_RESPONSE_1110_OK 3941 18.075 0.732 0.903
2021-06-29 TermAppISO authorisation_request_1100 AUTHORISATION_RESPONSE_1110_OK 13951 100.000 0.215 0.079
2021-07-16 TermAppISO authorisation_request_1100 AUTHORISATION_RESPONSE_1110_OK 14836 100.000 0.213 0.062
2021-07-19 TermAppISO authorisation_request_1100 AUTHORISATION_RESPONSE_1110_OK 15642 100.000 0.217 0.100
2021-07-22 TermAppISO authorisation_request_1100 AUTHORISATION_RESPONSE_1110_OK 4118 18.694 0.739 0.901
2021-07-23 TermAppISO authorisation_request_1100 AUTHORISATION_RESPONSE_1110_OK 12207 86.697 1.133 1.177
2021-07-27 TermAppISO authorisation_request_1100 AUTHORISATION_RESPONSE_1110_OK 15631 100.000 0.210 0.030
2021-07-28 TermAppISO authorisation_request_1100 AUTHORISATION_RESPONSE_1110_OK 15761 100.000 0.212 0.035
2021-08-02 TermAppISO authorisation_request_1100 AUTHORISATION_RESPONSE_1110_OK 14708 100.000 0.216 0.061
2021-08-02 TermAppISO authorisation_request_1100 AUTHORISATION_RESPONSE_1110_OK 13947 100.000 0.217 0.068
2021-10-20 TermAppISO authorisation_request_1100 AUTHORISATION_RESPONSE_1110_OK 13826 100.000 0.213 0.047
2021-11-01 TermAppISO authorisation_request_1100 AUTHORISATION_RESPONSE_1110_OK 15686 100.000 0.213 0.039

Performance of authorisation_request_1100 with outcome: disconnect

The following table shows the performance descriptive statistics for authorisation_request_1100 when the outcomes are disconnect.

TestDate Description Basename Outcome Count Percent Resp StdDev
2020-12-09 TermAppISO Int Routing Regression authorisation_request_1100 disconnect 304 1.602 4.166 0.205
2020-12-16 TermAppISO Integrator Routing Changes authorisation_request_1100 disconnect 3 0.015 4.318 0.124
2020-12-16 TermAppISO with 40 provider threads authorisation_request_1100 disconnect 2 0.010 4.258 0.060
2020-12-16 TermAppISO with 80 provider threads authorisation_request_1100 disconnect 5 0.026 4.191 0.021
2021-05-10 TermAppISO authorisation_request_1100 disconnect 7173 50.000 0.027 0.037
2021-05-20 TermAppISO authorisation_request_1100 disconnect 7089 49.979 0.016 0.024
2021-06-28 TermAppISO authorisation_request_1100 disconnect 467 2.142 4.241 0.721
2021-07-22 TermAppISO authorisation_request_1100 disconnect 568 2.579 3.758 0.901
2021-07-23 TermAppISO authorisation_request_1100 disconnect 177 1.257 34.187 1.177

Performance of authorisation_request_1100 with outcome: timeout

The following table shows the performance descriptive statistics for authorisation_request_1100 when the outcomes are timeout.

TestDate Description Basename Outcome Count Percent Resp StdDev
2020-12-09 TermAppISO Int Routing Regression authorisation_request_1100 timeout 41 0.216 99.999 0.000
2020-12-16 TermAppISO Integrator Routing Changes authorisation_request_1100 timeout 6 0.031 99.999 0.000
2020-12-16 TermAppISO with 40 provider threads authorisation_request_1100 timeout 3 0.015 99.999 0.000
2020-12-16 TermAppISO with 80 provider threads authorisation_request_1100 timeout 6 0.031 99.999 0.000
2021-06-28 TermAppISO authorisation_request_1100 timeout 17396 79.784 99.999 0.000
2021-07-22 TermAppISO authorisation_request_1100 timeout 17342 78.727 99.999 0.901
2021-07-23 TermAppISO authorisation_request_1100 timeout 1696 12.045 99.999 1.177

Performance of transaction_advice_response_1230 with outcome: disconnect

The following table shows the performance descriptive statistics for transaction_advice_response_1230 when the outcomes are disconnect.

TestDate Description Basename Outcome Count Percent Resp StdDev
2021-07-23 TermAppISO transaction_advice_response_1230 disconnect 2 0.016 21.704 1.177

Performance of transaction_advice_response_1230 with outcome: timeout

The following table shows the performance descriptive statistics for transaction_advice_response_1230 when the outcomes are timeout.

TestDate Description Basename Outcome Count Percent Resp StdDev
2020-12-07 TermAppISO Integrator Routing Regression transaction_advice_response_1230 timeout 33 0.211 99.999 0.000
2020-12-08 TermAppISO Integrator Routing Regression transaction_advice_response_1230 timeout 16 0.101 99.999 0.000
2020-12-09 TermAppISO Int Routing Regression transaction_advice_response_1230 timeout 15 0.081 99.999 0.000
2020-12-09 TermAppISO transaction_advice_response_1230 timeout 35 0.214 99.999 0.000
2020-12-16 TermAppISO Integrator Routing Changes transaction_advice_response_1230 timeout 28 0.145 99.999 0.000
2020-12-16 TermAppISO with 40 provider threads transaction_advice_response_1230 timeout 16 0.082 99.999 0.000
2020-12-16 TermAppISO with 80 provider threads transaction_advice_response_1230 timeout 57 0.296 99.999 0.000
2021-01-15 TermAppISO transaction_advice_response_1230 timeout 20 0.122 99.999 0.000
2021-05-10 TermAppISO transaction_advice_response_1230 timeout 14 0.196 99.999 0.000
2021-05-20 TermAppISO transaction_advice_response_1230 timeout 13 0.184 99.999 0.000
2021-06-28 TermAppISO transaction_advice_response_1230 timeout 14 0.357 99.999 0.000
2021-06-29 TermAppISO transaction_advice_response_1230 timeout 17 0.122 99.999 0.000
2021-07-16 TermAppISO transaction_advice_response_1230 timeout 23 0.156 99.999 0.062
2021-07-19 TermAppISO transaction_advice_response_1230 timeout 32 0.205 99.999 0.100
2021-07-22 TermAppISO transaction_advice_response_1230 timeout 18 0.439 99.999 0.901
2021-07-23 TermAppISO transaction_advice_response_1230 timeout 17 0.140 99.999 1.177
2021-07-27 TermAppISO transaction_advice_response_1230 timeout 16 0.103 99.999 0.030
2021-07-28 TermAppISO transaction_advice_response_1230 timeout 15 0.095 99.999 0.035
2021-08-02 TermAppISO transaction_advice_response_1230 timeout 28 0.191 99.999 0.061
2021-08-02 TermAppISO transaction_advice_response_1230 timeout 35 0.252 99.999 0.068
2021-10-20 TermAppISO transaction_advice_response_1230 timeout 12 0.087 99.999 0.047
2021-11-01 TermAppISO transaction_advice_response_1230 timeout 20 0.128 99.999 0.039

Performance of transaction_advice_response_1230 with outcome: TRANSACTION_ADVICE_RESPONSE_1230_OK

The following table shows the performance descriptive statistics for transaction_advice_response_1230 when the outcomes are TRANSACTION_ADVICE_RESPONSE_1230_OK.

TestDate Description Basename Outcome Count Percent Resp StdDev
2020-12-07 TermAppISO Integrator Routing Regression transaction_advice_response_1230 TRANSACTION_ADVICE_RESPONSE_1230_OK 15570 99.789 0.167 0.097
2020-12-08 TermAppISO Integrator Routing Regression transaction_advice_response_1230 TRANSACTION_ADVICE_RESPONSE_1230_OK 15780 99.899 0.162 0.094
2020-12-09 TermAppISO Int Routing Regression transaction_advice_response_1230 TRANSACTION_ADVICE_RESPONSE_1230_OK 18553 99.919 0.736 0.462
2020-12-09 TermAppISO transaction_advice_response_1230 TRANSACTION_ADVICE_RESPONSE_1230_OK 16358 99.786 0.164 0.091
2020-12-16 TermAppISO Integrator Routing Changes transaction_advice_response_1230 TRANSACTION_ADVICE_RESPONSE_1230_OK 19299 99.855 0.600 0.426
2020-12-16 TermAppISO with 40 provider threads transaction_advice_response_1230 TRANSACTION_ADVICE_RESPONSE_1230_OK 19503 99.918 0.602 0.428
2020-12-16 TermAppISO with 80 provider threads transaction_advice_response_1230 TRANSACTION_ADVICE_RESPONSE_1230_OK 19227 99.704 0.653 0.493
2021-01-15 TermAppISO transaction_advice_response_1230 TRANSACTION_ADVICE_RESPONSE_1230_OK 16335 99.878 0.110 0.046
2021-05-10 TermAppISO transaction_advice_response_1230 TRANSACTION_ADVICE_RESPONSE_1230_OK 7138 99.804 0.116 0.065
2021-05-20 TermAppISO transaction_advice_response_1230 TRANSACTION_ADVICE_RESPONSE_1230_OK 7053 99.816 0.110 0.035
2021-06-28 TermAppISO transaction_advice_response_1230 TRANSACTION_ADVICE_RESPONSE_1230_OK 3912 99.643 0.115 0.037
2021-06-29 TermAppISO transaction_advice_response_1230 TRANSACTION_ADVICE_RESPONSE_1230_OK 13875 99.878 0.110 0.058
2021-07-16 TermAppISO transaction_advice_response_1230 TRANSACTION_ADVICE_RESPONSE_1230_OK 14757 99.844 0.109 0.062
2021-07-19 TermAppISO transaction_advice_response_1230 TRANSACTION_ADVICE_RESPONSE_1230_OK 15552 99.795 0.113 0.100
2021-07-22 TermAppISO transaction_advice_response_1230 TRANSACTION_ADVICE_RESPONSE_1230_OK 4085 99.561 0.125 0.901
2021-07-23 TermAppISO transaction_advice_response_1230 TRANSACTION_ADVICE_RESPONSE_1230_OK 12126 99.844 0.111 1.177
2021-07-27 TermAppISO transaction_advice_response_1230 TRANSACTION_ADVICE_RESPONSE_1230_OK 15559 99.897 0.108 0.030
2021-07-28 TermAppISO transaction_advice_response_1230 TRANSACTION_ADVICE_RESPONSE_1230_OK 15703 99.905 0.109 0.035
2021-08-02 TermAppISO transaction_advice_response_1230 TRANSACTION_ADVICE_RESPONSE_1230_OK 14615 99.809 0.110 0.061
2021-08-02 TermAppISO transaction_advice_response_1230 TRANSACTION_ADVICE_RESPONSE_1230_OK 13859 99.748 0.110 0.068
2021-10-20 TermAppISO transaction_advice_response_1230 TRANSACTION_ADVICE_RESPONSE_1230_OK 13765 99.913 0.109 0.047
2021-11-01 TermAppISO transaction_advice_response_1230 TRANSACTION_ADVICE_RESPONSE_1230_OK 15609 99.872 0.109 0.039

Session details

sessionInfo();
## R version 3.6.0 (2019-04-26)
## Platform: x86_64-redhat-linux-gnu (64-bit)
## Running under: CentOS Linux 7 (Core)
## 
## Matrix products: default
## BLAS/LAPACK: /usr/lib64/R/lib/libRblas.so
## 
## locale:
##  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
##  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
##  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
##  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
##  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
## 
## attached base packages:
## [1] grid      stats     graphics  grDevices utils     datasets  methods  
## [8] base     
## 
## other attached packages:
## [1] pander_0.6.3    doBy_4.6.7      cmlrutils_1.19  XML_3.98-1.20  
## [5] scales_1.1.1    ggplot2_3.3.2   BSDA_1.2.0      lattice_0.20-38
## 
## loaded via a namespace (and not attached):
##  [1] Rcpp_1.0.7       highr_0.8        compiler_3.6.0   pillar_1.4.6    
##  [5] rmdformats_1.0.0 class_7.3-15     tools_3.6.0      digest_0.6.25   
##  [9] evaluate_0.14    lifecycle_0.2.0  tibble_3.0.3     gtable_0.3.0    
## [13] pkgconfig_2.0.3  rlang_0.4.7      Matrix_1.2-17    yaml_2.2.1      
## [17] xfun_0.17        e1071_1.7-4      withr_2.2.0      stringr_1.4.0   
## [21] dplyr_1.0.2      knitr_1.30       generics_0.0.2   vctrs_0.3.2     
## [25] tidyselect_1.1.0 glue_1.4.1       R6_2.4.1         rmarkdown_2.6   
## [29] bookdown_0.20    farver_2.0.3     tidyr_1.1.2      purrr_0.3.4     
## [33] magrittr_1.5     backports_1.1.8  MASS_7.3-51.4    ellipsis_0.3.1  
## [37] htmltools_0.5.0  colorspace_1.4-1 Deriv_4.0.1      labeling_0.3    
## [41] stringi_1.5.3    munsell_0.5.0    broom_0.7.0      crayon_1.3.4