Introduction

There are three elements to this cross-test performance summary report. The first shows a summary of the percentage success of each function/operation/call. The second element compares the performance of the latest test(s) to the pooled performance of previous tests for each of the functions/operations/calls and outcomes. And the third element of this report compares the performance by function/operation/call by the outcome across multiple NFT result sets.

The percentage successful outcomes are presented as a summary for the latest test(s). This summary is ranked in increasing order of the percentage of good outcomes of that scenario against all attempts of that scenario in the test.

As a summary and for ranking the performance of the last tests results against previous test results, for each function/operation/call and outcome, the tests in the last test session are compared against the tests in previous sessions. This is accomplished by pooling the sample mean of the response times and pooling the sample standard deviations of response times across all prior tests, and then comparing the latest test(s) with the previous tests using tsum.test. The results are by ranked by the corresponding \(p\)-values in increasing order and tabulated. For each function/operation/call request, three comparison tests are made: The first determines a measure of the difference between the respective response time distributions; the second determines a measure of those response times that could be considered worse in the latest test(s) as compared to the pooled previous test; and the third determines a measure of those response times that could be considered better in the latest test(s) as compared to the pooled previous tests.

In addition to tabulating the response time means and standard deviations against function/operation/call and outcomes across the tests, box-plots are produced to visually compare the performance/outcomes over the various tests. In each case, the box-plots show up the 15 most extreme functions/operations/calls that are most different to the historic response time distributions, and then a box-plot each showing those that have response times greatest increase and decrease in their response times when compared to their respective historic counterparts.

The last section of the report compares the performance by function/operation/call by the outcome across multiple NFT result sets. The summary results have been taken from the application performance sections of the individual NFT sessions. The Resp value is the sample mean of the response times in seconds and the StdDevis the corresponding sample standard deviation. In each case only those values where the customer or business function arrival rate did not materially exceed the peak observed/production target are included in the calculation.

Summary of successful outcomes for latest testing

Comparison of latest tests to pooled previous tests

The last test date in the summary data is used to delimit the prior tests from the tests in the last test session. This section compares the tests performed on testdate to the tests that ran in sessions prior to this date. Comparisons are made only for the successful outcomes, and only the performance data where the rate in each of the tests included in the comparison did not exceed the target rate is included in the comparison.

## Loading required package: grid

Comparison across all tests individually

This section compares the performance between the NFT tests to date for each of the functions/operations/calls included in the corresponding test.

In the box-plots that follow, in each case, the centre is the sample mean response time value in seconds for that particular function/operation/call qualified by the outcome of that function/operation/call. The lower edge of the box is the corresponding sample mean response time value less the sample standard deviation, and the upper edge of the box is the corresponding sample mean response time value plus the standard deviation. The minimum and maximum values are calculated by taking two times the standard deviation in a similar manner.

Session details

sessionInfo();

## R version 3.6.0 (2019-04-26)
## Platform: x86_64-redhat-linux-gnu (64-bit)
## Running under: CentOS Linux 7 (Core)
## 
## Matrix products: default
## BLAS/LAPACK: /usr/lib64/R/lib/libRblas.so
## 
## locale:
##  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
##  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
##  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
##  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
##  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
## 
## attached base packages:
## [1] grid      stats     graphics  grDevices utils     datasets  methods  
## [8] base     
## 
## other attached packages:
## [1] pander_0.6.3    doBy_4.6.7      cmlrutils_1.18  XML_3.98-1.20  
## [5] scales_1.1.1    ggplot2_3.3.2   BSDA_1.2.0      lattice_0.20-38
## 
## loaded via a namespace (and not attached):
##  [1] Rcpp_1.0.5       highr_0.8        compiler_3.6.0   pillar_1.4.6    
##  [5] rmdformats_1.0.0 class_7.3-15     tools_3.6.0      digest_0.6.25   
##  [9] evaluate_0.14    lifecycle_0.2.0  tibble_3.0.3     gtable_0.3.0    
## [13] pkgconfig_2.0.3  rlang_0.4.7      Matrix_1.2-17    yaml_2.2.1      
## [17] xfun_0.17        e1071_1.7-4      withr_2.2.0      stringr_1.4.0   
## [21] dplyr_1.0.2      knitr_1.30       generics_0.0.2   vctrs_0.3.2     
## [25] tidyselect_1.1.0 glue_1.4.1       R6_2.4.1         rmarkdown_2.6   
## [29] bookdown_0.20    farver_2.0.3     tidyr_1.1.2      purrr_0.3.4     
## [33] magrittr_1.5     backports_1.1.8  MASS_7.3-51.4    ellipsis_0.3.1  
## [37] htmltools_0.5.0  colorspace_1.4-1 Deriv_4.0.1      labeling_0.3    
## [41] stringi_1.5.3    munsell_0.5.0    broom_0.7.0      crayon_1.3.4