Introduction

There are two elements to this cross test resource usage comparison report. The first compares the current test sessions test(s) estimate of resource and measure for each server group to the corresponding pooled estimates of the resource and measure for that group. The second compares each of the tests current and earlier tests individually for each resource and measure for each server group.

For the pooled comparison, the resource usage data for each resource type and measure of each server group (as grouped by the corresponding server classification file) for the current test(s) (the last set of tests by date) are isolated as individual tests and compared to the corresponding pooled resource usage distribution using a Welch modified two-sample t-test (tsum.test) based on the distribution parameters. Each resource type and measure, the estimation of the background usage (Bias, or load independent resource usage), for that resource and measure for the current test(s) are compared to the estimated pooled background usage of that resource type and measure. In a similar way, the estimation of the usage rate (Rate, or load dependant resource usage, the measure of the cost in terms of the corresponding resource for a rate of one customer arrival per second). For the pooling, the previous test results are filtered so that only the tests for which there is some confidence in the corresponding coefficient estimate are included in the pool statistics. A set resource usage coefficient estimate is included in the pooling of the corresponding \(p\)-value does not exceed \(\alpha\) = 0.05 (so that there is some confidence in the pooled coefficients). The Bias and Rate together with the actual customer arrival rate (ArrivalRate) give an estimation of the cost in terms of the resource being considered as \(\mbox{TotalCost} = \mbox{Bias}+\mbox{ArrivalRate}*\mbox{Rate}\).

For each of the resource types and measures (Percent CPU Utilisation) and for each of the coefficient estimates (Bias and Rate), three tsum.tests are performed: One two-sided test to for differences in the corresponding distributions, and two one-sided tests, one to test for increases in the resource usage, and one to test for decreases in resource usage (of each of the current tests over pooled prior tests). These comparisons of the tests are ranked by the corresponding \(p\)-values. The items for which the \(p\)-value does not exceed the cut-off \(\alpha\) value (0.05) are high-lighted by reporting them in a ranked table. are included in the corresponding comparison table. Up to the first 15 are included in an accompanying box-plot as well.

This high-lighting is intended to quickly bring outliers or significant changes in resource usage to the readers attention, but the second part of the report includes a full cross test comparison of current and earlier tests each server group, resource and measure, for both Bias and Rate.

These summary results have been taken from the analysis sections of the individual NFT sessions. For the Rate coefficient, only those results which are considered significant in the response of the resource usage to the applied load are included. The Bias parameter is included wherever there was some significance in the estimated intercept value.

The coefficients are determined from linear models regressing the resource usage onto the throughput observed. A model is generated for each server class in the server classification list for the specific project’s server landscape. In each case, the resource usage is a measure of the resource usage, and these metrics are added up for all servers in the class. In addition, for the z/OS workload, the RMF measured CPU for each address space of interest is totalled and used as resource measure to regress onto the throughput observed (field R791TCPU of RMF record type 79 subtype one is used for this). A class is simply a group of servers intended to deliver the function to the application. The class name is intended to indicate the function provided (for example, TFIM_CLIENT). For each linear model, two coefficients are estimated. The Bias/Intercept is an estimate of the background resource usage, which is independent of the load applied. This load could be unrelated or overhead workload on a shared server or overhead workload on dedicated servers. For dedicated servers, this value is expected to be reasonably small as it should only reflect the overhead in managing and monitoring the workload. For shared servers, the value should ideally be reasonably low, and where this is the case, this indicates a reasonably controlled test environment (making the shared server behave like a dedicated server for the test duration). Where there is workload on a shared server/servers unrelated to the system under test during a test, the additional workload may confound the estimation of the Rate coefficient, making the required resource usage over or understated.

The Rate is the coefficient on the achieved load. This is usually the session rate or customer arrival rate, measure in customer arrivals per second. Depending on the NFT setup, a session would do either a chain of application functions on behalf of a single business/user-level function or may do a single function or operation. This distinction usually correlates to simulating end-users (multiple calls/operations within a session) or single function/call/operation within a session (system to system or device to device calls). In either case, the Rate coefficient is the session arrival rate achieved.

The interpretation of the Rate coefficient is that it is the resource cost in (one of) the appropriate metric for the server type, and hence is the estimated cost in terms that metric of a single session per second (or customer arrival per second).

Server classification from application landscape

Comparison of current tests to pooled earlier tests

The last test date in the summary data is used to delimit the prior tests from the tests in the last test session. This section compares the resource usage for each resource type and measure for each server group to the corresponding pooled reource usage and measire for the server group. The current test(s) are the tests performed on 2021-08-02. Resource uage metrics are pooled only where the corresponding \(p\)-value is demonstrates some confidence in the estimated coefficient.

## Loading required package: grid

Comparison of resource usage across tests

This section compares the resource usage between the NFT tests to date for each of the groups/servers in the classification list for which data has been collected.

In the box-plots that follow, in each case, the centre is the estimated value of the coefficient calculated for the particular test. The lower edge of the box is the corresponding estimated value less the standard error, and the upper edge of the box is the corresponding estimated value plus the standard error. The minimum and maximum values are calculated by taking two times the standard error values in a similar manner.

Session details

sessionInfo();

## R version 3.6.0 (2019-04-26)
## Platform: x86_64-redhat-linux-gnu (64-bit)
## Running under: CentOS Linux 7 (Core)
## 
## Matrix products: default
## BLAS/LAPACK: /usr/lib64/R/lib/libRblas.so
## 
## locale:
##  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
##  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
##  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
##  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
##  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
## 
## attached base packages:
## [1] grid      stats     graphics  grDevices utils     datasets  methods  
## [8] base     
## 
## other attached packages:
## [1] pander_0.6.3    doBy_4.6.7      cmlrutils_1.19  XML_3.98-1.20  
## [5] scales_1.1.1    ggplot2_3.3.2   BSDA_1.2.0      lattice_0.20-38
## 
## loaded via a namespace (and not attached):
##  [1] Rcpp_1.0.5       highr_0.8        compiler_3.6.0   pillar_1.4.6    
##  [5] rmdformats_1.0.0 class_7.3-15     tools_3.6.0      digest_0.6.25   
##  [9] evaluate_0.14    lifecycle_0.2.0  tibble_3.0.3     gtable_0.3.0    
## [13] pkgconfig_2.0.3  rlang_0.4.7      Matrix_1.2-17    yaml_2.2.1      
## [17] xfun_0.17        e1071_1.7-4      withr_2.2.0      stringr_1.4.0   
## [21] dplyr_1.0.2      knitr_1.30       generics_0.0.2   vctrs_0.3.2     
## [25] tidyselect_1.1.0 glue_1.4.1       R6_2.4.1         rmarkdown_2.6   
## [29] bookdown_0.20    farver_2.0.3     tidyr_1.1.2      purrr_0.3.4     
## [33] magrittr_1.5     backports_1.1.8  MASS_7.3-51.4    ellipsis_0.3.1  
## [37] htmltools_0.5.0  colorspace_1.4-1 Deriv_4.0.1      labeling_0.3    
## [41] stringi_1.5.3    munsell_0.5.0    broom_0.7.0      crayon_1.3.4