TermAppISONFT: Orkhestra Cross Test Resource Usage Summary

## Loading required package: lattice
## 
## Attaching package: 'BSDA'
## The following object is masked from 'package:datasets':
## 
##     Orange
##  : starts:  Wed Jul 28 12:23:18 2021

Introduction

There are two elements to this cross test resource usage comparison report. The first compares the current test sessions test(s) estimate of resource and measure for each server group to the corresponding pooled estimates of the resource and measure for that group. The second compares each of the tests current and earlier tests individually for each resource and measure for each server group.

For the pooled comparison, the resource usage data for each resource type and measure of each server group (as grouped by the corresponding server classification file) for the current test(s) (the last set of tests by date) are isolated as individual tests and compared to the corresponding pooled resource usage distribution using a Welch modified two-sample t-test (tsum.test) based on the distribution parameters. Each resource type and measure, the estimation of the background usage (Bias, or load independent resource usage), for that resource and measure for the current test(s) are compared to the estimated pooled background usage of that resource type and measure. In a similar way, the estimation of the usage rate (Rate, or load dependant resource usage, the measure of the cost in terms of the corresponding resource for a rate of one customer arrival per second). For the pooling, the previous test results are filtered so that only the tests for which there is some confidence in the corresponding coefficient estimate are included in the pool statistics. A set resource usage coefficient estimate is included in the pooling of the corresponding \(p\)-value does not exceed \(\alpha\) = 0.05 (so that there is some confidence in the pooled coefficients). The Bias and Rate together with the actual customer arrival rate (ArrivalRate) give an estimation of the cost in terms of the resource being considered as \(\mbox{TotalCost} = \mbox{Bias}+\mbox{ArrivalRate}*\mbox{Rate}\).

For each of the resource types and measures (Percent CPU Utilisation) and for each of the coefficient estimates (Bias and Rate), three tsum.tests are performed: One two-sided test to for differences in the corresponding distributions, and two one-sided tests, one to test for increases in the resource usage, and one to test for decreases in resource usage (of each of the current tests over pooled prior tests). These comparisons of the tests are ranked by the corresponding \(p\)-values. The items for which the \(p\)-value does not exceed the cut-off \(\alpha\) value (0.05) are high-lighted by reporting them in a ranked table. are included in the corresponding comparison table. Up to the first 15 are included in an accompanying box-plot as well.

This high-lighting is intended to quickly bring outliers or significant changes in resource usage to the readers attention, but the second part of the report includes a full cross test comparison of current and earlier tests each server group, resource and measure, for both Bias and Rate.

These summary results have been taken from the analysis sections of the individual NFT sessions. For the Rate coefficient, only those results which are considered significant in the response of the resource usage to the applied load are included. The Bias parameter is included wherever there was some significance in the estimated intercept value.

The coefficients are determined from linear models regressing the resource usage onto the throughput observed. A model is generated for each server class in the server classification list for the specific project’s server landscape. In each case, the resource usage is a measure of the resource usage, and these metrics are added up for all servers in the class. In addition, for the z/OS workload, the RMF measured CPU for each address space of interest is totalled and used as resource measure to regress onto the throughput observed (field R791TCPU of RMF record type 79 subtype one is used for this). A class is simply a group of servers intended to deliver the function to the application. The class name is intended to indicate the function provided (for example, TFIM_CLIENT). For each linear model, two coefficients are estimated. The Bias/Intercept is an estimate of the background resource usage, which is independent of the load applied. This load could be unrelated or overhead workload on a shared server or overhead workload on dedicated servers. For dedicated servers, this value is expected to be reasonably small as it should only reflect the overhead in managing and monitoring the workload. For shared servers, the value should ideally be reasonably low, and where this is the case, this indicates a reasonably controlled test environment (making the shared server behave like a dedicated server for the test duration). Where there is workload on a shared server/servers unrelated to the system under test during a test, the additional workload may confound the estimation of the Rate coefficient, making the required resource usage over or understated.

The Rate is the coefficient on the achieved load. This is usually the session rate or customer arrival rate, measure in customer arrivals per second. Depending on the NFT setup, a session would do either a chain of application functions on behalf of a single business/user-level function or may do a single function or operation. This distinction usually correlates to simulating end-users (multiple calls/operations within a session) or single function/call/operation within a session (system to system or device to device calls). In either case, the Rate coefficient is the session arrival rate achieved.

The interpretation of the Rate coefficient is that it is the resource cost in (one of) the appropriate metric for the server type, and hence is the estimated cost in terms that metric of a single session per second (or customer arrival per second).

Coef Description
Rate Resource usage per customer arrival per second.
Bias Load idependent rource usage or background load

Server classification from application landscape

Classification Server IP_Address OSType Description Hardware_Support Server_Support Application_Support SOURCE_FILE
LOAD LOAD0 176.67.166.86 Linux CML EcoSystem Patrick Hayward Patrick Hayward Patrick Hayward CMLEcoSystem_Server_Classification.csv
LOAD LOAD1 176.67.166.89 Linux CML EcoSystem Patrick Hayward Patrick Hayward Patrick Hayward CMLEcoSystem_Server_Classification.csv
MODEL MODELAPP0 176.67.166.20 Linux CML EcoSystem Patrick Hayward Patrick Hayward Patrick Hayward CMLEcoSystem_Server_Classification.csv
MODEL MODELAPP1 176.67.166.72 Linux CML EcoSystem Patrick Hayward Patrick Hayward Patrick Hayward CMLEcoSystem_Server_Classification.csv
NETWORK NETWORK 109.123.111.17 Linux CML EcoSystem Patrick Hayward Patrick Hayward Patrick Hayward CMLEcoSystem_Server_Classification.csv

Comparison of current tests to pooled earlier tests

The last test date in the summary data is used to delimit the prior tests from the tests in the last test session. This section compares the resource usage for each resource type and measure for each server group to the corresponding pooled reource usage and measire for the server group. The current test(s) are the tests performed on 2021-07-28. Resource uage metrics are pooled only where the corresponding \(p\)-value is demonstrates some confidence in the estimated coefficient.

Differences in resource usage

The following show the comparisons of the tests performed on 2021-07-28 as compared to the tests performed before this date. The tables are ranked in increasing order of the \(p\)-values from the corresponding Welch Modified Two-Sample t-Test (two.sided), starting from server group where the resource usage distribution differences are the greatest. Results are only shown for which the \(p\)-value is less than or equal to the cutoff value (\(\alpha\) = 0.05).

Test 1 - TermAppISO

Resource class: CPU

The following compares the Rate coefficent for the CPU resource usage differences from the test started at 2021-07-28 09:48:01 to the tests from previous test sessions.

Class Coef Count Metric Units Estimate Std..Error PrevCount PrevEst PrevSEM pvalue.d
LOAD Rate 78 cpu.cpu.system.user Percent single processor 0.989 0.013 1145 3.904 0.089 0
MODEL Rate 78 cpu.cpu.system.user Percent single processor 1.165 0.147 1047 2.938 0.102 0
NETWORK Rate 78 cpu.cpu.system.user Percent single processor 1.245 0.316 333 1.744 0.104 0
## Loading required package: grid

Key Group
A LOAD Rate Percent single processor CPU Usage
B MODEL Rate Percent single processor CPU Usage
C NETWORK Rate Percent single processor CPU Usage

The following compares the Bias coefficent for the CPU resource usage differences from the test started at 2021-07-28 09:48:01 to the tests from previous test sessions.

Class Coef Count Metric Units Estimate Std..Error PrevCount PrevEst PrevSEM pvalue.d
LOAD Bias 78 cpu.cpu.system.user Percent single processor 2.737 0.279 1123 44.76 1.705 0
MODEL Bias 78 cpu.cpu.system.user Percent single processor 26.395 3.137 1145 170.42 3.301 0

Key Group
A LOAD Bias Percent single processor CPU Usage
B MODEL Bias Percent single processor CPU Usage

Resource class: Network

The following compares the Rate coefficent for the Network resource usage differences from the test started at 2021-07-28 09:48:01 to the tests from previous test sessions.

Class Coef Count Metric Units Estimate Std..Error PrevCount PrevEst PrevSEM pvalue.d
MODEL Rate 78 recv bits per second 12301.59 150.338 1067 17701.03 481.545 0
LOAD Rate 78 recv bits per second 23208.79 285.686 1145 17100.41 195.445 0
MODEL Rate 78 sent bits per second 12952.81 316.605 918 16487.95 271.858 0
LOAD Rate 78 sent bits per second 28088.95 298.640 1145 26405.65 350.288 0

Key Group
A MODEL Rate bits per second Network Usage recv
B LOAD Rate bits per second Network Usage recv
C MODEL Rate bits per second Network Usage sent
D LOAD Rate bits per second Network Usage sent

The following compares the Bias coefficent for the Network resource usage differences from the test started at 2021-07-28 09:48:01 to the tests from previous test sessions.

Class Coef Count Metric Units Estimate Std..Error PrevCount PrevEst PrevSEM pvalue.d
MODEL Bias 78 recv bits per second 119129.0 3197.513 1121 565515.3 11959.552 0
MODEL Bias 78 sent bits per second 161652.7 6733.818 1121 846623.8 17795.389 0
LOAD Bias 78 sent bits per second 410292.8 6351.720 1145 316057.7 6765.211 0
LOAD Bias 78 recv bits per second 227386.5 6076.202 1067 198631.4 2979.363 0

Key Group
A MODEL Bias bits per second Network Usage recv
B MODEL Bias bits per second Network Usage sent
C LOAD Bias bits per second Network Usage sent
D LOAD Bias bits per second Network Usage recv

Increases in resource usage

The following show the comparisons of the tests performed on 2021-07-28 as compared to the tests performed before this date. The tables are ranked in increasing order of the \(p\)-values from the corresponding Welch Modified Two-Sample t-Test (greater), starting from server group where the resource usage distribution increases are the greatest. Results are only shown for which the \(p\)-value is less than or equal to the cutoff value (\(\alpha\) = 0.05).

Test 1 - TermAppISO

Resource class: Network

The following compares the Rate coefficent for the Network resource usage increases from the test started at 2021-07-28 09:48:01 to the tests from previous test sessions.

Class Coef Count Metric Units Estimate Std..Error PrevCount PrevEst PrevSEM pvalue.g
LOAD Rate 78 recv bits per second 23208.79 285.686 1145 17100.41 195.445 0
LOAD Rate 78 sent bits per second 28088.95 298.640 1145 26405.65 350.288 0

Key Group
A LOAD Rate bits per second Network Usage recv
B LOAD Rate bits per second Network Usage sent

The following compares the Bias coefficent for the Network resource usage increases from the test started at 2021-07-28 09:48:01 to the tests from previous test sessions.

Class Coef Count Metric Units Estimate Std..Error PrevCount PrevEst PrevSEM pvalue.g
LOAD Bias 78 recv bits per second 227386.5 6076.202 1067 198631.4 2979.363 0
LOAD Bias 78 sent bits per second 410292.8 6351.720 1145 316057.7 6765.211 0

Key Group
A LOAD Bias bits per second Network Usage recv
B LOAD Bias bits per second Network Usage sent

Decreases in resource usage

The following show the comparisons of the tests performed on 2021-07-28 as compared to the tests performed before this date. The tables are ranked in increasing order of the \(p\)-values from the corresponding Welch Modified Two-Sample t-Test (less), starting from server group where the resource usage distribution decreases are the greatest. Results are only shown for which the \(p\)-value is less than or equal to the cutoff value (\(\alpha\) = 0.05).

Test 1 - TermAppISO

Resource class: CPU

The following compares the Rate coefficent for the CPU resource usage decreases from the test started at 2021-07-28 09:48:01 to the tests from previous test sessions.

Class Coef Count Metric Units Estimate Std..Error PrevCount PrevEst PrevSEM pvalue.l
LOAD Rate 78 cpu.cpu.system.user Percent single processor 0.989 0.013 1145 3.904 0.089 0
MODEL Rate 78 cpu.cpu.system.user Percent single processor 1.165 0.147 1047 2.938 0.102 0
NETWORK Rate 78 cpu.cpu.system.user Percent single processor 1.245 0.316 333 1.744 0.104 0

Key Group
A LOAD Rate Percent single processor CPU Usage
B MODEL Rate Percent single processor CPU Usage
C NETWORK Rate Percent single processor CPU Usage

The following compares the Bias coefficent for the CPU resource usage decreases from the test started at 2021-07-28 09:48:01 to the tests from previous test sessions.

Class Coef Count Metric Units Estimate Std..Error PrevCount PrevEst PrevSEM pvalue.l
LOAD Bias 78 cpu.cpu.system.user Percent single processor 2.737 0.279 1123 44.76 1.705 0
MODEL Bias 78 cpu.cpu.system.user Percent single processor 26.395 3.137 1145 170.42 3.301 0

Key Group
A LOAD Bias Percent single processor CPU Usage
B MODEL Bias Percent single processor CPU Usage

Resource class: Network

The following compares the Rate coefficent for the Network resource usage decreases from the test started at 2021-07-28 09:48:01 to the tests from previous test sessions.

Class Coef Count Metric Units Estimate Std..Error PrevCount PrevEst PrevSEM pvalue.l
MODEL Rate 78 recv bits per second 12301.59 150.338 1067 17701.03 481.545 0
MODEL Rate 78 sent bits per second 12952.81 316.605 918 16487.95 271.858 0

Key Group
A MODEL Rate bits per second Network Usage recv
B MODEL Rate bits per second Network Usage sent

The following compares the Bias coefficent for the Network resource usage decreases from the test started at 2021-07-28 09:48:01 to the tests from previous test sessions.

Class Coef Count Metric Units Estimate Std..Error PrevCount PrevEst PrevSEM pvalue.l
MODEL Bias 78 recv bits per second 119129.0 3197.513 1121 565515.3 11959.55 0
MODEL Bias 78 sent bits per second 161652.7 6733.818 1121 846623.8 17795.39 0

Key Group
A MODEL Bias bits per second Network Usage recv
B MODEL Bias bits per second Network Usage sent

Comparison of resource usage across tests

This section compares the resource usage between the NFT tests to date for each of the groups/servers in the classification list for which data has been collected.

In the box-plots that follow, in each case, the centre is the estimated value of the coefficient calculated for the particular test. The lower edge of the box is the corresponding estimated value less the standard error, and the upper edge of the box is the corresponding estimated value plus the standard error. The minimum and maximum values are calculated by taking two times the standard error values in a similar manner.

CPU Resource usage for LOAD by CPU Usage using cpu.cpu.system.user in Percent single processor

Server OSType Description
LOAD0 Linux CML EcoSystem
LOAD1 Linux CML EcoSystem

The following table shows the Rate coefficient values over the various tests performed. In this table the units of estimate are: Percent single processor.

TestDate Coef Estimate Std..Error t.value p.value Test
2020-12-07 Rate 4.852 0.356 13.627 0 TermAppISO Integrator Routing Regression
2020-12-08 Rate 5.021 0.271 18.501 0 TermAppISO Integrator Routing Regression
2020-12-09 Rate 7.444 0.602 12.360 0 TermAppISO Int Routing Regression
2020-12-09 Rate 5.138 0.277 18.561 0 TermAppISO
2020-12-16 Rate 9.780 0.574 17.034 0 TermAppISO Integrator Routing Changes
2020-12-16 Rate 10.198 0.584 17.447 0 TermAppISO with 40 provider threads
2020-12-16 Rate 8.606 0.718 11.993 0 TermAppISO with 80 provider threads
2021-01-15 Rate 0.973 0.022 44.250 0 TermAppISO
2021-05-10 Rate 0.680 0.066 10.297 0 TermAppISO
2021-05-20 Rate 0.710 0.013 52.850 0 TermAppISO
2021-06-28 Rate 0.834 0.035 23.721 0 TermAppISO
2021-06-29 Rate 0.963 0.014 66.737 0 TermAppISO
2021-07-16 Rate 1.059 0.029 37.102 0 TermAppISO
2021-07-19 Rate 1.041 0.034 30.497 0 TermAppISO
2021-07-22 Rate 0.786 0.018 44.845 0 TermAppISO
2021-07-23 Rate 1.139 0.014 81.322 0 TermAppISO
2021-07-27 Rate 0.915 0.011 85.841 0 TermAppISO
2021-07-28 Rate 0.989 0.013 75.418 0 TermAppISO

The following table shows the Bias coefficient values over the various tests performed. In this table the units of estimate are: Percent single processor.

TestDate Coef Estimate Std..Error t.value p.value Test
2020-12-07 Bias 59.012 7.451 7.921 0.000 TermAppISO Integrator Routing Regression
2020-12-08 Bias 60.847 5.668 10.735 0.000 TermAppISO Integrator Routing Regression
2020-12-09 Bias 124.874 11.476 10.882 0.000 TermAppISO Int Routing Regression
2020-12-09 Bias 57.803 5.740 10.071 0.000 TermAppISO
2020-12-16 Bias 110.050 10.485 10.496 0.000 TermAppISO Integrator Routing Changes
2020-12-16 Bias 101.540 10.390 9.773 0.000 TermAppISO with 40 provider threads
2020-12-16 Bias 124.248 13.091 9.491 0.000 TermAppISO with 80 provider threads
2021-01-15 Bias 4.653 0.462 10.075 0.000 TermAppISO
2021-05-20 Bias 3.599 0.216 16.641 0.000 TermAppISO
2021-06-28 Bias 1.532 0.539 2.841 0.006 TermAppISO
2021-06-29 Bias 1.399 0.205 6.822 0.000 TermAppISO
2021-07-16 Bias 1.648 0.283 5.824 0.000 TermAppISO
2021-07-19 Bias 4.109 0.698 5.884 0.000 TermAppISO
2021-07-22 Bias 3.138 0.269 11.687 0.000 TermAppISO
2021-07-23 Bias 1.224 0.295 4.150 0.000 TermAppISO
2021-07-27 Bias 4.675 0.229 20.451 0.000 TermAppISO
2021-07-28 Bias 2.737 0.279 9.813 0.000 TermAppISO

Network Resource usage for LOAD by Network Usage recv using recv in bits per second

Server OSType Description
LOAD0 Linux CML EcoSystem
LOAD1 Linux CML EcoSystem

The following table shows the Rate coefficient values over the various tests performed. In this table the units of estimate are: bits per second.

TestDate Coef Estimate Std..Error t.value p.value Test
2020-12-07 Rate 15663.585 818.945 19.127 0 TermAppISO Integrator Routing Regression
2020-12-08 Rate 15753.089 745.948 21.118 0 TermAppISO Integrator Routing Regression
2020-12-09 Rate 8938.586 692.304 12.911 0 TermAppISO Int Routing Regression
2020-12-09 Rate 19602.564 1989.670 9.852 0 TermAppISO
2020-12-16 Rate 12978.364 542.369 23.929 0 TermAppISO Integrator Routing Changes
2020-12-16 Rate 12795.930 585.131 21.868 0 TermAppISO with 40 provider threads
2020-12-16 Rate 11544.987 748.371 15.427 0 TermAppISO with 80 provider threads
2021-01-15 Rate 23224.409 297.261 78.128 0 TermAppISO
2021-05-10 Rate 14748.670 1755.623 8.401 0 TermAppISO
2021-05-20 Rate 14621.637 272.552 53.647 0 TermAppISO
2021-06-28 Rate 10629.033 511.557 20.778 0 TermAppISO
2021-06-29 Rate 23164.964 373.138 62.082 0 TermAppISO
2021-07-16 Rate 23741.774 1043.020 22.763 0 TermAppISO
2021-07-19 Rate 22942.579 183.455 125.058 0 TermAppISO
2021-07-22 Rate 15398.095 595.974 25.837 0 TermAppISO
2021-07-23 Rate 24244.471 460.658 52.630 0 TermAppISO
2021-07-27 Rate 23785.093 366.436 64.909 0 TermAppISO
2021-07-28 Rate 23208.786 285.686 81.239 0 TermAppISO

The following table shows the Bias coefficient values over the various tests performed. In this table the units of estimate are: bits per second.

TestDate Coef Estimate Std..Error t.value p.value Test
2020-12-07 Bias 117112.32 17136.631 6.834 0.000 TermAppISO Integrator Routing Regression
2020-12-08 Bias 128316.44 15580.341 8.236 0.000 TermAppISO Integrator Routing Regression
2020-12-09 Bias 164916.40 13191.587 12.502 0.000 TermAppISO Int Routing Regression
2020-12-09 Bias 79903.07 41255.594 1.937 0.056 TermAppISO
2020-12-16 Bias 124584.24 9904.489 12.579 0.000 TermAppISO Integrator Routing Changes
2020-12-16 Bias 123844.15 10400.926 11.907 0.000 TermAppISO with 40 provider threads
2020-12-16 Bias 140398.04 13653.452 10.283 0.000 TermAppISO with 80 provider threads
2021-01-15 Bias 456044.32 6245.241 73.023 0.000 TermAppISO
2021-05-10 Bias 177202.72 54997.316 3.222 0.004 TermAppISO
2021-05-20 Bias 153604.59 4390.310 34.987 0.000 TermAppISO
2021-06-28 Bias 150601.54 7845.304 19.196 0.000 TermAppISO
2021-06-29 Bias 154516.43 5300.697 29.150 0.000 TermAppISO
2021-07-16 Bias 233115.34 10338.395 22.549 0.000 TermAppISO
2021-07-19 Bias 353663.23 3754.451 94.198 0.000 TermAppISO
2021-07-22 Bias 203343.05 9128.552 22.275 0.000 TermAppISO
2021-07-23 Bias 221648.78 9699.374 22.852 0.000 TermAppISO
2021-07-27 Bias 248432.92 7859.754 31.608 0.000 TermAppISO
2021-07-28 Bias 227386.46 6076.202 37.422 0.000 TermAppISO

Network Resource usage for LOAD by Network Usage sent using sent in bits per second

Server OSType Description
LOAD0 Linux CML EcoSystem
LOAD1 Linux CML EcoSystem

The following table shows the Rate coefficient values over the various tests performed. In this table the units of estimate are: bits per second.

TestDate Coef Estimate Std..Error t.value p.value Test
2020-12-07 Rate 19028.66 995.145 19.121 0 TermAppISO Integrator Routing Regression
2020-12-08 Rate 19042.75 934.235 20.383 0 TermAppISO Integrator Routing Regression
2020-12-09 Rate 30219.17 2364.677 12.779 0 TermAppISO Int Routing Regression
2020-12-09 Rate 19579.56 958.369 20.430 0 TermAppISO
2020-12-16 Rate 32193.80 653.690 49.249 0 TermAppISO Integrator Routing Changes
2020-12-16 Rate 31183.83 720.365 43.289 0 TermAppISO with 40 provider threads
2020-12-16 Rate 29416.80 1431.859 20.544 0 TermAppISO with 80 provider threads
2021-01-15 Rate 26987.33 242.256 111.400 0 TermAppISO
2021-05-10 Rate 19612.35 1623.789 12.078 0 TermAppISO
2021-05-20 Rate 21117.29 377.348 55.962 0 TermAppISO
2021-06-28 Rate 28204.25 1788.951 15.766 0 TermAppISO
2021-06-29 Rate 29243.75 436.441 67.005 0 TermAppISO
2021-07-16 Rate 30122.86 815.291 36.947 0 TermAppISO
2021-07-19 Rate 25840.71 3321.221 7.780 0 TermAppISO
2021-07-22 Rate 28744.37 1284.023 22.386 0 TermAppISO
2021-07-23 Rate 28074.25 420.903 66.700 0 TermAppISO
2021-07-27 Rate 27699.57 255.067 108.597 0 TermAppISO
2021-07-28 Rate 28088.95 298.640 94.056 0 TermAppISO

The following table shows the Bias coefficient values over the various tests performed. In this table the units of estimate are: bits per second.

TestDate Coef Estimate Std..Error t.value p.value Test
2020-12-07 Bias 204158.9 20823.674 9.804 0.000 TermAppISO Integrator Routing Regression
2020-12-08 Bias 212375.4 19513.027 10.884 0.000 TermAppISO Integrator Routing Regression
2020-12-09 Bias 139054.3 45057.989 3.086 0.003 TermAppISO Int Routing Regression
2020-12-09 Bias 206257.3 19871.668 10.379 0.000 TermAppISO
2020-12-16 Bias 109170.1 11937.368 9.145 0.000 TermAppISO Integrator Routing Changes
2020-12-16 Bias 114274.4 12804.747 8.924 0.000 TermAppISO with 40 provider threads
2020-12-16 Bias 136247.9 26123.167 5.216 0.000 TermAppISO with 80 provider threads
2021-01-15 Bias 430030.4 5089.620 84.492 0.000 TermAppISO
2021-05-10 Bias 413513.8 50867.430 8.129 0.000 TermAppISO
2021-05-20 Bias 389686.3 6078.379 64.110 0.000 TermAppISO
2021-06-28 Bias 544271.4 27435.564 19.838 0.000 TermAppISO
2021-06-29 Bias 393824.0 6199.974 63.520 0.000 TermAppISO
2021-07-16 Bias 381222.7 8081.151 47.174 0.000 TermAppISO
2021-07-19 Bias 484032.6 67969.577 7.121 0.000 TermAppISO
2021-07-22 Bias 546152.8 19667.416 27.769 0.000 TermAppISO
2021-07-23 Bias 419787.0 8862.318 47.368 0.000 TermAppISO
2021-07-27 Bias 418372.5 5470.982 76.471 0.000 TermAppISO
2021-07-28 Bias 410292.8 6351.720 64.596 0.000 TermAppISO

CPU Resource usage for MODEL by CPU Usage using cpu.cpu.system.user in Percent single processor

Server OSType Description
MODELAPP0 Linux CML EcoSystem
MODELAPP1 Linux CML EcoSystem

The following table shows the Rate coefficient values over the various tests performed. In this table the units of estimate are: Percent single processor.

TestDate Coef Estimate Std..Error t.value p.value Test
2020-12-07 Rate 1.132 0.048 23.545 0.000 TermAppISO Integrator Routing Regression
2020-12-08 Rate 1.239 0.017 74.254 0.000 TermAppISO Integrator Routing Regression
2020-12-09 Rate 7.740 0.829 9.331 0.000 TermAppISO Int Routing Regression
2020-12-09 Rate 1.227 0.015 80.090 0.000 TermAppISO
2020-12-16 Rate 7.307 0.592 12.351 0.000 TermAppISO Integrator Routing Changes
2020-12-16 Rate 7.929 0.581 13.643 0.000 TermAppISO with 40 provider threads
2020-12-16 Rate 6.526 0.746 8.752 0.000 TermAppISO with 80 provider threads
2021-01-15 Rate 1.200 0.129 9.336 0.000 TermAppISO
2021-05-20 Rate 1.473 0.112 13.117 0.000 TermAppISO
2021-06-28 Rate 0.597 0.113 5.266 0.000 TermAppISO
2021-06-29 Rate 0.866 0.385 2.249 0.029 TermAppISO
2021-07-16 Rate 1.632 0.220 7.435 0.000 TermAppISO
2021-07-19 Rate 1.370 0.150 9.104 0.000 TermAppISO
2021-07-22 Rate 0.808 0.055 14.686 0.000 TermAppISO
2021-07-27 Rate 1.189 0.067 17.836 0.000 TermAppISO
2021-07-28 Rate 1.165 0.147 7.899 0.000 TermAppISO

The following table shows the Bias coefficient values over the various tests performed. In this table the units of estimate are: Percent single processor.

TestDate Coef Estimate Std..Error t.value p.value Test
2020-12-07 Bias 4.808 1.006 4.779 0 TermAppISO Integrator Routing Regression
2020-12-08 Bias 3.084 0.348 8.849 0 TermAppISO Integrator Routing Regression
2020-12-09 Bias 180.757 15.805 11.437 0 TermAppISO Int Routing Regression
2020-12-09 Bias 3.996 0.318 12.580 0 TermAppISO
2020-12-16 Bias 122.689 10.804 11.356 0 TermAppISO Integrator Routing Changes
2020-12-16 Bias 117.633 10.331 11.386 0 TermAppISO with 40 provider threads
2020-12-16 Bias 151.751 13.604 11.155 0 TermAppISO with 80 provider threads
2021-01-15 Bias 25.053 2.701 9.275 0 TermAppISO
2021-05-10 Bias 129.953 26.227 4.955 0 TermAppISO
2021-05-20 Bias 121.790 1.809 67.324 0 TermAppISO
2021-06-28 Bias 699.995 1.738 402.874 0 TermAppISO
2021-06-29 Bias 26.687 5.467 4.881 0 TermAppISO
2021-07-16 Bias 23.169 2.176 10.646 0 TermAppISO
2021-07-19 Bias 24.944 3.080 8.099 0 TermAppISO
2021-07-22 Bias 694.715 0.843 824.181 0 TermAppISO
2021-07-23 Bias 395.494 41.801 9.461 0 TermAppISO
2021-07-27 Bias 28.779 1.429 20.134 0 TermAppISO
2021-07-28 Bias 26.395 3.137 8.414 0 TermAppISO

Network Resource usage for MODEL by Network Usage recv using recv in bits per second

Server OSType Description
MODELAPP0 Linux CML EcoSystem
MODELAPP1 Linux CML EcoSystem

The following table shows the Rate coefficient values over the various tests performed. In this table the units of estimate are: bits per second.

TestDate Coef Estimate Std..Error t.value p.value Test
2020-12-07 Rate 24100.050 954.550 25.248 0.000 TermAppISO Integrator Routing Regression
2020-12-08 Rate 25548.990 271.665 94.046 0.000 TermAppISO Integrator Routing Regression
2020-12-09 Rate 14069.744 920.324 15.288 0.000 TermAppISO Int Routing Regression
2020-12-09 Rate 24302.467 713.257 34.073 0.000 TermAppISO
2020-12-16 Rate 17467.524 683.900 25.541 0.000 TermAppISO Integrator Routing Changes
2020-12-16 Rate 17365.660 680.340 25.525 0.000 TermAppISO with 40 provider threads
2020-12-16 Rate 17074.863 664.909 25.680 0.000 TermAppISO with 80 provider threads
2021-01-15 Rate 24560.174 304.261 80.721 0.000 TermAppISO
2021-05-10 Rate 17389.627 5929.025 2.933 0.008 TermAppISO
2021-05-20 Rate 16631.667 971.810 17.114 0.000 TermAppISO
2021-06-28 Rate 15585.305 6115.750 2.548 0.013 TermAppISO
2021-06-29 Rate 24051.354 1557.272 15.445 0.000 TermAppISO
2021-07-16 Rate 12907.062 539.876 23.907 0.000 TermAppISO
2021-07-19 Rate 12306.350 138.379 88.932 0.000 TermAppISO
2021-07-22 Rate 4888.864 1017.101 4.807 0.000 TermAppISO
2021-07-27 Rate 12716.180 200.060 63.562 0.000 TermAppISO
2021-07-28 Rate 12301.593 150.338 81.826 0.000 TermAppISO

The following table shows the Bias coefficient values over the various tests performed. In this table the units of estimate are: bits per second.

TestDate Coef Estimate Std..Error t.value p.value Test
2020-12-07 Bias 64596.51 19974.205 3.234 0.002 TermAppISO Integrator Routing Regression
2020-12-08 Bias 52159.33 5674.160 9.192 0.000 TermAppISO Integrator Routing Regression
2020-12-09 Bias 158388.92 17536.412 9.032 0.000 TermAppISO Int Routing Regression
2020-12-09 Bias 67665.26 14789.314 4.575 0.000 TermAppISO
2020-12-16 Bias 128789.60 12489.057 10.312 0.000 TermAppISO Integrator Routing Changes
2020-12-16 Bias 130831.54 12093.291 10.819 0.000 TermAppISO with 40 provider threads
2020-12-16 Bias 132636.80 11969.911 11.081 0.000 TermAppISO with 80 provider threads
2021-01-15 Bias 462642.65 6392.297 72.375 0.000 TermAppISO
2021-05-20 Bias 177120.97 15654.052 11.315 0.000 TermAppISO
2021-06-28 Bias 3552601.09 93791.875 37.877 0.000 TermAppISO
2021-06-29 Bias 180897.77 22122.205 8.177 0.000 TermAppISO
2021-07-16 Bias 118244.76 5351.244 22.097 0.000 TermAppISO
2021-07-19 Bias 180920.17 2831.954 63.885 0.000 TermAppISO
2021-07-22 Bias 1808374.69 15578.961 116.078 0.000 TermAppISO
2021-07-23 Bias 1275083.95 143254.541 8.901 0.000 TermAppISO
2021-07-27 Bias 129537.85 4238.183 30.564 0.000 TermAppISO
2021-07-28 Bias 119129.01 3197.513 37.257 0.000 TermAppISO

Network Resource usage for MODEL by Network Usage sent using sent in bits per second

Server OSType Description
MODELAPP0 Linux CML EcoSystem
MODELAPP1 Linux CML EcoSystem

The following table shows the Rate coefficient values over the various tests performed. In this table the units of estimate are: bits per second.

TestDate Coef Estimate Std..Error t.value p.value Test
2020-12-07 Rate 23083.902 924.847 24.960 0.000 TermAppISO Integrator Routing Regression
2020-12-08 Rate 24600.259 251.866 97.672 0.000 TermAppISO Integrator Routing Regression
2020-12-09 Rate 6362.325 852.572 7.463 0.000 TermAppISO Int Routing Regression
2020-12-09 Rate 23318.892 749.322 31.120 0.000 TermAppISO
2020-12-16 Rate 10756.811 769.527 13.978 0.000 TermAppISO Integrator Routing Changes
2020-12-16 Rate 10172.372 701.912 14.492 0.000 TermAppISO with 40 provider threads
2020-12-16 Rate 8940.326 678.158 13.183 0.000 TermAppISO with 80 provider threads
2021-01-15 Rate 24097.187 241.131 99.934 0.000 TermAppISO
2021-05-10 Rate 18230.092 7038.605 2.590 0.018 TermAppISO
2021-05-20 Rate 18196.215 1111.117 16.377 0.000 TermAppISO
2021-06-29 Rate 25419.663 2207.053 11.517 0.000 TermAppISO
2021-07-16 Rate 13989.533 569.403 24.569 0.000 TermAppISO
2021-07-19 Rate 13062.866 297.431 43.919 0.000 TermAppISO
2021-07-27 Rate 12926.270 314.888 41.050 0.000 TermAppISO
2021-07-28 Rate 12952.805 316.605 40.912 0.000 TermAppISO

The following table shows the Bias coefficient values over the various tests performed. In this table the units of estimate are: bits per second.

TestDate Coef Estimate Std..Error t.value p.value Test
2020-12-07 Bias 186452.2 19352.666 9.634 0 TermAppISO Integrator Routing Regression
2020-12-08 Bias 167394.2 5260.624 31.820 0 TermAppISO Integrator Routing Regression
2020-12-09 Bias 331861.2 16245.425 20.428 0 TermAppISO Int Routing Regression
2020-12-09 Bias 187742.4 15537.099 12.083 0 TermAppISO
2020-12-16 Bias 307108.4 14052.742 21.854 0 TermAppISO Integrator Routing Changes
2020-12-16 Bias 291049.7 12476.750 23.327 0 TermAppISO with 40 provider threads
2020-12-16 Bias 400012.4 12208.426 32.765 0 TermAppISO with 80 provider threads
2021-01-15 Bias 365642.3 5065.982 72.176 0 TermAppISO
2021-05-20 Bias 314872.1 17898.032 17.593 0 TermAppISO
2021-06-28 Bias 4491325.5 152240.008 29.502 0 TermAppISO
2021-06-29 Bias 320788.4 31352.822 10.232 0 TermAppISO
2021-07-16 Bias 146356.9 5643.910 25.932 0 TermAppISO
2021-07-19 Bias 157004.9 6086.997 25.793 0 TermAppISO
2021-07-22 Bias 3053866.8 18802.581 162.417 0 TermAppISO
2021-07-23 Bias 2045037.6 209529.611 9.760 0 TermAppISO
2021-07-27 Bias 163493.5 6670.780 24.509 0 TermAppISO
2021-07-28 Bias 161652.7 6733.818 24.006 0 TermAppISO

Network Resource usage for NETWORK by Network Usage recv using recv in bits per second

Server OSType Description
NETWORK Linux CML EcoSystem

The following table shows the Rate coefficient values over the various tests performed. In this table the units of estimate are: bits per second.

TestDate Coef Estimate Std..Error t.value p.value Test
2021-05-20 Rate 6289.748 592.594 10.614 0 TermAppISO
2021-06-28 Rate 2711.282 448.421 6.046 0 TermAppISO

The following table shows the Bias coefficient values over the various tests performed. In this table the units of estimate are: bits per second.

TestDate Coef Estimate Std..Error t.value p.value Test
2021-05-10 Bias 1617885.6 328037.628 4.932 0 TermAppISO
2021-05-20 Bias 651687.7 9991.711 65.223 0 TermAppISO
2021-06-28 Bias 674006.3 6877.032 98.008 0 TermAppISO
2021-06-29 Bias 629183.3 163624.078 3.845 0 TermAppISO

Network Resource usage for NETWORK by Network Usage sent using sent in bits per second

Server OSType Description
NETWORK Linux CML EcoSystem

The following table shows the Rate coefficient values over the various tests performed. In this table the units of estimate are: bits per second.

TestDate Coef Estimate Std..Error t.value p.value Test
2021-06-29 Rate 939.561 362.238 2.594 0.013 TermAppISO

The following table shows the Bias coefficient values over the various tests performed. In this table the units of estimate are: bits per second.

TestDate Coef Estimate Std..Error t.value p.value Test
2021-05-10 Bias 6171231.4 1472966.74 4.190 0.000 TermAppISO
2021-06-28 Bias 250236.8 90104.11 2.777 0.007 TermAppISO

CPU Resource usage for NETWORK by CPU Usage using cpu.cpu.system.user in Percent single processor

Server OSType Description
NETWORK Linux CML EcoSystem

The following table shows the Rate coefficient values over the various tests performed. In this table the units of estimate are: Percent single processor.

TestDate Coef Estimate Std..Error t.value p.value Test
2021-05-20 Rate 1.139 0.294 3.868 0 TermAppISO
2021-06-29 Rate 2.785 0.197 14.135 0 TermAppISO
2021-07-19 Rate 0.760 0.147 5.171 0 TermAppISO
2021-07-23 Rate 2.838 0.198 14.300 0 TermAppISO
2021-07-27 Rate 1.444 0.291 4.960 0 TermAppISO
2021-07-28 Rate 1.245 0.316 3.936 0 TermAppISO

The following table shows the Bias coefficient values over the various tests performed. In this table the units of estimate are: Percent single processor.

TestDate Coef Estimate Std..Error t.value p.value Test
2021-05-20 Bias 12.834 4.965 2.585 0.012 TermAppISO
2021-06-28 Bias 9.329 2.628 3.550 0.001 TermAppISO
2021-06-29 Bias 11.455 2.799 4.093 0.000 TermAppISO
2021-07-16 Bias 13.326 5.802 2.297 0.029 TermAppISO
2021-07-19 Bias 10.806 3.007 3.594 0.001 TermAppISO
2021-07-22 Bias 11.766 3.047 3.862 0.000 TermAppISO
2021-07-23 Bias 15.364 4.179 3.676 0.000 TermAppISO
2021-07-27 Bias 44.797 6.247 7.171 0.000 TermAppISO
2021-07-28 Bias 17.637 6.727 2.622 0.011 TermAppISO

Session details

sessionInfo();
## R version 3.6.0 (2019-04-26)
## Platform: x86_64-redhat-linux-gnu (64-bit)
## Running under: CentOS Linux 7 (Core)
## 
## Matrix products: default
## BLAS/LAPACK: /usr/lib64/R/lib/libRblas.so
## 
## locale:
##  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
##  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
##  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
##  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
##  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
## 
## attached base packages:
## [1] grid      stats     graphics  grDevices utils     datasets  methods  
## [8] base     
## 
## other attached packages:
## [1] pander_0.6.3    doBy_4.6.7      cmlrutils_1.19  XML_3.98-1.20  
## [5] scales_1.1.1    ggplot2_3.3.2   BSDA_1.2.0      lattice_0.20-38
## 
## loaded via a namespace (and not attached):
##  [1] Rcpp_1.0.5       highr_0.8        compiler_3.6.0   pillar_1.4.6    
##  [5] rmdformats_1.0.0 class_7.3-15     tools_3.6.0      digest_0.6.25   
##  [9] evaluate_0.14    lifecycle_0.2.0  tibble_3.0.3     gtable_0.3.0    
## [13] pkgconfig_2.0.3  rlang_0.4.7      Matrix_1.2-17    yaml_2.2.1      
## [17] xfun_0.17        e1071_1.7-4      withr_2.2.0      stringr_1.4.0   
## [21] dplyr_1.0.2      knitr_1.30       generics_0.0.2   vctrs_0.3.2     
## [25] tidyselect_1.1.0 glue_1.4.1       R6_2.4.1         rmarkdown_2.6   
## [29] bookdown_0.20    farver_2.0.3     tidyr_1.1.2      purrr_0.3.4     
## [33] magrittr_1.5     backports_1.1.8  MASS_7.3-51.4    ellipsis_0.3.1  
## [37] htmltools_0.5.0  colorspace_1.4-1 Deriv_4.0.1      labeling_0.3    
## [41] stringi_1.5.3    munsell_0.5.0    broom_0.7.0      crayon_1.3.4