TermAppISONFT: Orkhestra Cross Test Resource Usage Summary

## Loading required package: lattice
## 
## Attaching package: 'BSDA'
## The following object is masked from 'package:datasets':
## 
##     Orange
##  : starts:  Tue Jul 20 14:36:38 2021

Introduction

There are two elements to this cross test resource usage comparison report. The first compares the current test sessions test(s) estimate of resource and measure for each server group to the corresponding pooled estimates of the resource and measure for that group. The second compares each of the tests current and earlier tests individually for each resource and measure for each server group.

For the pooled comparison, the resource usage data for each resource type and measure of each server group (as grouped by the corresponding server classification file) for the current test(s) (the last set of tests by date) are isolated as individual tests and compared to the corresponding pooled resource usage distribution using a Welch modified two-sample t-test (tsum.test) based on the distribution parameters. Each resource type and measure, the estimation of the background usage (Bias, or load independent resource usage), for that resource and measure for the current test(s) are compared to the estimated pooled background usage of that resource type and measure. In a similar way, the estimation of the usage rate (Rate, or load dependant resource usage, the measure of the cost in terms of the corresponding resource for a rate of one customer arrival per second). For the pooling, the previous test results are filtered so that only the tests for which there is some confidence in the corresponding coefficient estimate are included in the pool statistics. A set resource usage coefficient estimate is included in the pooling of the corresponding \(p\)-value does not exceed \(\alpha\) = 0.05 (so that there is some confidence in the pooled coefficients). The Bias and Rate together with the actual customer arrival rate (ArrivalRate) give an estimation of the cost in terms of the resource being considered as \(\mbox{TotalCost} = \mbox{Bias}+\mbox{ArrivalRate}*\mbox{Rate}\).

For each of the resource types and measures (Percent CPU Utilisation) and for each of the coefficient estimates (Bias and Rate), three tsum.tests are performed: One two-sided test to for differences in the corresponding distributions, and two one-sided tests, one to test for increases in the resource usage, and one to test for decreases in resource usage (of each of the current tests over pooled prior tests). These comparisons of the tests are ranked by the corresponding \(p\)-values. The items for which the \(p\)-value does not exceed the cut-off \(\alpha\) value (0.05) are high-lighted by reporting them in a ranked table. are included in the corresponding comparison table. Up to the first 15 are included in an accompanying box-plot as well.

This high-lighting is intended to quickly bring outliers or significant changes in resource usage to the readers attention, but the second part of the report includes a full cross test comparison of current and earlier tests each server group, resource and measure, for both Bias and Rate.

These summary results have been taken from the analysis sections of the individual NFT sessions. For the Rate coefficient, only those results which are considered significant in the response of the resource usage to the applied load are included. The Bias parameter is included wherever there was some significance in the estimated intercept value.

The coefficients are determined from linear models regressing the resource usage onto the throughput observed. A model is generated for each server class in the server classification list for the specific project’s server landscape. In each case, the resource usage is a measure of the resource usage, and these metrics are added up for all servers in the class. In addition, for the z/OS workload, the RMF measured CPU for each address space of interest is totalled and used as resource measure to regress onto the throughput observed (field R791TCPU of RMF record type 79 subtype one is used for this). A class is simply a group of servers intended to deliver the function to the application. The class name is intended to indicate the function provided (for example, TFIM_CLIENT). For each linear model, two coefficients are estimated. The Bias/Intercept is an estimate of the background resource usage, which is independent of the load applied. This load could be unrelated or overhead workload on a shared server or overhead workload on dedicated servers. For dedicated servers, this value is expected to be reasonably small as it should only reflect the overhead in managing and monitoring the workload. For shared servers, the value should ideally be reasonably low, and where this is the case, this indicates a reasonably controlled test environment (making the shared server behave like a dedicated server for the test duration). Where there is workload on a shared server/servers unrelated to the system under test during a test, the additional workload may confound the estimation of the Rate coefficient, making the required resource usage over or understated.

The Rate is the coefficient on the achieved load. This is usually the session rate or customer arrival rate, measure in customer arrivals per second. Depending on the NFT setup, a session would do either a chain of application functions on behalf of a single business/user-level function or may do a single function or operation. This distinction usually correlates to simulating end-users (multiple calls/operations within a session) or single function/call/operation within a session (system to system or device to device calls). In either case, the Rate coefficient is the session arrival rate achieved.

The interpretation of the Rate coefficient is that it is the resource cost in (one of) the appropriate metric for the server type, and hence is the estimated cost in terms that metric of a single session per second (or customer arrival per second).

Coef Description
Rate Resource usage per customer arrival per second.
Bias Load idependent rource usage or background load

Server classification from application landscape

Classification Server IP_Address OSType Description Hardware_Support Server_Support Application_Support SOURCE_FILE
LOAD LOAD0 176.67.166.86 Linux CML EcoSystem Patrick Hayward Patrick Hayward Patrick Hayward CMLEcoSystem_Server_Classification.csv
LOAD LOAD1 176.67.166.89 Linux CML EcoSystem Patrick Hayward Patrick Hayward Patrick Hayward CMLEcoSystem_Server_Classification.csv
MODEL MODELAPP0 176.67.166.20 Linux CML EcoSystem Patrick Hayward Patrick Hayward Patrick Hayward CMLEcoSystem_Server_Classification.csv
MODEL MODELAPP1 176.67.166.72 Linux CML EcoSystem Patrick Hayward Patrick Hayward Patrick Hayward CMLEcoSystem_Server_Classification.csv
NETWORK NETWORK 109.123.111.17 Linux CML EcoSystem Patrick Hayward Patrick Hayward Patrick Hayward CMLEcoSystem_Server_Classification.csv

Comparison of current tests to pooled earlier tests

The last test date in the summary data is used to delimit the prior tests from the tests in the last test session. This section compares the resource usage for each resource type and measure for each server group to the corresponding pooled reource usage and measire for the server group. The current test(s) are the tests performed on 2021-07-16. Resource uage metrics are pooled only where the corresponding \(p\)-value is demonstrates some confidence in the estimated coefficient.

Differences in resource usage

The following show the comparisons of the tests performed on 2021-07-16 as compared to the tests performed before this date. The tables are ranked in increasing order of the \(p\)-values from the corresponding Welch Modified Two-Sample t-Test (two.sided), starting from server group where the resource usage distribution differences are the greatest. Results are only shown for which the \(p\)-value is less than or equal to the cutoff value (\(\alpha\) = 0.05).

Test 1 - TermAppISO

Resource class: CPU

The following compares the Rate coefficent for the CPU resource usage differences from the test started at 2021-07-16 17:25:00 to the tests from previous test sessions.

Class Coef Count Metric Units Estimate Std..Error PrevCount PrevEst PrevSEM pvalue.d
LOAD Rate 31 cpu.cpu.system.user Percent single processor 1.059 0.029 811 5.108 0.126 0
MODEL Rate 31 cpu.cpu.system.user Percent single processor 1.632 0.220 789 3.510 0.135 0
## Loading required package: grid

Key Group
A LOAD Rate Percent single processor CPU Usage
B MODEL Rate Percent single processor CPU Usage

The following compares the Bias coefficent for the CPU resource usage differences from the test started at 2021-07-16 17:25:00 to the tests from previous test sessions.

Class Coef Count Metric Units Estimate Std..Error PrevCount PrevEst PrevSEM pvalue.d
LOAD Bias 31 cpu.cpu.system.user Percent single processor 1.648 0.283 789 62.381 2.425 0.000
MODEL Bias 31 cpu.cpu.system.user Percent single processor 23.169 2.176 811 134.208 2.502 0.000
NETWORK Bias 31 cpu.cpu.system.user Percent single processor 13.326 5.802 179 10.996 2.047 0.034

Key Group
A LOAD Bias Percent single processor CPU Usage
B MODEL Bias Percent single processor CPU Usage
C NETWORK Bias Percent single processor CPU Usage

Resource class: Network

The following compares the Rate coefficent for the Network resource usage differences from the test started at 2021-07-16 17:25:00 to the tests from previous test sessions.

Class Coef Count Metric Units Estimate Std..Error PrevCount PrevEst PrevSEM pvalue.d
MODEL Rate 31 recv bits per second 12907.06 539.876 810 20029.89 626.740 0
LOAD Rate 31 recv bits per second 23741.77 1043.020 811 15152.32 261.287 0
MODEL Rate 31 sent bits per second 13989.53 569.403 735 17315.39 335.728 0
LOAD Rate 31 sent bits per second 30122.86 815.291 811 25828.48 354.591 0

Key Group
A MODEL Rate bits per second Network Usage recv
B LOAD Rate bits per second Network Usage recv
C MODEL Rate bits per second Network Usage sent
D LOAD Rate bits per second Network Usage sent

The following compares the Bias coefficent for the Network resource usage differences from the test started at 2021-07-16 17:25:00 to the tests from previous test sessions.

Class Coef Count Metric Units Estimate Std..Error PrevCount PrevEst PrevSEM pvalue.d
MODEL Bias 31 sent bits per second 146356.9 5643.910 788 683723.7 15118.599 0
MODEL Bias 31 recv bits per second 118244.8 5351.244 788 476971.3 9805.339 0
LOAD Bias 31 sent bits per second 381222.7 8081.151 811 257233.7 6650.782 0
LOAD Bias 31 recv bits per second 233115.3 10338.395 733 172717.9 3993.857 0

Key Group
A MODEL Bias bits per second Network Usage sent
B MODEL Bias bits per second Network Usage recv
C LOAD Bias bits per second Network Usage sent
D LOAD Bias bits per second Network Usage recv

Increases in resource usage

The following show the comparisons of the tests performed on 2021-07-16 as compared to the tests performed before this date. The tables are ranked in increasing order of the \(p\)-values from the corresponding Welch Modified Two-Sample t-Test (greater), starting from server group where the resource usage distribution increases are the greatest. Results are only shown for which the \(p\)-value is less than or equal to the cutoff value (\(\alpha\) = 0.05).

Test 1 - TermAppISO

Resource class: CPU

There are no significant increases for the Rate coefficient for the test that started at CPU when compared to the 2021-07-16 17:25:00 resource usage from previous pooled test results.

The following compares the Bias coefficent for the CPU resource usage increases from the test started at 2021-07-16 17:25:00 to the tests from previous test sessions.

Class Coef Count Metric Units Estimate Std..Error PrevCount PrevEst PrevSEM pvalue.g
NETWORK Bias 31 cpu.cpu.system.user Percent single processor 13.326 5.802 179 10.996 2.047 0.017

Key Group
A NETWORK Bias Percent single processor CPU Usage

Resource class: Network

The following compares the Rate coefficent for the Network resource usage increases from the test started at 2021-07-16 17:25:00 to the tests from previous test sessions.

Class Coef Count Metric Units Estimate Std..Error PrevCount PrevEst PrevSEM pvalue.g
LOAD Rate 31 recv bits per second 23741.77 1043.020 811 15152.32 261.287 0
LOAD Rate 31 sent bits per second 30122.86 815.291 811 25828.48 354.591 0

Key Group
A LOAD Rate bits per second Network Usage recv
B LOAD Rate bits per second Network Usage sent

The following compares the Bias coefficent for the Network resource usage increases from the test started at 2021-07-16 17:25:00 to the tests from previous test sessions.

Class Coef Count Metric Units Estimate Std..Error PrevCount PrevEst PrevSEM pvalue.g
LOAD Bias 31 recv bits per second 233115.3 10338.395 733 172717.9 3993.857 0
LOAD Bias 31 sent bits per second 381222.7 8081.151 811 257233.7 6650.782 0

Key Group
A LOAD Bias bits per second Network Usage recv
B LOAD Bias bits per second Network Usage sent

Decreases in resource usage

The following show the comparisons of the tests performed on 2021-07-16 as compared to the tests performed before this date. The tables are ranked in increasing order of the \(p\)-values from the corresponding Welch Modified Two-Sample t-Test (less), starting from server group where the resource usage distribution decreases are the greatest. Results are only shown for which the \(p\)-value is less than or equal to the cutoff value (\(\alpha\) = 0.05).

Test 1 - TermAppISO

Resource class: CPU

The following compares the Rate coefficent for the CPU resource usage decreases from the test started at 2021-07-16 17:25:00 to the tests from previous test sessions.

Class Coef Count Metric Units Estimate Std..Error PrevCount PrevEst PrevSEM pvalue.l
LOAD Rate 31 cpu.cpu.system.user Percent single processor 1.059 0.029 811 5.108 0.126 0
MODEL Rate 31 cpu.cpu.system.user Percent single processor 1.632 0.220 789 3.510 0.135 0

Key Group
A LOAD Rate Percent single processor CPU Usage
B MODEL Rate Percent single processor CPU Usage

The following compares the Bias coefficent for the CPU resource usage decreases from the test started at 2021-07-16 17:25:00 to the tests from previous test sessions.

Class Coef Count Metric Units Estimate Std..Error PrevCount PrevEst PrevSEM pvalue.l
LOAD Bias 31 cpu.cpu.system.user Percent single processor 1.648 0.283 789 62.381 2.425 0
MODEL Bias 31 cpu.cpu.system.user Percent single processor 23.169 2.176 811 134.208 2.502 0

Key Group
A LOAD Bias Percent single processor CPU Usage
B MODEL Bias Percent single processor CPU Usage

Resource class: Network

The following compares the Rate coefficent for the Network resource usage decreases from the test started at 2021-07-16 17:25:00 to the tests from previous test sessions.

Class Coef Count Metric Units Estimate Std..Error PrevCount PrevEst PrevSEM pvalue.l
MODEL Rate 31 recv bits per second 12907.06 539.876 810 20029.89 626.740 0
MODEL Rate 31 sent bits per second 13989.53 569.403 735 17315.39 335.728 0

Key Group
A MODEL Rate bits per second Network Usage recv
B MODEL Rate bits per second Network Usage sent

The following compares the Bias coefficent for the Network resource usage decreases from the test started at 2021-07-16 17:25:00 to the tests from previous test sessions.

Class Coef Count Metric Units Estimate Std..Error PrevCount PrevEst PrevSEM pvalue.l
MODEL Bias 31 sent bits per second 146356.9 5643.910 788 683723.7 15118.599 0
MODEL Bias 31 recv bits per second 118244.8 5351.244 788 476971.3 9805.339 0

Key Group
A MODEL Bias bits per second Network Usage sent
B MODEL Bias bits per second Network Usage recv

Comparison of resource usage across tests

This section compares the resource usage between the NFT tests to date for each of the groups/servers in the classification list for which data has been collected.

In the box-plots that follow, in each case, the centre is the estimated value of the coefficient calculated for the particular test. The lower edge of the box is the corresponding estimated value less the standard error, and the upper edge of the box is the corresponding estimated value plus the standard error. The minimum and maximum values are calculated by taking two times the standard error values in a similar manner.

CPU Resource usage for LOAD by CPU Usage using cpu.cpu.system.user in Percent single processor

Server OSType Description
LOAD0 Linux CML EcoSystem
LOAD1 Linux CML EcoSystem

The following table shows the Rate coefficient values over the various tests performed. In this table the units of estimate are: Percent single processor.

TestDate Coef Estimate Std..Error t.value p.value Test
2020-12-07 Rate 4.852 0.356 13.627 0 TermAppISO Integrator Routing Regression
2020-12-08 Rate 5.021 0.271 18.501 0 TermAppISO Integrator Routing Regression
2020-12-09 Rate 7.444 0.602 12.360 0 TermAppISO Int Routing Regression
2020-12-09 Rate 5.138 0.277 18.561 0 TermAppISO
2020-12-16 Rate 9.780 0.574 17.034 0 TermAppISO Integrator Routing Changes
2020-12-16 Rate 10.198 0.584 17.447 0 TermAppISO with 40 provider threads
2020-12-16 Rate 8.606 0.718 11.993 0 TermAppISO with 80 provider threads
2021-01-15 Rate 0.973 0.022 44.250 0 TermAppISO
2021-05-10 Rate 0.680 0.066 10.297 0 TermAppISO
2021-05-20 Rate 0.710 0.013 52.850 0 TermAppISO
2021-06-28 Rate 0.834 0.035 23.721 0 TermAppISO
2021-06-29 Rate 0.963 0.014 66.737 0 TermAppISO
2021-07-16 Rate 1.059 0.029 37.102 0 TermAppISO

The following table shows the Bias coefficient values over the various tests performed. In this table the units of estimate are: Percent single processor.

TestDate Coef Estimate Std..Error t.value p.value Test
2020-12-07 Bias 59.012 7.451 7.921 0.000 TermAppISO Integrator Routing Regression
2020-12-08 Bias 60.847 5.668 10.735 0.000 TermAppISO Integrator Routing Regression
2020-12-09 Bias 124.874 11.476 10.882 0.000 TermAppISO Int Routing Regression
2020-12-09 Bias 57.803 5.740 10.071 0.000 TermAppISO
2020-12-16 Bias 110.050 10.485 10.496 0.000 TermAppISO Integrator Routing Changes
2020-12-16 Bias 101.540 10.390 9.773 0.000 TermAppISO with 40 provider threads
2020-12-16 Bias 124.248 13.091 9.491 0.000 TermAppISO with 80 provider threads
2021-01-15 Bias 4.653 0.462 10.075 0.000 TermAppISO
2021-05-20 Bias 3.599 0.216 16.641 0.000 TermAppISO
2021-06-28 Bias 1.532 0.539 2.841 0.006 TermAppISO
2021-06-29 Bias 1.399 0.205 6.822 0.000 TermAppISO
2021-07-16 Bias 1.648 0.283 5.824 0.000 TermAppISO

Network Resource usage for LOAD by Network Usage recv using recv in bits per second

Server OSType Description
LOAD0 Linux CML EcoSystem
LOAD1 Linux CML EcoSystem

The following table shows the Rate coefficient values over the various tests performed. In this table the units of estimate are: bits per second.

TestDate Coef Estimate Std..Error t.value p.value Test
2020-12-07 Rate 15663.585 818.945 19.127 0 TermAppISO Integrator Routing Regression
2020-12-08 Rate 15753.089 745.948 21.118 0 TermAppISO Integrator Routing Regression
2020-12-09 Rate 8938.586 692.304 12.911 0 TermAppISO Int Routing Regression
2020-12-09 Rate 19602.564 1989.670 9.852 0 TermAppISO
2020-12-16 Rate 12978.364 542.369 23.929 0 TermAppISO Integrator Routing Changes
2020-12-16 Rate 12795.930 585.131 21.868 0 TermAppISO with 40 provider threads
2020-12-16 Rate 11544.987 748.371 15.427 0 TermAppISO with 80 provider threads
2021-01-15 Rate 23224.409 297.261 78.128 0 TermAppISO
2021-05-10 Rate 14748.670 1755.623 8.401 0 TermAppISO
2021-05-20 Rate 14621.637 272.552 53.647 0 TermAppISO
2021-06-28 Rate 10629.033 511.557 20.778 0 TermAppISO
2021-06-29 Rate 23164.964 373.138 62.082 0 TermAppISO
2021-07-16 Rate 23741.774 1043.020 22.763 0 TermAppISO

The following table shows the Bias coefficient values over the various tests performed. In this table the units of estimate are: bits per second.

TestDate Coef Estimate Std..Error t.value p.value Test
2020-12-07 Bias 117112.32 17136.631 6.834 0.000 TermAppISO Integrator Routing Regression
2020-12-08 Bias 128316.44 15580.341 8.236 0.000 TermAppISO Integrator Routing Regression
2020-12-09 Bias 164916.40 13191.587 12.502 0.000 TermAppISO Int Routing Regression
2020-12-09 Bias 79903.07 41255.594 1.937 0.056 TermAppISO
2020-12-16 Bias 124584.24 9904.489 12.579 0.000 TermAppISO Integrator Routing Changes
2020-12-16 Bias 123844.15 10400.926 11.907 0.000 TermAppISO with 40 provider threads
2020-12-16 Bias 140398.04 13653.452 10.283 0.000 TermAppISO with 80 provider threads
2021-01-15 Bias 456044.32 6245.241 73.023 0.000 TermAppISO
2021-05-10 Bias 177202.72 54997.316 3.222 0.004 TermAppISO
2021-05-20 Bias 153604.59 4390.310 34.987 0.000 TermAppISO
2021-06-28 Bias 150601.54 7845.304 19.196 0.000 TermAppISO
2021-06-29 Bias 154516.43 5300.697 29.150 0.000 TermAppISO
2021-07-16 Bias 233115.34 10338.395 22.549 0.000 TermAppISO

Network Resource usage for LOAD by Network Usage sent using sent in bits per second

Server OSType Description
LOAD0 Linux CML EcoSystem
LOAD1 Linux CML EcoSystem

The following table shows the Rate coefficient values over the various tests performed. In this table the units of estimate are: bits per second.

TestDate Coef Estimate Std..Error t.value p.value Test
2020-12-07 Rate 19028.66 995.145 19.121 0 TermAppISO Integrator Routing Regression
2020-12-08 Rate 19042.75 934.235 20.383 0 TermAppISO Integrator Routing Regression
2020-12-09 Rate 30219.17 2364.677 12.779 0 TermAppISO Int Routing Regression
2020-12-09 Rate 19579.56 958.369 20.430 0 TermAppISO
2020-12-16 Rate 32193.80 653.690 49.249 0 TermAppISO Integrator Routing Changes
2020-12-16 Rate 31183.83 720.365 43.289 0 TermAppISO with 40 provider threads
2020-12-16 Rate 29416.80 1431.859 20.544 0 TermAppISO with 80 provider threads
2021-01-15 Rate 26987.33 242.256 111.400 0 TermAppISO
2021-05-10 Rate 19612.35 1623.789 12.078 0 TermAppISO
2021-05-20 Rate 21117.29 377.348 55.962 0 TermAppISO
2021-06-28 Rate 28204.25 1788.951 15.766 0 TermAppISO
2021-06-29 Rate 29243.75 436.441 67.005 0 TermAppISO
2021-07-16 Rate 30122.86 815.291 36.947 0 TermAppISO

The following table shows the Bias coefficient values over the various tests performed. In this table the units of estimate are: bits per second.

TestDate Coef Estimate Std..Error t.value p.value Test
2020-12-07 Bias 204158.9 20823.674 9.804 0.000 TermAppISO Integrator Routing Regression
2020-12-08 Bias 212375.4 19513.027 10.884 0.000 TermAppISO Integrator Routing Regression
2020-12-09 Bias 139054.3 45057.989 3.086 0.003 TermAppISO Int Routing Regression
2020-12-09 Bias 206257.3 19871.668 10.379 0.000 TermAppISO
2020-12-16 Bias 109170.1 11937.368 9.145 0.000 TermAppISO Integrator Routing Changes
2020-12-16 Bias 114274.4 12804.747 8.924 0.000 TermAppISO with 40 provider threads
2020-12-16 Bias 136247.9 26123.167 5.216 0.000 TermAppISO with 80 provider threads
2021-01-15 Bias 430030.4 5089.620 84.492 0.000 TermAppISO
2021-05-10 Bias 413513.8 50867.430 8.129 0.000 TermAppISO
2021-05-20 Bias 389686.3 6078.379 64.110 0.000 TermAppISO
2021-06-28 Bias 544271.4 27435.564 19.838 0.000 TermAppISO
2021-06-29 Bias 393824.0 6199.974 63.520 0.000 TermAppISO
2021-07-16 Bias 381222.7 8081.151 47.174 0.000 TermAppISO

CPU Resource usage for MODEL by CPU Usage using cpu.cpu.system.user in Percent single processor

Server OSType Description
MODELAPP0 Linux CML EcoSystem
MODELAPP1 Linux CML EcoSystem

The following table shows the Rate coefficient values over the various tests performed. In this table the units of estimate are: Percent single processor.

TestDate Coef Estimate Std..Error t.value p.value Test
2020-12-07 Rate 1.132 0.048 23.545 0.000 TermAppISO Integrator Routing Regression
2020-12-08 Rate 1.239 0.017 74.254 0.000 TermAppISO Integrator Routing Regression
2020-12-09 Rate 7.740 0.829 9.331 0.000 TermAppISO Int Routing Regression
2020-12-09 Rate 1.227 0.015 80.090 0.000 TermAppISO
2020-12-16 Rate 7.307 0.592 12.351 0.000 TermAppISO Integrator Routing Changes
2020-12-16 Rate 7.929 0.581 13.643 0.000 TermAppISO with 40 provider threads
2020-12-16 Rate 6.526 0.746 8.752 0.000 TermAppISO with 80 provider threads
2021-01-15 Rate 1.200 0.129 9.336 0.000 TermAppISO
2021-05-20 Rate 1.473 0.112 13.117 0.000 TermAppISO
2021-06-28 Rate 0.597 0.113 5.266 0.000 TermAppISO
2021-06-29 Rate 0.866 0.385 2.249 0.029 TermAppISO
2021-07-16 Rate 1.632 0.220 7.435 0.000 TermAppISO

The following table shows the Bias coefficient values over the various tests performed. In this table the units of estimate are: Percent single processor.

TestDate Coef Estimate Std..Error t.value p.value Test
2020-12-07 Bias 4.808 1.006 4.779 0 TermAppISO Integrator Routing Regression
2020-12-08 Bias 3.084 0.348 8.849 0 TermAppISO Integrator Routing Regression
2020-12-09 Bias 180.757 15.805 11.437 0 TermAppISO Int Routing Regression
2020-12-09 Bias 3.996 0.318 12.580 0 TermAppISO
2020-12-16 Bias 122.689 10.804 11.356 0 TermAppISO Integrator Routing Changes
2020-12-16 Bias 117.633 10.331 11.386 0 TermAppISO with 40 provider threads
2020-12-16 Bias 151.751 13.604 11.155 0 TermAppISO with 80 provider threads
2021-01-15 Bias 25.053 2.701 9.275 0 TermAppISO
2021-05-10 Bias 129.953 26.227 4.955 0 TermAppISO
2021-05-20 Bias 121.790 1.809 67.324 0 TermAppISO
2021-06-28 Bias 699.995 1.738 402.874 0 TermAppISO
2021-06-29 Bias 26.687 5.467 4.881 0 TermAppISO
2021-07-16 Bias 23.169 2.176 10.646 0 TermAppISO

Network Resource usage for MODEL by Network Usage recv using recv in bits per second

Server OSType Description
MODELAPP0 Linux CML EcoSystem
MODELAPP1 Linux CML EcoSystem

The following table shows the Rate coefficient values over the various tests performed. In this table the units of estimate are: bits per second.

TestDate Coef Estimate Std..Error t.value p.value Test
2020-12-07 Rate 24100.05 954.550 25.248 0.000 TermAppISO Integrator Routing Regression
2020-12-08 Rate 25548.99 271.665 94.046 0.000 TermAppISO Integrator Routing Regression
2020-12-09 Rate 14069.74 920.324 15.288 0.000 TermAppISO Int Routing Regression
2020-12-09 Rate 24302.47 713.257 34.073 0.000 TermAppISO
2020-12-16 Rate 17467.52 683.900 25.541 0.000 TermAppISO Integrator Routing Changes
2020-12-16 Rate 17365.66 680.340 25.525 0.000 TermAppISO with 40 provider threads
2020-12-16 Rate 17074.86 664.909 25.680 0.000 TermAppISO with 80 provider threads
2021-01-15 Rate 24560.17 304.261 80.721 0.000 TermAppISO
2021-05-10 Rate 17389.63 5929.025 2.933 0.008 TermAppISO
2021-05-20 Rate 16631.67 971.810 17.114 0.000 TermAppISO
2021-06-28 Rate 15585.31 6115.750 2.548 0.013 TermAppISO
2021-06-29 Rate 24051.35 1557.272 15.445 0.000 TermAppISO
2021-07-16 Rate 12907.06 539.876 23.907 0.000 TermAppISO

The following table shows the Bias coefficient values over the various tests performed. In this table the units of estimate are: bits per second.

TestDate Coef Estimate Std..Error t.value p.value Test
2020-12-07 Bias 64596.51 19974.205 3.234 0.002 TermAppISO Integrator Routing Regression
2020-12-08 Bias 52159.33 5674.160 9.192 0.000 TermAppISO Integrator Routing Regression
2020-12-09 Bias 158388.92 17536.412 9.032 0.000 TermAppISO Int Routing Regression
2020-12-09 Bias 67665.26 14789.314 4.575 0.000 TermAppISO
2020-12-16 Bias 128789.60 12489.057 10.312 0.000 TermAppISO Integrator Routing Changes
2020-12-16 Bias 130831.54 12093.291 10.819 0.000 TermAppISO with 40 provider threads
2020-12-16 Bias 132636.80 11969.911 11.081 0.000 TermAppISO with 80 provider threads
2021-01-15 Bias 462642.65 6392.297 72.375 0.000 TermAppISO
2021-05-20 Bias 177120.97 15654.052 11.315 0.000 TermAppISO
2021-06-28 Bias 3552601.09 93791.875 37.877 0.000 TermAppISO
2021-06-29 Bias 180897.77 22122.205 8.177 0.000 TermAppISO
2021-07-16 Bias 118244.76 5351.244 22.097 0.000 TermAppISO

Network Resource usage for MODEL by Network Usage sent using sent in bits per second

Server OSType Description
MODELAPP0 Linux CML EcoSystem
MODELAPP1 Linux CML EcoSystem

The following table shows the Rate coefficient values over the various tests performed. In this table the units of estimate are: bits per second.

TestDate Coef Estimate Std..Error t.value p.value Test
2020-12-07 Rate 23083.902 924.847 24.960 0.000 TermAppISO Integrator Routing Regression
2020-12-08 Rate 24600.259 251.866 97.672 0.000 TermAppISO Integrator Routing Regression
2020-12-09 Rate 6362.325 852.572 7.463 0.000 TermAppISO Int Routing Regression
2020-12-09 Rate 23318.892 749.322 31.120 0.000 TermAppISO
2020-12-16 Rate 10756.811 769.527 13.978 0.000 TermAppISO Integrator Routing Changes
2020-12-16 Rate 10172.372 701.912 14.492 0.000 TermAppISO with 40 provider threads
2020-12-16 Rate 8940.326 678.158 13.183 0.000 TermAppISO with 80 provider threads
2021-01-15 Rate 24097.187 241.131 99.934 0.000 TermAppISO
2021-05-10 Rate 18230.092 7038.605 2.590 0.018 TermAppISO
2021-05-20 Rate 18196.215 1111.117 16.377 0.000 TermAppISO
2021-06-29 Rate 25419.663 2207.053 11.517 0.000 TermAppISO
2021-07-16 Rate 13989.533 569.403 24.569 0.000 TermAppISO

The following table shows the Bias coefficient values over the various tests performed. In this table the units of estimate are: bits per second.

TestDate Coef Estimate Std..Error t.value p.value Test
2020-12-07 Bias 186452.2 19352.666 9.634 0 TermAppISO Integrator Routing Regression
2020-12-08 Bias 167394.2 5260.624 31.820 0 TermAppISO Integrator Routing Regression
2020-12-09 Bias 331861.2 16245.425 20.428 0 TermAppISO Int Routing Regression
2020-12-09 Bias 187742.4 15537.099 12.083 0 TermAppISO
2020-12-16 Bias 307108.4 14052.742 21.854 0 TermAppISO Integrator Routing Changes
2020-12-16 Bias 291049.7 12476.750 23.327 0 TermAppISO with 40 provider threads
2020-12-16 Bias 400012.4 12208.426 32.765 0 TermAppISO with 80 provider threads
2021-01-15 Bias 365642.3 5065.982 72.176 0 TermAppISO
2021-05-20 Bias 314872.1 17898.032 17.593 0 TermAppISO
2021-06-28 Bias 4491325.5 152240.008 29.502 0 TermAppISO
2021-06-29 Bias 320788.4 31352.822 10.232 0 TermAppISO
2021-07-16 Bias 146356.9 5643.910 25.932 0 TermAppISO

Network Resource usage for NETWORK by Network Usage recv using recv in bits per second

Server OSType Description
NETWORK Linux CML EcoSystem

The following table shows the Rate coefficient values over the various tests performed. In this table the units of estimate are: bits per second.

TestDate Coef Estimate Std..Error t.value p.value Test
2021-05-20 Rate 6289.748 592.594 10.614 0 TermAppISO
2021-06-28 Rate 2711.282 448.421 6.046 0 TermAppISO

The following table shows the Bias coefficient values over the various tests performed. In this table the units of estimate are: bits per second.

TestDate Coef Estimate Std..Error t.value p.value Test
2021-05-10 Bias 1617885.6 328037.628 4.932 0 TermAppISO
2021-05-20 Bias 651687.7 9991.711 65.223 0 TermAppISO
2021-06-28 Bias 674006.3 6877.032 98.008 0 TermAppISO
2021-06-29 Bias 629183.3 163624.078 3.845 0 TermAppISO

Network Resource usage for NETWORK by Network Usage sent using sent in bits per second

Server OSType Description
NETWORK Linux CML EcoSystem

The following table shows the Rate coefficient values over the various tests performed. In this table the units of estimate are: bits per second.

TestDate Coef Estimate Std..Error t.value p.value Test
2021-06-29 Rate 939.561 362.238 2.594 0.013 TermAppISO

The following table shows the Bias coefficient values over the various tests performed. In this table the units of estimate are: bits per second.

TestDate Coef Estimate Std..Error t.value p.value Test
2021-05-10 Bias 6171231.4 1472966.74 4.190 0.000 TermAppISO
2021-06-28 Bias 250236.8 90104.11 2.777 0.007 TermAppISO

CPU Resource usage for NETWORK by CPU Usage using cpu.cpu.system.user in Percent single processor

Server OSType Description
NETWORK Linux CML EcoSystem

The following table shows the Rate coefficient values over the various tests performed. In this table the units of estimate are: Percent single processor.

TestDate Coef Estimate Std..Error t.value p.value Test
2021-05-20 Rate 1.139 0.294 3.868 0 TermAppISO
2021-06-29 Rate 2.785 0.197 14.135 0 TermAppISO

The following table shows the Bias coefficient values over the various tests performed. In this table the units of estimate are: Percent single processor.

TestDate Coef Estimate Std..Error t.value p.value Test
2021-05-20 Bias 12.834 4.965 2.585 0.012 TermAppISO
2021-06-28 Bias 9.329 2.628 3.550 0.001 TermAppISO
2021-06-29 Bias 11.455 2.799 4.093 0.000 TermAppISO
2021-07-16 Bias 13.326 5.802 2.297 0.029 TermAppISO

Session details

sessionInfo();
## R version 3.6.0 (2019-04-26)
## Platform: x86_64-redhat-linux-gnu (64-bit)
## Running under: CentOS Linux 7 (Core)
## 
## Matrix products: default
## BLAS/LAPACK: /usr/lib64/R/lib/libRblas.so
## 
## locale:
##  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
##  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
##  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
##  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
##  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
## 
## attached base packages:
## [1] grid      stats     graphics  grDevices utils     datasets  methods  
## [8] base     
## 
## other attached packages:
## [1] pander_0.6.3    doBy_4.6.7      cmlrutils_1.18  XML_3.98-1.20  
## [5] scales_1.1.1    ggplot2_3.3.2   BSDA_1.2.0      lattice_0.20-38
## 
## loaded via a namespace (and not attached):
##  [1] Rcpp_1.0.5       highr_0.8        compiler_3.6.0   pillar_1.4.6    
##  [5] rmdformats_1.0.0 class_7.3-15     tools_3.6.0      digest_0.6.25   
##  [9] evaluate_0.14    lifecycle_0.2.0  tibble_3.0.3     gtable_0.3.0    
## [13] pkgconfig_2.0.3  rlang_0.4.7      Matrix_1.2-17    yaml_2.2.1      
## [17] xfun_0.17        e1071_1.7-4      withr_2.2.0      stringr_1.4.0   
## [21] dplyr_1.0.2      knitr_1.30       generics_0.0.2   vctrs_0.3.2     
## [25] tidyselect_1.1.0 glue_1.4.1       R6_2.4.1         rmarkdown_2.6   
## [29] bookdown_0.20    farver_2.0.3     tidyr_1.1.2      purrr_0.3.4     
## [33] magrittr_1.5     backports_1.1.8  MASS_7.3-51.4    ellipsis_0.3.1  
## [37] htmltools_0.5.0  colorspace_1.4-1 Deriv_4.0.1      labeling_0.3    
## [41] stringi_1.5.3    munsell_0.5.0    broom_0.7.0      crayon_1.3.4