3) Stress Test

The next goal is to "break" the cluster. By gradually increasing the number of replicas of the deployment that sends requests to the php-apache application, one finds the breaking point, which obviously grows as the flavor of the VMs grows.

StressTest (xLarge)

$ kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=200
$ kubectl scale deploy infinite-calls --replicas=50

$ kubectl get deploy
NAME             READY     UP-TO-DATE   AVAILABLE   AGE
infinite-calls   50/50     50           50          6h6m
php-apache       112/127   127          112         22h

$ kubectl get hpa --watch
NAME         REFERENCE               TARGETS    MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   0%/50%     1         200       1          11m
php-apache   Deployment/php-apache   78%/50%    1         200       1          12m
php-apache   Deployment/php-apache   78%/50%    1         200       2          12m
php-apache   Deployment/php-apache   249%/50%   1         200       2          13m
php-apache   Deployment/php-apache   249%/50%   1         200       4          13m
php-apache   Deployment/php-apache   249%/50%   1         200       8          13m
php-apache   Deployment/php-apache   249%/50%   1         200       10         13m
php-apache   Deployment/php-apache   221%/50%   1         200       10         14m
php-apache   Deployment/php-apache   221%/50%   1         200       14         14m
php-apache   Deployment/php-apache   223%/50%   1         200       14         15m
php-apache   Deployment/php-apache   223%/50%   1         200       28         15m
php-apache   Deployment/php-apache   223%/50%   1         200       56         15m
php-apache   Deployment/php-apache   174%/50%   1         200       56         15m
php-apache   Deployment/php-apache   174%/50%   1         200       77         16m
php-apache   Deployment/php-apache   85%/50%    1         200       77         17m
php-apache   Deployment/php-apache   85%/50%    1         200       127        17m
php-apache   Deployment/php-apache   57%/50%    1         200       127        18m
php-apache   Deployment/php-apache   58%/50%    1         200       127        19m
php-apache   Deployment/php-apache   59%/50%    1         200       127        20m
php-apache   Deployment/php-apache   57%/50%    1         200       127        21m
php-apache   Deployment/php-apache   59%/50%    1         200       127        22m
php-apache   Deployment/php-apache   58%/50%    1         200       127        23m
php-apache   Deployment/php-apache   59%/50%    1         200       127        24m
php-apache   Deployment/php-apache   58%/50%    1         200       127        25m
php-apache   Deployment/php-apache   59%/50%    1         200       127        27m
php-apache   Deployment/php-apache   60%/50%    1         200       127        28m
php-apache   Deployment/php-apache   59%/50%    1         200       127        29m
php-apache   Deployment/php-apache   59%/50%    1         200       127        30m
php-apache   Deployment/php-apache   59%/50%    1         200       127        31m
php-apache   Deployment/php-apache   0%/50%     1         200       127        32m
php-apache   Deployment/php-apache   0%/50%     1         200       127        35m
php-apache   Deployment/php-apache   0%/50%     1         200       1          35m

StressTest (large)

$ kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=100
$ kubectl scale deploy infinite-calls --replicas=30

$ kubectl get deploy
NAME             READY   UP-TO-DATE   AVAILABLE   AGE
infinite-calls   30/30   30           30          25h
php-apache       52/85   85           52          25h

$ kubectl get hpa --watch
NAME         REFERENCE               TARGETS    MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   0%/50%     1         100       1          45s
php-apache   Deployment/php-apache   250%/50%   1         100       1          78s
php-apache   Deployment/php-apache   250%/50%   1         100       4          93s
php-apache   Deployment/php-apache   250%/50%   1         100       5          108s
php-apache   Deployment/php-apache   235%/50%   1         100       5          2m19s
php-apache   Deployment/php-apache   235%/50%   1         100       10         2m35s
php-apache   Deployment/php-apache   235%/50%   1         100       19         2m50s
php-apache   Deployment/php-apache   212%/50%   1         100       19         3m21s
php-apache   Deployment/php-apache   212%/50%   1         100       38         3m36s
php-apache   Deployment/php-apache   212%/50%   1         100       43         3m51s
php-apache   Deployment/php-apache   99%/50%    1         100       43         4m22s
php-apache   Deployment/php-apache   99%/50%    1         100       58         4m38s
php-apache   Deployment/php-apache   86%/50%    1         100       58         5m9s
php-apache   Deployment/php-apache   86%/50%    1         100       85         5m24s
php-apache   Deployment/php-apache   84%/50%    1         100       85         7m12s
php-apache   Deployment/php-apache   86%/50%    1         100       85         8m15s
php-apache   Deployment/php-apache   87%/50%    1         100       85         9m17s
php-apache   Deployment/php-apache   83%/50%    1         100       85         10m
php-apache   Deployment/php-apache   87%/50%    1         100       85         11m
php-apache   Deployment/php-apache   85%/50%    1         100       85         13m
php-apache   Deployment/php-apache   86%/50%    1         100       85         14m
php-apache   Deployment/php-apache   85%/50%    1         100       85         15m
php-apache   Deployment/php-apache   87%/50%    1         100       85         16m
php-apache   Deployment/php-apache   86%/50%    1         100       85         17m
php-apache   Deployment/php-apache   86%/50%    1         100       85         18m
php-apache   Deployment/php-apache   85%/50%    1         100       85         20m
php-apache   Deployment/php-apache   0%/50%     1         100       85         21m
php-apache   Deployment/php-apache   0%/50%     1         100       85         26m
php-apache   Deployment/php-apache   0%/50%     1         100       1          26m

StressTest (medium)

$ kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=50
$ kubectl scale deploy infinite-calls --replicas=15

$ kubectl get deploy
NAME             READY   UP-TO-DATE   AVAILABLE   AGE
infinite-calls   15/15   15           15          5h4m
php-apache       22/42   42           22          5h6m

$ kubectl get hpa --watch
NAME         REFERENCE               TARGETS    MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   0%/50%     1         50        1          49m
php-apache   Deployment/php-apache   250%/50%   1         50        1          50m
php-apache   Deployment/php-apache   250%/50%   1         50        4          50m
php-apache   Deployment/php-apache   250%/50%   1         50        5          51m
php-apache   Deployment/php-apache   212%/50%   1         50        5          51m
php-apache   Deployment/php-apache   212%/50%   1         50        9          51m
php-apache   Deployment/php-apache   197%/50%   1         50        9          52m
php-apache   Deployment/php-apache   197%/50%   1         50        18         52m
php-apache   Deployment/php-apache   197%/50%   1         50        36         52m
php-apache   Deployment/php-apache   100%/50%   1         50        36         53m
php-apache   Deployment/php-apache   100%/50%   1         50        42         53m
php-apache   Deployment/php-apache   99%/50%    1         50        42         54m
php-apache   Deployment/php-apache   102%/50%   1         50        42         55m
php-apache   Deployment/php-apache   102%/50%   1         50        42         56m
php-apache   Deployment/php-apache   101%/50%   1         50        42         57m
php-apache   Deployment/php-apache   100%/50%   1         50        42         58m
php-apache   Deployment/php-apache   99%/50%    1         50        42         59m
php-apache   Deployment/php-apache   101%/50%   1         50        42         60m
php-apache   Deployment/php-apache   100%/50%   1         50        42         61m
php-apache   Deployment/php-apache   98%/50%    1         50        42         63m
php-apache   Deployment/php-apache   100%/50%   1         50        42         64m
php-apache   Deployment/php-apache   101%/50%   1         50        42         65m
php-apache   Deployment/php-apache   97%/50%    1         50        42         66m
php-apache   Deployment/php-apache   101%/50%   1         50        42         67m
php-apache   Deployment/php-apache   100%/50%   1         50        42         68m
php-apache   Deployment/php-apache   0%/50%     1         50        42         69m
php-apache   Deployment/php-apache   0%/50%     1         50        42         74m
php-apache   Deployment/php-apache   0%/50%     1         50        1          74m

We can deduce from these graphs, observing the ordinate axis, the percentage of CPU used. The maximum CPU available is given by the sum of the CPUs of the worker nodes alone, which bear all the workload (it is preferable not to perform operations on the master to protect it from any crashes). We have respectively 24, 12 and 6 CPUs available based on flavor and, on average, a percentage equal to approximately 115%, 79% and 82% of use.

Page tree

3) Stress Test