Function Test

Introduction

Function test is conducted to verify the functionality of the Non-RT RIC as well as the individual components.

A test engine and a number of test script is used as a foundation for the test environment.  

The test environment uses docker images both the Non-RT RIC components as well as for the the needed simulators.

Function test is normally started in "docker-mode" meaning that images will be started as container in a private docker network. As an option, the same test case also executed in a minikube environment. Running in minikube is still in an experimental state so docker should be used as a the standard execution environment.

The basic idea of function test is to run the Non-RT RIC components and test the exposed interfaces using simulators. The test engine and the test scripts controls the behaviour of the simulators but also use the exposed interfaces for test and monitoring. 

The test scripts are designed in a way so that the same script can be used for testing of the functionality in the master branch as well as previous release(s). The supported branches/releases are indicated by the existence of a environment variable file in the test/common dir with the name of the branch. The environment for master has the name of coming release. 

The majority of the test scripts test one or several aspects of the functionality but there is also test scripts for stability tests over longer period of time (currently 3 days max intensity test).

Test setup

The picture below shows the setup used for function tests. Non-RT RIC components in orange, simulators in light blue and the test environment in dark blue. The purple arrow is outside the scope of the test but is used for demo test scripts.

Note: The Control Panel GUI consist of two parts, control panel and the gateway for accessing the nonrtric components.

Note: When running in kubernetes, the test engine uses a proxy (not shown) for accessing the nonrtric components.

Note: The helm manager, Dmaap Adaptor and Dmaap Mediator components are not yet included in the function test setup.

Note: Enrichement Coordinator has changed name to Information Coordinator

Test scripts/Test cases

The below table is a list of test cases. All scripts beginning with the letters "FTC" (Function Test Case) are considered test scripts to test functionality. The remaining test scripts are primarily used for demo purposes.
Note: Control Panel and its gateway are started in most test cases but not tested.

Test case

Tested 
apps

Slogan

FTC1 

PMS

Sanity test, create service and then create, update and delete a policy using http/https and Agent REST/DMAAP with/without SDNC controller

FTC10  

PMS​

Basic use case, register service, create/update policy, delete policy, de-register service using both STD and OSC interface while mixing REST and Dmaap

FTC100  

PMS​

Full agent API walkthrough using agent REST/DMAAP and with/without SDNC A1 Controller

FTC110  

PMS​

Testing of service registration timeouts and keepalive

FTC150  

SDNC​

Sample tests of the SDNC A1 controller restconf API using http/https (no PMS)

FTC300 

PMS​

Resync 10000 policies using OSC and STD interface

FTC310 

PMS​

Resync of RIC via changes in the consul config or pushed config

FTC350 

PMS​

Change supported policy types and reconfigure rics

FTC800  

PMS​

Create 10000 policies in sequence using http/https and Agent REST/DMAAP with/without SDNC controller

FTC810

PMS​

Repeatedly create and delete policies in each RICs for 24h (or configured number of days). Via agent REST/DMAAP/DMAAP_BATCH and SDNC using http or https

FTC850

PMS​

Create/delete policies in parallel over a number of ric using a number of child process

FTC900

PMS​

Preparation for test of the Control Panel and the Health Check app - populating a number of rics simulators with types and instances"

FTC1100 

ICS

ICS full interface walkthrough

FTC1800 

ICS

ICS Create 10000 jobs and restart, test job persistency

FTC2001PMS
ICS

Testing southbound proxy for PMS and ICS

FTC2002SDNC

Testing southbound proxy for SDNC - docker only

FTC2003DMAAP-ADP

Testing southbound proxy for Dmaap Adaptor

FTC3000DMAAP-ADP
DMAAP-MED

App test DMAAP Meditor and DMAAP Adapter

FTC3001DMAAP-ADP
DMAAP-MED

App test DMAAP Meditor and DMAAP Adapter with 100 jobs,types and topics

FTC4000HELM-MANAGER

Test of Helm Manager

PM_DEMO

PMS

Preparation demo setup - PMS - populating a number of ric simulators with types and instances

PM_EI_DEMO

PMS
ICS​
RAPP-CATALOGUE

Preparation demo setup - PMS and ICS - policy management and enrichment information

Test execution

All function test scripts can be executed in your local environment. It is proven to work on MacOS and Ubuntu (brief tests has been made on Windows using git bash).

Required environment:

  • Bash shell
  • docker (latest)
  • docker-compose (latest)
  • python3 (latest)
  • minikube (latest) or a local kubernetes cluster - only needed when running in kubernetes mode

The test engine as well as all available test scripts is available in the nonrtric repo. Clone the repo and go to the auto-test directory:

$ git clone "https://gerrit.o-ran-sc.org/r/nonrtric"
$ cd nonrtric/test/autotest

Start the appropriate test script with the command below. No further action is required. The tests are fully automated. Once the execution is completed a test report is printed. 

Example, run images from the staging repo - which is the default. To run only release images, add the parameter 'release' after the 'docker' parameter.

When running with images from the staging or release repo, it is still possible to point to images in the local repo or in the release, staging or snapshot repo for individual applications.

$ ./FTC100.sh remote docker --env-file ../common/test_env-oran-cherry.sh

For further information and details, see README in the dirs 'auto-test' and 'common'.

Tested image versions

Componentimage (release repo)tag for d-release profiletag for d-release profiletag for cherry profile
Policy Management Service
(new image name)

nexus3.o-ran-sc.org:10002/nonrtric-a1-policy-management-service

2.3.1--
Policy Management Service
(old image name)

nexus3.o-ran-sc.org:10002/o-ran-sc/nonrtric-policy-agent

-2.2.02.1.1
A1-controller

nexus3.o-ran-sc.org:10002/o-ran-sc/nonrtric-a1-controller

--2.0.1
SDNC A1 Controller
(Replacement for the
above A1-Controller)
nexus3.onap.org:10002/onap/sdnc-image2.1.62.1.2-
Information Coordinator Service
(Replaces Enrichment Coordinator Service)
 nexus3.o-ran-sc.org:10002/o-ran-sc/nonrtric-information-coordinator-service1.2.1--
Enrichment Coordinator Service nexus3.o-ran-sc.org:10002/o-ran-sc/nonrtric-enrichment-coordinator-service-1.1.01.0.1
Control Panel

nexus3.o-ran-sc.org:10002/o-ran-sc/nonrtric-controlpanel

2.3.02.2.02.1.0
Gatewaynexus3.o-ran-sc.org:10002/o-ran-sc/nonrtric-gateway 1.1.01.1.0-
A1-Simulator

nexus3.o-ran-sc.org:10002/o-ran-sc/a1-simulator

2.2.02.1.0 2.1.0
R-APP Cataloguenexus3.o-ran-sc.org:10002/o-ran-sc/nonrtric-r-app-catalogue1.0.21.0.11.0.1
Dmaap Adaptor

nexus3.o-ran-sc.org:10002/o-ran-sc/nonrtric-dmaap-adaptor

1.0.1--
Dmaap Mediator

nexus3.o-ran-sc.org:10002/o-ran-sc/nonrtric-dmaap-mediator-producer

1.0.1--

"profile" refers to the name of the environment variable profile used when executed a test case. The profiles has the pattern 'test_env-oran-<profile-name>.sh in the test/common dir.