How to Deploy a custom O-RAN xApp

2021/03/15

The official O-RAN documentation on how to write/compile/deploy an xApp (the xApp Writer’s Guide in the version from June 9, 2020) is partially useful, partially misleading to lacking. For instance, it describes an old way to deploy custom xApps, which does not work anymore. Also, there is the installation guide of the hello world xApp (hwxapp), but I did not find it very helpful. The On-boarding and Deploying xApps is also quite confusing for somebody who does not know yet how to deploy an application and without any links to further information. All in all, I stumbled over multiple problems along the way of deploying my custom xApp, running into problems because of out-dated information or incomplete tutorials. I wrote this article as a reference for myself, and hopefully it can also help others, as the road to set everything up was quite rocky.

Note that all information is relative to Ubuntu 18.04, being the only distribution and version officially supported by O-RAN-SC.

How to compile and run standalone

The documentation of the hwxapp says to simply follow the Dockerfile, but if you have less experience with Docker like me, you might easily miss that information, although it is helpful. Therefore, I put the installation steps below for clarity.

First, install some basic necessary packages:

$ apt-get update cmake git build-essential automake autoconf-archive autoconf pkg-config gawk libtool wget zlib1g-dev libffi-dev

Then, clone the repository (and make sure to use the cherry release)

$ git clone "https://gerrit.o-ran-sc.org/r/ric-app/hw"
$ cd hw
$ git checkout cherry

Proceed by installing the dependencies

At this point, all the necessary dependencies are installed, and it should be possible to compile (the project only has a single Makefile).

$ cd hw/src
$ make

How to configure and run the xApp

An xApp relies on environment variables to be configured and set up correctly before being started. There are two files that seem to be related here and that are present in the cherry branch of hwxapp: run_xapp.sh (which does not actually run the application) and xapp_env.sh. Both set some environment variables, and there is at least two that are critical: DBAAS_SERVICE_HOST and RMR_SEED_RT.

DBAAS_SERVICE_HOST is the host name or IP for Database as a service. There is not much documentation to find, and the best place seems to be the repository that tells you how to connect a test application to the Redis cluster if you want to try it. Anyway, the dbaas is a service that provides a database for storage, and that contains the gNB-List, i.e., the list of connected gNBs. It is important to have it set correctly to the right service, or otherwise the xApp fails when retrieving the list of base stations with:

GNB List in R-NIB {\"gnb_list\":null,\"error_msg\":\"dial tcp 127.0.0.1:6379: connect: connection refused\"}

So what I did was finding the pod IP address using kubectl get pods -nricplt -o wide and putting it for this variable.

If you load the environment variables and start the xApp, it will load the gNB-List (assuming you connected something), but the subscription it sends will not reach the subscription manager. The whole O-RAN RIC uses the RMR library, and I invite you to go and read at least the User Guide and the Route Table Guide. In brief, RMR is a message router based on message types, and it uses routing tables to determine where to send a message. The routing tables are typically distributed by the rtmgr, but it is also possible to supply one statically.

So the RMR_SEED_RT points to a routing table, and its contents is perfectly explained in the RMR documentation. In fact, what is sent in the beginning is a subscription request (ID 12010), but it will actually come back as is visible in the output of the xApp. So what we need to modify is the IP of the endpoint for the message ID 12010 in the routing table. Look up the submgr’s IP address in the k8s cluster as for dbaas before, and add the IP address in the line of the message 12010 in the routing table.

All the IDs and their mapping to messages are documented in the rmr repository in file src/rmr/common/include/RIC_message_types.h, or if you installed it via the rmr-dev deb package, it is probably under /usr/local/include/rmr/RIC_message_types.h.

So now it it should be possible to retrieve the gNB-List, send a subscription, and hopefully have an interaction with the RAN function below. I also merged the two environment variable exporting scripts into one, coming up with this run_xapp.sh:

#! /bin/bash

set -x
export RMR_SEED_RT="routes.txt"
export RMR_RTG_SVC="9999"
export HW_PORTS="4560"
export MSG_MAX_BUFFER="2072"
export THREADS="1"
export VERBOSE="1"
export CONFIG_FILE="config/config-file.json"
export XAPP_NAME="HELLOWORLD_XAPP"
export XAPP_ID="3489-er492k-92389"
export LOG_LEVEL="MDCLOG_INFO"
export DBAAS_SERVICE_HOST=$(kubectl describe pods -nricplt statefulset-ricplt-dbaas-server-0 | awk '/IP:/{print $2; exit}')
export DBAAS_SERVICE_PORT="6379"
#export GNODEB="NYC123"
#export A1_SCHEMA_FILE="schemas/hwxapp-policy.json"
#export VES_SCHEMA_FILE="schemas/hwxapp-ves.json"
#export VES_COLLECTOR_URL="127.0.0.1:6350"
#export VES_MEASUREMENT_INTERVAL="10"
#export LOG_LEVEL="MDCLOG_ERR"
#export OPERATING_MODE="CONTROL"

./hw_xapp_main

Why just compiling and running apparently is not enough

At this point, I thought I could run the xApp directly, also since this is mentioned in the repository of hwxapp. However, this does not seem to be enough: the application starts, can read the gNB-List, and send a subscription. However, the subscription manager complains with the following message

{"ts":1615839344149,"crit":"ERROR","id":"submgr","mdc":{"time":"2021-03-15T20:15:44"},"msg":"XAPP-SubReq: transxapp(trans(0/meid(RanName=gnb_734_733_b5c67788))/transkey(pyroclaste:4560/)/0) err(CREATE routeinfo(1/[pyroclaste:4560]) failed with error: [POST /handles/xapp-subscription-handle][400] provideXappSubscriptionHandleBadRequest )"}

(to see that, find the submgr in the output of kubectl get pods -nricplt, then do kubectl logs -nricplt <pod>)

Some further digging revealed that the routing manager did not know the endpoint:

{"ts":1615839565788,"crit":"ERROR","id":"rtmgr","mdc":{"time":"2021-03-15T20:19:25"},"msg":"XApp instance not found: pyroclaste:4560"}

so my interpretation is that the subscription manager cannot put a new route (towards where?) for this xApp, as the routing manager itself does not know it. Maybe if kubernetes could resolve pyroclaste to the host, it might work, or it is maybe possible to push custom routes to the routing manager.

The bottom line seems to be that if such application is not properly set up, ideally using the xAppManager, then there will be no routing info and the app cannot be reached by the other components, making it impossible to practically use it. If somebody knows more, please let me know, but therefore I proceeded to put the app into a container to deploy it “properly”.

How to run the docker file and put into a registry

As mentioned in the documentation, the Dockerfile contains the build instructions to build a container. However, in order to allow Kubernetes (triggered from the xAppManager) to download it, we need to have it in a registry; this does not seem to be mentioned in the documentation, and a major point of confusion for me. The registry could be the docker hub, but since

  1. I do everything on the same machine
  2. the resulting image will be large (~1GB) and
  3. I planned to probably push more than once,

I thought it would be best to keep it within a local registry.

For reasons that have to do with the verification of JSON (the JSON schema) when onboarding an application (I know, this comes from far), the registry’s name needs to be an URL, or at least consist of alphabetical characters with a point between them (e.g., a.b). Therefore, the first thing would be to make such a name, and I added localhost.eur to my /etc/hosts, so the first line there reads:

127.0.0.1 localhost localhost.eur

Next, we need to install such a local registry (no, you don’t need a docker account, at least I don’t have one). So go ahead and install using

$ docker run -d -p 5000:5000 --name registry registry:2.7

This runs a new container named “registry”, which is a registry of version 2.7. -d means to run in detached mode, and the internal port 5000 is mapped to 5000 on the host machine. Note that the registry is not restarted upon reboot, in which case you would do docker start registry (and you can stop it with stop).

Now, go back to the hwxapp and build the container using

$ docker build --tag localhost.eur:5000/robert-fantastic-hwxapp:v1.0.1 .

Note that we specify the localhost.eur:5000 prefix as the registry’s location. Then we have the name of the xapp (you can chose any, will be important later for onboarding) and also any version you like (if you don’t put it, it will be latest!).

Once it has been built, just push to the registry like so:

$ docker push localhost.eur:5000/robert-fantastic-hwxapp:v1.0.1

and verify it is in the registry with

$ curl localhost.eur:5000/v2/_catalog

How to deploy

Note that the xApp Writer’s Guide talks about having your own chartmuseum (a Helm chart repository), pushing information with some command line tool, and other stuff like JSON schemas etc… Forget all of that, you don’t need it, at least not for the Cherry release of O-RAN-SC RIC, since the xAppOnboarder has its own internal chartmuseum, and already has the JSON schema.

The only thing that needs to be done at this stage is to onboard as any other pre-compiled xApp) using a descriptor that has to refer to the new xApp in the local registry. The xApp Writer’s Guide covers this section by section, but for reference, here is the descriptor that I have used:

{
  "config-file.json": {
    "xapp_name": "robert-fantastic-hwxapp",
    "version": "1.0.1",
    "containers": [
      {
        "name": "robert-fantastic-hwxapp",
        "image": {
          "registry": "localhost.eur:5000",
          "name": "robert-fantastic-hwxapp",
          "tag": "v1.0.1"
        }
      }
    ],
    "messaging": {
      "ports": [
        {
          "name": "rmr-data",
          "container": "hwxapp",
          "port": 4560,
          "rxMessages": [
            "RIC_SUB_RESP",
            "A1_POLICY_REQ",
            "RIC_HEALTH_CHECK_REQ"
          ],
          "txMessages": [
            "RIC_SUB_REQ",
            "A1_POLICY_RESP",
            "A1_POLICY_QUERY",
            "RIC_HEALTH_CHECK_RESP"
          ],
          "policies": [
            1
          ],
          "description": "rmr receive data port for HWxapp"
        },
        {
          "name": "rmr-route",
          "container": "hwxapp",
          "port": 4561,
          "description": "rmr route port for hwxapp"
        }
      ]
    },
    "rmr": {
      "protPort": "tcp:4560",
      "maxSize": 2072,
      "numWorkers": 1,
      "txMessages": [
        "RIC_SUB_REQ",
        "A1_POLICY_RESP",
        "A1_POLICY_QUERY",
        "RIC_HEALTH_CHECK_RESP"
      ],
      "rxMessages": [
        "RIC_SUB_RESP",
        "A1_POLICY_REQ",
        "RIC_HEALTH_CHECK_REQ"
      ],
      "policies": [
        1
      ]
    }
  }
}

You can give any xapp_name, but note that it uses the registry localhost.eur:5000, the name of the container, and the tag. Also, if the registry name did not follow the schema a.b, you would get an “Input payload validation failed” error at the next step. Finally, the descriptor also defines some of the RMR routes that have to be set up, which are not setup automatically if not passing by the xAppManager (remember the issue I mentioned after compiling). Now, push the descriptor, and verify that it has been pushed:

$ curl -L -XPOST "http://10.101.27.2:32080/onboard/api/v1/onboard" -H 'Content-Type: application/json' -d @descriptor.json
$ curl -X GET "http://10.101.27.2:32080/onboard/api/v1/charts" | jq

(for the IP address, use kubectl get pods -nricplt -o wide to get the xapp-onboarder’s IP address, or do as in my previous blogpost).

Finally, you can simply deploy/undeploy the xApp like so (use the above call to get the appmgr’s IP address)

$ curl -XPOST "http://10.101.27.2:32080/appmgr/ric/v1/xapps" -d '{"xappName": "robert-fantastic-hwxapp"}' -H 'Content-Type: application/json'
$ curl -L -XDELETE 'http://10.101.27.2:32080/appmgr/ric/v1/xapps/robert-fantastic-hwxapp'

And that’s it, the app should have been deployed!

Acknowledgment

Thanks a lot to Alireza for helping me find my way through Kubernetes and Docker!