| Table of Contents |
|---|
...
S1.2 - Burn the image to a USB stick:
| Info | |||||||
|---|---|---|---|---|---|---|---|
Be sure to use the correct USB device name when copying the image.
|
...
- Install Kubernetes Platform on All-in-one Simplex:
- When install controller-0 from the image, please select ‘All-in-one (lowlatency) Controller Configuration’
S2.2 In addition to required StarlingX configuration, you must set up the Ceph backend for Kubernetes PVC, isolcpus and hugepages:
| Code Block | ||
|---|---|---|
| ||
source /etc/platform/openrc
NODE=controller-0
OAM_IF=<OAM-PORT>
# if you use flat oam network
system host-if-modify ${NODE} $OAM_IF -c platform
system interface-network-assign ${NODE} $OAM_IF oam
# if you use vlan oam network
VLANID=<VLAN-ID>
system host-if-modify -n pltif -c platform $NODE $OAM_IF
system host-if-add ${NODE} -V $VLANID -c platform oam0 vlan pltif
system interface-network-assign ${NODE} oam0 oam
system host-label-assign $NODE sriovdp=enabled
system host-label-assign $NODE kube-topology-mgr-policy=restricted
# Ceph backend for k8s pvc
system storage-backend-add ceph --confirmed
system host-disk-list ${NODE} | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add ${NODE} {}
# isolate cpus depends on number of the physical core
system host-cpu-modify -f application-isolated -p0 28 $NODE
# allocate/enable hugepages for DPDK usage
system host-memory-modify $NODE -1G 10 0
system host-unlock $NODE |
S2.3 After the system has been unlocked and available for the first time, configure ACC100 / Mount Bryce:
| Code Block | ||
|---|---|---|
| ||
source /etc/platform/openrc NODE=controller-0 system host-lock $NODE # get the device name of the Mount Bryce, we assume it is # pci_0000_8a_00_0 here. system host-device-list $NODE # Modify the Mount Bryce device to enable it, specify the base driver # and vf driver, and configure it for 1 VFs # NOTE: If this is the initial install and have not unlocked, you will # get following error message. # Cannot configure device 73b13ddf-99be-44c8-8fbe-db85eb8d99ba until host # controller-0 is unlocked for the first time. system host-device-modify $NODE pci_0000_8a_00_0 -e true --driver igb_uio --vf-driver vfio -N 1 system host-unlock $NODE |
...
The following steps provide a quick-start procedure for developers.
Create a PVC for FlexRAN build storage:
Info Note: ThePVCsize should be larger than 70G.
Code Block theme Midnight cat > volume-ceph.yaml << 'EOF' kind: PersistentVolumeClaim apiVersion: v1 metadata: name: flexran-storage spec: accessModes: - ReadWriteOnce resources: requests: storage: 80Gi storageClassName: general EOF kubectl create -f volume-ceph.yaml kubectl create -f volume-ceph.yaml persistentvolumeclaim/flexran-storage created controller-0:~$ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE flexran-storage Bound pvc-43e50806-785f-440b-8ed2-85bb3c9e8f79 80Gi RWO general 9sInstructions for FlexRAN building image creation
Code Block theme Midnight mkdir dockerbuilder && cd dockerbuilder # prepare the artifacts used for FlexRAN prebuilt binary Docker image mkdir docker-image-building cat > docker-image-building/readme << 'EOF' # Instructions of Docker image generation # Following steps are supposed to be executed inside building Pod, # after building FlexRAN from source code flxr_install_dir=/opt/fb/flexran/ # populate flexran related env var cd ${flxr_install_dir} source set_env_var.sh -d # prepare the FlexRAN binaries ./transport.sh # build the Docker image docker build -t flr-run -f Dockerfile . # tag and push orgname=somename docker tag flr-run ${orgname}/flr-run EOF cat > docker-image-building/transport.sh << 'EOF' #!/bin/bash # ICXPATH=/opt/fb/intel/oneapi/ echo "Make sure source setvars.sh first.(located in ICX oneapi installation directory)" echo "Make sure source set_env_var.sh -d first.(located in FlexRAN installation directory)" [[ -z "$MKLROOT" ]] && { echo "MKLROOT not set, exit..."; exit 1; } [[ -z "$IPPROOT" ]] && { echo "MKLROOT not set, exit..."; exit 1; } [[ -z "$CMPLR_ROOT" ]] && { echo "MKLROOT not set, exit..."; exit 1; } [[ -z "$DIR_WIRELESS_SDK_ROOT" ]] && { echo "DIR_WIRELESS_SDK_ROOT not set, exit..."; exit 1; } FLXPATH=`echo $DIR_WIRELESS_SDK_ROOT| awk -F '/sdk' '{print $1}'` [[ -d stuff ]] && { echo "Directory stuff exists, move it to old."; mv -f stuff stuff.old; } mkdir stuff; cd stuff mkdir libs cp -a $MKLROOT/lib/intel64/libmkl_intel_lp64.so* libs cp -a $MKLROOT/lib/intel64/libmkl_core.so* libs cp -a $MKLROOT/lib/intel64/libmkl_intel_thread.so* libs cp -a $MKLROOT/lib/intel64/libmkl_avx512.so.* libs cp -a $MKLROOT/lib/intel64/libmkl_avx2.so* libs cp -a $MKLROOT/lib/intel64/libmkl_avx.so* libs cp -a $IPPROOT/lib/intel64/libipps.so* libs cp -a $IPPROOT/lib/intel64/libippe.so* libs cp -a $IPPROOT/lib/intel64/libippcore.so* libs cp -a $IPPROOT/lib/intel64/libippee9.so* libs cp -a $IPPROOT/lib/intel64/libippse9.so* libs cp -a $CMPLR_ROOT/linux/compiler/lib/intel64_lin/libiomp5.so* libs cp -a $CMPLR_ROOT/linux/compiler/lib/intel64_lin/libirc.so* libs cp -a $CMPLR_ROOT/linux/compiler/lib/intel64_lin/libimf.so* libs cp -a $CMPLR_ROOT/linux/compiler/lib/intel64_lin/libsvml.so* libs cp -a $CMPLR_ROOT/linux/compiler/lib/intel64_lin/libintlc.so* libs cp -a $CMPLR_ROOT/linux/compiler/lib/intel64_lin/libirng.so* libs cp -a $FLXPATH/libs/cpa/bin/libmmwcpadrv.so* libs cp -a $FLXPATH/wls_mod/libwls.so* libs mkdir -p flexran/sdk/build-avx512-icx/ cp -rf $FLXPATH/sdk/build-avx512-icx/source flexran/sdk/build-avx512-icx/ cp -rf $FLXPATH/sdk/build-avx512-icx/install flexran/sdk/build-avx512-icx/ cp -rf $FLXPATH/bin flexran/ cp -rf $FLXPATH/set_env_var.sh flexran/ # testcase files mkdir -p tests/nr5g/ cd tests/nr5g/ for cfg in $FLXPATH/bin/nr5g/gnb/testmac/icelake-sp/*.cfg do cat $cfg | grep TEST_FD > /tmp/$$.testfile while IFS= read line do array=($(echo "$line" | sed 's/5GNR,/ /g')) for i in "${array[@]}"; do if [[ "$i" =~ \.cfg ]]; then casedir=`echo "$i"| cut -d / -f 1-3 | xargs` caseabsdir=$FLXPATH/tests/nr5g/$casedir [[ ! -d $casedir ]] && { mkdir -p $casedir; cp -rf $caseabsdir/* $casedir; } fi done done < /tmp/$$.testfile done echo "Transportation Completed." EOF chmod a+x docker-image-building/transport.sh cat > docker-image-building/set-l1-env.sh << 'EOF' # source this script to l1 binary location export WORKSPACE=/root/flexran export isa=avx512 cd $WORKSPACE source ./set_env_var.sh -i ${isa} MODE=$1 [[ -z "$MODE" ]] && read -p "Enter the MODE(LTE or 5G): " MODE if [ $MODE = LTE ]; then cd $WORKSPACE/bin/lte/l1/ fi if [ $MODE = 5G ]; then cd $WORKSPACE/bin/nr5g/gnb/l1 fi EOF cat > docker-image-building/set-l2-env.sh << 'EOF' # source this script to l2 binary location export WORKSPACE=/root/flexran export isa=avx512 cd $WORKSPACE source ./set_env_var.sh -i ${isa} MODE=$1 [[ -z "$MODE" ]] && read -p "Enter the MODE(LTE or 5G): " MODE if [ $MODE = LTE ]; then cd $WORKSPACE/bin/lte/testmac/ fi if [ $MODE = 5G ]; then cd $WORKSPACE/bin/nr5g/gnb/testmac fi EOF cat > docker-image-building/res-setup.sh << 'EOF' #!/bin/bash [[ -z "$PCIDEVICE_INTEL_COM_INTEL_ACC100_FEC" ]] && { echo "ACC100 not used, sleep..."; sleep infinity; } sed -i 's#.*dpdkBasebandFecMode.*# <dpdkBasebandFecMode\>1</dpdkBasebandFecMode>#' /root/flexran/bin/nr5g/gnb/l1/phycfg_timer.xml sed -i 's#.*dpdkBasebandDevice.*# <dpdkBasebandDevice\>'"$PCIDEVICE_INTEL_COM_INTEL_ACC100_FEC"'</dpdkBasebandDevice>#' /root/flexran/bin/nr5g/gnb/l1/phycfg_timer.xml echo "Resource setup Completed, sleep..." sleep infinity EOF chmod a+x docker-image-building/res-setup.sh mkdir docker-image-building/rootdir mv docker-image-building/res-setup.sh docker-image-building/rootdir mv docker-image-building/set-l1-env.sh docker-image-building/rootdir mv docker-image-building/set-l2-env.sh docker-image-building/rootdir cat > docker-image-building/Dockerfile << 'EOF' FROM centos:7.9.2009 RUN [ -e /etc/yum.conf ] && sed -i '/tsflags=nodocs/d' /etc/yum.conf || true RUN yum install -y libhugetlbfs* libstdc++* numa* gcc g++ iproute \ module-init-tools kmod pciutils python libaio libaio-devel \ numactl-devel nettools ethtool RUN yum clean all COPY stuff/libs/* /usr/lib64/ WORKDIR /root/ COPY stuff/flexran ./flexran COPY stuff/tests ./flexran/tests COPY rootdir/* ./ CMD ["/root/res-setup.sh"] EOF cat > Dockerfile << 'EOF' FROM centos:7.9.2009 RUN [ -e /etc/yum.conf ] && sed -i '/tsflags=nodocs/d' /etc/yum.conf || true RUN yum groupinstall -y 'Development Tools' RUN yum install -y vim gcc-c++ libhugetlbfs* libstdc++* kernel-devel numa* gcc git mlocate \ cmake wget ncurses-devel hmaccalc zlib-devel binutils-devel elfutils-libelf-devel \ numactl-devel libhugetlbfs-devel bc patch git patch tar zip unzip python3 sudo docker RUN yum install -y gtk3 mesa-libgbm at-spi2-core libdrm xdg-utils libxcb libnotify RUN yum install -y centos-release-scl RUN yum install -y devtoolset-8 RUN yum clean all RUN pip3 install meson && \ pip3 install ninja pyelftools # ENV HTTP_PROXY="" # ENV HTTPS_PROXY="" WORKDIR /usr/src/ RUN git clone https://github.com/pkgconf/pkgconf.git WORKDIR /usr/src/pkgconf RUN ./autogen.sh && ./configure && make && make install WORKDIR /usr/src/ RUN git clone git://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git WORKDIR /usr/src/rt-tests RUN git checkout stable/v1.0 RUN make all && make install COPY docker-image-building /root/docker-image-building WORKDIR /opt # Set default command CMD ["/usr/bin/bash"] EOF # build the Docker image for FlexRAN building environment orgname=somename sudo docker build -t flexran-builder . sudo docker tag flexran-builder ${orgname}/flexran-builder:22.03 sudo docker login sudo docker push ${orgname}/flexran-builder:22.03Launch the quick start building Pod attaching to the PVC:
Info Note: This pod is assumed to be assigned enough resources to launch quickly after FlexRAN is built. If you don’t have isolated CPU, hugepage and accelerator resources configured as part of the system used for building, feel free to remove related content from the yaml spec file. Hugepages-1Gi and intel.com/intel_acc100_fec are not required to perform the build.Code Block theme Midnight cat > flexran-buildpod.yml << 'EOF' apiVersion: v1 kind: Pod metadata: name: buildpod annotations: spec: restartPolicy: Never containers: - name: buildpod image: wrsnfv/flexran-builder:22.03 imagePullPolicy: IfNotPresent volumeMounts: - name: usrsrc mountPath: /usr/src - mountPath: /hugepages name: hugepage - name: lib-modules mountPath: /lib/modules - name: pvc1 mountPath: /opt/fb - name: docker-sock-volume mountPath: /var/run/docker.sock command: ["/bin/bash", "-ec", "sleep infinity"] securityContext: privileged: true capabilities: add: ["IPC_LOCK", "SYS_ADMIN"] resources: requests: memory: 32Gi hugepages-1Gi: 10Gi intel.com/intel_acc100_fec: '1' limits: memory: 32Gi intel.com/intel_acc100_fec: '1' hugepages-1Gi: 10Gi volumes: - name: usrsrc hostPath: path: /usr/src - name: lib-modules hostPath: path: /lib/modules - name: hugepage emptyDir: medium: HugePages - name: docker-sock-volume hostPath: path: /var/run/docker.sock type: Socket - name: pvc1 persistentVolumeClaim: claimName: flexran-storage EOF kubectl create -f flexran-buildpod.yml
...