...
S1.2 - Burn the image to a USB stick:
Info |
---|
Be sure to use the correct USB device name when copying the image. Code Block |
---|
language | bash |
---|
theme | Midnight |
---|
| dd if=inf-image-centos-all-x86-64.iso of=/dev/sdc bs=1M |
|
...
- Install Kubernetes Platform on All-in-one Simplex:
- When install controller-0 from the image, please select ‘All-in-one (lowlatency) Controller Configuration’Image Modified
S2.2 In addition to required StarlingX configuration, you must set up the Ceph backend for Kubernetes PVC, isolcpus and hugepages:
Code Block |
---|
|
source /etc/platform/openrc
NODE=controller-0
OAM_IF=<OAM-PORT>
# if you use flat oam network
system host-if-modify ${NODE} $OAM_IF -c platform
system interface-network-assign ${NODE} $OAM_IF oam
# if you use vlan oam network
VLANID=<VLAN-ID>
system host-if-modify -n pltif -c platform $NODE $OAM_IF
system host-if-add ${NODE} -V $VLANID -c platform oam0 vlan pltif
system interface-network-assign ${NODE} oam0 oam
system host-label-assign $NODE sriovdp=enabled
system host-label-assign $NODE kube-topology-mgr-policy=restricted
# Ceph backend for k8s pvc
system storage-backend-add ceph --confirmed
system host-disk-list ${NODE} | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add ${NODE} {}
# isolate cpus depends on number of the physical core
system host-cpu-modify -f application-isolated -p0 28 $NODE
# allocate/enable hugepages for DPDK usage
system host-memory-modify $NODE -1G 10 0
system host-unlock $NODE |
Code Block |
---|
|
source /etc/platform/openrc
NODE=controller-0
system host-lock $NODE
# get the device name of the Mount Bryce, we assume it is
# pci_0000_8a_00_0 here.
system host-device-list $NODE
# Modify the Mount Bryce device to enable it, specify the base driver
# and vf driver, and configure it for 1 VFs
# NOTE: If this is the initial install and have not unlocked, you will
# get following error message.
# Cannot configure device 73b13ddf-99be-44c8-8fbe-db85eb8d99ba until host
# controller-0 is unlocked for the first time.
system host-device-modify $NODE pci_0000_8a_00_0 -e true --driver igb_uio --vf-driver vfio -N 1
system host-unlock $NODE |
...
The following steps provide a quick-start procedure for developers.
S3.1 Create a PVC for FlexRAN build storage:
Info |
---|
Note: ThePVCsize should be larger than 70G. |
Code Block |
---|
|
cat > volume-ceph.yaml << 'EOF'
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: flexran-storage
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 80Gi
storageClassName: general
EOF
kubectl create -f volume-ceph.yaml
kubectl create -f volume-ceph.yaml
persistentvolumeclaim/flexran-storage created
controller-0:~$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
flexran-storage Bound pvc-43e50806-785f-440b-8ed2-85bb3c9e8f79 80Gi RWO general 9s |
S3.2 Instructions for FlexRAN building image creation
Code Block |
---|
|
mkdir dockerbuilder && cd dockerbuilder
# prepare the artifacts used for FlexRAN prebuilt binary Docker image
mkdir docker-image-building
cat > docker-image-building/readme << 'EOF'
# Instructions of Docker image generation
# Following steps are supposed to be executed inside building Pod,
# after building FlexRAN from source code
flxr_install_dir=/opt/fb/flexran/
# populate flexran related env var
cd ${flxr_install_dir}
source set_env_var.sh -d
# prepare the FlexRAN binaries
./transport.sh
# build the Docker image
docker build -t flr-run -f Dockerfile .
# tag and push
orgname=somename
docker tag flr-run ${orgname}/flr-run
EOF
cat > docker-image-building/transport.sh << 'EOF'
#!/bin/bash
# ICXPATH=/opt/fb/intel/oneapi/
echo "Make sure source setvars.sh first.(located in ICX oneapi installation directory)"
echo "Make sure source set_env_var.sh -d first.(located in FlexRAN installation directory)"
[[ -z "$MKLROOT" ]] && { echo "MKLROOT not set, exit..."; exit 1; }
[[ -z "$IPPROOT" ]] && { echo "MKLROOT not set, exit..."; exit 1; }
[[ -z "$CMPLR_ROOT" ]] && { echo "MKLROOT not set, exit..."; exit 1; }
[[ -z "$DIR_WIRELESS_SDK_ROOT" ]] && { echo "DIR_WIRELESS_SDK_ROOT not set, exit..."; exit 1; }
FLXPATH=`echo $DIR_WIRELESS_SDK_ROOT| awk -F '/sdk' '{print $1}'`
[[ -d stuff ]] && { echo "Directory stuff exists, move it to old."; mv -f stuff stuff.old; }
mkdir stuff; cd stuff
mkdir libs
cp -a $MKLROOT/lib/intel64/libmkl_intel_lp64.so* libs
cp -a $MKLROOT/lib/intel64/libmkl_core.so* libs
cp -a $MKLROOT/lib/intel64/libmkl_intel_thread.so* libs
cp -a $MKLROOT/lib/intel64/libmkl_avx512.so.* libs
cp -a $MKLROOT/lib/intel64/libmkl_avx2.so* libs
cp -a $MKLROOT/lib/intel64/libmkl_avx.so* libs
cp -a $IPPROOT/lib/intel64/libipps.so* libs
cp -a $IPPROOT/lib/intel64/libippe.so* libs
cp -a $IPPROOT/lib/intel64/libippcore.so* libs
cp -a $IPPROOT/lib/intel64/libippee9.so* libs
cp -a $IPPROOT/lib/intel64/libippse9.so* libs
cp -a $CMPLR_ROOT/linux/compiler/lib/intel64_lin/libiomp5.so* libs
cp -a $CMPLR_ROOT/linux/compiler/lib/intel64_lin/libirc.so* libs
cp -a $CMPLR_ROOT/linux/compiler/lib/intel64_lin/libimf.so* libs
cp -a $CMPLR_ROOT/linux/compiler/lib/intel64_lin/libsvml.so* libs
cp -a $CMPLR_ROOT/linux/compiler/lib/intel64_lin/libintlc.so* libs
cp -a $CMPLR_ROOT/linux/compiler/lib/intel64_lin/libirng.so* libs
cp -a $FLXPATH/libs/cpa/bin/libmmwcpadrv.so* libs
cp -a $FLXPATH/wls_mod/libwls.so* libs
mkdir -p flexran/sdk/build-avx512-icx/
cp -rf $FLXPATH/sdk/build-avx512-icx/source flexran/sdk/build-avx512-icx/
cp -rf $FLXPATH/sdk/build-avx512-icx/install flexran/sdk/build-avx512-icx/
cp -rf $FLXPATH/bin flexran/
cp -rf $FLXPATH/set_env_var.sh flexran/
# testcase files
mkdir -p tests/nr5g/
cd tests/nr5g/
for cfg in $FLXPATH/bin/nr5g/gnb/testmac/icelake-sp/*.cfg
do
cat $cfg | grep TEST_FD > /tmp/$$.testfile
while IFS= read line
do
array=($(echo "$line" | sed 's/5GNR,/ /g'))
for i in "${array[@]}"; do
if [[ "$i" =~ \.cfg ]]; then
casedir=`echo "$i"| cut -d / -f 1-3 | xargs`
caseabsdir=$FLXPATH/tests/nr5g/$casedir
[[ ! -d $casedir ]] && { mkdir -p $casedir; cp -rf $caseabsdir/* $casedir; }
fi
done
done < /tmp/$$.testfile
done
echo "Transportation Completed."
EOF
chmod a+x docker-image-building/transport.sh
cat > docker-image-building/set-l1-env.sh << 'EOF'
# source this script to l1 binary location
export WORKSPACE=/root/flexran
export isa=avx512
cd $WORKSPACE
source ./set_env_var.sh -i ${isa}
MODE=$1
[[ -z "$MODE" ]] && read -p "Enter the MODE(LTE or 5G): " MODE
if [ $MODE = LTE ]; then
cd $WORKSPACE/bin/lte/l1/
fi
if [ $MODE = 5G ]; then
cd $WORKSPACE/bin/nr5g/gnb/l1
fi
EOF
cat > docker-image-building/set-l2-env.sh << 'EOF'
# source this script to l2 binary location
export WORKSPACE=/root/flexran
export isa=avx512
cd $WORKSPACE
source ./set_env_var.sh -i ${isa}
MODE=$1
[[ -z "$MODE" ]] && read -p "Enter the MODE(LTE or 5G): " MODE
if [ $MODE = LTE ]; then
cd $WORKSPACE/bin/lte/testmac/
fi
if [ $MODE = 5G ]; then
cd $WORKSPACE/bin/nr5g/gnb/testmac
fi
EOF
cat > docker-image-building/res-setup.sh << 'EOF'
#!/bin/bash
[[ -z "$PCIDEVICE_INTEL_COM_INTEL_ACC100_FEC" ]] && { echo "ACC100 not used, sleep..."; sleep infinity; }
sed -i 's#.*dpdkBasebandFecMode.*# <dpdkBasebandFecMode\>1</dpdkBasebandFecMode>#' /root/flexran/bin/nr5g/gnb/l1/phycfg_timer.xml
sed -i 's#.*dpdkBasebandDevice.*# <dpdkBasebandDevice\>'"$PCIDEVICE_INTEL_COM_INTEL_ACC100_FEC"'</dpdkBasebandDevice>#' /root/flexran/bin/nr5g/gnb/l1/phycfg_timer.xml
echo "Resource setup Completed, sleep..."
sleep infinity
EOF
chmod a+x docker-image-building/res-setup.sh
mkdir docker-image-building/rootdir
mv docker-image-building/res-setup.sh docker-image-building/rootdir
mv docker-image-building/set-l1-env.sh docker-image-building/rootdir
mv docker-image-building/set-l2-env.sh docker-image-building/rootdir
cat > docker-image-building/Dockerfile << 'EOF'
FROM centos:7.9.2009
RUN [ -e /etc/yum.conf ] && sed -i '/tsflags=nodocs/d' /etc/yum.conf || true
RUN yum install -y libhugetlbfs* libstdc++* numa* gcc g++ iproute \
module-init-tools kmod pciutils python libaio libaio-devel \
numactl-devel nettools ethtool
RUN yum clean all
COPY stuff/libs/* /usr/lib64/
WORKDIR /root/
COPY stuff/flexran ./flexran
COPY stuff/tests ./flexran/tests
COPY rootdir/* ./
CMD ["/root/res-setup.sh"]
EOF
cat > Dockerfile << 'EOF'
FROM centos:7.9.2009
RUN [ -e /etc/yum.conf ] && sed -i '/tsflags=nodocs/d' /etc/yum.conf || true
RUN yum groupinstall -y 'Development Tools'
RUN yum install -y vim gcc-c++ libhugetlbfs* libstdc++* kernel-devel numa* gcc git mlocate \
cmake wget ncurses-devel hmaccalc zlib-devel binutils-devel elfutils-libelf-devel \
numactl-devel libhugetlbfs-devel bc patch git patch tar zip unzip python3 sudo docker
RUN yum install -y gtk3 mesa-libgbm at-spi2-core libdrm xdg-utils libxcb libnotify
RUN yum install -y centos-release-scl
RUN yum install -y devtoolset-8
RUN yum clean all
RUN pip3 install meson && \
pip3 install ninja pyelftools
# ENV HTTP_PROXY=""
# ENV HTTPS_PROXY=""
WORKDIR /usr/src/
RUN git clone https://github.com/pkgconf/pkgconf.git
WORKDIR /usr/src/pkgconf
RUN ./autogen.sh && ./configure && make && make install
WORKDIR /usr/src/
RUN git clone git://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git
WORKDIR /usr/src/rt-tests
RUN git checkout stable/v1.0
RUN make all && make install
COPY docker-image-building /root/docker-image-building
WORKDIR /opt
# Set default command
CMD ["/usr/bin/bash"]
EOF
# build the Docker image for FlexRAN building environment
sudo docker build -t flexran-builder .
sudo docker tag flexran-builder registry.local:9001/flexran-builder:22.03
# push to registry.local:9001
# user: admin
sudo docker login registry.local:9001 sudo-u dockeradmin push registry-p <your_sysadmin_passwd>
sudo docker push registry.local:9001/flexran-builder:22.03
|
S3.3 Launch the building pod attaching to the PVC:
Info |
---|
Note: This pod is assumed to be assigned enough resources to launch quickly after FlexRAN is built. If you don’t have isolated CPU, hugepage and accelerator resources configured as part of the system used for building, feel free to remove related content from the yaml spec file. Hugepages-1Gi and intel.com/intel_acc100_fec are not required to perform the build. |
Code Block |
---|
|
cat > flexran-buildpod.yml << 'EOF'
apiVersion: v1
kind: Pod
metadata:
name: buildpod
annotations:
spec:
restartPolicy: Never
containers:
- name: buildpod
image: registry.local:9001/flexran-builder:22.03
imagePullPolicy: IfNotPresent
volumeMounts:
- name: usrsrc
mountPath: /usr/src
- mountPath: /hugepages
name: hugepage
- name: lib-modules
mountPath: /lib/modules
- name: pvc1
mountPath: /opt/fb
- name: docker-sock-volume
mountPath: /var/run/docker.sock
command: ["/bin/bash", "-ec", "sleep infinity"]
securityContext:
privileged: true
capabilities:
add:
["IPC_LOCK", "SYS_ADMIN"]
resources:
requests:
memory: 32Gi
hugepages-1Gi: 10Gi
intel.com/intel_acc100_fec: '1'
limits:
memory: 32Gi
intel.com/intel_acc100_fec: '1'
hugepages-1Gi: 10Gi
volumes:
- name: usrsrc
hostPath:
path: /usr/src
- name: lib-modules
hostPath:
path: /lib/modules
- name: hugepage
emptyDir:
medium: HugePages
- name: docker-sock-volume
hostPath:
path: /var/run/docker.sock
type: Socket
- name: pvc1
persistentVolumeClaim:
claimName: flexran-storage
EOF
kubectl create -f flexran-buildpod.yml |
S4 Build FlexRAN in Pod
S4.1 Use a shell inside Pod to build FlexRAN:
Code Block |
---|
|
kubectl exec -it buildpod -- bash |
Code Block |
---|
|
mkdir -p /opt/fb/scratch && cd /opt/fb/scratch
scp <options> FlexRAN-22.03-L1.tar.gz_part00 .
scp <options> FlexRAN-22.03-L1.tar.gz_part01 .
scp <options> dpdk_patch-22.03.patch .
cat FlexRAN-22.03-L1.tar.gz_part00 FlexRAN-22.03-L1.tar.gz_part01 > FlexRAN-22.03-L1.tar.gz
rm FlexRAN-22.03-L1.tar.gz_part00
rm FlexRAN-22.03-L1.tar.gz_part01
|
...
S5 Generate Docker image with FlexRAN binaries
Info |
---|
Note: Since host path/var/run/docker.sock has been mounted into the building pod, you can build the Docker image using the FlexRAN binaries from the previous step inside the building pod. The artifacts used by docker build have been integrated into the build image and are ready to use. |
S5.1 Prepare the env var for the script in /root/docker-image-building/transport.sh
:
Code Block |
---|
|
source /opt/fb/intel/oneapi/setvars.sh
cd /opt/fb/flexran && source ./set_env_var.sh -d |
S5.2 Prepare binaries and scripts for Docker build:
Code Block |
---|
|
cd /root/docker-image-building
./transport.sh |
S5.3 Build Docker image which will be saved in local host:
Code Block |
---|
|
docker build -t flr-run -f Dockerfile . |
S6 Run the FlexRAN Test cases in Pod
After the build and Docker image generation steps above, you can launch the FlexRAN execution pod from the host.
S6.1 Push the Docker image to registry.local:9001
Code Block |
---|
|
# change to host side, in this case, it should be controller-0 host
sudo docker login registry.local:9001 -u admin -p <your_sysadmin_passwd>
sudo docker tag flr-run registry.local:9001/flxrun:22.03
sudo docker push registry.local:9001/flxrun:22.03
|
S6.2 Launch the FlexRAN Pod.