while scaling for large no of indications /seconds (throughput). The xApp rmr library is crashing, that causes xApp to crash.

While scaling indication throughput, I am facing run time error in SIsendt function  in si95/sisendt.c file (when my application sends health check response to near RT RIC, from xApp framework ). This is happening  when the indication throughput to my xApp reaches around 2000-3000 msg /second. I am getting stack smashing detected run time error.

The maximum E2AP indication payload size is 1368 Bytes (as seen in Wireshark packet capture) .

Let say , if I roughly consider all packets to have this maximum payload , then the maximum throughput before xApp crashing  comes out to be  1368(bytes)2500(no of message/second)=3.25 Mb/second{*}

Below i the xApp binary dump log 

 
** *stack smashing detected ***: terminated
AddressSanitizer:DEADLYSIGNAL
=================================================================
==1==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7f041cd22941 bp 0x7f04197fdde0 sp 0x7f04197fdb80 T1)
==1==The signal is caused by a READ memory access.
==1==Hint: address points to the zero page.
Messages per second: 978
    #0 0x7f041cd22940 in __GI_abort /build/glibc-LcI20x/glibc-2.31/stdlib/abort.c:106
    #1 0x7f041cd8d26d in __libc_message ../sysdeps/posix/libc_fatal.c:155
    #2 0x7f041ce2fcd9 in _GI__fortify_fail /build/glibc-LcI20x/glibc-2.31/debug/fortify_fail.c:26
    #3 0x7f041ce2fca5 in __stack_chk_fail /build/glibc-LcI20x/glibc-2.31/debug/stack_chk_fail.c:24
    #4 0x7f041dc03cd7 in SIsendt /w/workspace/ric-plt-lib-rmr-rt-cmake-packagecloud-stage-master/src/rmr/si/src/si95/sisendt.c:117 
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /build/glibc-LcI20x/glibc-2.31/stdlib/abort.c:106 in __GI_abort
Thread T1 created by T0 here:
    #0 0x7f041e3ed805 in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x3a805)
    #1 0x7f041dc00540 in init /w/workspace/ric-plt-lib-rmr-rt-cmake-packagecloud-stage-master/src/rmr/si/src/rmr_si.c:812
    #2 0x7f041dc00614 in rmr_init /w/workspace/ric-plt-lib-rmr-rt-cmake-packagecloud-stage-master/src/rmr/si/src/rmr_si.c:843
    #3 0x7f041e015b68 in xapp::Messenger::Messenger(char const*, bool) /w/workspace/ric-plt-xapp-frame-cpp-rt-cmake-packagecloud-stage-master/src/messaging/messenger.cpp:75
    #4 0x7f041e023899 in Xapp::Xapp(char const*, bool) /w/workspace/ric-plt-xapp-frame-cpp-rt-cmake-packagecloud-stage-master/src/xapp/xapp.cpp:57
    #5 0x5636caea3156 in main /lite/src/main.cpp:643
    #6 0x7f041cd24082 in __libc_start_main ../csu/libc-start.c:308

As of now I fixed this issue by commenting the xApp default health check callback in xApp c++ framework .

Now my xApp facing run time error in SIwait function in si95/siwait.c file. This is happening  when the indication throughput to my xApp reaches around 2500-3000 msg /second. I am getting segmentation fault.

I have also tried allocating more computing resources to xApp (4 cpu , 12 gb ram  just for my  xApp pod only). I also  added cpu affinity functionality   in c++ xApp framework  to run  4 receiving threads in 4 different cores  . But the problem is persistent.