Atlassian uses cookies to improve your browsing experience, perform analytics and research, and conduct advertising. Accept all cookies to indicate that you agree to our use of cookies on your device. Atlassian cookies and tracking notice, (opens new window)
Crash in E2T buildPrometheusList Function
Description
Hello, during my testing of the E2T, I encountered the following crash in the E2T. The entire log is quite large as it contains non-related traffic, but the relevant part is provided below. I set the log level to "Debug" and used gdb to trace the root cause for the error.
{"ts":1706046263587,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737236219648","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-z78rk","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"Start EPOLL Wait. Timeout = -1"}
{"ts":1706046263784,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737236219648","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-z78rk","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"handling epoll event 1 out of 1"}
{"ts":1706046263784,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737236219648","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-z78rk","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"new message from SCTP, epoll flags are : 1"}
{"ts":1706046263784,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737236219648","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-z78rk","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"Start Read from SCTP 19 fd"}
{"ts":1706046263784,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737236219648","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-z78rk","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"Start Read from SCTP fd 19 stream 0 "}
{"ts":1706046263784,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737236219648","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-z78rk","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"Finish Read from SCTP 19 fd message length = 7"}
{"ts":1706046263784,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737236219648","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-z78rk","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"Before Encoding E2AP PDU for : gnb_734_373_16b8cef1, Read time is : 0 seconds, 29556 nanoseconds"}
{"ts":1706046263784,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737236219648","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-z78rk","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"PDU buffer length = 7, data = : 0001400"}
{"ts":1706046263784,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737236219648","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-z78rk","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"After Encoding E2AP PDU for : gnb_734_373_16b8cef1, Read time is : 0 seconds, 14486 nanoseconds"}
{"ts":1706046263784,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737236219648","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-z78rk","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"Encoding E2AP PDU past : InitiatingMessage ::= { procedureCode: 1 criticality: 1 (ignore) value: E2setupRequest ::= { protocolIEs: ProtocolIE-Container ::= { } } } "}
{"ts":1706046263784,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737236219648","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-z78rk","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"Initiating message 1 "}
{"ts":1706046263784,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737236219648","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-z78rk","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"Got E2setup"}
{"ts":1706046263784,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737236219648","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-z78rk","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"Start from SCTP 19 fd"}
{"ts":1706046263784,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737236219648","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-z78rk","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"SCTP status assoc id 116 instrms 2 outstrms 2"}
terminate called after throwing an instance of 'std::invalid_argument'
what(): Invalid label nameThread 8 "e2" received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffff0f8b700 (LWP 40)]
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x00007ffff7dd3859 in __GI_abort () at abort.c:79
#2 0x00007ffff71a08d1 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#3 0x00007ffff71ac37c in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#4 0x00007ffff71ac3e7 in std::terminate() () from /lib/x86_64-linux-gnu/libstdc++.so.6
#5 0x00007ffff71ac699 in __cxa_throw () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6 0x000000000053fcd8 in prometheus::Family<prometheus::Counter>::Add(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::unique_ptr<prometheus::Counter, std::default_delete<prometheus::Counter> >) ()
#7 0x0000000000442afb in prometheus::Family<prometheus::Counter>::Add<>(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&) (this=0x65b910, labels=Python Exception <class 'AttributeError'> 'NoneType' object has no attribute 'pointer':
std::map with 2 elements) at /opt/e2/RIC-E2-TERMINATION/3rdparty/prometheus-cpp/core/include/prometheus/family.h:113
#8 0x0000000000429c03 in buildPrometheusList (peerInfo=0x7fffdc002b60, prometheusFamily=0x65b910) at /opt/e2/RIC-E2-TERMINATION/sctpThread.cpp:1998
#9 0x00000000004282f8 in asnInitiatingRequest (pdu=0x7fffdc04e390, sctpMap=0x656700, message=..., rmrMessageBuffer=..., streamId=0) at /opt/e2/RIC-E2-TERMINATION/sctpThread.cpp:2231
#10 0x000000000042594e in receiveDataFromSctp (events=0x7fffdc000e40, sctpMap=0x656700, numOfMessages=<optimized out>, rmrMessageBuffer=..., ts=...) at /opt/e2/RIC-E2-TERMINATION/sctpThread.cpp:1618
#11 0x000000000041f509 in listener (params=0x7fffffff8fb8) at /opt/e2/RIC-E2-TERMINATION/sctpThread.cpp:1110
#12 0x00007ffff71d8df4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#13 0x00007ffff72ec609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#14 0x00007ffff7ed0353 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
The crash seems to stem from the buildPrometheusList function, particularly at the following line:
Please let me know if you need any additional information, or if you have any idea on what might be the root cause for the crash. I can also provide the complete log (16.3MB) upon request. Thanks for your help!
Hello, during my testing of the E2T, I encountered the following crash in the E2T. The entire log is quite large as it contains non-related traffic, but the relevant part is provided below. I set the log level to "Debug" and used gdb to trace the root cause for the error.
{"ts":1706046263587,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737236219648","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-z78rk","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"Start EPOLL Wait. Timeout = -1"} {"ts":1706046263784,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737236219648","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-z78rk","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"handling epoll event 1 out of 1"} {"ts":1706046263784,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737236219648","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-z78rk","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"new message from SCTP, epoll flags are : 1"} {"ts":1706046263784,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737236219648","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-z78rk","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"Start Read from SCTP 19 fd"} {"ts":1706046263784,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737236219648","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-z78rk","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"Start Read from SCTP fd 19 stream 0 "} {"ts":1706046263784,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737236219648","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-z78rk","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"Finish Read from SCTP 19 fd message length = 7"} {"ts":1706046263784,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737236219648","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-z78rk","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"Before Encoding E2AP PDU for : gnb_734_373_16b8cef1, Read time is : 0 seconds, 29556 nanoseconds"} {"ts":1706046263784,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737236219648","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-z78rk","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"PDU buffer length = 7, data = : 0001400"} {"ts":1706046263784,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737236219648","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-z78rk","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"After Encoding E2AP PDU for : gnb_734_373_16b8cef1, Read time is : 0 seconds, 14486 nanoseconds"} {"ts":1706046263784,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737236219648","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-z78rk","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"Encoding E2AP PDU past : InitiatingMessage ::= { procedureCode: 1 criticality: 1 (ignore) value: E2setupRequest ::= { protocolIEs: ProtocolIE-Container ::= { } } } "} {"ts":1706046263784,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737236219648","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-z78rk","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"Initiating message 1 "} {"ts":1706046263784,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737236219648","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-z78rk","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"Got E2setup"} {"ts":1706046263784,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737236219648","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-z78rk","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"Start from SCTP 19 fd"} {"ts":1706046263784,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737236219648","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-z78rk","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"SCTP status assoc id 116 instrms 2 outstrms 2"} terminate called after throwing an instance of 'std::invalid_argument' what(): Invalid label nameThread 8 "e2" received signal SIGABRT, Aborted. [Switching to Thread 0x7ffff0f8b700 (LWP 40)] __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 #1 0x00007ffff7dd3859 in __GI_abort () at abort.c:79 #2 0x00007ffff71a08d1 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6 #3 0x00007ffff71ac37c in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6 #4 0x00007ffff71ac3e7 in std::terminate() () from /lib/x86_64-linux-gnu/libstdc++.so.6 #5 0x00007ffff71ac699 in __cxa_throw () from /lib/x86_64-linux-gnu/libstdc++.so.6 #6 0x000000000053fcd8 in prometheus::Family<prometheus::Counter>::Add(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::unique_ptr<prometheus::Counter, std::default_delete<prometheus::Counter> >) () #7 0x0000000000442afb in prometheus::Family<prometheus::Counter>::Add<>(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&) (this=0x65b910, labels=Python Exception <class 'AttributeError'> 'NoneType' object has no attribute 'pointer': std::map with 2 elements) at /opt/e2/RIC-E2-TERMINATION/3rdparty/prometheus-cpp/core/include/prometheus/family.h:113 #8 0x0000000000429c03 in buildPrometheusList (peerInfo=0x7fffdc002b60, prometheusFamily=0x65b910) at /opt/e2/RIC-E2-TERMINATION/sctpThread.cpp:1998 #9 0x00000000004282f8 in asnInitiatingRequest (pdu=0x7fffdc04e390, sctpMap=0x656700, message=..., rmrMessageBuffer=..., streamId=0) at /opt/e2/RIC-E2-TERMINATION/sctpThread.cpp:2231 #10 0x000000000042594e in receiveDataFromSctp (events=0x7fffdc000e40, sctpMap=0x656700, numOfMessages=<optimized out>, rmrMessageBuffer=..., ts=...) at /opt/e2/RIC-E2-TERMINATION/sctpThread.cpp:1618 #11 0x000000000041f509 in listener (params=0x7fffffff8fb8) at /opt/e2/RIC-E2-TERMINATION/sctpThread.cpp:1110 #12 0x00007ffff71d8df4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6 #13 0x00007ffff72ec609 in start_thread (arg=<optimized out>) at pthread_create.c:477 #14 0x00007ffff7ed0353 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95The crash seems to stem from the
buildPrometheusListfunction, particularly at the following line:peerInfo->counters[IN_INITI][MSG_COUNTER][(ProcedureCode_id_E2setup)] = &prometheusFamily->Add({{peerInfo->enodbName, "IN"}, {"SetupRequest", "Messages"}});As a temporary fix, I implemented a non-null check and wrapped the metric collection code in a try-catch block:
void buildPrometheusList(ConnectedCU_t *peerInfo, Family<Counter> *prometheusFamily) { if (peerInfo == nullptr || prometheusFamily == nullptr) { return; } if (peerInfo->enodbName[0] == '\0') { return; } try { peerInfo->counters[IN_INITI][MSG_COUNTER][(ProcedureCode_id_E2setup)] = &prometheusFamily->Add({{peerInfo->enodbName, "IN"}, {"SetupRequest", "Messages"}}); // rest of the metric collecting code } catch (const std::exception& e) { mdclog_write(MDCLOG_ERR, "Exception caught in %s: %s", __FUNCTION__, e.what()); } }Please let me know if you need any additional information, or if you have any idea on what might be the root cause for the crash. I can also provide the complete log (16.3MB) upon request. Thanks for your help!