Some errors after integrating GDS

[sheershak-agarwal]

Hello,

when I was done with a little flight code, I tried to integrate the GDS to my App. I saw a few things that didn't make sense and I wanted to see if that is really a problem.

1. About SimpleObjRegistry: In the Ref example, we have SimpleObjRegistry but not in the RPI example. When I include SimpleObjRegistry in my Topology.cpp, it throws an segmentation fault error while initializing the second custom made component. I removed the SimpleObjRegistry and everything is fine. Is this a problem?

2. About priority in GPIO_INT: while opening the GPIO as interrupt, I am getting this warning.

pthread_create: Insufficient permissions to create thread. May not set thread priority without permission. Operation not permitted
[WARNING] Insufficient Permissions:
[WARNING] Insufficient permissions to set task priority or set task CPU affinity on task TP_GPINT_gpio26Drv. Creating task without priority nor affinity.
[WARNING] Please use no-argument <component>.start() calls, set priority/affinity to TASK_DEFAULT or ensure user has correct permissions for operating system.
[WARNING]      Note: future releases of fprime will fail when setting priority/affinity without sufficient permissions 

[WARNING] Task priority set and permissions unavailable. Discarding priority.

and this is the code I am using for it. Should I not give 99 as input argument?

if (not gpio26Drv.open(26, Drv::LinuxGpioDriverComponentImpl::GPIO_INT))
  {
    printf("[ERROR] Failed to open PS GPIO 26\n");
    if (driv.gpio26 < MAX_ERRORS) {
      driv.gpio26++;
      writefile(driv);
      // //return true;
    }
  }
  else
  {
    gps_pps = gpio26Drv.startIntTask(99);
    FW_ASSERT(Os::Task::TASK_OK == gps_pps, gps_pps);
    printf("[INFO] Opened PS GPIO 26\n");
  }

3. Framer Error: I am getting multiple [ERROR] Failed to send framed data: 1 and I don't know why. I have cross check my topology and xml files and it looks just like the examples. Is there a hint on why I am seeing this?

Thanks in advance

[LeStarch]

I can answer question 2 and question 3:

2. In order to start a thread with priority, you need to have permissions with the OS to do so. Either add the thread priority permission to your user, or run the software as root. e.g. sudo...

3. This occurs when the framed data is being sent but the communication driver is not up. Once the communication driver connects, this error will stop.

[sheershak-agarwal]

I figured the second question when I ran it with sudo.

For the third one, I am seeing this output

[ERROR] Failed to send framed data: 1
[ERROR] Failed to send framed data: 1
[ERROR] Failed to send framed data: 1
[ERROR] Failed to send framed data: 1
[ERROR] Failed to send framed data: 1
[ERROR] Failed to send framed data: 1
[ERROR] Failed to send framed data: 1
[ERROR] Failed to send framed data: 1
[ERROR] Failed to send framed data: 1
[ERROR] Failed to send framed data: 1
[ERROR] Failed to send framed data: 1
[ERROR] Failed to send framed data: 1
[ERROR] Failed to send framed data: 1
[ERROR] Failed to send framed data: 1
[ERROR] Failed to send framed data: 1
Connected to 127.0.0.1:50000 as a tcp client
[ERROR] Deframing checksum validation failed
[ERROR] Deframing checksum validation failed

I only want use gds for ground testing so when it is on the satellite then there wont be any tcp connection and it will keep throwing the framed data ERROR. is it fine to have this error in background and do the normal flight operations?

[sheershak-agarwal]

PS the last two deframing error came when I sent commands from the GDS html.

[LeStarch]

Deframing error likely means a divergence between the GDS protocol and the F´  protocol. Which version of F´ and which version of fprime-gds?

[LeStarch]

When you go to flight, you'll replace the TCP layer with another comm implementation, correct? Currently our com layer looks like this:

ActiveLoger or TlmChan -> Framer -> Tcp Driver

In flight this might become:

ActiveLogger ... -> Framer -> Comm Driver

In this case Framer is producing ERRORs as the Tcp or Comm driver is not connected. It produces Fw::Logger errors as the "downlink" cannot send, so it produces this data in hopes that someone might see them.

[sheershak-agarwal]

Deframing error likely means a divergence between the GDS protocol and the F´ protocol. Which version of F´ and which version of fprime-gds?

yeah it was the wrong version. so was able to fix this

[timcanham]

If you start the app without the networking parameters, it won't try to connect to the GDS.

[sheershak-agarwal]

What I understood: So there is another thread that is always trying to connect to the TCP driver. (and this won't effect the functionality of the other software in f').

I recently switch to the 2.0 version. I was using 1.5.3 and never saw this message. Also, I didn't want to consume my computer resources to keep trying for a tcp protocol.
What happens when the connection is lost? can the connection be regained or the f' need to be restarted again? I am thinking of using the gds for our ground system as well. So trying to understand it a little better so that I can connect my radio to this. Any threads that can help me?

[LeStarch]

In 1.5.3 we had a unified ground testing component that did 2 things:

1. Frame the data going out to the ground
2. Connect to the ground via TCP

Item 1 is reusable, item 2 specific to Tcp communication. In F´ 2.0 we broke the two into two components: Framer, and TcpDriver.

If you run without a host and port, the tcp driver does not connect and it does not run. However, you keep sending telemetry to the ground regardless (events are emitted, as are channels). Thus in this case the Framer is trying to pass data to the ground, finds a non-started Tcp driver, prints an error, and waits for the next ground packet....which then errors again. If you switched to a radio, or UART, but also chose to not start those components the same error would occur.

This error also happens "at startup" even when running a driver because some components come online before the driver is fully online. Once you get the "connected" message these errors stop.

Independently, the Tcp driver is designed to attempt a reconnection after an error. That is, once it has connected, an error puts it back into connecting state as it tries to restore communication. This only happens when the address/port are supplied. Otherwise, it always reports an error.

[LeStarch]

To clarify, the Tcp driver always returns an error to the caller (Framer) component, who prints the error.

[sheershak-agarwal]

That was a really good explanation. Thank you.

[timcanham]

It seems like if we intentionally don't connect (by not supplying the command line parameters), the errors should be silent.

[sheershak-agarwal]

now I know how the GDS side works on the space craft side, I have a few questions. I don't know if I should create a new thread or not so putting all of them here.

1. In the Topology.cpp we are using the TcpClient as comm but in the xml files we are using ByteStreamDriverModel. How are both of them are connected? or in other words how does deframer and framer get/set data to Tcp?

For my second question, I want first tell you my goal. I want two kinds of functionality from the GDS. First, I should be able to command and get telemetry/events from the f' running hardware through TCP/IP connection in real time.(this feature is already there) Second, I want to use the same framing, deframing, file uplink and downlink infrastructure to connect to the radio hardware. I can only transmit when we are near the gnd station and that too only if it receives a request to send telemetry. (the satellite will be always listening and in a slave mode)

2. How should I change the Ref example to have both these functionality?
   My idea is to add a new COM Component and connect the recv and send port from the deframer and framer to it in addition to the TCP client. The COM component will take the byte stream, and convert it into the form that the radio expects and also do the decoding.

So in the end when we run the flight software without supplying the command line parameters it should just use the COM component and when we provide the parameters then it should use both TCP and COM/radio. This should also take care of the error since the message will now go to COM component.

Does this make sense?

[rawinza555]

I have done this in the past. I used both TCP/IP and a 868 MHz Transceiver module. Basically I made a component that interface with the transceiver. I tried to connect both bytestreammodel and RF interface but there's an error in buffer allocation and deallocation. I believe TCP allocate the buffer and let the deframer to deallocate. The problem I had was that the transceiver module I used was connected to another MCU via UART and I believe buffer allocation and deallocation was done in UART driver.

So, what I did was connecting the RF interface to groundinterface and connect ground interface to chantlm, cmddisp and eventlogger. It worked like a charm. Groundinterface component has framer and deframer built in in one component.

Ideally, I believe you can just swap out the framer, deframer and bytestreammodel and replace it with groundinterface + TCP + your RF component but I was too lazy to do it back then lol. (It was just a proof of concept so I didn't do much)

[sheershak-agarwal]

Oh nice! Does ground interface even take care of FileUplink and FileDownlink or I need to connect separately? I also saw there is a new component TlmPacketizer which kind of replaces chanTlm. Do you have the code/diagram of the groundinteface connections?

What I understood:
(Maybe) FileUplink/FileDownlink, eventLogger, chanTlm, PktTlm -> groundInteface -> TCP/ByteStreamModel
(Maybe) FileUplink/FileDownlink, eventLogger, chanTlm, PktTlm -> groundInteface -> Custom COM Component

[timcanham]

TlmPacketizer is meant as an alternative to TlmChan, but we haven't fully tested it in the new build yet.

[LeStarch]

ByteStreamDriverModel is a design for generic byte streams. TcpClient implements this design. In the future we could write a UART driver that also implements that design. That way, in topology, you could drop in Tcp or UART easily.

The GroundInterface component was replaced with Framer and Defamer. I would caution against it, as it is we are not supporting it. If you wanted to run 2 ground interfaces (as opposed to just choosing at compile time) you'd need to do the following:

1. Implement a radio/comm component that implements ByteStreamDriverModel's design, or a new design that at least has the same ports.
2. Are you running both in parallel, or is one always idle when the other is working and this is purely coding convenience?
3. If they run in parallel, use BufferManager for the memory allocation. If they run one or the other, use the same port index for allocation in the staticmemory component.
4. Wire both up to framer/deframer

That should work correctly.

Note: FileUplink and FileDownlink components connect to Framer/Deframer as TlmChan and ActiveLogger do.

[sheershak-agarwal]

Alright that makes sense. I will be looking forward to the future update but I will also try the steps you mentioned.

I won't use the Ground Interface then. Yes I do want two ground interface. The COM interface will always be on and tcp can be chosen through terminal commands (Image for a little more clarification). The goal for the umbilical testing is to make sure each component is running as expected including the COM/radio system. So both should be able to run in parallel.

Last question before I start the implementation of the steps, should I use the ref or rpi example? because ref has tlmPacketizer and simpleObjregistry and other extra stuff which might not be needed for my project.

[sheershak-agarwal]

Hello,
I added all the components for GDS (CmdSeq, CmdDisp etc) , created a component linkBufferManager to replace the staticMemory and also add all the ByteStreamModel ports to my COM Manager component.

I then connected them as suggested. Here is a diagram to explain the connections. I i have only drwan the connections for comm and ComManger. (If you want me include more components, let me know.)

After all these connection I am facing two problems:
First, It always throws a segmentation fault in framing component. I am getting these error codes:

[ERROR] Failed to send framed data: 1434840560
[ERROR] Failed to send framed data: 1434840560
[ERROR] Failed to send framed data: 1434840560
[ERROR] Failed to send framed data: 1434840560

Assert file "/home/sheershak/Desktop/fprime/Svc/FramingProtocol/FprimeProtocol.cpp": Line: 37 2
Assert file "/home/sheershak/Desktop/fprime/Svc/FramingProtocol/FprimeProtocol.cpp": Line: 40 2
Assert file "/home/sheershak/Desktop/fprime/Svc/FramingProtocol/FprimeProtocol.cpp": Line: 49 2
Assert file "/home/sheershak/Desktop/fprime/Utils/Hash/libcrc/CRC32.cpp": Line: 33 

Thread 16 "App" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffeb7fe700 (LWP 21523)]
0x0000555555614c18 in Utils::Hash::hash(void const*, int, Utils::HashBuffer&) ()
(gdb) bt
#0  0x0000555555614c18 in Utils::Hash::hash(void const*, int, Utils::HashBuffer&) ()
#1  0x00005555555fd2ed in Svc::FprimeFraming::frame(unsigned char const*, unsigned int, Fw::ComPacket::ComPacketType) ()
#2  0x000055555557d9d5 in Svc::FramerComponentBase::m_p_comIn_in(Fw::PassiveComponentBase*, int, Fw::ComBuffer&, unsigned int) ()
#3  0x000055555560e238 in Fw::OutputComPort::invoke(Fw::ComBuffer&, unsigned int) ()
#4  0x00005555555771da in Svc::TlmChanImpl::Run_handler(int, unsigned int) ()
#5  0x00005555555773c3 in Svc::TlmChanComponentBase::doDispatch() ()
#6  0x00005555556030c9 in Fw::ActiveComponentBase::s_baseTask(void*) ()
#7  0x0000555555612a07 in pthread_entry_wrapper(void*) ()
#8  0x00007ffff7bbb6db in start_thread (arg=0x7fffeb7fe700) at pthread_create.c:463
#9  0x00007ffff6fa571f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

and the Second problem is I still see [ERROR] Failed to send framed data: -1647944208 even after I after connecting with TCP. since the data should be coming to my comManager so can't I do an acknowledge of frame in the comManager? if yes, how?

Connected to 127.0.0.1:50000 as a tcp client
[ERROR] Failed to send framed data: -1647944208
[ERROR] Failed to send framed data: -1647944208
[ERROR] Failed to send framed data: -1647944208
[ERROR] Failed to send framed data: -1647944208
[ERROR] Failed to send framed data: -1647944208

Thank you in advance

[rawinza555]

Normally TCP allocate a buffer in itself and deallocate at Deframer. When you send commands through GDS, the buffer get allocated and deallocated normally. However, there might be something wrong with your allocation in your second COM channel.

It's one of the reason why I just use groundinterface back then. I was in the time crunch and didn't have time to dive into buffer allocation and deallocation. There's no buffer allocation and deallocation in that component and for buffer send and return is already done in LinuxSerialDriver so I believe it should be somewhat safe for a demo. I could be dead wrong though

[sheershak-agarwal]

this line allocates the buffer: Fw::Buffer buffer = this->allocate_out(0, 24) which is the exact way used in TcpClientImpl. I am super close to the implementation I wanted.

[LeStarch]

BufferManager allocates memory from it's internal pool for you, and expects you return a buffer back to it from the same pool. The setData and getData calls set and retrieve the pointer to that allocated data. Thus, you are clobbering the pointer below, which leads to the assert you are seeing.

U8 temp[] = {0xde, 0xad, 0xbe, 0xef, 0x0, 0x0, 0x0, 0xc, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0x91, 0x0, 0x0, 0x0, 0xb, 0xc2, 0x46, 0xa7, 0xeb};
Fw::Buffer buffer = this->allocate_out(0, 24);
buffer.setData(temp);  // Pointer clobbered here
this->recv_out(0, buffer, Drv::RECV_OK);

I would rework this as below. Here we get the class-representation of the buffer as something that can serialize/deserialize data. Then we serialize your data into it. This prevents the clobbering of the data pointer but otherwise carries out your desired goal.

Note: the below function calls are from-memory, so there might be minor misspellings in the names used.

U8 temp[] = {0xde, 0xad, 0xbe, 0xef, 0x0, 0x0, 0x0, 0xc, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0x91, 0x0, 0x0, 0x0, 0xb, 0xc2, 0x46, 0xa7, 0xeb};
Fw::Buffer buffer = this->allocate_out(0, 24);
buffer.getSerializeRepr().serialize(temp, sizeof(temp));
this->recv_out(0, buffer, Drv::RECV_OK);

[LeStarch]

As for the Sched call: you are likely failing to engage the system. The Sched port is used to drive the whole system and so disabling the call likely causes messages to not be processed.

[sheershak-agarwal]

Thank you @LeStarch for the answers.

buffer.getSerializeRepr().serialize(temp, sizeof(temp)); doesn't work. I can't see it perform the GDS command but buffer.setData(temp); works with the ASSERT output.

for the TlmChan, I understand that I have to connect it with the Sched to make it work but I still want to understand how it works under the hood. When does create the telemetry? Does it create it every one second or as soon as I output a tlm channel from one of my components. The reason for this because I don't want to send a lot of messages through radio (its half duplex).