ReadFromBlockPipeOut(...) times out on large reads

FrontPanel
1

Hi all,

I’ve written some multithreaded C++ software for a ZEM4310 (USB3.0ss) that receives large amounts of data (length = 32,768 MB) from a block pipe. I call it like this in what is essentially an asynchronous loop:

result = dev_->ReadFromBlockPipeOut(address, 16384, 32768, (unsigned char *)data);

Sometimes this works fine, in which case it takes approximately 800us to receive the data. That is fine. However, after a random number of calls, the function will hang and then return after 10.1s. When it hangs, it always takes 10.1s, so I assume the default timeout time is 10 seconds or so. After it hangs the first time, any subsequent call to ReadFromBlockPipeOut(…) hangs as well, like this

dev_->ReadFromBlockPipeOut(address, 16384, 32768, data); // takes 800us
dev_->ReadFromBlockPipeOut(address, 16384, 32768, data); // takes 800us
dev_->ReadFromBlockPipeOut(address, 16384, 32768, data); // takes 800us
dev_->ReadFromBlockPipeOut(address, 16384, 32768, data); // TIMEOUT: takes 10.1s
dev_->ReadFromBlockPipeOut(address, 16384, 32768, data); // TIMEOUT: takes 10.1s
dev_->ReadFromBlockPipeOut(address, 16384, 32768, data); // TIMEOUT: takes 10.1s
…
…

Can anyone provide any advice on why this is happening? Thanks for your time

2

See section 7 here regarding multi-threading.
https://docs.opalkelly.com/display/FPSDK/FrontPanel+API

3

Thanks for the fast reply! I have read this before but I guess I failed to consider the implications of this. So I create dev_ (the frontpanel object) in my ‘main’ thread. Later when I want to access the data from the pipe out, I do this using an asynchronous thread created with std::async(…). So this is exactly the kind of problem associated with what section 7 says in your link?

That means that, while the API will allow you to create another instance and communicate with the same device, there are likely going to be problems with doing so.

The thing is, I’m not creating another instance of the frontpanel object, I’m sharing the same instance but in a different thread. Is this still likely to cause issues? Should I create the frontpanel object in the same thread instead?

My program flow I guess could be seen like this:
//thread 1, on setup only:

dev_ = new okCFrontPanel();
/*...do some other things...*/
std::async(...) // create thread 2...

//thread 2, some time later:

dev_->readFromBlockPipeOut(...)

Is this an issue? My understanding of section 7, it seems like the multithreading part refers to constructing multiple instances of dev_ (i.e dev_1_, dev_2_) and have them try and access the same physical device

4

Simply put, you cannot make overlapping calls to the same instance of the FrontPanel object. You can use multithreading so long as you have locks in place to avoid this.

5

Understood. I will check if this is the issue, although I’m fairly sure what I’ve done is fine. I’ll report back with my findings in case others have this problem. Thanks for your time.

6

I’ve since tested the same okCFrontPanel calls/structure running off of a single thread only. The problem persists - I get a seemingly random number of very fast transfers until one of them times out and then all future transfers also time out.

The least amount of transfers I get before the API falls over is 1, the maximum amount of transfers I got were 112.

Is it possible that I’ve done something incorrect on the hardware side? Or is this kind of problem usually related to misusing the C++ API?

For reference, here is how my control signals work:

  1. Host triggers FPGA (asserting InTrigger) to gather some data. Host then calls IsTriggered repeatedly and waits for FPGA ack
  2. FPGA gathers data, fills up FIFO
  3. FIFO is full, assert OutTrigger to signal ack back to Host
  4. Host now calls readFromBlockPipeOut(…)
    /* loop */

In the meantime, I will check and make sure my signals are working correctly on the FPGA side, although any advice would be much appreciated.

One thing I might add is that my OpalKelly source files (.cpp/.h) are quite old. If there is a difference between old .cpp/.h entrypoints and a newer .so, this could maybe be causing issues

Thanks for your time

7

The issue you have described could be caused by the FPGA design. If your design holds the ep_ready signal low for too long it will result in a timeout and subsequent failures of the readFromBlockPipeOut API. Have you tried measuring this signal with the Vivado ILA to ensure that it is not held low for significantly long periods of time?

8

Hi Tom, thanks for the response. I’m using an Altera/Intel FPGA, but no, I haven’t tried using SignalTap, I have timing closure issues if I attempt to probe some of the signals. Instead I am using an oscilloscope to measure some of the signals on the board. Thanks for the suggestion regarding ep_ready, I will check this (I think you are correct though and the issue is more related to my hardware now that I’ve checked the software over). I’ll let you know what I find - thanks I appreciate the suggestion

9

Hi Tom, much thanks for your response. The issue turned out to be ep_ready as you suggested. At some point, the readFromBlockPipeOut(…) call in software would sync up with ep_ready being asserted low in hardware, resulting in a timeout.

Best regards,
Aaron