Handling Real Time Data in Python

FrontPanel
1

I’m using XEM7320 with SYZYGY ADC to make a high-speed data acquisition application in python without losing any samples. Since the sampling rate is high (more than 1MSps), I put the pipe function into another thread to call it as soon as it returns. However, I realize that other data processing procedures in the main thread increase the delay between two consecutive pipe function calls. I would like you to share some recommendations with me about how to handle the data as soon as it arrives from the pipe function.

The procedure that I am using now is as follows and it works fine until 800kSps:

  1. User initiates data acquisition.
  2. A thread starts a function.
  3. This function has a ReadFromBlockPipeOut function inside while loop.
  4. As soon as FPGA prepares data, pipeout returns and data is appended to a queue, and pipeout is recalled again.
  5. In main thread, I process the data or just save it as txt file.

In theory by USB3.0 I can transfer ((300MB * 8 Mbit/MB) / 32 Mbit/Msample) 75 MegaSample/sec. How can I store that much data properly in Python? Also, my application should handle plotting a block of data.

1 Like
2

Hi rustem!

I would recommend checking out using the features in Buffer Protocol — Python 3.11.1 documentation to help reduce latency handling the data from your pipe function.

3

Dear hayden,
Thanks for your response. I know that the pipe function gets bytearray parameter to store data. Actually, my question is the next steps. Let me explain with an example.

A byte array continually filled with 2048 bit => 32 bit each = 64 sample. I want to calculate the sum of those 64 samples and append the result in a plot. Simultaneously, I want to save those 64 samples into a file. In this case, could you describe or give an example code of what I am trying to achieve.

I am trying to achieve something similar to okCameraApp. Since it is C++ code, it is really hard for me to understand. I know that there is also a second thread running to collect each frame. I need a similar approach in Python.