Pipe Timing

Spartan-3 USB 2.0 Modules
1

Hello Everyone,

After working through a few logic bugs :o , I recently discovered that there is a potential timing problem (hold violation) with the buffered pipe modules and this could also apply to the unbuffered pipe modules as well.

Background:
I am using the most recent installation of the OK software release v1.3.0.
A portion of my design implements two independent SERDES blocks. The SERDES blocks are instantiations of one design which includes a bufPipeIn and bufPipeOut set of modules along with status and control registers.

Symptoms:

  1. I would generate traffic on one SERDES channel and the results were identical to the expected results verified in simulation. I would use the exact same sequence on the other SERDES channel and it would fail. Looking at the data in the status/control registers, it became apparent to me that the data was not transferring properly between the fifo and my logic.
  2. The symptom above would appear intermittently depending on the image file that was generated. In other words, I would make a tweak and have to generate a new programming file for the FPGA which would introduce this failure. Other images would not. This is a first clue that it was more dependent on place and route.

Diagnosis:

  1. Some of the status/control registers are direct copies of the fifo data getting pulled from the bufPipeOut module. These values are corrupted when the symptoms appear. I connected a logic analyzer and brought out the fifo data and read signal. Indeed the read strobe, data out is as described in Xilinx XAPP131. However, the data coming out was already corrupted.
  2. I then brought out the ti_control and ti_data lines to see what could be happening. Again, the waveforms resemble the ones in the XAPP131 doc.
  3. I deduced that the data is actually latched on the falling edge of the WRITE strobe (ti_control[0]). In some instances I saw that the data started to change prior to the WRITE strobe’s falling edge as it transitioned back to the idle state (0xFF). This is of course dependent on the placement and routing of the logic within the FPGA which changes everytime one generates a new programming image.
  4. Reducing the clock frequency does not alleviate the problem. This is another strong indicator that we are dealing with a hold time violation.

Possible Remedies:

  1. I have attached a sample waveform of what I use in my design. I am essentially transitioning the data on the falling edge of the clock and generate the WRITE strobe as spec’d in the XAPP131 note. As shown in the diagram, the yellow marker shows a write strobe will latch 0xef into the bufPipeIn module. Likewise, I sample the bufPipeOut data on the falling edge of the clock cycle. In the diagram, you can see the data presented by the pipe (0xc3) and the data appears in the shift register (serdesOut) on the next half cycle. This makes my part of the design robust, but what about the Opal Kelly side? I created a 1/2 cycle delay off of ti_data and feed this into the bufPipeOut module. I am able to run the clock in my portion of the design at 32MHz with plenty of margin. I am borrowing from the setup time to do this, so higher frequency designs may not benefit from this technique.
  2. I am not an FPGA design expert, so it may take me a while to figure out how to specify constraints in the Xilinx suite. Could Opal Kelly provide a constraint file to help the Xilinx tool place and route the Opal Kelly portions to meet timing? With a set of constraints, we may not have to resort to these techniques. Given my relative slow frequency operation, it is faster for me to code verilog for a robust design than to deal with timing problems.
  3. Any other suggestions???

Aside:
If this problem can happen between the okHostInterface and okBufPipeOut modules (things that should be black boxes), what about the other modules? Has anyone seen this problem on the other interfaces? Jake, could you evaluate the other interfaces as well? It is critical for the data to transfer properly even between wire and trigger blocks on the first attempt. Other than this issue, I think the kit offers everything we need for the moment.

bufPipeTiming.gif (10.4 KB)

2

Hi Adrian-

Everything within the Opal Kelly HDL modules is synchronous in design. That is, the only significant event is the rising edge of TI_CLK (or EP_CLK for blocks that operate with an endpoint clock).

Therefore, the data is actually latched, not on the falling edge of WRITE, but on the rising edge of TI_CLK whenever WRITE is asserted during at that time. In logic terms, the data is the ‘D’ input to a D flip-flop, TI_CLK is the ‘CLK’ input, and WRITE is the ‘ENABLE’ input.

There should be no constraints required since the logic synthesis and clock tree guarantee that this will work. In other words – the clock tree within the device is designed to maintain minimal skew between CLK inputs of the FFs across the device. Routing delays only -help- the hold time because it means that a signal coming out of a flip flop at the rising edge will be held at that value longer (while the signal propagates to the input of the next FF)

Does this make sense? I don’t mean to suggest you have a phantom problem at all. I’m just wondering if you’ve found the right cause. If you run the timing analyzer on your design, you should be able to check out the signals in question and see what their timing looks like.

How large is your design right now? i.e. what percentage of the FPGA are you consuming? Finally, are you using the latest NGC (HDL modules) files?

3

Hi Jake,

I understand your points, but I still need to understand what is causing the data corruption into the bufPipeOut module. I have a known byte stream transmitted to the fifo from my software application and some of the bytes comes out of the fifo corrupted. I have seen this using a logic analyzer (external). For some reason, the data on ti_data seems to start switching before the write pulse edge. Any other suggestions on where to look? I will try and figure out how to use the timing analyzer.

I am only using about 25% of the FPGA resources for logic, so I did not think placement/routing should have been an issue. I grabbed the NGC files from the latest installation set of folders, XilinxISE61_v1 in particular. I have an 8-bit version of the kit.

Thanks.

4

Hi Adrian-

Any chance you could provide a test fixture for this? Logic analyzer plots could be useful too.

How is the LA setup? i.e. is it setup on a free-running clock or is it being clocked by one of the signals in the design?

Remember that you’re looking for how TI_DATA switches relative to the TI_CLK, depending on the level of WRITE at that clock edge. It is irrelevant what level TI_DATA holds at the edge of WRITE.

If you are willing to send an email containing some slice of your code, I’d be happy to take a look at it. You can email it to [email protected].

I understand how frustrating some of these on again / off again problems can be and am willing to help if I can.

5

The few cases that I experienced this were early in the development phase, i.e. I was changing the design often. I did not have hooks to capture a trace until later. I now have hooks in the design so that I can attempt to capture a trace with a logic analyzer if it crops up in a future change in the design.

The logic analyzer setup is a free-running clock with 1/2 ns granularity. I brought out TI_DATA and TI_CLK since I was suspecting data corruption going into the fifo from the USB controller. The one trace I did capture (but not save) was TI_DATA starting to change before TI_CLK edge transitions. I don’t know if this is even possible since you mentioned that the data is synchronous.

I will definitely send you a trace if I see this problem in the future.