Sequence errors during transmit

Can't talk to your radio? This is the place to ask!
User avatar
n2gq
Posts: 139
Joined: Sun Dec 18, 2022 7:26 pm

Sequence errors during transmit

Postby n2gq » Sun Mar 05, 2023 10:07 pm

Hi
I received my Anan-7000dle mk3 tuesday. I installed protocol 2 firmware and latest thetis and installed the patch as well for fm/eer.
I have a dedicated tplink network card for the radio. I also followed w1aex static ip instructions. I also configured card for ipv4 only with the npcap only.
Okay when receiving I am not getting any sequence errors, only on transmit. I have tried most suggestions here on the board. I have reset the database. The static ip gave good results. I can't get the nic to stay in private. It always revert back to public.
I have program the radio back to protocol 1 to see how it performs. And it works great. I am wondering if I missed something?
Computer is a AMD ryzen5 5600x 6 core processor and 16 gigs of ram.
Jay N2GQ
User avatar
w-u-2-o
Posts: 5540
Joined: Fri Mar 10, 2017 1:47 pm

Re: Sequence errors during transmit

Postby w-u-2-o » Mon Mar 06, 2023 2:33 pm

Are the seq errors resulting in any noticeable degradation of your transmit audio? If not then you can ignore them.

Public/private doesn't really matter.

In Thetis Setup > General > H/W Select have you enabled Network Throttle Index Tweak?

You can try disabling Windows Firewall on the dedicated TP-Link NIC and see if that helps.

Finally, you can try messing with some of the advanced properties of the NIC, e.g. flow control, etc. and see if that makes any difference. There is no specific guidance on this, every NIC is different.
Carajo
Posts: 57
Joined: Tue Feb 28, 2023 11:13 pm
Location: Tampa, Florida

Re: Sequence errors during transmit

Postby Carajo » Mon Mar 06, 2023 9:25 pm

I feel your pain, this is unsat. also W2UO wrote

There is no "issue" with the Mark III and Protocol 2 per se. There are issues with Protocol 2 overall. P2 firmware puts greater demands on both the FPGA (to achieve timing closure) and the Ethernet connection (Gigabit speeds and UDP packet loss rates). Some ANAN units have difficulty with it, although this tends to be rare. More often it's network related and it can fixed as described above.

It's unlikely that there will be any more versions of Protocol 2 firmware made.

I am contacting apache about sending me a radio that works with PROTOCOL II< I did not invest the money to go back to Protocol 1. or they can start sending me a refund.
:x
User avatar
w-u-2-o
Posts: 5540
Joined: Fri Mar 10, 2017 1:47 pm

Re: Sequence errors during transmit

Postby w-u-2-o » Mon Mar 06, 2023 10:02 pm

As it happens, there will be another P2 firmware version coming out for EP4CE and 5CEFA based hardware because a bug was found in the QSK code in the current builds. So that might (or might not!) fix any problems. When the new firmware will be available I cannot say. It has to go through some alpha testing first before any beta might be posted.

However, it still begs the question: are any of these seq errors causing audible issues? If not then while they are annoying they are not functionally significant.
User avatar
n2gq
Posts: 139
Joined: Sun Dec 18, 2022 7:26 pm

Re: Sequence errors during transmit

Postby n2gq » Mon Mar 06, 2023 10:12 pm

w-u-2-o wrote:Are the seq errors resulting in any noticeable degradation of your transmit audio? If not then you can ignore them.

Public/private doesn't really matter.

In Thetis Setup > General > H/W Select have you enabled Network Throttle Index Tweak?

You can try disabling Windows Firewall on the dedicated TP-Link NIC and see if that helps.

Finally, you can try messing with some of the advanced properties of the NIC, e.g. flow control, etc. and see if that makes any difference. There is no specific guidance on this, every NIC is different.


Hi,
Thanks for replying. I have the network throttle index tweak on. I had disabled the firewall on norton 360 and had checked all the rules. And on the advanced side of the nic I had turned flow control and most of the other parameters that were in a flex radio bullentin. The static ip was a lot of help. Before most of the adjustments I had sequence errors on both rec and trans. Then after all adjustments I had the receive working great but when I transmitted I would see a power output drop and sometimes a small spike in output power along with the sequence errors alerts popping on the screen.
Jay N2GQ
User avatar
w-u-2-o
Posts: 5540
Joined: Fri Mar 10, 2017 1:47 pm

Re: Sequence errors during transmit

Postby w-u-2-o » Mon Mar 06, 2023 10:17 pm

You are already doing everything in the cookbook. The only thing I have left to offer is to try messing with some of the advanced settings. Over the years I've found that the advanced settings that have had the most dramatic effects for me were flow control and interrupt moderation. For instance, on my old NIC (that got killed by lightning--RIP!) it worked much better with interrupt moderation turned off. But my new NIC works best with all of the default settings. It's just something you'll have to experiment with and any changes have a low probability of success.
User avatar
n2gq
Posts: 139
Joined: Sun Dec 18, 2022 7:26 pm

Re: Sequence errors during transmit

Postby n2gq » Tue Mar 07, 2023 12:12 am

Okay I will try some of those settings I hope I get something going. Will keep you posted.
Thank you
Jay n2gq.....
Jay N2GQ
User avatar
n2gq
Posts: 139
Joined: Sun Dec 18, 2022 7:26 pm

Re: Sequence errors during transmit

Postby n2gq » Tue Mar 07, 2023 12:18 am

Carajo wrote:I feel your pain, this is unsat. also W2UO wrote

There is no "issue" with the Mark III and Protocol 2 per se. There are issues with Protocol 2 overall. P2 firmware puts greater demands on both the FPGA (to achieve timing closure) and the Ethernet connection (Gigabit speeds and UDP packet loss rates). Some ANAN units have difficulty with it, although this tends to be rare. More often it's network related and it can fixed as described above.

It's unlikely that there will be any more versions of Protocol 2 firmware made.

I am contacting apache about sending me a radio that works with PROTOCOL II< I did not invest the money to go back to Protocol 1. or they can start sending me a refund.
:x


Hi
I would not give up on this radio none are perfect. But it does work nice on protocol 1. And hopefully they can resolve the the timing issue. I have made about five on air transmissions with this radio. Not much fun with the radio yet but I like it so far.
Jay n2gq
Jay N2GQ
User avatar
n2gq
Posts: 139
Joined: Sun Dec 18, 2022 7:26 pm

Re: Sequence errors during transmit

Postby n2gq » Wed Mar 08, 2023 11:53 pm

Good evening,

Okay this is what I did and the sequence errors have gone away. I went into tp-link network card properties and disabled most of the parameters that were enable. Now what actually helped is the parameter [Speed & Duplex] which I set it to 100mbps full duplex not 1000 gbps. And I swiitched back and forth between the two 100 then 1000 and everytime I chose 1000 gbps and sequence error popped up. Give it a try. Hope it helps.

Jay
N2GQ
Jay N2GQ
User avatar
w-u-2-o
Posts: 5540
Joined: Fri Mar 10, 2017 1:47 pm

Re: Sequence errors during transmit

Postby w-u-2-o » Thu Mar 09, 2023 12:42 pm

n2gq wrote:Okay this is what I did and the sequence errors have gone away. I went into tp-link network card properties and disabled most of the parameters that were enable.

It would probably be better to leave all the off-loading options turned on. That makes things easier on the NIC.

Now what actually helped is the parameter [Speed & Duplex] which I set it to 100mbps full duplex not 1000 gbps. And I swiitched back and forth between the two 100 then 1000 and everytime I chose 1000 gbps and sequence error popped up. Give it a try. Hope it helps.

Unfortunately that approach has two problems.

The first problem is that there will be limitations on the max. number of receivers, sample rates, PureSignal and diversity. Depending upon what you are doing you can quickly exceed 100Mbit/s on the Ethernet interface.

For example, I normally run 768KHz sampling rate on both RX1 and RX2. When I have both receivers enabled and transmit with PureSignal the total data rate exceeds 100Mbit/s.

You can watch your data rates using the "resmon" tool built into Windows. It's like Task Manager but better. Just hit Windows Key + R on your keyboard and type resmon into the run box.

The second problem is that the ability to run at 100Mbit/s is an undocumented/unsupported feature. It only exists in the EP4CE and 5CEFA firmware builds. And because it made it so difficult to achieve firmware timing closure it is going to be removed as part of the upcoming firmware update to fix the QSK bug in those builds.
User avatar
n2gq
Posts: 139
Joined: Sun Dec 18, 2022 7:26 pm

Re: Sequence errors during transmit

Postby n2gq » Thu Mar 09, 2023 3:15 pm

Yes you are correct about sample rate. When at the largest sample I can't enable diversity. But when I lower to 768.... sample rate then it works.
I don't have a sampler for the pure signal yet so I don't know how it affects it but i'm sure as you said there will be a limitation.

Jay n2gq
Jay N2GQ
K1LSB
Posts: 639
Joined: Wed Feb 05, 2020 5:25 pm

Re: Sequence errors during transmit

Postby K1LSB » Thu Mar 09, 2023 9:46 pm

You don't need an external sampler to test for proper PS operation, you can just use the internal sampler:

Gen Ant-Filters Antenna.jpg
Gen Ant-Filters Antenna.jpg (116.49 KiB) Viewed 9200 times

Mark
User avatar
w-u-2-o
Posts: 5540
Joined: Fri Mar 10, 2017 1:47 pm

Re: Sequence errors during transmit

Postby w-u-2-o » Thu Mar 09, 2023 10:08 pm

K1LSB wrote:You don't need an external sampler to test for proper PS operation, you can just use the internal sampler:

Gen Ant-Filters Antenna.jpg
Mark

Unless he's using an external amplifier...
K1LSB
Posts: 639
Joined: Wed Feb 05, 2020 5:25 pm

Re: Sequence errors during transmit

Postby K1LSB » Thu Mar 09, 2023 10:54 pm

Regardless of whether he's using an external amplifier or not, he can test for proper PS operation by using my suggestion.

His PS won't correct for an external amp but it will still correct for the internal PA, so he can in fact test for proper PS operation.

Mark
User avatar
n2gq
Posts: 139
Joined: Sun Dec 18, 2022 7:26 pm

Re: Sequence errors during transmit

Postby n2gq » Fri Mar 10, 2023 1:21 pm

I am going to try a PS test today. Hopefully if I get time. Are any of you guys operating with a new mk3 chip and had any success?

Jay n2gq
Jay N2GQ
User avatar
w-u-2-o
Posts: 5540
Joined: Fri Mar 10, 2017 1:47 pm

Re: Sequence errors during transmit

Postby w-u-2-o » Fri Mar 10, 2023 4:50 pm

n2gq wrote:I am going to try a PS test today. Hopefully if I get time. Are any of you guys operating with a new mk3 chip and had any success?

Jay n2gq

Jay--AFAIK, you are the only person with the most recent FPGA that has reported an intractable problem with a GigE connection.
User avatar
n2gq
Posts: 139
Joined: Sun Dec 18, 2022 7:26 pm

Re: Sequence errors during transmit

Postby n2gq » Wed Mar 15, 2023 12:47 pm

Now I am back to this issue.
I am able to run at a sample rate of 384000 with Pure signal and Diversity enabled and no sequence errors. The network adapter set at 100mbps.

Jay n2gq
Jay N2GQ
K1CWD
Posts: 10
Joined: Wed Jan 18, 2023 3:51 pm

Re: Sequence errors during transmit

Postby K1CWD » Mon Mar 20, 2023 2:43 pm

w-u-2-o wrote:
n2gq wrote:I am going to try a PS test today. Hopefully if I get time. Are any of you guys operating with a new mk3 chip and had any success?

Jay n2gq

Jay--AFAIK, you are the only person with the most recent FPGA that has reported an intractable problem with a GigE connection.


I am also having the problem with my MKIII. It just started a few weeks ago, and was concerned it was a windows OS update issue, but it seems like this is more about the FPGA. I have done a number of large FPGA designs, and this gets me worried since the sequence errors have a large enough impact on operation to make me want to shut the radio down and use something else. Also concerned this just started happening, so not sure what might be drifting to cause a timing problem in the ethernet subsystem.

Any more insight into the next FPGA firmware drop?
User avatar
w-u-2-o
Posts: 5540
Joined: Fri Mar 10, 2017 1:47 pm

Re: Sequence errors during transmit

Postby w-u-2-o » Mon Mar 20, 2023 3:26 pm

K1CWD wrote:Any more insight into the next FPGA firmware drop?

I know that Rick, N1GP, is working on a new drop for the second and third FPGA revisions in order to fix the QSK bug. There is no schedule for this, and since the code for the MAC and PHY interface on the FPGA will not be touched there's no expectation that it will attain better timing closure.

If you have done a large number of FPGA designs in a professional development environment, then you are better than able than most to appreciate the challenges, which are:

- The area of timing closure that is at issue is that of the MAC-PHY GigE interface. The remainder of the code runs at substantially slower speeds and has not proven to be a problem.

- openHPSDR firmware development has always used the free version of Quartus. This version does not allow for floor planning or incremental builds. Thus the critical GigE timing closure becomes disturbed for even the smallest changes elsewhere in the code.

- There has never been a formal thermal/voltage study done on the hardware design, so inputs to the Quartus timing model in this area are only a rough estimate. Rick has made substantial improvements in this area but the model settings are still a guess.

- The MAC-PHY interface code was homegrown (not by Rick, he inherited it) and does not use commercial IP. Therefore it quite literally is "amateur built".

As a result, whenever there is a new build of P2 firmware the best Rick can hope for is that if it closes timing on the single development system he has that it will also do the same for a small alpha test group, and then subsequently for a larger beta test group, the latter essentially being everyone. This used to rarely occur (you can look through the older P2 development threads on this forum to see that). Rick has developed an amazing feel for what will work, however, and his latest builds tend work on the vast majority of systems with few revisions. However there is no guarantee they will work on every serial number.

All in all, this is obviously not the most structured or desirable development environment, but it is what it is. If you feel like you might like to assist with firmware development please reach out to me.

73,

Scott
K1CWD
Posts: 10
Joined: Wed Jan 18, 2023 3:51 pm

Re: Sequence errors during transmit

Postby K1CWD » Mon Mar 20, 2023 4:19 pm

w-u-2-o wrote:
K1CWD wrote:Any more insight into the next FPGA firmware drop?

I know that Rick, N1GP, is working on a new drop for the second and third FPGA revisions in order to fix the QSK bug. There is no schedule for this, and since the code for the MAC and PHY interface on the FPGA will not be touched there's no expectation that it will attain better timing closure.

If you have done a large number of FPGA designs in a professional development environment, then you are better than able than most to appreciate the challenges, which are:

- The area of timing closure that is at issue is that of the MAC-PHY GigE interface. The remainder of the code runs at substantially slower speeds and has not proven to be a problem.

- openHPSDR firmware development has always used the free version of Quartus. This version does not allow for floor planning or incremental builds. Thus the critical GigE timing closure becomes disturbed for even the smallest changes elsewhere in the code.

- There has never been a formal thermal/voltage study done on the hardware design, so inputs to the Quartus timing model in this area are only a rough estimate. Rick has made substantial improvements in this area but the model settings are still a guess.

- The MAC-PHY interface code was homegrown (not by Rick, he inherited it) and does not use commercial IP. Therefore it quite literally is "amateur built".

As a result, whenever there is a new build of P2 firmware the best Rick can hope for is that if it closes timing on the single development system he has that it will also do the same for a small alpha test group, and then subsequently for a larger beta test group, the latter essentially being everyone. This used to rarely occur (you can look through the older P2 development threads on this forum to see that). Rick has developed an amazing feel for what will work, however, and his latest builds tend work on the vast majority of systems with few revisions. However there is no guarantee they will work on every serial number.

All in all, this is obviously not the most structured or desirable development environment, but it is what it is. If you feel like you might like to assist with firmware development please reach out to me.

73,

Scott


Thanks Scott,

I do appreciate the technical issues you enumerated. I had no real understanding of the engineering/production environment Apache Labs is working with until reading some of your most recent comments. I have to admit though I'm a little surprised. I thought everything up to 2.5Gbit/s these days was fairly low risk and everyone implemented the reference design without significant modification. Would think too they would contract out for thermal qual.

All the SOC stuff I did was with Xilinx in VHDL, but it's possible I could help out. I would have to think about that a little since I retired recently, am a little hesitant to commit to new work, and most of my recent work was in the project management role (although I have been perusing some development boards to play around with).

73, Craig
User avatar
w-u-2-o
Posts: 5540
Joined: Fri Mar 10, 2017 1:47 pm

Re: Sequence errors during transmit

Postby w-u-2-o » Mon Mar 20, 2023 5:00 pm

Craig,

You wouldn't be committing to anything. One does not commit to open source, at least not in any fiduciary way ;)

You might not fully appreciate how strange or unique the situation is. Apache does not have an environment for software or firmware development. They don't do any of that. See also:

viewtopic.php?f=9&t=4531

and

viewtopic.php?f=32&t=4004&p=17441

73,

Scott
K1LSB
Posts: 639
Joined: Wed Feb 05, 2020 5:25 pm

Re: Sequence errors during transmit

Postby K1LSB » Mon Mar 20, 2023 7:26 pm

Nor will you get paid for it.
User avatar
n2gq
Posts: 139
Joined: Sun Dec 18, 2022 7:26 pm

Re: Sequence errors during transmit

Postby n2gq » Sun Apr 02, 2023 12:53 pm

K1CWD wrote:
w-u-2-o wrote:
n2gq wrote:I am going to try a PS test today. Hopefully if I get time. Are any of you guys operating with a new mk3 chip and had any success?

Jay n2gq

Jay--AFAIK, you are the only person with the most recent FPGA that has reported an intractable problem with a GigE connection.


I am also having the problem with my MKIII. It just started a few weeks ago, and was concerned it was a windows OS update issue, but it seems like this is more about the FPGA. I have done a number of large FPGA designs, and this gets me worried since the sequence errors have a large enough impact on operation to make me want to shut the radio down and use something else. Also concerned this just started happening, so not sure what might be drifting to cause a timing problem in the ethernet subsystem.

Any more insight into the next FPGA firmware drop?


Hi Craig
How is it going with the radio and sequence errors? Any improvements?
Jay
Jay N2GQ
K1CWD
Posts: 10
Joined: Wed Jan 18, 2023 3:51 pm

Re: Sequence errors during transmit

Postby K1CWD » Fri Apr 07, 2023 1:59 am

n2gq wrote:
K1CWD wrote:
w-u-2-o wrote:Jay--AFAIK, you are the only person with the most recent FPGA that has reported an intractable problem with a GigE connection.


I am also having the problem with my MKIII. It just started a few weeks ago, and was concerned it was a windows OS update issue, but it seems like this is more about the FPGA. I have done a number of large FPGA designs, and this gets me worried since the sequence errors have a large enough impact on operation to make me want to shut the radio down and use something else. Also concerned this just started happening, so not sure what might be drifting to cause a timing problem in the ethernet subsystem.

Any more insight into the next FPGA firmware drop?


Hi Craig
How is it going with the radio and sequence errors? Any improvements?
Jay


Hi Jay,

My sequence errors were apparently due to a change in my network environment somewhere. I had been running fine connected to my home lan, but I think a windows update (or loading my lan with some additional devices) caused the sequence errors. Once I went back to direct connection from computer to radio everything was fine.

I guess it's understood by most everyone, but not me apparently, that sequence errors are primarily a network loss issue caused by network problems external to the radio. Network problems would include things like computer I/O or CPU loading that may cause UDP packet loss or out of order packet delivery.

If not a network issue, then the the next most likely cause might be FPGA timing issues due to drift in FPGA timing. I think FPGA issues have a much lower probability of occurrence compared to FPGA causes.

I kind of wrongly latched onto the FPGA as a culprit because of course I can't believe a modern gigabit ethernet home lan would have something going on to cause UDP streaming problems. Live and learn.

-Craig

-Craig
User avatar
kc2rgw
Posts: 165
Joined: Mon Jun 22, 2020 5:44 pm

Re: Sequence errors during transmit

Postby kc2rgw » Fri Apr 07, 2023 12:18 pm

A dense stream of UDP like we generate is actually kind of unusual and brings out flaws in a lot of hardware/IP stacks.

To the OP, I'm curious what FPS rates and the FFT size being run. I've found people running some crazy combinations there and once adjusted to more sane values, their issues went away.
User avatar
w-u-2-o
Posts: 5540
Joined: Fri Mar 10, 2017 1:47 pm

Re: Sequence errors during transmit

Postby w-u-2-o » Fri Apr 07, 2023 12:36 pm

Tools for testing network performance, both IP and UDP:

https://www.tamos.com/products/throughput-test/
https://iperf.fr/

Both require a high performing server machine at one end. Running these tools can be quite sobering. UDP truly is not a lossless protocol.

Also, there is a new version of the firmware out for EP4CE and 5CEFA equipped units:

https://community.apache-labs.com/viewtopic.php?f=32&t=4588
W4WMT
Posts: 325
Joined: Sun Apr 09, 2017 10:12 pm

Re: Sequence errors during transmit

Postby W4WMT » Fri Apr 07, 2023 5:11 pm

Once the new Saturn SDR boards become available and get into common use, I guess we'll finally prove whether or not it was FPGA issues causing all the sequence errors.

The "rewind" style sequence errors seem very hard to blame on the stack. How would such a thing even happen???
User avatar
w-u-2-o
Posts: 5540
Joined: Fri Mar 10, 2017 1:47 pm

Re: Sequence errors during transmit

Postby w-u-2-o » Fri Apr 07, 2023 5:21 pm

We know the out of sequence stuff is the Windows networking stack. Dropped or malformed packets can be the FPGA. Dropped packets can also be the network.
W4WMT
Posts: 325
Joined: Sun Apr 09, 2017 10:12 pm

Re: Sequence errors during transmit

Postby W4WMT » Fri Apr 07, 2023 5:47 pm

Right, but what I'm thinking of is the type of sequence error where the sequential numbers jump back (say by 100) and then start over again from that point, counting forward and repeating the sequence of numbers it just ran through. It's hard to imagine how the network stack could do that?!?
User avatar
w-u-2-o
Posts: 5540
Joined: Fri Mar 10, 2017 1:47 pm

Re: Sequence errors during transmit

Postby w-u-2-o » Fri Apr 07, 2023 7:34 pm

If it's doing that then it's probably firmware. I've not seen or heard of that error modality, though. Is this something you've actually seen?

Return to “Network Connections & Network Hardware”