iPXE drops iSCSI connection when flooded with network packets #1714

dkiper · 2026-05-20T15:46:36Z

dkiper
May 20, 2026

Environment: An UEFI platform which loads iPXE which loads the GRUB over iSCSI. The GRUB load fails/hangs, sometimes with read errors, when you flood network interface with tons of packets.

You can flood network interface with following commands:

nping --tcp --flags syn --dest-port 46017 --rate 1000 --win 62580 --mtu 9000 --count 0 <ip_addr>
nping --udp --dest-port 46017 --rate 1000 --count 0 <ip_addr>
nping --icmp --icmp-type 8 --icmp-code 0 --rate 1000 --count 0 <ip_addr>

In practice the most important parameter is rate. The other arguments are almost not relevant. I was able to reproduce the issue in my environment on every boot by rising rate to 1000.

Output from iPXE with some debug enabled:

  TCP 0x7e65c1c8 timer fired in SYN_SENT for 0ffeb765..0ffeb766 00000000
  IPv4 TX 192.0.2.2->192.0.2.1 len 64 proto 6 id 8d79 csum 01f1
  NETDEV net0 transmission 0x7e65c0e8 failed: Error 0x2a376089 (https://ipxe.org/2a376089)

iSCSI server: 192.0.2.1
iPXE client: 192.0.2.2

The problem is that it is very easy to stop machine network boot by flooding network interface to which iPXE is bound. As I said earlier, type of packets does not matter. Only packets generation speed matters. I think backtrace below explains that:

  #0  netdev_tx_err (netdev=0x7dddc6a8, iobuf=0x7dddda28, rc=rc@entry=-708272265) at net/netdevice.c:450
  #1  0x000000007dd9ed20 in neighbour_destroy (neighbour=0x7dddda88, rc=-708272265) at net/neighbour.c:309
  #2  0x000000007dd9edef in neighbour_discard () at net/neighbour.c:520
  #3  0x000000007dd9cffc in discard_cache (size=<optimized out>) at core/malloc.c:233
  #4  0x000000007dd9d20e in heap_alloc_block (heap=heap@entry=0x7ddd2d80, size=size@entry=1608, align=<optimized out>, offset=0) at core/malloc.c:380
  #5  0x000000007dd9d309 in malloc_phys_offset (size=size@entry=1608, phys_align=<optimized out>, offset=<optimized out>) at core/malloc.c:688
  #6  0x000000007dd99784 in alloc_iob_raw (len=<optimized out>, align=<optimized out>, offset=offset@entry=0) at core/iobuf.c:91
  #7  0x000000007dd99820 in alloc_iob (len=len@entry=1518) at core/iobuf.c:145
  #8  0x000000007dd99860 in alloc_rx_iob (len=len@entry=1518, dma=0x7dddc108) at core/iobuf.c:193
  #9  0x000000007ddafaca in virtnet_refill_rx_virtqueue (netdev=netdev@entry=0x7dddc6a8) at drivers/net/virtio-net.c:166
  #10 0x000000007ddaff0a in virtnet_process_rx_packets (netdev=0x7dddc6a8) at drivers/net/virtio-net.c:405
  #11 virtnet_poll (netdev=0x7dddc6a8) at drivers/net/virtio-net.c:427
  #12 0x000000007dda119f in netdev_poll (netdev=netdev@entry=0x7dddc6a8) at net/netdevice.c:625
  #13 0x000000007dda2f10 in net_poll () at net/netdevice.c:1146
  #14 0x000000007dda4612 in step () at core/process.c:117
  #15 0x000000007ddc471c in sandev_command (sandev=sandev@entry=0x7dddd5a8, command=command@entry=0x7ddc4304 <sandev_command_rw>, params=params@entry=0x7fefd138)
      at core/sanboot.c:552
  #16 0x000000007ddc47a4 in sandev_rw (sandev=0x7dddd5a8, sandev@entry=0x7dddc7c8, lba=lba@entry=3586695031, count=<optimized out>, buffer=buffer@entry=0x7baba600,
      block_rw=<optimized out>) at core/sanboot.c:624
  #17 0x000000007ddc47f7 in sandev_read (sandev=sandev@entry=0x7dddc7c8, lba=lba@entry=3586695031, count=<optimized out>, buffer=buffer@entry=0x7baba600)
      at core/sanboot.c:652
  #18 0x000000007ddba032 in efi_block_rw (sandev_rw=<optimized out>, len=131072, data=0x7baba600, lba=2304, sandev=0x7dddd5a8) at interface/efi/efi_block.c:116
  #19 efi_block_io_read (block_io=<optimized out>, media=<optimized out>, lba=2304, len=131072, data=0x7baba600) at interface/efi/efi_block.c:167

AIUI huge influx of packets leads to memory exhaustion by allocating tons of I/O buffers to service unestablished connections. Then memory allocator is not able to get any longer more memory from the heap and it has to release some caches. Unfortunately the allocator is not aware what is happening in TCP layer and simply it kills active iSCSI session by deallocating, among others, TCP I/O buffer. Then TCP transmission, and subsequently iSCSI session, from iPXE POV fails...

A fix for this probably is not easy and requires quite significant changes in the iPXE code. I am happy to work on it but I am not very familiar with iPXE internals. So, I need some guidance.

mcb30 · 2026-05-20T15:59:17Z

mcb30
May 20, 2026
Maintainer

From the stack backtrace, I see that you are using the virtio-net driver. This driver was replaced in its entirety a week or so ago. Are you able to reproduce with the current master branch?

2 replies

dkiper May 20, 2026
Author

My observation shows that network driver does not matter at all. I was able to reproduce the issue on physical machine too. Though I will run test for you again. Do you want backtrace from it too?

dkiper May 20, 2026
Author

Latest master -> the same issue...

mcb30 · 2026-05-20T22:51:50Z

mcb30
May 20, 2026
Maintainer

There aren't many places in iPXE where queued I/O buffers can accumulate without limit. Received packets get processed immediately (in net_poll()), and processing should in all cases completely consume and free the buffer. Transmitted packets (including transmissions in response to received packets) are almost always naturally limited by the size of the device's transmission queue: once that small queue is full, transmissions will usually simply be dropped.

The neighbour resolution deferred transmission queue is one of the few queues that doesn't have a limit. It would be interesting to see if simply limiting the size of the deferred transmission queue would make the problem go away.

Another option would be to add a second cache discarder mechanism (not marked as expensive) to neighbour.c which would attempt to drop a queued packet from the oldest cache entry (rather than dropping the entirety of the oldest cache entry). This should allow memory pressure to naturally control the number of queued packets, as we do with e.g. the TCP out-of-order receive queue.

It's worth noting that under packet flood conditions it's quite likely that the TCP connection will stall. iPXE uses small receive ring fill levels, and a sustained flood is likely to cause packets to be dropped by the NIC. The above suggestions should help ensure that iPXE recovers more gracefully after a flood, but probably won't allow progress to be made while the flood is ongoing.

3 replies

dkiper May 21, 2026
Author

First of all, do we have, even (a bit) outdated document, describing iPXE internals including packet flows? It would be very helpful...

Anyway, do I understand correctly that net/neighbour.c is responsible for keeping an an eye on active connection at network/transport layer?

I had an idea to add a member to struct io_buffer saying that this buffer belongs to an active transmission. Then cache discarder should omit such entries. Would that work?

Another idea is to restore session when it was dropped for some reason by the iPXE itself. Though this one, at least for now, looks less compelling for me.

mcb30 May 21, 2026
Maintainer

First of all, do we have, even (a bit) outdated document, describing iPXE internals including packet flows? It would be very helpful...

Probably not, sorry.

Anyway, do I understand correctly that net/neighbour.c is responsible for keeping an an eye on active connection at network/transport layer?

No, net/neighbour.c exists solely to perform neighbour resolution (e.g. ARP for IPv4 addresses or NDP for IPv6 addresses). It maintains a queue of packets that could not be transmitted because the neighbour was not yet resolved (i.e. the MAC address was not yet known), and transmits these once the neighbour resolution has completed.

As a secondary responsibility, net/neighbour.c also provides a fault-injection capability for emulating high-latency links. The rationale behind this is described in commit ff6d612e7.

I had an idea to add a member to struct io_buffer saying that this buffer belongs to an active transmission. Then cache discarder should omit such entries. Would that work?

I wouldn't want to start adding that kind of imprecisely defined heuristic.

Another idea is to restore session when it was dropped for some reason by the iPXE itself. Though this one, at least for now, looks less compelling for me.

iPXE has done this since commit ff6d612e7 in 2017. Any failed SAN device operation will be retried several times, and if you're using a multipath SAN device then all paths will be retried.

dkiper May 21, 2026
Author

WRT net/neighbour.c, so, AIUI the transmission breaks because ARP entry is removed during cache eviction. If yes then it totally makes sense.

mcb30 · 2026-05-21T12:11:41Z

mcb30
May 21, 2026
Maintainer

There aren't many places in iPXE where queued I/O buffers can accumulate without limit. Received packets get processed immediately (in net_poll()), and processing should in all cases completely consume and free the buffer. Transmitted packets (including transmissions in response to received packets) are almost always naturally limited by the size of the device's transmission queue: once that small queue is full, transmissions will usually simply be dropped.

The neighbour resolution deferred transmission queue is one of the few queues that doesn't have a limit. It would be interesting to see if simply limiting the size of the deferred transmission queue would make the problem go away.

Another option would be to add a second cache discarder mechanism (not marked as expensive) to neighbour.c which would attempt to drop a queued packet from the oldest cache entry (rather than dropping the entirety of the oldest cache entry). This should allow memory pressure to naturally control the number of queued packets, as we do with e.g. the TCP out-of-order receive queue.

I've pushed commit 2d8236995 which implements this approach. I have tested by flood pinging with sudo nping --icmp --icmp-type 8 --icmp-code 0 --rate 1000 --count 0 -S <garbage source IP> <dest IP> while iPXE attempts to perform an HTTP download in a tight loop. Before this commit, I see the ARP cache entry for the HTTP server getting regularly discarded, which disrupts that connection. After this commit, the HTTP server ARP cache entry is never discarded.

This may not fix every aspect of your issue, but it will definitely change the underlying behaviour. Could you please retest with the current master branch (commit 2d8236995) and report back?

12 replies

dkiper May 22, 2026
Author

It is not possible to reproduce the issue with a HTTP transfer... Even with sleep 10 after :loop label...

So, I tried iSCSI again but this time with very simple UEFI application reading in loop 128/256 MiB file from ESP on iSCSI disk...

tftp://192.0.2.3/autoexec.ipxe... ok
autoexec.ipxe : 361 bytes [script]
TCP 0x7dde2988 allocated
TCP 0x7dde2988 transitioned from CLOSED to SYN_SENT
TCP 0x7dde2988 bound to port 18592
TCP 0x7dde2988 timer fired in SYN_SENT for 3ed53179..3ed53179 00000000
NEIGHBOUR net0 IP 192.0.2.1 created
NEIGHBOUR net0 IP 192.0.2.1 discovering via ARP
NEIGHBOUR net0 IP 192.0.2.1 is Ethernet ba:98:76:54:32:10
TCP 0x7dde2988 using timestamps, SACK, TX window x128, RX window x512
TCP 0x7dde2988 transitioned from SYN_SENT to ESTABLISHED
NEIGHBOUR net0 IP 192.0.2.3 is Ethernet 01:23:45:67:89:ab
Registered SAN device 0x80
Booting from SAN device 0x80
Stall...
NEIGHBOUR net0 IP 192.0.2.3 destroyed: Error 0x2a376089 (https://ipxe.org/2a376089)
NEIGHBOUR net0 IP 192.0.2.1 destroyed: Error 0x2a376089 (https://ipxe.org/2a376089)
Read...
NEIGHBOUR net0 IP 192.0.2.1 created
NEIGHBOUR net0 IP 192.0.2.1 discovering via ARP
NEIGHBOUR net0 IP 192.0.2.1 destroyed: Error 0x2a376089 (https://ipxe.org/2a376089)
NEIGHBOUR net0 IP 192.0.2.3 created
NEIGHBOUR net0 IP 192.0.2.3 discovering via ARP
NEIGHBOUR net0 IP 192.0.2.1 created
NEIGHBOUR net0 IP 192.0.2.1 discovering via ARP
NEIGHBOUR net0 IP 192.0.2.3 destroyed: Error 0x2a376089 (https://ipxe.org/2a376089)
NEIGHBOUR net0 IP 192.0.2.1 destroyed: Error 0x2a376089 (https://ipxe.org/2a376089)
NEIGHBOUR net0 IP 192.0.2.3 created
NEIGHBOUR net0 IP 192.0.2.3 discovering via ARP
NEIGHBOUR net0 IP 192.0.2.3 destroyed: Error 0x2a376089 (https://ipxe.org/2a376089)
NEIGHBOUR net0 IP 192.0.2.3 created
NEIGHBOUR net0 IP 192.0.2.3 discovering via ARP
NEIGHBOUR net0 IP 192.0.2.3 destroyed: Error 0x2a376089 (https://ipxe.org/2a376089)
NEIGHBOUR net0 IP 192.0.2.3 created
...

The key thing to reproduce the issue is stall/sleep. When loop is executed without stall/sleep then everything works as expected. Even under constant flood...

Interestingly breakage can be reproduced even when flood is stopped after following messages:

...
Booting from SAN device 0x80
Stall...
NEIGHBOUR net0 IP 192.0.2.3 destroyed: Error 0x2a376089 (https://ipxe.org/2a376089)
NEIGHBOUR net0 IP 192.0.2.1 destroyed: Error 0x2a376089 (https://ipxe.org/2a376089)
...

I hope that helps...

If you still need more tests drop me a line...

mcb30 May 24, 2026
Maintainer

Two things:

Introducing arbitrary stalls during which no RX packet processing is allowed to take place is always going to result in dropped packets. (Legacy BIOS USB keyboard support can result in similar issues, as mentioned in commit e0fc8fe78.) I'm not particularly worried about any kind of problem that arises only when something outside of iPXE is inducing stalls and thereby preventing iPXE from doing its job.
The behaviour here:
```
NEIGHBOUR net0 IP 192.0.2.3 destroyed: Error 0x2a376089 (https://ipxe.org/2a376089)
NEIGHBOUR net0 IP 192.0.2.1 destroyed: Error 0x2a376089 (https://ipxe.org/2a376089)
```
does not look as though it corresponds to the code on the current master branch. As of commit 2d8236995 it should be basically impossible to ever see a NEIGHBOUR ... destroyed: Error 0x2a376089 (https://ipxe.org/2a376089) with this precise error number. Please retest with the current master branch.

dkiper May 25, 2026
Author

WRT stalls, I can see at least one use case when loaded application introduce sleeps or waits for an event. It is a GRUB load over iSCSI with menu and timeout enabled. So, I think these kinds of setups should be supported.

WRT NEIGHBOUR debug messages, I have pulled master again and dropped my iPXE changes. Nothing has changed...

mcb30 May 26, 2026
Maintainer

WRT stalls, I can see at least one use case when loaded application introduce sleeps or waits for an event. It is a GRUB load over iSCSI with menu and timeout enabled. So, I think these kinds of setups should be supported.

That kind of "stall" shouldn't cause a problem, since it doesn't happen when iPXE is attempting to download data. It is only stalls that happen during a data transfer that would be expected to affect behaviour. For example, if a UEFI application were to set up a TPL_NOTIFY callback attached to a high frequency periodic timer, then that would block iPXE's execution and could be expected to cause problems.

WRT NEIGHBOUR debug messages, I have pulled master again and dropped my iPXE changes. Nothing has changed...

I don't see any way that can happen, sorry. The only code path that can lead to that error message is via neighbour_discard(), since nothing else passes -ENOBUFS to neighbour_destroy(). The only way neighbour_discard() can end up being called is if all other cache discarders have already freed up everything they can. This means that the TCP out-of-order receive queue has been discarded, and all deferred neighbour transmissions have been discarded. I don't see what you could possibly have in this scenario that is using up so much memory. Unless some part of GRUB is trying to write an excessively large block to disk, but why would GRUB need to be doing that?

I think you need to find a way to reproduce the issue without GRUB involved.

dkiper May 26, 2026
Author

The GRUB cannot write anything, except small environment block/file, because it does not have write functionality/infrastructure.

Additionally, the GRUB is out of the picture when I run latest tests. I have just wrote simple UEFI application doing roughly this:

while (1) {
  Print(L"Stall...\n");
  uefi_call_wrapper(BS->Stall, 1, 10000000);
  Print(L"Read...\n");
  uefi_call_wrapper(Volume->Open, 5, Volume, &FileHandle, FileName, EFI_FILE_MODE_READ, EFI_FILE_READ_ONLY | EFI_FILE_HIDDEN | EFI_FILE_SYSTEM);
  ReadSize = FileSize(FileHandle);
  Buffer = AllocatePool(ReadSize);
  uefi_call_wrapper(FileHandle->Read, 3, FileHandle, &ReadSize, Buffer);
  uefi_call_wrapper(FileHandle->Close, 1, FileHandle);
  FreePool(Buffer);
  Print(L"Done!\n");
}

If you need full code or binary drop me a line. I can also create a backtrace of neighbour_destroy() call...

dkiper · 2026-06-08T18:35:13Z

dkiper
Jun 8, 2026
Author

Here is a backtrace...

#0  neighbour_queue_discard () at net/neighbour.c:539
#1  0x000000007dd90a91 in discard_cache (size=<optimized out>) at core/malloc.c:233
#2  0x000000007dd90ca3 in heap_alloc_block (heap=0x7ddc9d60, size=<optimized out>, align=<optimized out>, offset=0) at core/malloc.c:380
#3  0x000000007dd8d234 in alloc_iob_raw (len=<optimized out>, align=<optimized out>, offset=offset@entry=0) at core/iobuf.c:91
#4  0x000000007dd8d2d0 in alloc_iob (len=<optimized out>) at core/iobuf.c:145
#5  0x000000007dd8d310 in alloc_rx_iob (len=<optimized out>, dma=0x7ddd3108) at core/iobuf.c:193
#6  0x000000007dda6bcf in virtio_net_refill_rx (vnet=0x7ddd3a70) at drivers/net/virtio-net.c:282
#7  0x000000007dd94ecb in netdev_poll (netdev=0x7ddd36a8) at net/netdevice.c:625
#8  0x000000007dd85183 in efi_snp_poll (snpdev=0x7ddd3dc8, snpdev@entry=0x660) at interface/efi/efi_snp.c:173
#9  0x000000007dd853c3 in efi_snp_receive (snp=0x7ddd3df0, ll_header_len=0x7fefdaf0, len=0x7fefdaf8, data=0x7e53301e, ll_src=0x0, ll_dest=0x0, net_proto=0x0)
    at interface/efi/efi_snp.c:790

... and another one...

#0  neighbour_discard () at net/neighbour.c:567
#1  0x000000007dd90a91 in discard_cache (size=<optimized out>) at core/malloc.c:233
#2  0x000000007dd90ca3 in heap_alloc_block (heap=0x7ddc9d60, size=<optimized out>, align=<optimized out>, offset=0) at core/malloc.c:380
#3  0x000000007dd8d234 in alloc_iob_raw (len=<optimized out>, align=<optimized out>, offset=offset@entry=0) at core/iobuf.c:91
#4  0x000000007dd8d2d0 in alloc_iob (len=<optimized out>) at core/iobuf.c:145
#5  0x000000007dd8d310 in alloc_rx_iob (len=<optimized out>, dma=0x7ddd3108) at core/iobuf.c:193
#6  0x000000007dda6bcf in virtio_net_refill_rx (vnet=0x7ddd3a70) at drivers/net/virtio-net.c:282
#7  0x000000007dd94ecb in netdev_poll (netdev=0x7ddd36a8) at net/netdevice.c:625
#8  0x000000007dd85183 in efi_snp_poll (snpdev=0x7ddd3dc8, snpdev@entry=0x7ddcb960) at interface/efi/efi_snp.c:173
#9  0x000000007dd853c3 in efi_snp_receive (snp=0x7ddd3df0, ll_header_len=0x7fefdaf0, len=0x7fefdaf8, data=0x7e53869e, ll_src=0x0, ll_dest=0x0, net_proto=0x0)
    at interface/efi/efi_snp.c:790

0 replies

dkiper · 2026-06-30T09:54:14Z

dkiper
Jun 30, 2026
Author

@mcb30 Ping? Is there any chance you can take look at this? Or at least give some guidance?

3 replies

mcb30 Jun 30, 2026
Maintainer

If you can get me a way to reproduce this locally, then I can take a look. Assume that I have a bootable iPXE UEFI VM (in QEMU) and an iSCSI target already set up. What I would need from you is:

A minimal EFI application to place on the iSCSI target disk as BOOTX64.EFI. This should be in source code form, so that I can modify it as needed for debugging. Ideally as a standalone EDK2 package (see e.g. https://github.com/ipxe/ProxyLoaderPkg for an example of such a package), but a single source file that can be built with gnu-efi would be acceptable.
The command that you expect to be run to induce the packet flood to illustrate the problem.

dkiper Jun 30, 2026
Author

Here it is: efitest.c

If you have any questions please drop me a line...

mcb30 Jun 30, 2026
Maintainer

Thank you for the test code. I have built it and performed an iSCSI boot whilst running the flood ping command stated within efitest.c.

The problem does not reproduce for me. I see your test code go through multiple Stall... / Read... / Done! cycles with no issues.

I also tried reverting commit 2d8236995 and rerunning the test. Even with this commit reverted, I still don't ever see the NEIGHBOUR net0 IP x.x.x.x destroyed message. Your test code continues to go through multiple Stall... / Read... / Done! cycles with no issues.

I have run these tests in a QEMU VM, using an iPXE ROM (bin-x86_64-efi/8086100e.efirom) connected to a tap0 network device, which is bridged to the host. The host is itself the iSCSI target.

I am not able to reproduce your issue. Everything is behaving as expected.

dkiper · 2026-07-01T12:45:48Z

dkiper
Jul 1, 2026
Author

Hmmm... Interesting... I was able to reproduce the issue on hardware and QEMU. Though I was using in QEMU virtio net device, iPXE in ROM, iPXE as an EFI application served from TFTP server and iSCSI target hosted on separate virtual machine. Could you try virtio net driver?

Did you flood iPXE machine with packates during Stall... phase?

Could you share OVMF version which you are using in QEMU?

2 replies

mcb30 Jul 1, 2026
Maintainer

Hmmm... Interesting... I was able to reproduce the issue on hardware and QEMU. Though I was using in QEMU virtio net device, iPXE in ROM, iPXE as an EFI application served from TFTP server and iSCSI target hosted on separate virtual machine.

That's a lot of moving parts. Could you try simplifying your test scenario?

Could you try virtio net driver?

Tested. Same result, though it does run noticeably slower.

Did you flood iPXE machine with packates during Stall... phase?

Yes.

Could you share OVMF version which you are using in QEMU?

Built from source, most likely from commit c4d29cb62187060493a1f595083ddfb6dd346f39.

mcb30 Jul 2, 2026
Maintainer

Could you try virtio net driver?

Tested. Same result, though it does run noticeably slower.

In the process of tracking down why it was running slower, I encountered an unrelated issue that can result in large numbers of dropped transmissions when using virtio-net. This issue is fixed in commit a392b4f6d. It has nothing to do with the neighbour cache discard mechanism but you may well see different behaviour from the driver after this commit, so it would be worth retesting with the current master branch.

ajyoung-oracle · 2026-07-21T15:53:00Z

ajyoung-oracle
Jul 21, 2026

Hi Michael, I was asked to investigate this issue as we have customers hitting the issue.

I was able to reproduce the issue - very easily and consistently - with the following setup:

HOST: x86_64 desktop tower
OS: Oracle Linux 9
VM stack: QEMU standard stack provided by OL
iPXE version: latest upstream (commit 05e6256)

Here are the detailed reproduction steps:

I configured 2 QEMU VMs on the host:
* VM1 configured as an iSCSI server to provide disk image with EFI System Partition. The boot loader (/EFI/BOOT/BOOTX64.EFI) is replaced with Daniel's efi app (modified to stall for 30 secs instead of 10).

* VM2 is booted with iPXE built as an e1000 Option ROM (i.e. 8086100e.efirom). An iPXE embedded script attempts to sanhook/sanboot the iSCSI disk and run the boot loader (i.e. Daniel's app).

use nping from the host to flood VM2 using this command: sudo nping --icmp --icmp-type 8 --icmp-code 0 --rate 1000 --count 0

This will result in iPXE going into a doom-loop and the EFI app is effectively hung and the VM eventually reset after several minutes. We believe this is the BUG/issue our customers are experiencing.

More details:

The VMs and host are bridged with tap0/tap1 devices.

VM2 is booting using the following QEMU options:

/usr/libexec/qemu-kvm
-name guest=VM2_guest -m 4G
-smp 8
-cpu host
-machine q35,accel=kvm,kernel_irqchip=split
-drive file=/usr/share/OVMF/OVMF_CODE.pure-efi.fd,index=0,if=pflash,format=raw,readonly=on
-drive file=pwd/OVMF_VARS.pure-efi.fd.VM2_guest,index=1,if=pflash,format=raw
-netdev tap,id=net0,ifname=tap1,script=no,downscript=no
-device e1000,netdev=net0,mac=52:54:00:12:34:57,romfile=pwd/8086100e.efirom
-monitor stdio
-serial telnet:127.0.0.1:4557,server
-vga std

IPXE was built using:
make bin-x86_64-efi/8086100e.efidrv EMBED=./embed.ipxe DEBUG=netdevice,ipv4,tcp,neighbour:3
EfiRom -f 0x8086 -i 0x100e -e bin-x86_64-efi/8086100e.efidrv -o 8086100e.efirom

Embedded script is:
#!ipxe
ifconf net0
sanhook iscsi:192.168.0.50::::iqn.2026-07.com.test1:t1
sanboot --keep --filename \EFI\BOOT\BOOTX64.EFI

NOTE: Daniel's app does a stall followed by a file read in an infinite loop. I increased the stall timeout from 10 to 30 to more easily determine when the app is stalling or reading from the block file.

Something very important to note is, from my observations, the nping packet flood must happen concurrently with the file read period of the app. If the packet flood is limited to just during the stall period, iPXE recovers. So, this possibly points to some sort of concurrency issue in iPXE packet handling while accessing the iSCSI block device.

If you need another config details, please let me know.

A key difference between my setup and yours could be the iSCSI server on a separate VM (mine) versus the host (yours)?

Perhaps you can try to reproduce again using the above more specific configuration?

If I can be of any further assistance (to obtain debug information, run specific tests, etc.), let me know.

Thanks!

-Aaron

@mcb30

5 replies

mcb30 Jul 21, 2026
Maintainer

Hi Michael, I was asked to investigate this issue as we have customers hitting the issue.

Understood. I have already spent quite a bit of time looking into this at my own expense. Since this is a customer issue for Oracle, I am happy to continue further on a commercial basis: please contact me directly by email to arrange.

mcb30 Jul 29, 2026
Maintainer

@ajyoung-oracle Just checking that you received my reply with the suggested route forward?

dkiper Jul 29, 2026
Author

@mcb It looks @ajyoung-oracle and I identified a culprit. Reverting commit 1295b4a ([efi] Allow initialisation via SNP interface even while claimed) fixes the issue for iPXE with SNP driver on bare metal. However, for VirtIO driver we had to "revert" c70b3e0 ([efi] Always enable recursion when calling ConnectController()) too (we could not simply revert this commit; so we replaced ConnectController() last argument with FALSE where possible). Additionally, we had to apply bounded_cache.patch generated by Codex with following properties:

Four queued packets per neighbour.
32 entries globally and eight per device/protocol.
Completed gateway and iSCSI neighbours are never allocator victims.

We were also able to reproduce the issue with e1000 driver in VM but in this case we had to revert commit e10a40d ([efi] Avoid dropping below TPL as at entry to iPXE) instead to fix the issue.

We are almost sure that reverting the commits mentioned above and applying Codex generated code (with rough review only) are not real fixes. They are just showing where we should look for problems. So, if you could share a few hints with us where to look at then we are happy to work on real fix/fixes for the issues.

mcb30 Jul 29, 2026
Maintainer

We are almost sure that reverting the commits mentioned above and applying Codex generated code (with rough review only) are not real fixes. They are just showing where we should look for problems. So, if you could share a few hints with us where to look at then we are happy to work on real fix/fixes for the issues.

They are absolutely not real fixes, and unfortunately I don't think they even show where to look for problems. All that they really show is that if you make large numbers of apparently arbitrary and highly intrusive changes that cause substantial changes in behaviour then the behaviour changes, and it happens to change enough that the behaviour you were previously observing no longer occurs.

I'm not going to comment further on the LLM-authored patch beyond saying that adding complexity to temporarily hide a problem is never a good idea.

Rather than letting an agentic LLM chaos monkey poke semi-randomly at the code, I think the only sensible way forward is to find a way that reliably reproduces the issue outside of Oracle. Get a reproduction that involves vanilla upstream (or non-Oracle distro) QEMU and EDK2, a vanilla upstream (or non-Oracle distro) iSCSI target, the current unmodified iPXE master branch, and the simplest possible set of reproduction instructions. That's where any engineering effort ought to be focused.

dkiper Sep 8, 2026
Author

Sorry for late reply but I was on vacation. Now I should be able to respond on more timely manner. Below you can find all needed details about environment in which you can reproduce the issue.

Host: Intel NUC7i5DNHE/NUC7i5DNB, BIOS DNKBLi5v.86A.0066.2019.0710.1158 07/10/2019, 8 GiB RAM
OS: Debian GNU/Linux 13.6 (trixie)

Additional packages: dosfstools gcc git gnu-efi kpartx make nmap ovmf qemu-system-x86 targetcli-fb

New EFI test app which should be installed at VM iSCSI target ESP:/efi/boot/bootx64.efi: efitest.c

Create 256 MiB file at VM iSCSI target ESP:/file.bin.

Latest iPXE master: make bin-x86_64-efi-rom/virtio-net.efirom EMBED=autoexec.ipxe DEBUG=tcp,neighbour

autoexec.ipxe:

#!ipxe
dhcp
sanboot iscsi:<IP_addr_of_iSCSI_server>::::<IQN>:<target>

iSCSI target is served from the host on which QEMU runs.

VM start:

/usr/bin/qemu-system-x86_64 -nodefaults -cpu host -smp 4 -m 5G -vga vmware -vnc 0.0.0.0:<display> \
  -serial telnet:0.0.0.0:<TCP_port>,server,nowait -parallel none -net nic,model=virtio,macaddr=<macaddr> \
  -net bridge,br=<bridge> -monitor stdio -M q35,accel=kvm \
  -drive file=/usr/share/OVMF/OVMF_CODE_4M.fd,format=raw,if=pflash,unit=0,readonly=on \
  -drive file=<path_to_OVMF_VARS_4M.fd_copy>,format=raw,if=pflash,unit=1 \
  -global virtio-net-pci.romfile=bin-x86_64-efi-rom/virtio-net.efirom

Flood command started from the QEMU host: nping --tcp --flags syn --dest-port 46017 --rate 1000 --win 62580 --count 0 <VM_IP_addr>

If you still have any questions please drop me a line...

Uh oh!

Uh oh!

iPXE drops iSCSI connection when flooded with network packets #1714

Uh oh!

dkiper May 20, 2026

Replies: 7 comments · 27 replies

Uh oh!

mcb30 May 20, 2026 Maintainer

Uh oh!

Uh oh!

dkiper May 20, 2026 Author

Uh oh!

dkiper May 20, 2026 Author

Uh oh!

mcb30 May 20, 2026 Maintainer

Uh oh!

dkiper May 21, 2026 Author

Uh oh!

mcb30 May 21, 2026 Maintainer

Uh oh!

dkiper May 21, 2026 Author

Uh oh!

Uh oh!

mcb30 May 21, 2026 Maintainer

Uh oh!

dkiper May 22, 2026 Author

Uh oh!

mcb30 May 24, 2026 Maintainer

Uh oh!

dkiper May 25, 2026 Author

Uh oh!

mcb30 May 26, 2026 Maintainer

Uh oh!

dkiper May 26, 2026 Author

Uh oh!

Uh oh!

dkiper Jun 8, 2026 Author

Uh oh!

dkiper Jun 30, 2026 Author

Uh oh!

mcb30 Jun 30, 2026 Maintainer

Uh oh!

dkiper Jun 30, 2026 Author

Uh oh!

mcb30 Jun 30, 2026 Maintainer

Uh oh!

dkiper Jul 1, 2026 Author

Uh oh!

mcb30 Jul 1, 2026 Maintainer

Uh oh!

mcb30 Jul 2, 2026 Maintainer

Uh oh!

Uh oh!

ajyoung-oracle Jul 21, 2026

Uh oh!

mcb30 Jul 21, 2026 Maintainer

Uh oh!

mcb30 Jul 29, 2026 Maintainer

Uh oh!

dkiper Jul 29, 2026 Author

Uh oh!

mcb30 Jul 29, 2026 Maintainer

Uh oh!

dkiper Sep 8, 2026 Author

dkiper
May 20, 2026

Replies: 7 comments 27 replies

mcb30
May 20, 2026
Maintainer

dkiper May 20, 2026
Author

dkiper May 20, 2026
Author

mcb30
May 20, 2026
Maintainer

dkiper May 21, 2026
Author

mcb30 May 21, 2026
Maintainer

dkiper May 21, 2026
Author

mcb30
May 21, 2026
Maintainer

dkiper May 22, 2026
Author

mcb30 May 24, 2026
Maintainer

dkiper May 25, 2026
Author

mcb30 May 26, 2026
Maintainer

dkiper May 26, 2026
Author

dkiper
Jun 8, 2026
Author

dkiper
Jun 30, 2026
Author

mcb30 Jun 30, 2026
Maintainer

dkiper Jun 30, 2026
Author

mcb30 Jun 30, 2026
Maintainer

dkiper
Jul 1, 2026
Author

mcb30 Jul 1, 2026
Maintainer

mcb30 Jul 2, 2026
Maintainer

ajyoung-oracle
Jul 21, 2026

mcb30 Jul 21, 2026
Maintainer

mcb30 Jul 29, 2026
Maintainer

dkiper Jul 29, 2026
Author

mcb30 Jul 29, 2026
Maintainer

dkiper Sep 8, 2026
Author