Multiple Execution Contexts in lwIP code

The most common source of lwIP problems is to have multiple execution contexts inside the lwIP code.

lwIP can be used in two basic modes: Mainloop mode ("NO_SYS") (no OS/RTOS running on target system) or OS mode (TCPIP thread) (there is an OS running on the target system).

See also: Multithreading (especially the part about LWIP_ASSERT_CORE_LOCKED()!)

Mainloop Mode

In mainloop mode, only "raw" APIs can be used. The user has two possibilities to ensure there is only one exection context at a time in lwIP:

1) Deliver RX ethernet packets directly in interrupt context to lwIP by calling netif->input directly in interrupt. This implies all lwIP callback functions are called in IRQ context, which may cause further problems in application code: IRQ is blocked for a long time, multiple execution contexts in application code etc. When the application wants to call lwIP, it only needs to disable interrupts during the call. If timers are involved, even more locking code is needed to lock out timer IRQ and ethernet IRQ from each other, assuming these may be nested.

2) Run lwIP in a mainloop. There is example code here: Mainloop mode ("NO_SYS"). lwIP is ONLY called from mainloop callstacks here. The ethernet IRQ has to put received telegrams into a queue which is polled in the mainloop. Ensure lwIP is NEVER called from an interrupt, e.g. some SPI IRQ wants to forward data to udp_send() or tcp_write()!

OS Mode

In OS mode, "raw" APIs AND Sequential-style APIs can be used. Sequential-style APIs are designed to be called from threads other than the TCPIP thread, so there is nothing to consider here. But "raw" APIs functions must ONLY be called from TCPIP thread. It is a common error to call these from other threads or from IRQ contexts. Ethernet RX needs to deliver incoming packets in the correct way by sending a message to TCPIP thread, this is implemented in tcpip_input(). Again, ensure lwIP is NEVER called from an interrupt, e.g. some SPI IRQ wants to forward data to udp_send() or tcp_write()!

1) tcpip_callback() can be used get called back from TCPIP thread, it is safe to call any "raw" APIs from there.

2) Use LWIP_TCPIP_CORE_LOCKING. All "raw" APIs functions can be called when lwIP core lock is aquired, see LOCK_TCPIP_CORE() and UNLOCK_TCPIP_CORE(). These macros cannot be used in an interrupt context! Note the OS must correctly handle priority inversion for this.

Cache / DMA issues

DMA-capable ethernet hardware and zero-copy RX

lwIP changes the content of RECEIVED pbufs in the TCP code path. This implies one or more cacheline(s) of the RX pbuf become dirty and need to be flushed before the memory is handed over to the DMA ethernet hardware for the next telegram to be received. See http://lists.nongnu.org/archive/html/lwip-devel/2017-12/msg00070.html for a more detailed explanation. Also keep in mind the user application may also write into pbufs, so it is generally a bug not to flush the data cache before handing a buffer to DMA hardware.

DMA-capable ethernet hardware and cacheline alignment

Nice description about DMA capable hardware and buffer handling: http://www.pebblebay.com/a-guide-to-using-direct-memory-access-in-embedded-systems-part-two/ Read especially sections "Cache coherency" and "Buffer alignment".