OK. When I re-enable the functionality, I now see that when I mouse over the IPv4 on a server with 10 IP addresses on one port, I see a list of that ports subnet repeated 37K times. So, there's the source of the memory issue.
Clearly that value is greater when there are more addresses.
On a server with one IP on a port, the subnet is listed just once.
With the change I proposed, none are listed, so clearly that's not a good solution, just as you suggest.
If I change the line to this:
$int_links_v4[$new['port_id']][$new['ipv4_network']] = $new['ipv4_network'];
everything seems to work correctly, for servers with any number of IPs.
Assuming this change is the right one, the IPv6 line might need to be:
$int_links_v6[$new['port_id']][$new['ipv6_network']] = $new['ipv6_network'];
Fixed in r5763.