Details
-
Improvement
-
Resolution: Unresolved
-
Major
-
None
-
Professional Edition
-
None
Description
We had an issue with always increasing memory demand of the ports endpoint on Observium, it was caused by loading the entire ports table into memory, modifying it in place, and then encoding the whole array as json, and only then finally sending it to the requester.
Attached is code I made as a proof of concept. it is compatible with the current version with the only json difference being that the "count" key appears last, since json key order is not important that should be fine.
The attached script selects the same ports, and then uses an iterator to fetch the result, encode and send it to the requester.
I agree the attached script is a ... bit hacky, a proper script might include a streaming json encoder, but it's mostly a proof of concept, that works and is in production one instance.
Attachments
Activity
I can understand that, but in our case, we need all ports for billing reasons.
Do you mean you're also adding pagination?
haven't wanted to switch away from our existing functions and style, but i have added the ability to limit fields returned in the API, does this help? 
Not exactly. You can now specify a subset of fields for specific use cases, say only port_name, ifIndex and ifAlias.
Cuts down data transfer a lot, not sure if it helps in your case.