Graphics on demand and load.

dirty_al

08-06-2004 23:05:55

I am monitoring a storage and it has 47 LUNs. Each LUN has 3 graphics and for each graphics 2 queries are being done.
The load of my server stays at 3, as a lot of RRDs get updated ate each polling, that is done each 15 seconds, because of the storage design.
I wonder if the collector could put all the information it collets in the mysql database and the graphics would be generate on demand when a click on the device tree.
I know that this is another aproach to the project, but I thing if could make NetMRG scale a lot more on the same hardware, I think.

Thanks.

silfreed

09-06-2004 09:16:49

Every 15 seconds? That's crazy.

Seriously, NetMRG's design is to use RRD as the backend for storing data and displaying graphs. Graphs actually aren't stored when the gatherer runs; it only updates the data in the rrd files; graphs are rendered 'on the fly' when you view them in the device tree.
MRTG actually did it the way you're describing; it would write 4 images plus the data file; this would literally kill servers in my experience (at least two a year).
The actual act of updating the RRD files is very quick; they are a form of database themselves, they're just in much smaller files than MySQL and aren't cached in RAM. To save the data that is stored in the rrd files in MySQL would involve a significant amount of work; the beauty of RRD is in it's name - 'Round Robin Database'; it's design is to age out data in a controlled fashion so you're not keeping it around forever (which would imply ever-increasing storage requirements).

Hopefully this explains why we chose to do things the way we did.

-Doug

dirty_al

09-06-2004 13:18:50

I understand the project design.
My dream graphic system would be the NetMRg interface, with the the metmrg poller and the Nisca way of generating graphics on demand by time period.
I understand that the round robin database makes thinks easier.
Thanks and NetMrg is really good and works very well, we should onlye have thread by subdevices, as a topic that a opened some time ago. regarding the same storage with 47 LUNs.
[]s.

silfreed

09-06-2004 14:09:59

[quoteb10fad418f="dirty_al"]I understand the project design.
My dream graphic system would be the NetMRg interface, with the the metmrg poller and the Nisca way of generating graphics on demand by time period.
I understand that the round robin database makes thinks easier.[/quoteb10fad418f]
We [ib10fad418f]do[/ib10fad418f] generate graphs (graphics) on demand, we just store the database in a file.

[quoteb10fad418f="dirty_al"]Thanks and NetMrg is really good and works very well, we should onlye have thread by subdevices, as a topic that a opened some time ago. regarding the same storage with 47 LUNs.[/quoteb10fad418f]
Yep, I remember that then, too. You do have an interesting application (and those are good!). I'll bug balleman to jump on this thread and comment on the per-subdevice threads; I'm not sure if he was following the thread to the point where you meantioned about subdevice threads.

BTW, look for a release in the next day or so with the 'ignore snmp uptime' option.

-Doug

balleman

09-06-2004 14:36:30

With regard to threading based on sub-devices... we have lots of reasons against it, but most of them depend a lot on the circumstances.

1. Some (many) SNMP agents, especially those in special-purpose hardware, handle multiple simultaneous queries poorly. IIRC, the Cisco 5509 is one such example, where the queries seem to be serialized before being responded to.

2. Each thread needs lots of resources. For each thread, we make a MySQL connection, initialize an SNMP session, and other things that by themselves are simple, but when done in greater quantity generate a lot of unnecessary overhead.

In special instances such as those you have described, the best I can suggest is to arbitrarily split the subdevices between two device entries for your server. It may not be semantically correct, but it will give you the benefits of threaded polling on a single device.

If we do add the feature to thread on subdevices, it will have to be an optional one. A gatherer can easily become tarpitted if an 'evil' device is being processed, and each thread is waiting for the same device to respond.

Well, keep the suggestions coming. What you're doing sounds very impressive. Thanks for monitoring with NetMRG. -)

dirty_al

11-06-2004 09:27:38

Based on what you said, subdevice based threading can be very dangerous. I remember that even some old UCD-SNMP based linux boxes used hang the agent when multiple queries would be done.
I already divide the storage into 3 virtual hosts each collecting a different OID, so I am getting 3 threads now. Another was the SNMP UpTime that did not allowed to use the netmrg snmp collector. I am calling a extrenal script, which loads the system a little more.
I will upgrade to 0.16 and try to move all collections to netmrg SNMP collector.
Thanks a lot for all the answers.

[]s.