2017-10-19 05:25 CEST

View Issue Details Jump to Notes ]
IDProjectCategoryView StatusLast Update
0001079openmediavaultFeaturepublic2017-10-18 00:05
ReporterWastlJ 
Assigned Tovotdev 
PrioritylowSeverityfeatureReproducibilityN/A
StatusassignedResolutionopen 
Product VersionSardaukar (0.5.x) 
Target VersionNextFixed in Version 
Summary0001079: Disk IO graphs in System Information
DescriptionThe various graphs in system info are really useful, but they're lacking disk io which I always found useful in other distributions.
Additional Informationhttp://forums.openmediavault.org/index.php/Thread/5294-Disk-IO-graphs-in-System-Information/?postID=47053#post47053
TagsNo tags attached.
Product build
Attached Files
  • zip file icon disk.zip (1,654 bytes) 2014-10-27 13:23
  • txt file icon disk.txt (3,345 bytes) 2017-10-12 16:27 -
    #!/bin/sh
    #
    # This file is part of OpenMediaVault.
    #
    # @license   http://www.gnu.org/licenses/gpl.html GPL Version 3
    # @author    Volker Theile <volker.theile@openmediavault.org>
    # @copyright Copyright (c) 2009-2014 Volker Theile
    #
    # OpenMediaVault is free software: you can redistribute it and/or modify
    # it under the terms of the GNU General Public License as published by
    # the Free Software Foundation, either version 3 of the License, or
    # any later version.
    #
    # OpenMediaVault is distributed in the hope that it will be useful,
    # but WITHOUT ANY WARRANTY; without even the implied warranty of
    # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    # GNU General Public License for more details.
    #
    # You should have received a copy of the GNU General Public License
    # along with OpenMediaVault. If not, see <http://www.gnu.org/licenses/>.
    
    # Documentation/Howto:
    # http://www.korbinian-pauli.de/artikel/mit-lm-sensors-und-rrd-cpu-und-mb-temperatur-uberwachen
    # http://www.uvm.edu/~bcodding/ticker/archives/33
    # http://blog.stefan-betz.net/2009/1/6/collectd-und-rrdtool-im-einsatz
    # http://wiki.lugbz.de/HowTo/SelfMadeMonitoring
    # https://jeremy.visser.name/2010/02/23/enable-caching-in-collectd
    
    set -e
    
    . /etc/default/openmediavault
    . /usr/share/openmediavault/scripts/helper-functions
    
    OMV_COLLECTD_CONFIG=${OMV_COLLECTD_CONFIG:-"/etc/collectd/collectd.conf"}
    OMV_COLLECTD_RRDTOOL_MKGRAPH=${OMV_COLLECTD_RRDTOOL_MKGRAPH:-"/usr/sbin/omv-mkgraph"}
    
    xmlstarlet sel -t \
      -o "LoadPlugin disk" -n \
      ${OMV_CONFIG_FILE} | xmlstarlet unesc >> ${OMV_COLLECTD_CONFIG}
    
    DISKS_SDX="$(ls -l /dev/disk/by-uuid/ | awk '{print $11}' | sed -e 's/.*\/\([a-z]*\).*/\1/g' | sort -u | tr '\n' ' ')"
      
    cat <<EOF >> ${OMV_COLLECTD_RRDTOOL_MKGRAPH}
    # Plugin: disk
    TITLE_DF="Disk I/O"
    COLOR_LINE_DISK_READ="#00cc00" # green
    COLOR_LINE_DISK_WRITE="#ff0000" # red
    
    for dirname in $DISKS_SDX; do
    	[ ! -e "\${DATA}/disk-\${dirname}" ] && continue
    	# only for the first partition! (see grep below)
    	UUID_DISK="\$(ls -l /dev/disk/by-uuid | grep \${dirname}1 | awk '{print \$9}')"
    EOF
    for ucperiod in HOUR DAY WEEK MONTH YEAR; do
    	lcperiod=$(echo ${ucperiod} | tr '[:upper:]' '[:lower:]')
    	cat <<EOF >> ${OMV_COLLECTD_RRDTOOL_MKGRAPH}
    	rrdtool graph \${IMGDIR}/disk-media-\${UUID_DISK}-${lcperiod}.png --start \${${ucperiod}START} \${DEFAULTS} --title "\${TITLE_DF}\${${ucperiod}TITLE}" --slope-mode --lower-limit 0 --vertical-label "Bytes/s" \
    	  DEF:ravg=\${DATA}/disk-\${dirname}/disk_octets.rrd:read:AVERAGE \
    	  DEF:rmin=\${DATA}/disk-\${dirname}/disk_octets.rrd:read:MIN \
    	  DEF:rmax=\${DATA}/disk-\${dirname}/disk_octets.rrd:read:MAX \
    	  DEF:wavg=\${DATA}/disk-\${dirname}/disk_octets.rrd:write:AVERAGE \
    	  DEF:wmin=\${DATA}/disk-\${dirname}/disk_octets.rrd:write:MIN \
    	  DEF:wmax=\${DATA}/disk-\${dirname}/disk_octets.rrd:write:MAX \
    	  LINE1:ravg\${COLOR_LINE_DISK_READ}:"Read" \
    	  GPRINT:rmin:MIN:"%5.1lf%sB Min" \
    	  GPRINT:ravg:AVERAGE:"%5.1lf%sB Avg" \
    	  GPRINT:rmax:MAX:"%5.1lf%sB Max" \
    	  GPRINT:ravg:LAST:"%5.1lf%sB Last\l" \
    	  LINE1:wavg\${COLOR_LINE_DISK_WRITE}:"Write" \
    	  GPRINT:wmin:MIN:"%5.1lf%sB Min" \
    	  GPRINT:wavg:AVERAGE:"%5.1lf%sB Avg" \
    	  GPRINT:wmax:MAX:"%5.1lf%sB Max" \
    	  GPRINT:wavg:LAST:"%5.1lf%sB Last\l" \
    	  COMMENT:"\${LASTUPDATE}" >/dev/null
    EOF
    done
    cat <<EOF >> ${OMV_COLLECTD_RRDTOOL_MKGRAPH}
    done
    EOF
    
    txt file icon disk.txt (3,345 bytes) 2017-10-12 16:27 +
  • txt file icon disk-2.txt (3,345 bytes) 2017-10-13 17:30 -
    #!/bin/sh
    #
    # This file is part of OpenMediaVault.
    #
    # @license   http://www.gnu.org/licenses/gpl.html GPL Version 3
    # @author    Volker Theile <volker.theile@openmediavault.org>
    # @copyright Copyright (c) 2009-2014 Volker Theile
    #
    # OpenMediaVault is free software: you can redistribute it and/or modify
    # it under the terms of the GNU General Public License as published by
    # the Free Software Foundation, either version 3 of the License, or
    # any later version.
    #
    # OpenMediaVault is distributed in the hope that it will be useful,
    # but WITHOUT ANY WARRANTY; without even the implied warranty of
    # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    # GNU General Public License for more details.
    #
    # You should have received a copy of the GNU General Public License
    # along with OpenMediaVault. If not, see <http://www.gnu.org/licenses/>.
    
    # Documentation/Howto:
    # http://www.korbinian-pauli.de/artikel/mit-lm-sensors-und-rrd-cpu-und-mb-temperatur-uberwachen
    # http://www.uvm.edu/~bcodding/ticker/archives/33
    # http://blog.stefan-betz.net/2009/1/6/collectd-und-rrdtool-im-einsatz
    # http://wiki.lugbz.de/HowTo/SelfMadeMonitoring
    # https://jeremy.visser.name/2010/02/23/enable-caching-in-collectd
    
    set -e
    
    . /etc/default/openmediavault
    . /usr/share/openmediavault/scripts/helper-functions
    
    OMV_COLLECTD_CONFIG=${OMV_COLLECTD_CONFIG:-"/etc/collectd/collectd.conf"}
    OMV_COLLECTD_RRDTOOL_MKGRAPH=${OMV_COLLECTD_RRDTOOL_MKGRAPH:-"/usr/sbin/omv-mkgraph"}
    
    xmlstarlet sel -t \
      -o "LoadPlugin disk" -n \
      ${OMV_CONFIG_FILE} | xmlstarlet unesc >> ${OMV_COLLECTD_CONFIG}
    
    DISKS_SDX="$(ls -l /dev/disk/by-uuid/ | awk '{print $11}' | sed -e 's/.*\/\([a-z]*\).*/\1/g' | sort -u | tr '\n' ' ')"
      
    cat <<EOF >> ${OMV_COLLECTD_RRDTOOL_MKGRAPH}
    # Plugin: disk
    TITLE_DF="Disk I/O"
    COLOR_LINE_DISK_READ="#00cc00" # green
    COLOR_LINE_DISK_WRITE="#ff0000" # red
    
    for dirname in $DISKS_SDX; do
    	[ ! -e "\${DATA}/disk-\${dirname}" ] && continue
    	# only for the first partition! (see grep below)
    	UUID_DISK="\$(ls -l /dev/disk/by-uuid | grep \${dirname}1 | awk '{print \$9}')"
    EOF
    for ucperiod in HOUR DAY WEEK MONTH YEAR; do
    	lcperiod=$(echo ${ucperiod} | tr '[:upper:]' '[:lower:]')
    	cat <<EOF >> ${OMV_COLLECTD_RRDTOOL_MKGRAPH}
    	rrdtool graph \${IMGDIR}/disk-media-\${UUID_DISK}-${lcperiod}.png --start \${${ucperiod}START} \${DEFAULTS} --title "\${TITLE_DF}\${${ucperiod}TITLE}" --slope-mode --lower-limit 0 --vertical-label "Bytes/s" \
    	  DEF:ravg=\${DATA}/disk-\${dirname}/disk_octets.rrd:read:AVERAGE \
    	  DEF:rmin=\${DATA}/disk-\${dirname}/disk_octets.rrd:read:MIN \
    	  DEF:rmax=\${DATA}/disk-\${dirname}/disk_octets.rrd:read:MAX \
    	  DEF:wavg=\${DATA}/disk-\${dirname}/disk_octets.rrd:write:AVERAGE \
    	  DEF:wmin=\${DATA}/disk-\${dirname}/disk_octets.rrd:write:MIN \
    	  DEF:wmax=\${DATA}/disk-\${dirname}/disk_octets.rrd:write:MAX \
    	  LINE1:ravg\${COLOR_LINE_DISK_READ}:"Read" \
    	  GPRINT:rmin:MIN:"%5.1lf%sB Min" \
    	  GPRINT:ravg:AVERAGE:"%5.1lf%sB Avg" \
    	  GPRINT:rmax:MAX:"%5.1lf%sB Max" \
    	  GPRINT:ravg:LAST:"%5.1lf%sB Last\l" \
    	  LINE1:wavg\${COLOR_LINE_DISK_WRITE}:"Write" \
    	  GPRINT:wmin:MIN:"%5.1lf%sB Min" \
    	  GPRINT:wavg:AVERAGE:"%5.1lf%sB Avg" \
    	  GPRINT:wmax:MAX:"%5.1lf%sB Max" \
    	  GPRINT:wavg:LAST:"%5.1lf%sB Last\l" \
    	  COMMENT:"\${LASTUPDATE}" >/dev/null
    EOF
    done
    cat <<EOF >> ${OMV_COLLECTD_RRDTOOL_MKGRAPH}
    done
    EOF
    
    txt file icon disk-2.txt (3,345 bytes) 2017-10-13 17:30 +
  • zip file icon omv-disk.zip (5,019 bytes) 2017-10-18 00:05

-Relationships
+Relationships

-Notes

~0003066

votdev (administrator)

https://collectd.org/wiki/index.php/Plugin:Disk

~0003316

solo0816 (reporter)

The disk.zip file contains a collectd script for disk-IO in OMV. It saves the PNGs as "disk-UUID-of-the-disk-$time.png"
the script converts e.g. sda to the system-UUID (only the first partition at the moment), so the PNG can easily be assigned.

~0003317

votdev (administrator)

Does this collectd plugin wakeup drives that are in standby?

~0003318

solo0816 (reporter)

haven't tried.
It writes only the IO to the rrd-db.

~0003320

votdev (administrator)

The directory naming schema of the 'disk' plugin is not really good. The device name may change during a reboot. Any idea if this is configurable in any way. I think a device ID is better than the name.

~0003334

solo0816 (reporter)

Don't know it /dev/disk/by-uuid is possible in the disk plugin
Will try and report back

~0003335

solo0816 (reporter)

I can't get it to work with:
/dev/disk/by-uuid/
/dev/disk/by-uuid/1234-5678-etc
/media/1234-5678-etc
Should we drop this, until collectd supports UUIDs?

~0003848

raulfg3 (reporter)

some news?

~0004956

Silberlog (reporter)

Any news about adding disk-io stats graphs in OMV ?

~0004957

votdev (administrator)

Last edited: 2017-10-12 14:25

View 2 revisions

Not from my side because it's too much work (other things have higher prio), but hey, this is an OSS project, everyone can contribute.

~0004958

Silberlog (reporter)

Thanks for answering.
Yes of course i know this is an OSS project and that everyone can contribute, keep cool ;-)
I'd just want to know if there was some progress on this works that's all ...
So i'll try to have a look at this an restart from the disk.zip attached in this thread.

Best regards,

~0004959

votdev (administrator)

I don't know because i can not open the archive, but maybe it's a good starting point. Please note that the collectd/RRD implementation has been changed in OMV4, see https://github.com/openmediavault/openmediavault/commit/2db2970e78a169c90207802624a00e632d765389.

We should also take care that this collectd plugin does not prevent devices from spin down or wake them up when AAP/AAM options are enabled.

~0004960

Silberlog (reporter)

I was able to open the ZIP file, here it is

~0004961

votdev (administrator)

This is only 1/3 of the work to do.

This is the list of things to be done:
- The collectd config must be adapted
- Write a plugin for omv-mkrrdgraph
- Adapt the WebUI to display the graphs in a new tab

~0004962

Silberlog (reporter)

I think it's okay for the 2 first steps.
How can we adapt the WebUI to display graphs ?
I'm a bit lost to find the way to do this ...

~0004963

votdev (administrator)

Have a look here
https://github.com/openmediavault/openmediavault/tree/master/deb/openmediavault/var/www/openmediavault/js/omv/module/admin/diagnostic/system
https://github.com/openmediavault/openmediavault/tree/master/deb/openmediavault/var/www/openmediavault/js/omv/module/admin/diagnostic/system/plugin

~0004964

Silberlog (reporter)

Here is a working version for OMV 3.X mkgraph plugion for disk statistics.
Tested on a 3 disks VM (sda sdb and sdc).
RRD datas and Graphs are well generated, maybe graphs should be perfectible.
I tried to display the graphs in a new tab but don't succeed for the moment, harder and i'm not a webdev ...

~0004965

votdev (administrator)

Last edited: 2017-10-13 17:37

View 2 revisions

Please keep in mind that new features only go into OMV4 if you plan to contribute your work.

~0004966

Silberlog (reporter)

Sure.
Is there an ISO or a documentation for installing OMV 4.X on my VM please ?
Thx

~0004968

Silberlog (reporter)

Okay just installed a new Debian 9 VM and OMV 4.0.6
FYI, folder /var/lib/openmedivault/rrd isn't created by setup and omv-initsystem said :

root@omv4:~# omv-initsystem
W: mdadm: /etc/mdadm/mdadm.conf defines no arrays.
ERROR: Could not save png to '/var/lib/openmediavault/rrd/load-hour.png'
ERROR: Could not save png to '/var/lib/openmediavault/rrd/memory-hour.png'
ERROR: Could not save png to '/var/lib/openmediavault/rrd/df-root-hour.png'
ERROR: Could not save png to '/var/lib/openmediavault/rrd/cpu-0-hour.png'
ERROR: Could not save png to '/var/lib/openmediavault/rrd/load-day.png'
ERROR: Could not save png to '/var/lib/openmediavault/rrd/memory-day.png'
ERROR: Could not save png to '/var/lib/openmediavault/rrd/df-root-day.png'
ERROR: Could not save png to '/var/lib/openmediavault/rrd/cpu-0-day.png'
ERROR: Could not save png to '/var/lib/openmediavault/rrd/load-week.png'
ERROR: Could not save png to '/var/lib/openmediavault/rrd/memory-week.png'
ERROR: Could not save png to '/var/lib/openmediavault/rrd/df-root-week.png'
ERROR: Could not save png to '/var/lib/openmediavault/rrd/cpu-0-week.png'
ERROR: Could not save png to '/var/lib/openmediavault/rrd/load-month.png'
ERROR: Could not save png to '/var/lib/openmediavault/rrd/memory-month.png'
ERROR: Could not save png to '/var/lib/openmediavault/rrd/df-root-month.png'
ERROR: Could not save png to '/var/lib/openmediavault/rrd/cpu-0-month.png'
ERROR: Could not save png to '/var/lib/openmediavault/rrd/load-year.png'
ERROR: Could not save png to '/var/lib/openmediavault/rrd/memory-year.png'
ERROR: Could not save png to '/var/lib/openmediavault/rrd/df-root-year.png'
ERROR: Could not save png to '/var/lib/openmediavault/rrd/cpu-0-year.png'
root@omv4:~#

Creating the foilder manually solved the problem and graphs are generated.
Now i'm going to see to rewrite my plugin for OMV4 ...

~0004973

votdev (administrator)

>Okay just installed a new Debian 9 VM and OMV 4.0.6
>FYI, folder /var/lib/openmedivault/rrd isn't created by setup and omv-initsystem said :

Fixed with https://github.com/openmediavault/openmediavault/commit/f29fc025a9dc4f6e610c421bea1aa3e62786552f

~0004976

Silberlog (reporter)

Another bug i think, there are identical multiple entries in /etc/collectd/collectd.conf.d/*.conf files. Each time we run omv-mkconf collectd, entries for plugins are added instead of replaced :

E.g. for df plugin :

root@omv4:/usr/share/openmediavault/mkrrdgraph/plugins.d# cat /etc/collectd/collectd.conf.d/df.conf
LoadPlugin df
<Plugin df>
  MountPoint "/"
  IgnoreSelected false
</Plugin>
LoadPlugin df
<Plugin df>
  MountPoint "/"
  IgnoreSelected false
</Plugin>
LoadPlugin df
<Plugin df>
  MountPoint "/"
  IgnoreSelected false
</Plugin>
LoadPlugin df
<Plugin df>
  MountPoint "/"
  IgnoreSelected false
</Plugin>
LoadPlugin df
<Plugin df>
  MountPoint "/"
  IgnoreSelected false
</Plugin>
LoadPlugin df
<Plugin df>
  MountPoint "/"
  IgnoreSelected false
</Plugin>
LoadPlugin df
<Plugin df>
  MountPoint "/"
  IgnoreSelected false
</Plugin>
root@omv4:/usr/share/openmediavault/mkrrdgraph/plugins.d# omv-mkconf collectd
root@omv4:/usr/share/openmediavault/mkrrdgraph/plugins.d# cat /etc/collectd/collectd.conf.d/df.conf
LoadPlugin df
<Plugin df>
  MountPoint "/"
  IgnoreSelected false
</Plugin>
LoadPlugin df
<Plugin df>
  MountPoint "/"
  IgnoreSelected false
</Plugin>
LoadPlugin df
<Plugin df>
  MountPoint "/"
  IgnoreSelected false
</Plugin>
LoadPlugin df
<Plugin df>
  MountPoint "/"
  IgnoreSelected false
</Plugin>
LoadPlugin df
<Plugin df>
  MountPoint "/"
  IgnoreSelected false
</Plugin>
LoadPlugin df
<Plugin df>
  MountPoint "/"
  IgnoreSelected false
</Plugin>
LoadPlugin df
<Plugin df>
  MountPoint "/"
  IgnoreSelected false
</Plugin>
LoadPlugin df
<Plugin df>
  MountPoint "/"
  IgnoreSelected false
</Plugin>
root@omv4:/usr/share/openmediavault/mkrrdgraph/plugins.d#

Regards,

~0004977

Silberlog (reporter)

Fixed with :

${OMV_CONFIG_FILE} | xmlstarlet unesc > "${OMV_COLLECTD_CONFIG_DIR}/df.conf"

instead of
${OMV_CONFIG_FILE} | xmlstarlet unesc >> "${OMV_COLLECTD_CONFIG_DIR}/df.conf"

~0004978

Silberlog (reporter)

Same thing for other plugins, ">" instead of ">>" ...

~0004979

Silberlog (reporter)

Here is the OMV 4 version.

disk.sh goes in /usr/share/openmediavault/mkconf/collectd.d/

disk-octets.py , disk-time.py and disk-ios.py goes in /usr/share/openmediavault/mkrrdgraph/plugins.d
+Notes

-Issue History
Date Modified Username Field Change
2014-08-16 10:54 WastlJ New Issue
2014-08-16 10:54 WastlJ Status new => assigned
2014-08-16 10:54 WastlJ Assigned To => votdev
2014-08-17 19:16 votdev Note Added: 0003066
2014-10-27 13:23 solo0816 File Added: disk.zip
2014-10-27 13:28 solo0816 Note Added: 0003316
2014-10-27 14:03 votdev Note Added: 0003317
2014-10-27 14:10 solo0816 Note Added: 0003318
2014-10-28 08:50 votdev Note Added: 0003320
2014-11-01 12:49 solo0816 Note Added: 0003334
2014-11-02 08:40 solo0816 Note Added: 0003335
2014-11-03 21:14 votdev Target Version => Next
2015-09-05 12:39 raulfg3 Note Added: 0003848
2017-10-12 13:10 Silberlog Note Added: 0004956
2017-10-12 14:24 votdev Note Added: 0004957
2017-10-12 14:25 votdev Note Edited: 0004957 View Revisions
2017-10-12 15:06 Silberlog Note Added: 0004958
2017-10-12 15:14 votdev Note Added: 0004959
2017-10-12 16:27 Silberlog File Added: disk.txt
2017-10-12 16:27 Silberlog Note Added: 0004960
2017-10-12 18:19 votdev Note Added: 0004961
2017-10-13 11:27 Silberlog Note Added: 0004962
2017-10-13 12:02 votdev Note Added: 0004963
2017-10-13 17:30 Silberlog File Added: disk-2.txt
2017-10-13 17:30 Silberlog Note Added: 0004964
2017-10-13 17:37 votdev Note Added: 0004965
2017-10-13 17:37 votdev Note Edited: 0004965 View Revisions
2017-10-13 17:49 Silberlog Note Added: 0004966
2017-10-14 19:13 Silberlog Note Added: 0004968
2017-10-15 20:28 votdev Note Added: 0004973
2017-10-16 23:25 Silberlog Note Added: 0004976
2017-10-16 23:29 Silberlog Note Added: 0004977
2017-10-16 23:31 Silberlog Note Added: 0004978
2017-10-18 00:05 Silberlog File Added: omv-disk.zip
2017-10-18 00:05 Silberlog Note Added: 0004979
+Issue History