[FEAT] Allow option to pass device type so devices not found in scan can be included #45

madajczyk · 2020-09-28T00:48:42Z

First, I'd like to say that Scrutiny is such a great idea. Thank for you for creating this!!

Feature Request
It would be awesome if there was a way to pass the device type to Scrutiny so that it can collect details on drives that the smartctl utility can't identify on it's own (due to USB bridge identification limits, etc).

Context

I deployed this in Docker to a little home server I run, and mapped in 3 devices in the config file as such:

  - device=/dev/sda
  - device=/dev/sdb
  - device=/dev/sdc

When I opened the Scrutiny UI after doing a scan, only the first two internal drives were visible. The last (/dev/sdc) is an external USB drive. I poked around and saw that it can be problematic for smartctl to get the details for external drives. However, if I could find and pass the correct device type for the USB bridge, it might work.

I was able to identify the correct value to pass for my particular drive, which allowed the retrieval of SMART data from my external drive from terminal. If I could provide configuration detail that the device is of type sat, hopefully I'll be able to monitor that external drive as well with Scrutiny. Below are the outputs of various commands I ran as I tested my theory.

sudo smartctl --scan

/dev/sda -d scsi # /dev/sda, SCSI device
/dev/sdb -d scsi # /dev/sdb, SCSI device

sudo smartctl -i /dev/sdc

smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.4.0-42-generic] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

/dev/sdc: Unknown USB bridge [0x04e8:0x6123 (0x202)]
Please specify device type with the -d option.

Use smartctl -h to get a usage summary

sudo smartctl -i /dev/sdc -d sat

smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.4.0-42-generic] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 7200.14 (AF)
Device Model:     ST2000DM001-1E6164
Serial Number:    W1E86G1H
LU WWN Device Id: 5 000c50 073eb6239
Firmware Version: SC48
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Sun Sep 27 19:10:19 2020 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

The text was updated successfully, but these errors were encountered:

AnalogJ · 2020-09-28T02:30:54Z

This is a great idea! I actually went down the route of filtering out removable disks, since I wanted to ignore thumbdrives, but I should have thought about external hard drives connected over USB.

Let me think about this a bit. Eventually I plan on adding support for multi-host collector device grouping (and filtering) so I want to make sure that any include functionality I add is compatible with that deployment model.

develroo · 2020-10-01T13:53:00Z

It is not just removable disks. Some array controllers need parameters. so

Smartctl open device: /dev/sdf failed: DELL or MegaRaid controller, please try adding '-d megaraid,N'

optional driver flags will need to be passed especially if it is to be run in server environments (and lets face it multiple disk scenarios means servers)

Otherwise it is a great little tool and I have already installed it on non controller card servers. Cheers!

AnalogJ · 2020-10-04T15:53:03Z

Scrutiny uses smartctl --scan to try and auto-detect the correct device type. It has worked for some users with RAID controllers, but yeah in other cases smartctl just doesn't know what to do.
A config file is definitely necessary, I'm working on this off-and-on while waiting for a couple of users to verify the notifications code in a different branch is working. Once that's done I can dedicate some time to this issue.

develroo · 2020-10-04T16:46:57Z

S'ok. I understand. Thanks for a great project and look forward to seeing improvements when they land. No worries.

Cheers.

EDIT:

FWIW I think I can see where the confusion comes in. 2 sets of devices are listed.


smartctl --scan
/dev/sda -d scsi # /dev/sda, SCSI device
/dev/sdb -d scsi # /dev/sdb, SCSI device
/dev/sdc -d scsi # /dev/sdc, SCSI device
/dev/sdd -d scsi # /dev/sdd, SCSI device
/dev/sde -d scsi # /dev/sde, SCSI device
/dev/sdf -d scsi # /dev/sdf, SCSI device
/dev/bus/0 -d megaraid,14 # /dev/bus/0 [megaraid_disk_14], SCSI device
/dev/bus/0 -d megaraid,15 # /dev/bus/0 [megaraid_disk_15], SCSI device
/dev/bus/0 -d megaraid,18 # /dev/bus/0 [megaraid_disk_18], SCSI device
/dev/bus/0 -d megaraid,19 # /dev/bus/0 [megaraid_disk_19], SCSI device
/dev/bus/0 -d megaraid,20 # /dev/bus/0 [megaraid_disk_20], SCSI device
/dev/bus/0 -d megaraid,21 # /dev/bus/0 [megaraid_disk_21], SCSI device


I suspect that some extra parsing will be needed in these edge cases.

PrplHaz4 · 2020-10-06T02:19:29Z

Scrutiny uses smartctl --scan to try and auto-detect the correct device type. It has worked for some users with RAID controllers, but yeah in other cases smartctl just doesn't know what to do.
A config file is definitely necessary, I'm working on this off-and-on while waiting for a couple of users to verify the notifications code in a different branch is working. Once that's done I can dedicate some time to this issue.

Definitely looks like lots of inconsistencies/special cases - I have an H240 card in HBA mode, and it appears that the only way --scan identifies any drives is by specifying a driver (eg. smartctl --scan -d sat). smartctl itself actually recommends using -d ccis,N for these devices, but both seem to work at the drive device level.

For this case it'd be useful to specify devices and/or drivers in the config file (similar to how you've proposed)...

smarctl --scan
<no results>

smartctl --scan -d sat
/dev/sda -d scsi # /dev/sda, SCSI device
/dev/sdb -d scsi # /dev/sdb, SCSI device
/dev/sdc -d scsi # /dev/sdc, SCSI device
/dev/sdd -d scsi # /dev/sdd, SCSI device
/dev/sde -d scsi # /dev/sde, SCSI device

mrtumnus · 2020-10-06T12:30:02Z

I'm in a similar boat here - I have a 3ware 9650 RAID card, and it requires the following form to identify the device to smartctl:
smartctl -d 3ware,<N> /dev/twa0
where <N> is the index of the drive on the RAID card

smartctl --scan doesn't automatically populate the information needed for this:

/dev/sda -d scsi # /dev/sda, SCSI device
/dev/sdb -d scsi # /dev/sdb, SCSI device
/dev/sdc -d scsi # /dev/sdc, SCSI device
/dev/sdd -d scsi # /dev/sdd, SCSI device
/dev/sde -d scsi # /dev/sde, SCSI device
/dev/sdf -d scsi # /dev/sdf, SCSI device
/dev/sdg -d scsi # /dev/sdg, SCSI device
/dev/sdh -d scsi # /dev/sdh, SCSI device

AnalogJ · 2020-10-08T00:54:01Z

Hey,
I'm working on this issue right now. Just wanted to give you all a preview of what the collector config file will look like:

# Commented Scrutiny Configuration File
#
# The default location for this file is /scrutiny/config/collector.yaml.
# In some cases to improve clarity default values are specified,
# uncommented. Other example values are commented out.
#
# When this file is parsed by Scrutiny, all configuration file keys are
# lowercased automatically. As such, Configuration keys are case-insensitive,
# and should be lowercase in this file to be consistent with usage.


######################################################################
# Version
#
# version specifies the version of this configuration file schema, not
# the scrutiny binary. There is only 1 version available at the moment
version: 1

# This block allows you to override/customize the settings for devices detected by
# Scrutiny via `smartctl --scan`
# See the "--device=TYPE" section of https://linux.die.net/man/8/smartctl
# type can be a 'string' or a 'list'
devices:
  # example for forcing device type detection for a single disk
  - device: /dev/sda
    type: 'sat'

  # example to show how to ignore a specific disk/device.
  - device: /dev/sda
    ignore: true

  # examples showing how to force smartctl to detect disks inside a raid array/virtual disk
  - device: /dev/bus/0
    type:
      - megaraid,14
      - megaraid,15
      - megaraid,18
      - megaraid,19
      - megaraid,20
      - megaraid,21

  - device: /dev/twa0
    type:
      - 3ware,0
      - 3ware,1
      - 3ware,2
      - 3ware,3
      - 3ware,4
      - 3ware,5

Feedback is welcome & appreciated :)

PrplHaz4 · 2020-10-08T02:20:51Z

If the devices are being explicitly specified, I think they should be under their own key, to be used instead of or in addition to --scan results.

devices:
  - device: /dev/sda
    type: 'scsi'
  - device: /dev/sdb
    type: 'scsi'
  - device: /dev/sdc
    type: 'sat'
  - device: /dev/sdd
    type: 'sat'

I have more testing to verify this makes sense across different versions of smartctl, but wanted to get you some quick feedback.

AnalogJ · 2020-10-08T03:13:26Z

that's a good point. I'll change the keyname.

AnalogJ · 2020-10-08T04:59:47Z

Hey everyone,

I just released a beta version of the Scrutiny docker image with support for overriding the collector device detection.

The instructions for how to create the collector config file, and the new docker image tag are available in the PR description:

#88

All feedback (success & failure) is appreciated :)

madajczyk · 2020-10-08T17:05:58Z

I tested it out with my original use case, and the external drive lit up right away:

The configuration file was dead simple to use. In the PR, I also appreciated how you set up the example.collector.yaml file with clear instructions that covered all of the use cases discussed here. Thank you very much!!

malyyigor34 · 2023-01-05T19:49:16Z

Hello. My config is:
devices:
- device: /dev/sda
type: 'scsi'
- device: /dev/sdb
type: 'scsi'
- device: /dev/sdc
type: 'sat'
- device: /dev/sdd
type: 'sat'

and i am getting error: failed to deploy a stack: services.scrutiny.devices.0 must be a string

bertlp5 · 2023-04-09T09:34:00Z

and i am getting error: failed to deploy a stack: services.scrutiny.devices.0 must be a string

Had the same Problem. The Device overwrite config ist not in the docker-compose.yaml instead the device config musst be in the config folder in the file collector.yaml

tristan-k · 2024-07-13T10:15:43Z

Does so the ghcr.io/analogj/scrutiny:master-omnibus docker image also allows reading the collector.yaml. I just used the example.collector.yaml as a base config and changed the name and deviced accordingly to my setup but scrutiny still recognize my /dev/sda as a ATA device event though I defined:

version: 1
host:
  id: ""
- device: /dev/sda
  type: 'SCSI'

$ smartctl --scan
/dev/sda -d scsi # /dev/sda, SCSI device

AnalogJ added the enhancement New feature or request label Sep 28, 2020

AnalogJ mentioned this issue Oct 8, 2020

adding support for a collector config file. #88

Merged

AnalogJ closed this as completed in #88 Oct 8, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FEAT] Allow option to pass device type so devices not found in scan can be included #45

[FEAT] Allow option to pass device type so devices not found in scan can be included #45

madajczyk commented Sep 28, 2020

AnalogJ commented Sep 28, 2020

develroo commented Oct 1, 2020

AnalogJ commented Oct 4, 2020

develroo commented Oct 4, 2020 •

edited

Loading

PrplHaz4 commented Oct 6, 2020

mrtumnus commented Oct 6, 2020 •

edited

Loading

AnalogJ commented Oct 8, 2020 •

edited

Loading

PrplHaz4 commented Oct 8, 2020 •

edited

Loading

AnalogJ commented Oct 8, 2020

AnalogJ commented Oct 8, 2020

madajczyk commented Oct 8, 2020

malyyigor34 commented Jan 5, 2023 •

edited

Loading

bertlp5 commented Apr 9, 2023

tristan-k commented Jul 13, 2024 •

edited

Loading

[FEAT] Allow option to pass device type so devices not found in scan can be included #45

[FEAT] Allow option to pass device type so devices not found in scan can be included #45

Comments

madajczyk commented Sep 28, 2020

AnalogJ commented Sep 28, 2020

develroo commented Oct 1, 2020

AnalogJ commented Oct 4, 2020

develroo commented Oct 4, 2020 • edited Loading

PrplHaz4 commented Oct 6, 2020

mrtumnus commented Oct 6, 2020 • edited Loading

AnalogJ commented Oct 8, 2020 • edited Loading

PrplHaz4 commented Oct 8, 2020 • edited Loading

AnalogJ commented Oct 8, 2020

AnalogJ commented Oct 8, 2020

madajczyk commented Oct 8, 2020

malyyigor34 commented Jan 5, 2023 • edited Loading

bertlp5 commented Apr 9, 2023

tristan-k commented Jul 13, 2024 • edited Loading

develroo commented Oct 4, 2020 •

edited

Loading

mrtumnus commented Oct 6, 2020 •

edited

Loading

AnalogJ commented Oct 8, 2020 •

edited

Loading

PrplHaz4 commented Oct 8, 2020 •

edited

Loading

malyyigor34 commented Jan 5, 2023 •

edited

Loading

tristan-k commented Jul 13, 2024 •

edited

Loading