Proposal: ChatOps User Survey

[blag]

This is a follow up to #8.

I'm opening this proposal to gather questions we would like to ask users about ChatOps.

Motivation

We would like to gather feedback from users about exactly how they use ChatOps to determine if a more integrated, Python-based implementation would better serve their use cases, or if we should continue to develop the current separate, Javascript-based hubot-stackstorm implementation.

Potential Survey Questions

Here is the initial list of questions that I have come up with:

• How long have you been using StackStorm and/or EWC?
  • 0-6-months
  • 6-12 months
  • 1-2 years
  • 2+ years
• How large is your organization?
  • 1-10
  • 10-100
  • 100-1000
  • 1000-10000
  • 10000+
• How many ChatOps users within your organization do you have?
  • 1-10
  • 10-100
  • 100-1000
  • 1000-10000
  • 10000+
• Do you use StackStorm-flavored ChatOps?
  • Yes
  • No
• If you don't use ChatOps, what are the blockers in your adoption? (select all that apply)
  • Missing chat adapter
  • No RBAC
  • No HA
  • Buggy, not stable enough
  • Haven't tried it yet
  • Didn't work
  • Other (please specify)
• Do you currently use any chat bots outside of StackStorm?
  • Yes
  • No
• Have you written any chat bots outside of StackStorm?
  • Yes
  • No
• Which domain/s are you using ChatOps in? (select all that apply)
  • Network
  • Security
  • Infrastructure
  • Cloud/SaaS
  • Other (please specify)
• How do you use StackStorm's ChatOps? (select all that apply)
  • Action aliases
  • Integrating hubot-stackstorm into a custom hubot bot
  • Using err-stackstorm with Errbot
  • Something else (please explain what you do)
• What types of tasks do you commonly execute/use via ChatOps? (select all that apply)
  • One-way notifications
  • Querying systems/statistics
  • Perform arbitrary StackStorm actions
  • Teach/collaborate with other team members
  • Start deployments
• How important is user-defined ChatOps in your organization?
  • Mission critical
  • Makes things easier/faster/more efficient
  • Just another way to run commands
  • We have a few aliases, but nobody uses them
  • ChatOps is configured, but nobody has made any aliases
  • ChatOps isn't even configured or installed
  • Hinders actual work
• What chat provider/s do you currently use with StackStorm ChatOps? (select all that you currently use)
  • Slack
  • Mattermost v4
  • Mattermost v5
  • Rocket.chat
  • Cisco Spark or Webex Teams
  • Microsoft Teams (via BotFramework)
  • HipChat
  • Something else (please explain)
• If you are considering changing your chat provider/s which one/s are you currently considering? (select all that apply)
  • Slack
  • Mattermost v4
  • Mattermost v5
  • Rocket.chat
  • Cisco Spark or Webex Teams
  • Microsoft Teams (via BotFramework)
  • HipChat - not including this one, since it's reaching EOL on June 30th, 2020 and we're ripping out the adapter then as well
  • Something else (please explain)

Excessive for Survey. Click to expand.

• What ChatOps alias features have you used/are currently using? (select all that apply)
  • Using the !help command in a ChatOps channel
  • Alias formats

    # Simple usage
    formats:
      - run {{ cmd }} on {{ hosts }}

    # Complex usage
    formats:
      - "execute order 51"
      - display: "run {{ cmd }} on {{ hosts }}"
        representation:
          - "(run|execute) {{ cmd }}( on {{ hosts=localhost }})?[!.]?"  # timeout parameter not explicitly listed
          - "run remote command {{ cmd }} on {{ hosts }}"
    immutable_parameters:
      timeout: 30  # seconds
      sudo_password: "{{ st2kv('system.dev_server_sudo_password', decrypt=true) }}"

  • Using additional ChatOps parameters in workflows

    tasks:
      task1:
        action: core.echo
        input:
          message: "[<% ctx().source_channel %>] <% ctx().action_context.api_user %> (<% ctx().action_context.user %>)"

  • Automatic acknowledgements

    # Simple usage
    ack:
      format: "acknowledged!"

    # Complex usage
      ack:
        format: "Executing `{{ actionalias.ref }}`, the execution ID is `{{ execution.id }}`"
        enabled: false

  • Result formats

    # Simple usage
    result:
      format: |
        Ran command `{{ execution.parameters.cmd }}` and {{ execution.result | length }} hosts.
    
        Details are as follows:
        {% for host in execution.result -%}
            Host: `{{ host }}`
            ---> stdout: {{ execution.result[host].stdout }}
            ---> stderr: {{ execution.result[host].stderr }}
        {%+ endfor %}

    # Complex usage
    result:
      format: |
        Action completed!
        {~}
        Ran command `{{ execution.parameters.cmd }}` and {{ execution.result | length }} hosts.
    
        Details are as follows:
        {% for host in execution.result -%}
            Host: `{{ host }}`
            ----> stdout: {{ execution.result[host].stdout }}
            ----> stderr: {{ execution.result[host].stderr }}
        {%+ endfor %}
      enabled: true

  • Attachments with extra (Slack, Mattermost, and Rocket.chat)

    # Simple usage
    result:
      # ...
      extra:
        slack:
          image_url: "https://i.imgur.com/Gb9kAYK.jpg"
          fields:
            - title: Kitten headcount
              value: Eight.
              short: true
            - title: Number of boxes
              value: A bunch.
              short: true
          color: "#FF0000"

    # Complex usage
    result:
      # ...
      extra:
        slack:
          image_url: "https://i.imgur.com/Gb9kAYK.jpg"
          fields:
            - title: Kitten headcount
              value: Eight.
              short: true
            - title: Number of boxes
              value: A bunch.
              short: true
          color: "{{ execution.parameters.color }}"
        mattermost:
          color: "{{ execution.parameters.mm_color }}"
        rocketchat:
          color: "{{ execution.parameters.rc_color }}"
        audit: true  # additional property, passed directly to chat provider adapter

  • Inquiries with ChatOps
  • Looked through ChatOps logs
  • Using ChatOps behind a proxy
  • Using an external chat adapter like hubot-yammer
  • BYO Hubot (eg: Bring Your Own Hubot) by using hubot-stackstorm in your own hubot instance
• For any current ChatOps features that you have not used yet, do you expect to use any features in the future? (select all that apply)
  • Using the !help command in a ChatOps channel
  • Alias formats

    # Simple usage
    formats:
      - run {{ cmd }} on {{ hosts }}

    # Complex usage
    formats:
      - "execute order 51"
      - display: "run {{ cmd }} on {{ hosts }}"
        representation:
          - "(run|execute) {{ cmd }}( on {{ hosts=localhost }})?[!.]?"  # timeout parameter not explicitly listed
          - "run remote command {{ cmd }} on {{ hosts }}"
    immutable_parameters:
      timeout: 30  # seconds
      sudo_password: "{{ st2kv('system.dev_server_sudo_password', decrypt=true) }}"

  • Using additional ChatOps parameters in workflows

    tasks:
      task1:
        action: core.echo
        input:
          message: "[<% ctx().source_channel %>] <% ctx().action_context.api_user %> (<% ctx().action_context.user %>)"

  • Automatic acknowledgements

    # Simple usage
    ack:
      format: "acknowledged!"

    # Complex usage
      ack:
        format: "Executing `{{ actionalias.ref }}`, the execution ID is `{{ execution.id }}`"
        enabled: false

  • Result formats

    # Simple usage
    result:
      format: |
        Ran command `{{ execution.parameters.cmd }}` and {{ execution.result | length }} hosts.
    
        Details are as follows:
        {% for host in execution.result -%}
            Host: `{{ host }}`
            ---> stdout: {{ execution.result[host].stdout }}
            ---> stderr: {{ execution.result[host].stderr }}
        {%+ endfor %}

    # Complex usage
    result:
      format: |
        Action completed!
        {~}
        Ran command `{{ execution.parameters.cmd }}` and {{ execution.result | length }} hosts.
    
        Details are as follows:
        {% for host in execution.result -%}
            Host: `{{ host }}`
            ----> stdout: {{ execution.result[host].stdout }}
            ----> stderr: {{ execution.result[host].stderr }}
        {%+ endfor %}
      enabled: true

  • Attachments with extra (Slack, Mattermost, and Rocket.chat)

    # Simple usage
    result:
      # ...
      extra:
        slack:
          image_url: "https://i.imgur.com/Gb9kAYK.jpg"
          fields:
            - title: Kitten headcount
              value: Eight.
              short: true
            - title: Number of boxes
              value: A bunch.
              short: true
          color: "#FF0000"

    # Complex usage
    result:
      # ...
      extra:
        slack:
          image_url: "https://i.imgur.com/Gb9kAYK.jpg"
          fields:
            - title: Kitten headcount
              value: Eight.
              short: true
            - title: Number of boxes
              value: A bunch.
              short: true
          color: "{{ execution.parameters.color }}"
        mattermost:
          color: "{{ execution.parameters.mm_color }}"
        rocketchat:
          color: "{{ execution.parameters.rc_color }}"
        audit: true  # additional property, passed directly to chat provider adapter

  • Inquiries with ChatOps
  • Looked through ChatOps logs
  • Using ChatOps behind a proxy
  • Using an external chat adapter like hubot-yammer
  • BYO Hubot (eg: Bring Your Own Hubot) by using hubot-stackstorm in your own hubot instance

• What new feature/s would you like to see in ChatOps?
  • Better integration with the rest of StackStorm
  • Easier configuration/troubleshooting
  • HA
  • Threaded conversations/responses
  • Uploading files via ChatOps
  • Ability to query and post graphs and diagrams
  • RBAC for ChatOps
  • Better inquiries support
  • NLP (natural language processing) or NLU (natural language understanding)
  • More interactivity options in action aliases (Slack blocks, MS Teams cards, etc.)
  • Other (please specify)
• Rank these new features, with the most important feature at the top (if you don't care, you can leave the ranking blank)
  • Better integration with the rest of StackStorm
  • Easier configuration/troubleshooting
  • HA
  • Threaded conversations/responses
  • Uploading files via ChatOps
  • RBAC for ChatOps
  • Better inquiries support
  • NLP (natural language processing) or NLU (natural language understanding)
  • More interactivity options in action aliases (Slack blocks, MS Teams cards, etc.)
  • Other (please specify)
• What does StackStorm get right with its ChatOps implementation?
• What does StackStorm get wrong with its ChatOps implementation?
• Is there anything we should do to make it easier to use ChatOps with StackStorm?
• Is there anything we absolutely should NOT do regarding ChatOps in StackStorm?
• Would you be willing to have a deeper discussion with us about your ChatOps use case/s?
• Would you be willing to help us refactor, redesign, develop, or test new ChatOps features?

[punkrokk]

I’d suggest we also ask

• what providers people are likely to use in the future.
• Add NLP to the ranking
• Are there any modern features you can think of that have a ST2 use case
• would you be willing to have a deeper exploratory discussion about your chatops use cases?
• what does ST2 get right in regards to chatops?
• what does ST2 get wrong with chatops?
• Would you be willing to also have to install a slack app or similar for full functionality?

[blag]

Are there any modern features you can think of that have a ST2 use case

Is this is already covered by the "Other (please specify)" option in the

What new feature/s would you like to see in ChatOps?"

question? And if not, is there a better way to word this to explain the difference to users?

what does ST2 get wrong with chatops?

This is fairly close to "Is there anything we should do to make it easier to use ChatOps with StackStorm?" I'm a little afraid that we're going to get the same answers for each question unless we explain the difference between the two questions a bit more.

I incorporated almost all of your questions though. Thanks!

[cognifloyd]

Reading through that list, there were a few features I forgot about that I've been meaning to use. And a few others that I didn't know exist (inquiries can use buttons in slack?).

Maybe, after the survey, offer the list of features again, with deeplinks into the docs for each feature to facilitate learning about or using features they weren't aware of.
The biased side of me says "but no links for BYO-hubot; I want to drop hubot" ... 😄

That ranking question is hard (it's a great question, but I'm so conflicted on which feature would benefit people where I work the most). In my mind there is a technical progression of some features need to be implemented before others so they can use them: Better inquiries would ideally use threaded conversations and/or interactive components. I'm not sure if we need to do/say anything about possible technical dependencies between the features.

Could we add one or more (optional) questions inviting participation in the community?

• Would you or someone you know be able and willing to help us implement these features?
• Would you be more likely to help if StackStorm's chatops was based on JavaScript or Python?
• For this question, assume you want to participate: Where would you go to learn how to contribute to StackStorm's ChatOps solution?

And maybe some tangential queries about chat bots in general:

• Have you written any chat bots outside of StackStorm?
• If yes, what would StackStorm need to support the use case covered by your other bot(s)?
• Have you used any other chat bots, especially for ChatOps use-cases, that you felt worked better in some way than StackStorm's ChatOps implementation? If yes, how was it better?

[arm4b]

That's a lot of work put in! 👍 Great job!
I really liked the first 5 questions from this draft ^^.

Let's gather more ideas and then fine-tune and cut it. Remember that as longer or weirder the survey becomes, - less users will reach the end. So it's within our interest to get more useful info with less questions.

A few suggestions:

• How long have you been using StackStorm?
  • 0-6 months
  • 6-12 months
  • 1-2 years
  • 2+ years

^^ This entry question will help us to understand how experienced is the user with the platform. We asked something like that before.

• Org size
  • 1-10
  • 10-100
  • 100-1000
  • 1000-10000
  • 10000+
• ChatOps users:
  • 1
  • 1-10
  • 10-100
  • 100+

^^ Here we could then compare if there are any big org users and how large are the groups interacting with chatops. I've heard from many orgs that chatops doesn't work for bigger teams. We may try to find the answer why it doesn't scale. Feel free to adjust, but you got an idea.

• If you don't use ChatOps, what are the blockers in your adoption?
  • Missing chat adapter
  • Bad Security
  • Missing RBAC
  • No HA
  • Buggy, not stable enough
  • We didn't try it yet
  • It didn't work for us
  • ___ (custom answer)

^^ Here we could branch out if user responded "No, I don't use chatops". We should continue with this thread a few steps further to know what didn't work for them and why chatops didn't fly. Don't forget to allow them to fully express their difficulties/confusion in the final free text form.

• Please add "Ability to query/draw graphs with diagrams in Chat" to "What new feature/s would you like to see in ChatOps?" That's a big deal in monitoring/ops.

• We need to formulate the questions about use mechanics around chatops. Is it just notifications, bi-directional, etc. Do people just query some system or perform any actions, do deployments, etc.

• On the final page:

Thank you!
We're considering to modernize the ChatOps platform and invite you to collaborate further around the possible design improvements: #8.

^^ Text could be different, but you got the point.

• Besides of that I think it's better to omit the part where code samples started to be involved: don't ask about very granular/specific features they use. We don't want to dig so deep and that'll scare out people to continue the survey.

• I'd also like to note to avoid any strange, rare or very specific questions that may work only for 1% of potential users or when users are not familiar with the specific answer/too technical/too extraordinary.

Thinking out loud, if we'll consider changing the platform in future (is it OpsDroid, ErrBot or whatever), which areas will likely lead to possible breaking changes or difficulties?

Let me chime in more ...

[blag]

@armab Incorporated your suggestions as best I could. 👍 Let me know if I need to tweak anything further.

[blag]

@cognifloyd

Better inquiries would ideally use threaded conversations

I actually disagree - I think inquiries should use private messages (when supported by the chat provider, there is some complication with this in Slack). And yes, interactive components like Slack's (now deprecated buttons) is great for inquiries.

Would you be more likely to help if StackStorm's chatops was based on JavaScript or Python?

I don't like this question at all. Almost all of our current ChatOps users shouldn't have to know or care, because they don't have to write any code to get ChatOps working. They just have to write action aliases, which abstracts them from all of the implementation details like this. The only people who will know or care about this are people who are interested in developing ChatOps. This is the ChatOps User Survey, not the ChatOps Developer Survey.

For this question, assume you want to participate: Where would you go to learn how to contribute to StackStorm's ChatOps solution?

Same problem with this question. Most users don't, and aren't going to want to, contribute.

If yes, what would StackStorm need to support the use case covered by your other bot(s)?

Have you used any other chat bots, especially for ChatOps use-cases, that you felt worked better in some way than StackStorm's ChatOps implementation? If yes, how was it better?

I think these are covered in the question about what features they would like to see, and what StackStorm can do to make ChatOps easier for people to use.

Otherwise, I've incorporated your questions - let me know what you think! 😃

[nzlosh]

Everyones already done a solid job fleshing out the survey. Would it be of interest to ask which domain chatops is being primarily used in?

• Network
• Security
• Infrastructure
• Cloud/SaaS
• Other

This could be used to help determine if chatops specialisations could be made to the bot/API.

[arm4b]

FYI I OpenSourced a couple of archive discussions around StackStorm ChatOps:

• [2014] Discussion: ChatOps #24
• [2015] ChatOps Requirements #25

Please take a look, - the content in there (surprisingly) is still actual! This all might help you more.

[blag]

Results for the 2020 ChatOps User Survey are in #48. Closing this issue.