Feedback form(s) for the SEP

[Forest]

In another thread, @mike2014 wrote,

Hi Forest, just a suggestion, but is there a way of including a basic feedback questionnaire/form at the end of the programs?

I think that that is a great idea. Thanks for suggesting it.

Systematically gathering feedback has been on my mind for a while. The nonprofit is getting large and complex enough that it is getting harder and harder for me to keep on top of everything. For example, there is content that I would like to add to the SEP, but I don't know which of the 42 days are working well and which days are working less well. Concretely, Dr. David Clarke has donated an essay about self-care and I can't add it without removing something else. The current plan is to add it on Day 37 to replace content on that day that people are having trouble accessing. (This will take a while because I don't have anyone to delegate things to who already knows how to edit Wikipedia-style wiki pages. Also, coming up with professional language that communicates the right things can be very time consuming.)

I've found that the best decisions result from not just one piece of feedback, but many. Feedback forms could help us gather enough information that we can be more responsive by focusing on the most important needs.

I've gone ahead and drafted a form:
http://www.tmswiki.org/forum/forms/feedback-on-the-structured-educational-program.3/respond

I've gone ahead and added it to the footer at the bottom of each day since day-specific feedback is so valuable. It will take a bit longer to add it to the final day, as it may take a while to come up with the right language. Ideally, it seems like an opportunity to have people reflect back on the program and get some closure on the process. Of course, we want to keep it very concise and professional as well. If anyone has any thoughts on how to present it, I'd love to hear them.

Another thought that has been marinating in my brain for a couple of years is collecting outcome information before and after people take the program. One idea that comes up periodically for how we could expand the nonprofit would be to apply for grants. Research has suggested that our chances of getting a grant will be greatly enhanced if we collect data on outcomes. Similarly, experience from the comments to the recent HuffPo article suggest that gathering data is one of the few things that we can do to significantly build credibility.

Using validated measures would, one would think, enhance the chances of this data being persuasive. The following came from an email I sent, but captures some of my current thinking on this:

At the wiki we evaluated the Roland Morris and the Oswestry scales for measuring progress by people completing our Structured Educational Program. The idea would be to have people complete the questionnaire on the first day of the program and on the last. Results would be collected via a Google Form.

We decided that the Roland Morris scale would be better for us because it was shorter and therefore more likely to be completed. It is also completely royalty free and available online at http://www.rmdq.org/

By the way, PPD practitioner and SIRPA executive board member Mags Clark-Smith has done a very compelling poster using the Morris scale:
http://www.resolvingchronicpain.com/s/Mags-Clark-Smith-SBPR-poster.pdf
To any individual practitioner who is thinking of collecting data, I'd highly recommend taking a look at Mags' poster. Her presentation at the 2015 SIRPA conference has the same title as the poster, so I'd bet she's presenting it there.

One problem with the Morris scale for us is that it focuses on back pain. In contrast, many PPD consumers are concerned with symptoms beyond just back symptoms. However, from what I gather, it is appropriate to replace "because of my back pain" with "because of my pain" or even "because of my pain or other symptom" to expand the applicability.

According to [1], "A number of researchers have proposed modifications to the RDQ. The simplest modification to the wording of the questionnaire has been to change the terminal phrase of each statement from “because of my back” to “because of my back or leg problem.” [98] This makes the questionnaire more suitable for use in a population of patients with sciatica and is an acceptable modification."

References:
[1] Roland M, Fairbank J. The Roland-Morris Disability Questionnaire and the Oswestry Disability Questionnaire. Spine 2000; 25: 3115-3124
[98] Patrick D, Deyo R, Atlas S, et al. Assessing health-related quality of life in patients with sciatica. Spine 1995; 20:1899–909.

Regarding the wiki's data collection, Dr. Clarke suggested that it would be valuable to survey people 6/12/24 months later to see if gains are maintained. I don't know if we have resources for that.​

 

[Ellen]

Forest,

This is all great, and as we have discussed before, I'm interested in both initiatives and willing to help in any way I can.

However, I am unable to respond in detail at the moment, as I have monumental jet lag and travel fatigue having just arrived back home from Japan.

But count me in, and I'll give specific feedback once I'm in a more coherent state--assuming that occurs

 

[mike2014]

Hi Forest,

That's great @Forest - the form looks good.

I hope you have recovered @Ellen, I can imagine that's quite a flight, at least you have made it back in time for Christmas

One item I think is missing on the Morris scale, but would be important to capture by person (in a sample group) is the number of working days lost, by country as a result of PPD illnesses, so therefor we could ask how many days have you taken off from work as a result of a mindbody disorder within the 6, 12 months prior to starting the program, then ask a further 6, 12 months after starting the program.

This will not only help as chart progress from a symptoms perspective, but also allow us to quantify the change in days lost, as you can imagine the economy loses billions a year through PPD type illnesses. It's just a thought, but could help us strengthen our case when getting funding.

All the best,

Mike

 

[Ellen]

This will take a while because I don't have anyone to delegate things to who already knows how to edit Wikipedia-style wiki pages.

​

Is this difficult to learn? Is it something that you could teach in a Fuze meeting or two to several of us who are willing to assist with this task? I am willing but my tech skills are limited.

As to obtaining feedback so that the SEP can be revised and updated (BTW great suggestion @mike2014):

I think you might get more useful information and increase the likelihood of responses if you use a Likert Scale and a couple of specific questions. For example,

How helpful/informative did you find Day X of the SEP?
0 (not helpful/informative); 1 (somewhat helpful/informative); 2 (very helpful/informative); 3 (extremely helpful/informative)

What was most helpful/informative?

What do think could be improved or changed?

Some thoughts about measuring the effectiveness of the SEP:

• All revisions to the SEP would need to be completed before this is started
• People using other forms of TMS treatment (e.g. seeing a TMS therapist) would have to be excluded from the study
• It would be useful to use the same questionnaires/measures that the TMS therapist group decides to use, if possible. Then we could compare a self-help program to other treatments
• Since outcome independence is something we promote, outcome measures should not be focused only on pain reduction, but also on Quality of Life measures
• To get the 6, 12, and 24 month questionnaires out, is it possible for automated emails to be sent to the respondents based on the date they fill out the post-SEP questionnaire?

There has been a lot of research on the effectiveness of self-help online programs and other self-help methodologies for other conditions. We could benefit from reading their protocols. Here is the summary of one example:

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2483843/


Looking forward to assisting with these efforts. Should we form a committee if others are interested?

 

[Becca]

@mike2014, I also think this is a great idea, and I completely agree with @Forest on the benefits of collecting information. @Ellen, I love your suggestions.

I think you might get more useful information and increase the likelihood of responses if you use a Likert Scale and a couple of specific questions. For example,

How helpful/informative did you find Day X of the SEP?
0 (not helpful/informative); 1 (somewhat helpful/informative); 2 (very helpful/informative); 3 (extremely helpful/informative)

What was most helpful/informative?

What do think could be improved or changed?

I agree that using a Likert scale combined with more specific questions would provide more useful information than an open feedback form. Considering the form is currently on the end of each day, though, I wonder if it might be a little much to ask all the questions (the Likert scale ones and the open-ended ones) each day and if it would make more sense to do it at the end of each week instead. It may help to have some time between completing the day itself and evaluating the day's effectiveness. It might also be helpful to have a general feedback form like the one Forest drew up at the end of each day anyhow, so if people have immediate reactions or thoughts they can share them easily. Not having gone through the SEP, though, I don't know what would be best here. Anyone else have thoughts on this?

Since outcome independence is something we promote, outcome measures should not be focused only on pain reduction, but also on Quality of Life measures

YES! That's such an important point! Thank you for proposing that. If you couldn't tell, I wholeheartedly agree

People using other forms of TMS treatment (e.g. seeing a TMS therapist) would have to be excluded from the study

I definitely understand why excluding people who are using other forms of TMS treatment from the study makes sense in measuring the effectiveness of the program. On the other hand, I wonder if including them could provide valuable information about TMS treatment. It may not be productive to evaluating the SEP specifically, but I can see a potential value in having the results of both groups of people.

This probably depends on how frequently we're asking people to complete the questionnaire, though. If we are looking at results on a more frequent (i.e. weekly) basis, having the information from those who also are seeing a TMS therapist, for instance, may allow us to see where the SEP is most impactful on its own, and where additional forms of TMS treatment, like therapy, make a significant difference. If we don't evaluate people's progress while they go through the program (which it sounds like that's what we'd be leaning towards), I think you are right that we may need to restrict the results to those without additional forms of treatment or support. It also may be too large of a project to include and interpret results of people going through the SEP while also seeking other forms of TMS treatment.

(I suppose I wrote a lot of that out for nothing, since reading the thread over, it seems we'd be leaning towards a model where folks answer questions after they've completed the SEP, not while they are going through it. I guess I got a little excited about the possibilities of research/data!)

Agreed on the revisions to the SEP, too, though I suspect we will need to pitch in and help @Forest with that. Editing Wikipedia-style pages is fairly straightforward and pretty easy to learn. Plus, we know Forest is a great teacher!

I'm excited to see what we all can come up with!

 

[Forest]

Wow, that was quick, Ellen! When I came back from Korea, I formulated the hypothesis that it takes one day per hour of time difference to recover from jetlag. Make sure you get your rest!

> This will take a while because I don't have anyone to delegate things to who already knows how to edit Wikipedia-style wiki pages.
Is this difficult to learn? Is it something that you could teach in a Fuze meeting or two to several of us who are willing to assist with this task?

It's pretty straightforward. We use the same software as Wikipedia uses, so it is exactly as hard as editing Wikipedia (strictly speaking, it is as hard as it was to edit Wikipedia two years ago). When we transitioned from our old site to this new one two years ago, I made a video to show people how to do it and people seemed to pick it up pretty readily.

We don't have the capability to send out automated emails right now. However, there is a service called "LetterMeLater" that we could use to schedule emails to be sent later.

The ideas that both of you have described sound great, as does the idea of having a teleconference. My inclination is to keep the group rather small as there will be significant preparation involved for each member and smaller groups can often move more quickly.

What I've found in previous projects like this is that you learn so much just by jumping in and getting started. Given this, it might be good to think of this as a multi-phase operation. In the first phase (say the first 6 months), we put up a simple measure or collection of measures at the beginning and end of the project. We just try to do the simplest thing possible because we will learn things in the process of doing that that we couldn't have possibly anticipated otherwise.

In general, right now I'm pretty tapped out in terms of my ability to put much time into anything with any complexity to it. I have a huge list of projects that I would love to work on and hope to get to eventually, but can't work on right now. However, if you are interested in working on it, Ellen, I think that that sounds great. I could definitely find time to coordinate, support and brainstorm, but, after that, especially given your obvious expertise, I would want to get out of the way. If that works for you, it sounds great to me!