Takashi Yamashita, PhD, is an assistant professor in the department of sociology at the University of Nevada, Las Vegas. His research focus is on geographic access to health resources, including healthcare services, health literacy over the life course, fall risk factors, and chronic disease self-management behaviors in older populations. Heather Cooke is a PhD candidate at the Centre on Aging at the University of Victoria in British Columbia with almost 20 years’ experience in the field of dementia care. Her research interests include quality of life and quality of care issues within long-term care; staff caregiving practices within long-term care; therapeutic design of environments for individuals with dementia, and long-term care policy.
ElderBranch interviewed Dr. Yamashita and Ms. Cooke to discuss their recently published study, “An international comparison of the Ohio department of aging-resident satisfaction survey: applicability in a U.S. And Canadian sample,” which they wrote with J. Scott Brown and Jane Straker of the Scripps Gerontology Center at Miami University, and Susan Baiton Wilkinson of the Fraser Health Authority.
What led you to research the applicability of a U.S. aging resident satisfaction survey in Canadian assisted living facilities? Why is this important?
In 2008, the Fraser Health Authority (the largest health authority in British Columbia, Canada) was looking for a satisfaction survey to use as part of their quality indicator process in assisted living. A review of existing surveys revealed that the majority of assisted living satisfaction surveys were developed for use within the United States.
In the absence of a valid and reliable Canadian survey, the decision was made to use the Ohio Department of Aging’s Resident Satisfaction Survey (ODS-RSS). However, the challenge in using a satisfaction survey developed for use in another country is whether the survey questions remain contextually relevant.
On the face of it, the ODS-RSS appeared relevant and applicable to assisted living tenants in B.C., but we believed it was important to conduct further analyses to ensure that this was indeed the case.
Please describe your study. What were your methods? What were your in-going hypotheses?
Our study examined the validity (i.e., are we measuring what we intend to measure?) and reliability (i.e., are we repeatedly and consistently measuring the same thing?) of the Ohio Department of Aging Resident Satisfaction Survey (ODA-RSS) when used with a sample of U.S. and Canadian assisted living residents.
The ODA-RSS was developed for use in Ohio residential care facilities, a licensing category that covers a diverse range of facilities, including assisted living. Analyses conducted following its inaugural statewide administration in 2007 led to some minor modifications; the resultant survey revealed good validity and reliability.
In its current form, the survey consists of 46 questions in 11 domains including (1) Activities, (2) Choice, (3) Care and Services, (4) Employee Relations, (5) Employee Responsiveness, (6) Communications, (7) Meals and Dining, (8) Laundry, (9) Facility Environment, (10) Resident Environment, and (11) General Satisfaction. Responses are scored on a 4-point scale (Yes, Always; Yes, Sometimes; No, Hardly Ever; No, Never).
Several minor changes were made to the ODA-RSS prior to its use within Fraser Health. These included removal of the General Satisfaction domain, along with some minor wording changes (e.g., substitution of “staff” for “employee” and “tenant” for “resident”). The response set remained unchanged. For the purposes of our study, analyses were conducted using the 10 domains & 42 questions common to both the Ohio and B.C. (Fraser Health) samples.
Given the considerable overlap in terms of the services provided by Ohio residential care facilities and B.C. assisted living facilities and the straightforward language used in the survey, we hypothesized that the psychometric properties of the ODA-RSS would be similar for the both the Ohio & B.C. samples.
Data for our study were drawn from Ohio’s 2009 survey, which was completed by 9,739 residents, and Fraser Health’s 2010 survey, which was completed by 938 residents.
We used a statistical modeling technique called confirmatory factor analysis to estimate the validity of eight of the individual satisfaction domains for both the Ohio and Fraser Health samples. (We could not use the technique to assess validity of the Activities or Laundry domains as they had too few questions.) We used a statistic called Cronbach’s alpha (commonly used in survey development) to assess the reliability of the survey.
What did you find?
In the Ohio sample, all eight domains demonstrated good or adequate validity. However, in the Fraser Health sample, only four domains (Care and Services, Employee Relations, Employee Responsiveness, and Communications) demonstrated good validity. Most domains in both samples exhibited good reliability.
We concluded that the domains of Choice, Meals and Dining, Facility Environment, and Resident Environment do not appear to capture the nuances of the assisted living context in Fraser Health as well as they do in Ohio.
Possible reasons for this include: (a) the variation in opportunities for resident choice that potentially exists across Fraser Health sites; (b) uniformity of meal service (i.e., restaurant-style) across Fraser Health sites and absence of questions regarding more potentially pressing issues (such as type of food, appropriateness of menu items for special diets); and, (c) more standardized physical environments of Fraser Health sites, many of which were purpose-built in the last decade with readily accessible outdoor space.
What are the implications of your findings?
Our findings illustrate the danger in assuming that rigorous development and testing of a survey tool ensures its applicability in states or countries other than that in which it was initially developed.
They also highlight the importance of establishing the similarity or differences in care settings (e.g., policy, infrastructure) prior to an instrument’s use. We encourage anyone who plans to adopt the ODA-RSS or any other existing satisfaction survey to pilot-test first and make appropriate modifications accordingly.
What are the next steps to further your work in this area?
Following our study, focus groups were conducted with residents at several B.C. assisted living sites to identify aspects of the Choice, Meals and Dining, Resident Environment, and Facility Environment domains considered most relevant to residents’ assisted living experience.
Although several identified aspects were reflected in existing items (e.g., privacy, maintenance of building and grounds, cleanliness), the majority were not. Survey items within each of these domains were subsequently revised, and cognitive interviews were conducted with tenants to explore their understanding of the revised items.
In August 2012, the revised survey was distributed to 1,351 assisted living tenants within Fraser Health. Preliminary confirmatory factor analyses conducted on data from the 903 completed surveys revealed improved validity and reliability for the modified domains. We are currently in the process of writing up our results for publication.
As a next step, we hope to test the revised survey in a sample of Ohio assisted living residents. This will help us determine whether the revised survey works in both B.C. and Ohio. If it does, we would seek to expand our testing to other states and/or provinces.
Ultimately, we would like to develop a universal survey that offers a comprehensive, valid, and reliable means for assisted living residents to submit their opinions and experiences of the care and services they receive, in the hopes of improving care quality in assisted living facilities across geographic locations.