Sense, Active Together report 2023
Feasibility of data collection with children with complex disabilities during virtual activity sessions
Executive summary
Sense is a UK charity supporting people who are deafblind or who have complex disabilities. They work to empower people to communicate, connect and experience the world. Their Sport programme is an important way Sense ensure that no one is left out of life.
‘Sense, Active Together’ was a project run by the Sense Active team between 2019 and 2023, with funding from Sport England and the National Lottery. It aims to provide lifelong opportunities for people who are deafblind or have complex disabilities to be active. The programme runs at a national level to establish meaningful, local opportunities to take part in sport and physical activity.
The original evaluation of ‘Sense, Active Together’ aimed to measure health and wellbeing outcomes of children and young people taking part in the in-person delivery of activities. Due to Covid-19 Sense’s physical activity delivery was adapted to virtual activity sessions, run on Zoom, and the evaluation was also adapted to align with this.
Collecting data from children who are deafblind or who have complex disabilities had not been undertaken in this way (e.g. virtually, via video) before as far as the Sense Active and Children and Young People teams and ukactive Research Institute were aware. Therefore, the aim of the evaluation changed. The focus shifted from measuring health and wellbeing data in children who are deafblind or who have complex disabilities to developing and testing a data collection methodology in a virtual activity setting.
Using modified interactive image cards, co-created by the ukactive Research institute and the Sense Active team, data was collected during the Sense, Active Together online sessions between May 2022 and December 2022. During the session, three questions were asked (How do you feel today?; Did you like today’s session?; Do you like taking part sport /exercise /physical activity /movement?), to the children directly, with their parents and / or carers supporting with the responses. Answers were provided using the interactive cards or by another preferred object or way of communicating. Feedback on the feasibility of implementing the data collection methodology – using the interactive cards – was captured through written comments from the Sense Active and Children and Young People teams delivering the sessions and through an online focus group.
For each of the three questions, the interactive cards were used to respond by around a third of children and their families. The majority chose to respond using another preferred method of communication. For example, parents provided observational data on how their child engaged with the said activities, which helped identify the children’s interest. Children showed their interest through, ‘thumbs up’, smiling, laughing, gesturing, moving body parts (like their arms or head), and dancing or moving to the rhythm of music.
The benefits of capturing this additional observational data were highlighted by the Sense Active and Children and Young People teams. They felt that it allowed for a better understanding of the individual abilities and experiences of the children they were working with, and the unique ways they expressed themselves. While the interactive cards were found to be useful as a standardised approach, the design of them prevented them from being fully adopted and they were thought to be too complex for some of the children.
Including a standardised element to the data collection was agreed to be useful but this was thought to be better done through a standardised set questions rather than a set response method. This would then give children, parents and carers, or Sense session leaders the flexibility of using a response method that best suited and represented the individual children which might be the use of interactive cards and / or through observation. Having a mixed approach to data collection was suggested by including refined interactive cards for those children that could use them but to be combined with observations, to avoid bias of parents and carers responding for the children, a challenge that was noted in the introduction.
Overall, this report highlights the development of a data collection approach for research with disabled children in a virtual environment. Although the approach was not fully adopted, the flexibility in the approach allowed for the uptake of different response methods which further evolved the approach that was initially proposed. Co-creation between researchers and delivers was important in the development but future work, such as the refinement of interactive cards, should look to include parents or carers of disabled children to further refine the approach. Continued research in this area is suggested to ensure a level of co-design with disabled children, and that any services being developed for them include, as much as possible, their value insight and perspectives.
Acknowledgements
We would like to thank the parents and carers of children with disabilities who contributed to this research, through their participation in the data collection through the online Sense family sessions. We would also like to thank the Sense Active and Children and Young People Teams who tested out the data collection methodology and without them this report would not be possible.
Introduction
Sense is a UK charity supporting people who are deafblind or who have complex disabilities. They work to empower people to communicate, connect and experience the world. Their Sport programme is an important way Sense ensure that no one is left out of life.
‘Sense, Active Together’ was a project run by the Sense Active team between 2019 and 2023, with funding from Sport England and the National Lottery. It aims to provide lifelong opportunities for people who are deafblind or have complex disabilities to be active by establishing meaningful, local opportunities to take part in sport and physical activity. The project had three core objectives:
- Provide lifelong opportunities for people with complex disabilities to take part in sport and physical activity in more areas of England.
- Influence and equip the sports and health sector to enable more suitable opportunities for people with complex disabilities to be active throughout their lives.
- Develop insight in measuring and evaluating the benefits of sport and physical activity for people with complex disabilities across the sports and health sector.
The original evaluation aims of ‘Sense, Active Together’ were to measure health and wellbeing outcomes of children and young people taking part in the in-person delivery of activities. However, less than a year into the project the Covid-19 pandemic hit and altered Sense’s activity delivery plan.
Sense adapted their approach to support children to be active at home as well as re-establishing opportunities that were previously in place, such as providing weekly drumming to music, dance and yoga sessions through Zoom. Through innovation, collaboration and determination Sense worked with the health, social care and sports sectors to provide opportunities for children to be active in new ways. This change in support from Sense is detailed in the ‘Sense, Active Together Impact Report’1.
With the ‘Sense, Active Together’ delivery moving from in-person to virtual sessions via Zoom, the evaluation aims, and process were also adapted to align with this new method of delivery. The data was to be collected virtually during the activity session. This came with additional challenge of collecting data from children who are deafblind or who have complex disabilities and called for a more flexible and adaptable data collection approach. As far as the research team and Sense teams were aware, this had not been undertaken before.
Considerations for working with disabled people
Research involving children and young people provides valuable insights into their perspectives, however, it is less common for children with disabilities to be involved in research compared to non-disabled children2. Children with complex disabilities, as those supported by Sense, may require flexible approaches in order to communicate effectively and participate in research2. Focusing purely on speech, for example, can be problematic for people who do not use this as their primary form of communication3.
A previous systematic review of how disabled children and young people might be involved in research identified a range of successful alternative communication methods. These included drawing, photography, talking mats, cue cards, pictures, and tape recordings4. However, this research focused on data collection methods delivered in person – less exists about the methods used in a virtual setting and their success rates.
Aims
Therefore, the aim of the evaluation changed. The focus became testing the data collection methodology used to measure health and wellbeing outcomes in a virtual activity setting in children who are deafblind or have complex disabilities, in a virtual setting.
Although the focus was on testing the feasibility of a data collection approach, this report also summarises the data collected through the ‘Sense, Active Together’ delivery as part of the virtual sessions delivered. This, and the feasibility outcomes, have been looked at in combination to provide learnings around next steps and further development for future data collection with children with complex disabilities for Sense.
Methodology
To meet the aims of this project, a feasibility evaluation was conducted by the ukactive Research Institute, from April 2022 – January 2023, working alongside the Sense Active and Children and Young People teams. This included the development of a data collection method, and the testing and evaluation of the feasibility of delivering and using this method. The data collected as part of the evaluation is also presented.
Development of data collection approach
As described above, there are numerous ways for people to communicate beyond speech. To determine the most appropriate approach to use with the children in the sessions, the ukactive Research Institute led a scoping exercise with the Sense Active team in April 2022. This involved exploring literature on collecting data from disabled children, reviewing previously used methods and combining this with the delivery expertise of Sense. Two key elements were identified:
- Flexible, so that the approach could be used with the all the children that Sense support and their varying communication skills and preferences.
- Simple, so that data collection could take place virtually with different members of the Sense teams, whilst also allowing parents or carers to support the children to respond.
The table below shows the three main approaches that were discussed and examples of how data could be collected.
Table 1. Data collection approaches
| Method | Description | Examples |
| Image cards5 | Data capture through age and ability appropriate images and pictures | After the activity session ends, children would be given a range of images that represent activities, settings and equipment used as part of the Sense activity programme, including they had not taken part in. They could then use emotion cards to indicate how they feel about the activity, setting, or piece of equipment By drawing too, the children could express how they feel about the activities and tell us “what they did that day” With post-it notes, the children could show their activity preference and react to given statements, by Sticking a post-it on their favourite image Giving a short sentence/statement in response to a question Using different colours to answer/indicate a response Creating a ‘Graffiti Wall’ to write messages and answer questions (‘what was your favourite thing today’; ‘what has made you happy’) collected after each session |
| Photos6 | Photos can be used to document the types of activities participated in | Photographs could be taken from a child’s, parent’s or deliverer’s perspective. They should include good and bad aspects of the sessions (incorporating emotion cards, positive/negative objects) and could include individuals, families and groups together. Photos of the children both taking part in activities and of just the activity, setting, or equipment could then be used as part of the image cards exercise, so that the images resonate with children. Use of photos representing types of equipment and activities to capture activity preference through pointing or placing stickers. Commentary from all parties can accompany photos to add context. |
| Activities* | Data collection built into age and ability appropriate activities and games. | At the end of the delivery, children could be asked to collect activity equipment from the sessions they liked Children could throw a coloured object to reflect their response to a question or statement – green is good/agree; red is bad/disagree. Children could be asked to move to an area that represents their answer to a given question – in-person only |
* Developed through Sense Active team and ukactive Research Institute’s prior knowledge.
As Sense have expertise in the area, it was agreed that a modified version of image cards would be used (Figure 1.). These cards were built to target multiple sensory forms, catering for the children’s different engagement styles. The cards therefore included different images, colours, and textures. The prototyped cards used simple facial expressions (happy, sad, and neutral), as these are well-known images often associated with the emotion expressed.
These faces were also layered with the traffic light colours (red for sad, orange for neutral, and green for happy) often associated to these three perspectives. Finally, each of the coloured facial expressions had raised textures so the shapes could be made out by touch. During the ‘Sense, Active Together’ sessions, these cards were to be used to obtain responses from the children when asked questions around their preference for and enjoyment of the Sense Activity.
Data collection during the activity sessions
Data was collected during the ‘Sense, Active Together’ online sessions between May 2022 and December 2022. Questions were asked at the beginning and end of the session as to not interrupt delivery or participation. It was agreed that the Sense Active and Children and Young People teams would collect the data, due to their established rapport and relationship with the children and parents or carers. The data was collected following detailed training provided by the ukactive Research Institute.
Parents or carers were sent the interactive cards alongside information on the research when they signed up to take part in the sessions. This information covered why the research was taking place, what would be done with the data, and stressed that participation was optional.
During the session, there were three questions for the children and their parents or carers to respond to,
- Question 1: How do you feel today?
- Question 2: Did you like today’s session?
- Question 3: Do you like taking part sport /exercise /physical activity /movement?
Question 1 was asked at the start of the session, and questions 2 and 3 towards the end. Answers were given using the interactive cards provided or by another preferred communicating. All data was recorded by the Sense Active and Children and Young People teams in a secure, password protected, online database.
Analysis was conducted by the ukactive Research Institute. The aims were to understand the success of the data collection cards across the sessions, the experiences of using them as part of delivery and what the given responses were.
Focus group with Sense deliverers
Feedback on the feasibility of implementing this data collection method was captured through written comments from the Sense Active and Children and Young People teams. Feedback was also gathered through a single online focus group held with members of these teams on 13th December 2022. This focus group lasted 60 minutes and was conducted with eight members of the Sense Active and Children and Young People teams who had been involved in the data collection process. Questions centred around understanding how the team members found the data collection approach, what went well and what did not go well, and specific thoughts on the type of method (the cards) used, and suggestions for future alternations.
The focus group was recorded, transcribed, and analysed using thematic content analysis. This is a process of coding, theming and sub-theming which identifies re-occurring patterns in the data and categorises those patterns into prevalent groups or themes7.
The themed data from the focus groups, the observational notes and group feedback were combined to identify overall key reoccurring themes, develop findings and to inform refinement opportunities for Sense’s services. This data is also appropriate for use with the public, policy makers, and wider stakeholders.
Results
How was data collected?
Data collection took place during seven ‘Sense, Active Together’ online activity sessions between May 2022 and December 2022. Activities that were delivered during these sessions included drumming to music, boxing, dance, sensory sport, and a mini games session. A total of 22 children took part in the research.
Overall, 52 data collection points occurred (data was collected on more than one occasion from some children) with 39 using the set questions and 37 observational data points. Observational data only was provided for 4 children. The average age of the children that took part was 11.7 years – table 2 provides a further break down of ages.
| Age category | Frequency |
| Under 5 years | 1 |
| 5-10 years | 8 |
| 11-15 years | 8 |
| 16-20 years | 3 |
| 21-25 years | 2 |
As described in the methodology, three questions were asked to the children and their parents or carers. They were able to respond using the interactive card or by another preferred way of communicating.
The first question asked at the start of the session was ‘How do you feel today?’. This was answered by all participants (n=39/39). The interactive card was used in 31% of responses (n=12/39) but another preferred communication method was used in 69% of responses (n=27/39).
The next two questions were asked at the end of the sessions. The second question, ‘Did you like today’s session?’, was answered 35 times (n=35/39). Of these 35, another preferred communication method was again popular and used in 64% of responses (n=25/35), with the interactive card being used in 26% of responses (n=10/35).
The same trend was followed for the final question, ‘Do you like taking part sport /exercise /physical activity /movement?’. This was also answered 35 times (n=35/39). Another preferred communication method was used in 77% of responses (n=27/35), and the interactive card used in 23% of responses (n=8/35).
Overall, the same response method was used by, or for, each child for each question. In two cases the second question was answered using a different method to the first and remained as the response method for the third question.
In another two cases, the same response was used for the first two questions and changed for the third. In these instances, the method started as the interactive card and changed to another preferred method of communication.
The observational data collected provided an individualised understanding of how the children responded. In some instances, this was to indicate the ‘other preferred communication’ used as part of the data collection. These methods of communication were recorded by session leaders, either through observation or information provided by parents and carers ahead of the session. For example, some children showed their interest or engagement through, giving a thumbs up, smiling, laughing, gesturing, moving body parts, like their arms or head, dancing or moving to the rhythm of music.
Sense teams provided feedback on the communication style information that had been supplied by parents. The feedback stated that it was sometimes difficult, in a virtual setting, to spot the types of behaviours which parents and carers had described. It was, therefore, easier and more effective to rely on the feedback of the parent in that present moment, rather than ahead of the session.
Signs of positive behaviour and engagement from children differed greatly and were not always obvious to those leading the sessions. For example, it was reported by a parent that their child ‘will attend, go away, and come back’, as part of engaging with the session. Here the parent understood their child was engaged, however, to a Sense session leader, they may have interpreted going away as disengagement. This indicates the importance of having this bespoke knowledge from parents and carers of their children’s regular behaviour. This supplementary information was extremely helpful as to understanding how the children participate and engage in the activities themselves and how they preferred to respond to questions.
How did the children respond to the questions?
The majority of responses to each of the three questions asked were positive. For the first question, ‘How do you feel today?’, 90% (n=35/39) were happy, 8% (n=3/39) were neither happy nor sad and 3% (n=1/39) were sad. For question two, ‘Did you like today’s session?’, 97% (n=34/35) liked the session with just one response neither linking nor disliking it (3%). Finally, for question three, ‘Do you like taking part sport /exercise /physical activity /movement?,’ 74% (n=26/35) like physical activity with 26% (n=9/35) neither linking nor disliking it.
While these results are positive, the amount of data collected currently provides too small a sample size to allow conclusions as to the impact of the sessions.
What was the Sense deliverers feedback?
Based on the feedback from the focus group, three key aspects of evaluation feasibility were highlighted. These are:
- Data Collection Types
- Style of Delivery and Quality of Feedback
- Standardised Approaches
Each of these areas are explained in detail below, with aligning suggested next steps highlighted in the section after.
Data Collection Types
Cards
As highlighted above, during the data collection, there were some difficulties with using the interactive cards. Most often, this was because of the different ways the children interpreted and connected to the card colours, images, and textures.
Whilst the interactive cards had been designed to reflect common classic conditioning (such as a red, sad face meaning a negative response), this was not necessarily the associations the children made. Each were attracted to different aspects of the cards. To some for example, the neutral smiley face was not orange but a glittery gold which many children found attractive and were therefore more likely to select. However, this selection was not necessarily because they associated it with a positive experience within the session. It was also noted by Sense session leaders that because of these associations the cards required a process that was too difficult for some of the children’s cognitive abilities, as described in the below:
“To be able to symbolically represent how they’re [the children] feeling or how they’ve understood or how they’ve enjoyed something, it’s quite a high level, cognitive skill. To be able to say, I’m enjoying this, I’m having this feeling, I’m feeling this emotion, this emotion translates to a smile on my face, that smile translates to an upturned circle on a piece of card… that’s quite a high level. The cards weren’t really accessible for our young people, [because] we have so many different communication methods, and [there are so many] different ways [they can] express themselves and a lot of our young people are still on very early expressive stages.” – Member of the Sense Children and Young People Team
The above learnings indicate an assumption may have been made by the initial ukactive and Sense Active team, based on their experiences, of what the children would understand and associate to. This indicates the importance of having individuals who represent the children and their experiences who can provide feedback into any future development or adaptation of interactive data collection approaches from the offset.
Observations
Sense session leaders felt that the use of observations alongside the delivery and use of the cards – and sometimes in replace of them – worked as a more accurate way to understand the children’s experience of the sessions and responses to the questions.
“From my perspective, [observation] is the best way that we’re going to gain insight from the people that we’re supporting. There’s going to be more of a reliance on observation from kind of the people that know [the children] the best in order to understand what that impact is going to look like. And that’s a challenge really, because it’s means that any kind of evaluation, it’s going to be a lot more labour intensive than it would if someone was self-reporting. But I think from my perspective, observation is the way to go if we’re going to dig down a little bit deeper into impact and stuff. And it’s just about how we embed that into our evaluation, into our sessions, given [our team] capacity.” – session leader from of Sense Children and Young People team
Parents and carers provided information to the Sense Active and Children and Young People Activity teams who were running the sessions, enabling them to better determine what positive and negative responses might look from each child. This was easier for members of Sense Children and Young People Team than the Sense Active Team because the latter had less exposure and knowledge of working with the children and their families. This indicates the importance of an ongoing personalised approach working and getting to know the children, with support from the parents and carers to support the quality of the data that can be collected.
Style of Delivery and Quality of Feedback
Style of session (virtual versus in person)
All delivery and implementation of data collection took place during the online sessions. This was adapted from the original in person delivery because of the adaptation Sense made to its activity delivery following the Covid-19 pandemic. There were challenges specifically associated to using both the cards and conducting observation for data collection online.
Online delivery made it harder to collect data in any form, due to the complications of online interaction (such as participants not having a microphone or turning cameras off) meaning that questions could not be properly communicated. Furthermore, in a virtual setting, deliverers had less control over helping the children, parents and carers use the cards because they could not move around and interact with each of them individually as would be possible in a face-to-face session.
Finally, it was also felt that the approach, using the cards and asking the individual questions to everyone, was not scalable to larger group sizes in a virtual setting due to the time it took to ask each child the questions and get responses. A size of around 5-10 individuals was thought to be feasible, but the approach was not executed in groups that were larger than that. Generally, the online sessions did not have more than 10 children participating.
Feedback quality
The accuracy of responses using the cards improved over the course of the research. Sense session leaders, parents and carers got used to using the cards in a virtual setting. This was considered both a positive and negative.
While it meant that the number of children questioned and quality of the data increased, it also meant that it was not an approach that any member of the team could immediately pick up an execute without prior knowledge or learning. This equated to a potentially more labour-intensive way of collecting data.
As a result, members of both the Sense Active and Children and Young People teams felt that better collaboration between the two teams was needed. Both hold key areas of expertise that contributed to better data collection – the Children and Young People team with their experience of the children and the Active team with their experience of the data collection method.
Due to the difficulties of using the cards, and because these were not appropriate for all children, the parents and carers were relied upon to provide answers on behalf of their children. Likewise, it also presented the potential of a subjectivity bias, whereby Sense session leaders provided responses to questions based on the observation information they had received from the parents and carers. This would rely on their individual interpretations of the observed behaviour, which may differ between team members and may lead to inconsistency if it was not the same session leader recording responses each time.
The above two challenges meant that responses were received from a parent or carer and not necessarily the children themselves, which is what this project set out to achieve. It was felt, by some Sense session leaders, that a potential courtesy bias was in place with parent’s responses, whereby parents and carers did not fully express when their dislike or unhappiness with a session for fear that the sessions may stop if they did.
Standardised Approaches
There was a consensus within the Sense Active and Children and Young People teams that a feasible data collection approach should be in some way standardised, but also with an element of flexibility. This was to allow consistency so the same approach could be adopted across all sessions and team members.
Based on this, it was suggested that the data collection questions could be standardised across all sessions and data collection, but the method through which responses were obtained could be varied, using, for example, a combination of cards, observations, and other preference communication. This was the approach that was, through natural evolution, adopted towards the end of the delivery. However, this requires further refinement from the two Sense teams, and the parents and carers of participating children.
“Really trying to sort of think about a way in which to make it standardised, but adaptable I think is probably the way to go. So that the questions that you want answers to a standardised but the way in which we find the answers to those are individualised I think would be the way forward.” – session leader from the Sense Children and Young People team
It was also noted that variability existed in day-to-day behaviours, moods, and reactions of the children. Any data collection, particularly through observation, should therefore be an aggregation of responses over time. Use of this approach would be to ensure responses were representative of the children’s overall engagement, rather than being dependent on the circumstances on any one given day.
“I suppose [another complication] is that all of the observations it’s always at like, very clear that it is [based on], this [specific] moment, at this time, on this day, in these conditions. I think, to come to any conclusions that you could confidently say was a child or young person’s consistent reaction to something that would have to be observation over time, I don’t think we could take sort of one session as a definite.” – session leader from the Sense Children and Young People team
In line with creating a formalised approach to data collection, it was also agreed that a more formalised method of recording observations was needed. This would allow different team members to more accurately and easily determine key behaviour points to look out for each child and help with the challenge associated inconsistency and individual interpretation.
Conclusion and Suggested Next Steps
This evaluation aimed to develop and test a new data collection method with disabled children and their families as part of the ‘Sense, Active Together’ virtual delivery. There is currently limited research to understand how best to include disabled children within research conducted in-person, let alone in virtual settings. The adaptations Sense made to the ‘Sense, Active Together’ delivery – by moving from in-person to virtual delivery due to the Covid-19 pandemic – presented both a challenge and opportunity for this type of data collection.
The ukactive Research Institute and Sense Active team developed an adapted interactive image card that aimed to be flexible and simple tool that could be used for online data collection with disabled children. Although the interactive cards were used during seven online activity sessions between May and December 2022, for each of the three questions asked around two-thirds responded by another preferred object or way of communicating relevant to the child. This indicated that there were some limitations with the use of the interactive cards on their own.
Feedback from the Sense Active and Children and Young People teams who were involved in using the cards, highlighted the benefit of also capturing observational data, as this allowed for a better understanding of the individual children they were working with. The interactive cards were found to be useful as a standardised approach, but the design of them prevented them from being fully adopted. This inspired the learning that, in any future development of interactive data collection methods, an individual who represents the child and understands their ability level of interaction should be included from the offset.
Nevertheless, including a standardised element to the data collection was agreed to be useful. However, this was thought to be done better through a standardised set of questions rather than response method. This would then give children, parents and carers, or Sense session leaders the option of using a response method that best suited and represented the individual children. The added benefit of this approach would be that it is also transferrable between virtual and in-person delivery, should a future project provide both options. Having a mixed approach to data collection was suggested by including refined interactive cards for those children that could use them, but to be combined with observations to avoid bias of parents and carers responding for the children – a challenge that was noted in the introduction.
Overall, this report highlights how a data collection approach can be developed for research with disabled children in a virtual environment. Although the developed approach was not fully adopted, the flexibility in the approach allowed for the uptake of different response methods which further evolved the approach that was initially proposed.
Co-creation between researchers and delivers was important in the development. Future work should look to include parents or carers of disabled children in this discussion, to further refine the approach.
Continued research in this area is suggested to ensure a level of co-design with disabled children, and that any services being developed for them include, as much as possible, their value insight and perspectives.