Survey-Based Research

Survey-based research is a good entry point into global teaching and learning research. It’s easy to manage, and most students and instructors are familiar with electronic surveys.

 

Unlike student feedback surveys, research-focused surveys are administered twice: before starting your COIL project and after completing it. The pre-survey highlights the project’s impact, ensuring any gains are due to the COIL project, not prior experiences.

 

The COIL toolkit offers data collection methods aligned with program outcomes, including six validated survey instruments used in global education studies.

Step 1: Choose Your Survey Instrument

If you would like to assess your students’ intercultural skills, you may wish to consider one (or more) the following survey instruments, each of which measures a different facet of intercultural competency.

CULTURAL INTELLIGENCE SCALE (CQ)*
This 20-item scale measures four dimensions of cultural intelligence: metacognitive, cognitive, motivational, and behavioral (Dynne, Ang, & Koh, 2015).

INTERCULTURAL SENSITIVITY SCALE (ISS)
This 24-item item scale measures awareness/recognition of intercultural differences. Including perspective taking.

INTERCULTURAL EFFECTIVENESS SCALE (IES)
This 40-item scale (with 6 sub-scales) measures intercultural effectiveness, including factors such as behavioral flexibility and identity maintenance.

INTERCULTURAL CRITICAL PERSPECTIVES (IntCrit)
This 12-item scale measures perceptions of critical thinking applied to global learning experiences. The IntCrit and IntComm are frequently disseminated together.

INTERCULTURAL COMMUNICATION (IntComm)
This 12-item scale measures perceptions of communications skills applied to global learning. The IntCrit and IntComm are frequently disseminated together.

GLOBAL ENGAGEMENT MEASUREMENT SCALE (GEMS)*
This 37-item scale is intended to be a comprehensive measure of intercultural competency, with an emphasis on facets such as cultural engagement, knowledge of the partner culture, tolerance for ambiguity and openness to diversity.

*These surveys may require additional fees to use.

Once you’ve selected your survey instrument(s), let us know. We will provide you with a Qualtrics version of your survey, ready to distribute to your students.

Step 2: Choose Your Demographic Questions

Adding demographic questions to your surveys helps differentiate the impact of the COIL project across various student populations. Each COIL toolkit survey includes default demographic questions: major, semester standing, gender identity, race/ethnicity, age range, and first-generation status. You can add or remove items as needed. While COIL Toolkit surveys are available, many other tools measure intercultural competencies.

Here are some considerations when choosing an instrument:

    • Validated Surveys: Use previously validated surveys to meet research and publication standards. Many validated surveys measure COIL learning outcomes.
    • Cost: Survey development and validation can be costly. Be prepared for potential fees, including setup and per-use charges.
    • Relevance: Ensure the survey’s language and constructs remain relevant and appropriate.
    • Translation: If working with non-English-speaking COIL partners, have surveys available in the language of instruction or arrange for translation.

Step 3: Choose Your Incentives

The use of incentives, such as extra credit or prizes, can increase the motivation of students to complete your survey.

The COIL toolkit offers four options to incentivize students to complete your selected survey. These options are listed in order of increasing logistical difficulty, starting with the easiest to manage.

You can provide students with a link or QR code to take the survey without additional incentives. Post it in your Canvas course shell or display it on the screen during class/Zoom.

Note: This is the easiest option but usually results in the lowest response rates.

 

You can offer students low-stakes extra credit for completing the survey. Note that this applies to completing the survey, not just consenting.

Here’s how it works:

  1. Ask students to take the survey using the provided link/QR code.
  2. The first question will ask for their consent to include their responses in the research study.
    • If they consent, they complete the survey and receive a six-digit random number.
    • If they do not consent, they answer open-ended questions about their COIL experience and receive a six-digit random number.
  3. Students upload a screenshot of the random number to receive extra credit.

You will only see the random numbers, ensuring students can freely choose whether to participate in the research study.

Tip: Extra credit often results in the highest response rates, especially in courses where students are concerned about their grades, such as introductory STEM courses.

If the survey is relevant to your course’s learning outcomes, you can include it as a course assignment.

Students will provide their name and PSU email address in the survey to receive credit. After final grades are submitted, you’ll receive a list of consenting students. You can then use the surveys from consenting students for research, excluding non-consenting responses.

If you want to offer a small prize or a drawing for a larger prize for completing the survey, we can set up a second survey for students to enter their name and contact information. This second survey will be linked to the primary research survey.

As the instructor, you should not have access to this second survey. Designate a non-instructor to monitor it, answer student questions, and distribute prizes.

Ensure the prize matches the effort required. For example, avoid offering $50 gift cards for a 10-minute survey.

Tip: e-Gift cards are convenient, but you can also use swag like t-shirts, mugs, or water bottles if you lack funding.

Tip: If conducting student interviews, use the same second survey to collect volunteer contact information.

Step 4: Email Your Information to Laura Cruz (The COIL Principal Investigator) at lxc601@psu.edu

Once you have completed the steps above, send the following information to Laura Cruz:

    • Which survey(s) you wish to implement
    • Who should have access to those survey (name and PSU email)
    • Which demographic questions you would like included in the survey
    • How you plan to incentivize your students

Dr. Cruz will send you the link and QR code for your pre-survey (usually within 1 business day). She will provide a separate link and QR code for your post-survey.