Each year, the Center for Measurement Justice offers Black, Brown, and Indigenous (BBI) PhD scholars in the field of educational measurement the opportunity to gain research experience by collaborating with one of our esteemed BBI mentors. This program aims to provide research and professional development support to BBI graduate students in educational measurement in an effort to create a more fair and equitable assessment system.
In the Graduate Summer Internship, scholars work alongside mentors on research projects, gaining hands-on experience in conducting cutting-edge research within the field. This year, we had three graduate summer interns who made significant contributions and developed essential research skills.
Read more about the scholars’ transformative experiences below.
Project Title: Culturally Relevant Assessment and Teacher Evaluation of Students
Wallace Nascimento Pinto Junior
Scholar
Dr. Kyndra Middleton
Mentor
Description of Research
Development of culturally relevant mathematics assessments for K-12 students; Literature review on the underrepresentation of Black and Brown American students in gifted programs; Review of Academic Acculturation Scale for Middle and High School Students items.
Summary of Activities
- Conducted a literature review on culturally relevant educational assessments for elementary and middle school students, identifying two key perspectives: antiracist framework (Randall et al.) and representativeness in samples for gifted student assessments (Maker et al.).
- Developed 14 culturally relevant and traditional math problems, modifying the PISA mathematical literacy construct definition based on Randall’s antiracist framework. Items were categorized using PISA, TIMSS frameworks, and Common Core Math Standards. Reviewed by Dr. Middleton and another intern.
- Conducted a literature review on the underrepresentation of Black and Brown students in gifted programs, summarizing Ford’s (2010) four roadblocks: teacher referral, test performance, outdated policies, and social-emotional concerns.
- Reviewed 39 items of the Academic Acculturation Scale for Middle and High School Students, categorizing them into 7 themes/sub-scales and using survey item writing guidelines to suggest revisions. Discussed findings with Dr. Middleton.
My favorite part of the internship was…
that my activities were mainly related to the development of instruments (test and questionnaire items) and validity issues, areas in which I have a lot of enthusiasm.
What did you learn from this internship?
I had the opportunity to learn a lot about culturally relevant educational assessments and the anti-racist assessment approach. Prior to this experience, I knew very little about these topics, and I was struck by the issues involved.
I realized that the goal to reduce or eliminate construct-irrelevant variance in assessment development is not only difficult to achieve but also problematic, depending on the way the construct is defined. By attempting to exclude cultural aspects from assessment items, developers may inadvertently assume an “average” student – more specifically, a white middle-class student. This can result in assessments that do not represent the experiences and knowledge of students from diverse cultural and social backgrounds.
In other words, the exclusion of cultural aspects from assessment items does not necessarily result in neutrality. Instead, it can perpetuate a hidden bias towards the mainstream culture. This can have negative consequences for students who do not fit the assumed cultural norm, as they may be disadvantaged by assessments that exclude their social and cultural identity, or even their knowledge and skills.
What’s more, this process can occur without the intention or awareness of assessment developers. So, it is essential to recognize and address these issues to create more inclusive, equitable and useful assessments.
Ultimately, my experience learning about culturally relevant educational assessments and the anti-racist assessment approach was eye-opening and thought-provoking. It highlighted the importance of considering the complex and nuanced issues involved in assessment development and the need for greater diversity, equity, and inclusion in this process.
Project Title: Displaced Livelihoods Initiative Narrative project on Rohingya population
Mehdi Rajeb
Scholar
Dr. Fatima Zahra
Mentor
Description of Research
During my internship, I contributed to three key research projects:
-
- Impacts of Climate Change on Cognitive Performance: I co-authored a manuscript analyzing the effects of rising temperatures and climate volatility on cognitive abilities using 20 years of country-level data. The findings, showing significant cognitive declines due to climate change, are being prepared for journal submission.
- Displaced Livelihoods Initiative on Rohingya Refugees: I developed research protocols, IRB documentation, and data collection tools for a study on the mental health and livelihoods of displaced Rohingya populations. The tools are under review for final approval.
- AI-Enhanced Assessments and Psychosocial Index Study: I conducted a literature review on AI’s role in improving assessments and initiated a feasibility study on the Psychosocial Index scale.
Summary of Activities
Research Article on Climate Change and Cognitive Performance
- Collaborated on a manuscript titled “Impacts of Climate Change on Cognitive Performance.”
- Collected country-level cognitive performance data and climate data from the past 20 years.
- Conducted literature reviews, identified methodologies, and performed data analysis using multilevel modeling.
- Findings showed a significant decline in cognitive function due to rising temperatures and climate volatility.
- Manuscript is currently being prepared for submission to a peer-reviewed journal.
Displaced Livelihoods Initiative Narrative Project on Rohingya Refugees
- Research Protocol Development
- Developed IRB documentation and research protocol for a project on the mental health and livelihoods of displaced Rohingya populations.
- Contributed to research methodologies, including data collection procedures, sampling plans, and analytical frameworks.
- IRB protocol is under review for submission by the implementing organization.
- Instrument and Interview Protocol Development
- Played a key role in designing the data and collection tool and interview protocols.
- Ensures alignment with ethical guidelines and research objectives.
- Instrument is in final review and awaiting approval for field testing.
Other Research Activities
- Conducted a literature review on AI-enhanced assessments to conceptualize innovative approaches for improving traditional assessment practices.
- Submitted the review for future studies.
- Conducted a feasibility study on the Psychosocial Index scale, though the project was not completed due to time constraints.
- Managed research budgets and coordinated communication with stakeholders for the Rohingya project.
My favorite part of the internship was…
First, my learning curve was very high. I never worked in research-based project management. In this internship, I could get the complete idea about the rigor and intensity of research-based project management. Second, I learned a great deal about communication from my adviser. I think it was important for me. Furthermore, my adviser always inspired me to collaborate with other researchers, scholars, and research professionals. I learned a great deal from these interactions.
What did you learn from this internship?
Rohingya Project : My involvement in the Rohingya project provided me with valuable experience in preparing Institutional Review Board (IRB) documentation for research initiatives. Additionally, I contributed to the development of the data collection instrument for this project, which enhanced my understanding of data collection methods for socially marginalized populations. Through this role, I became acquainted with various assessment scales, including the HTQ-5, PHQ-9, and SPIESS scales, which are utilized for psychosocial assessments. In this project, I also learned a great deal about how to communicate and manage a data collection agency.
Impact of Climate Change Project: In my research projects, I acquired the skill of compiling datasets from secondary sources to address specific hypotheses. Additionally, I investigated various methodologies for analyzing panel data. This experience provided me with substantial knowledge in panel data modeling,including the application of both fixed effect and random effect models.
AI in Assessment Project: I developed a literature review on AI-enhanced assessments. The literature review allowed me to be acquainted with the current practices and methodologies that can be applied to enhance AI-based assessment.
Project Title: Competency assessments in STEM and engineering education
Sandra Liliana Camargo Salamanca
Scholar
Dr. Malena Oliveri
Mentor
Description of Research
During this internship, I had the privilege of participating in a wide array of research areas, each offering unique insights and contributing to the broader field of educational measurement in engineering education. Working alongside the esteemed team of Dr. Oliveri and Dr. Lawless, we delved into the crucial topics of validity and fairness in higher education assessments. This diverse exposure not only broadened my understanding but also enriched my skills in educational assessment research within the field of engineering education.
Summary of Activities
- I wrote one article about measurement in engineering education and was starting to prepare another based on the ITC Technology-based Assessment Guidelines.
- One of my tasks was to finish the results and discussion sections of the paper “Fairness Is Essential for Interpreting Engineering Students’ Conceptual Understanding.” I conducted the Measurement Invariance, DIF analysis, and the estimation of the impact of DIF on female and male students. I also contribute to writing part of the introduction, method, results, and discussion sections. Finally, I reviewed and edited the paper to prepare it for submission.
- The second task was writing a journal paper discussing the ITC Technology-based Assessment Guidelines. I prepared the summary of the guidelines, proposed a visualization for the summary, and started writing the section about the overview of the ITC TBA guidelines chapters. We expect to finish this paper in October 2024.
- I prepared conference submissions for NCME and AERA 2025
- I submitted a proposal for the NCME 2025. This proposal will present the results of the study titled “Fairness Is Essential for Interpreting Engineering Students’ Conceptual Understanding.”
- I submitted a conference proposal for AERA 2025. This presentation will share the results of a meta-analysis of the effectiveness of rubric use on performance.
- I supported the edition and revision of conference presentations
- I reviewed and edited four conference presentations. The presentations share advances in research about cultural and linguistic responsiveness in workplace assessments for learning the International Baccalaureate (IB) exams’ procedure for using the successive adaptation model in their translation/adaptation processes global perspectives on higher education admissions, specifically navigating access, diversity, and equity challenges. SCALE: AI hardware assessments.
My favorite part of the internship was…
working with Dr. Oliveri. She is very clear and kind in communicating expectations and feedback, which helped me advance my work and contribute to the project on time. Additionally, the internship gave me the opportunity to expand my network and meet other professionals, such as Dr. Lawless and Dr. Zenisky. This experience broadened my perspective about working collaboratively and learning from different experts.
What did you learn from this internship?
Regarding knowledge in the field, I learnt more about cultural responsive education in engineeringeducation contexts, and the guidelines of technology-based assessments. About skills, the internship allowed me to improve my writing skills as well as improve my abilities to work with other experts in the field.
Project Title: Evaluation of a National-level Mentorship Program for Educational Measurement Doctoral Students
Roti Chakraborty
Scholar
Dr. Joseph Rios
Mentor
Description of Research
This summer I worked with Dr. Joseph Rios to develop and administer a survey to evaluate the supports that the Center for Measurement Justice (CMJ) provides to doctoral students for their academic and professional development. This survey asks CMJ scholars, at every stage of doctoral study, to provide feedback on their experience with CMJ’s various programs (mentoring circles, writing accountability circles, professional development webinars). The outcomes from this study will help to enhance CMJ’s activities to assist the doctoral student more effectively.
Summary of Activities
- Developed a survey focused on minoritized doctoral students’ engagement with CMJ’s mentorship program and additional needs.
- Shared the survey with five CMJ mentors and writing group facilitators for feedback on design.
- Survey piloted with five CMJ scholars and revised based on feedback.
- Final survey sent to 141 CMJ scholars; Received 39 responses.
- Quantitative data analyzed and proposal submitted for NCME 2025 Annual Meeting.
- Follow-up online interviews arranged with nine consenting participants for mid-October, with 30-60 minute interviews to expound on survey responses.
- Manuscript writing in progress.
My favorite part of the internship was…
the collaborative teamwork, the productive open discussions, and the respectful environment. I truly appreciated how my insights were valued and my contributions recognized. I am grateful for the consistent feedback and guidance from my mentors at each step, which enabled me to complete my tasks promptly.
What did you learn from this internship?
I have acquired advanced knowledge in mentorship for doctoral students, program evaluation research methodologies, survey item development, and manuscript writing. Specifically, I have gained expertise in designing surveys effectively to enhance response rates. Engaging in program evaluation studies and studying mentorship strategies represent significant achievements that will be instrumental in shaping my future career.