OER have the potential to improve learning in many contexts and they can contribute to limit the digital divide. Diverse quality criteria are applied for the evaluation of OERs and there is a need for more systematic approaches. This work reports a systematic literature review that synthesizes various aspects (theoretical, methodological, didactical, technical etc.) in regards to the evaluation of OERs. These are mapped into four categories: Learning Content and Learning Experience Design, Technological Issues and Learning Process and Pedagogy, Value-added Services. Furthermore, barriers for the adoption of OER were identified. The criteria and their order in terms of relevance were evaluated in a student survey. Finally, a group of international OER experts assessed and discussed the new instrument in a Delphi study focus group. The results show different ratings and revealed some weaknesses of the criteria list. Based on the discussion between the experts, an improved list was assembled.