The present study investigated the extent to which accent sensitivity provides the foundation for lexical knowledge and listening comprehension for Japanese university students learning English as a foreign language (EFL). On an English accent correctness (i.e., YES/NO response) decision task, 63 participants showed considerably high accuracy and speed in identifying correctly-accented nouns (82.14% and 1091 ms) and adverbs (86.67% and 959 ms). However, students were much less accurate in rejecting incorrectly-accented nouns (72.33% and 910 ms) and adverbs (67.67% and 933 ms). The results of a structural equation modeling (SEM) analysis showed that accent sensitivity makes no contribution to either participants’ lexical knowledge or listening comprehension skills. In contrast, a strong direct effect was found from lexical knowledge to listening comprehension. As such, accent knowledge by Japanese EFL students is isolated from their lexical knowledge, with no contribution to their ability for listening comprehension.