The protest march started from National Stadium where the Game's opening ceremony is planned to be held, and went through the busy shopping street of Omote Sando.
Protesters held banners and shouted slogans denouncing the Olympics and Paralympics.
The protest organiser Toshio Miyazaki voiced his concerns that the recent leadership change of Tokyo 2020 may have distracted people's attention from his final goal, the cancellation.
"Now they (Olympics organizers) are trying to hold Olympics and Paralympics as if holding itself is meaningful, even if without spectators. Today we gather here to against such an idea," Miyazaki said.
An opinion poll published this week by the Yomiuri newspaper showed 58% of people in Japan oppose holding the Games this year, although that figure was about 20 percentage points lower than earlier surveys.
The Olympics are scheduled for July 23 to Aug. 8 and the Paralympics from Aug. 24 to Sept. 5.