Modi, who is the chief guest, arrived on Friday morning at the venue and was received by Sri Lankan Prime Minister Ranil Wickremesinghe amid traditional fanfare.
Sri Lankan President Maithripala Sirisena was also present.
The Vesak Day marks the birth, enlightenment and passing away of Lord Buddha.
The visit follows an invitation from Sirisena. It is Modi’s second trip to Sri Lanka as Prime Minister after March 2015.
This is the first time that Sri Lanka is hosting the International Vesak Day, recognised by the UN. The theme of the event is “Buddhist teachings for social justice and sustainable world peace”.
Later in the day, Modi will open an India-funded 150-bed multi-speciality hospital at Dickoya in the central hill district which is home to Tamil tea plantation workers of Indian origin.
Modi will also pray at the Temple of the Sacred Tooth Relic at Kandy. The golden-roofed Kandy temple houses Sri Lanka’s most important Buddhist relic — a tooth of Lord Buddha.
In Pallekele, Modi will unveil a plaque commemorating the foundation laying of the Faculty of Kandyan Dance at the Sri Lankan International Buddhist Academy that will be constructed with India’s grant assistance.
Although no bilateral talks are scheduled, the visit is expected to give fresh boost to India-Sri Lanka ties.
Prime Minister Wickremesinghe had visited India in April.