There are accepted opinions that rule one is not obligated to eat bread by Shalosh Seudah. Based on this, one is no required to repeat Birchas Hamazon if Retzei was omitted.
There is an interesting discussion regarding what should be added when Rosh Chodesh falls out on Sunday. As a rule, we recite Retzei by Shalosh Sedudos even when the meal is continued after Shabbos. The only exception to this rule is if Maariv is Davened before reciting Birchas Hamazon. One cannot recite Retzei once Shabbos has ended with the prayer of Ata Chonantanu in Maariv.
According to the first opinion in the Mishna Berura, only Retzei should be recited, as it is contradictory to recite both Reztei and Yaaleh Viyavo, as it cannot be Shabbos and Rosh Chodesh at the same time. According to the second opinion in the Mishna Berura, one can add Yaaleh Viyavo, as we are not concerned with this apparent contradiction.
Reb Shlomo Gantzfreid in the Kitzur Shulchan Aruch writes that one should avoid eating a Zayis of bread after nightfall. This is in order to avoid dealing with this dilemma. Reb Shlomo Zalaman Aurbach rules that although one should avoid eating after nightfall, it is permissible to eat products other than bread. It is also permissible to eat bread, as long as one avoids eating the proper amount necessary in order to recite Birchas Hamazon after night begins.

