Justin Bieber is back to busking — this time for love!

On Tuesday, the singer, 24, was spotted singing with a guitar outside London’s Buckingham Palace, where he sang in front of a group of fans that included wife Hailey Baldwin.

In addition to telling the model, 21, that he loves her, Bieber serenaded her with an acoustic version of his 2016 collaboration with Major Lazer and MØ, titled “Cold Water.”

“Nobody recognized Justin at first because he had a hoodie on,” an onlooker tells PEOPLE. “Then he just took his guitar out and started serenading Hailey while she recorded him her phone. He played for around 20 minutes and dedicated “Cold Water” and a cover of [Tracy Chapman’s] “Fast Car” to her, saying, ‘This is for the love of my life.’”

The source adds, “You could see that Hailey was really touched — she put her hands over her mouth. The crowd was really touched too.”

Splash News

The pop star was dressed in black shorts and a multi-colored tie-dye sweater with the hood over his head while bride-to-be Baldwin wore a matching grey sweatsuit.

The pair, who legally tied the knot in New York on Sept. 13, was also seen walking hand-in-hand and Bieber wrapping his arms around Baldwin after visiting the London Eye.

Inside Justin Bieber and Hailey Baldwin’s Secret Marriage: ‘They Didn’t Want Anyone to Know’

On Monday, Bieber and Baldwin traveled to London for her Adidas fashion show and have been spotted packing on the PDA while sightseeing around town.

RELATED VIDEO: Inside Hailey Baldwin’s London Fashion Week Show Days After Secret Marriage to Justin Bieber

Meanwhile, that same day, across the pond, Hailey’s uncle, Alec Baldwin, seemingly confirmed the marriage news at the 2018 Emmy Awards in Los Angeles, telling Access: “They just went off and got married.”

How Justin Bieber & Hailey Baldwin ‘Found Their Way Back’ to One Another: Source Splash News

PEOPLE confirmed on Friday that Bieber and Baldwin had a courthouse wedding, just two months after getting engaged in the Bahamas on July 7.

“They didn’t want anyone but their families to know they got married. This wasn’t a wedding,” a source told PEOPLE exclusively, adding that the actual party with family and friends seems more important to the newlyweds than the legal act.

“They can’t wait to have a wedding,” the source said. “But for other reasons, it was easier for them to get the legal stuff out of the way now and to get married. It was more of a necessary ceremony to make it legal. It doesn’t seem like it was a big deal to them,” the source added of the pair who started dating (again) in May.

From ‘Just Friends’ to Married! A Timeline of Justin Bieber and Hailey Baldwin’s Whirlwind Relationship Neil Mockford/GC Images Justin Bieber & Hailey Baldwin’s Secret Marriage: ‘They Didn’t Want Anyone to Know’ Says Source

Bieber and Baldwin previously dated before a messy 2016 breakup and rekindled their relationship after he broke up with ex Selena Gomez.

“Both Justin and Hailey’s families are very excited about them finding their way back to each other. It all seems meant to be,” an insider recently told PEOPLE.

“They are now in the right place and this is the time. It was never like that before. They each needed to take individual journeys to get to this point. Now they are able to continue a journey together,” the insider shared.