This article makes it seem like there was quite a bit of genuine Union sentiment in Norfolk, Virginia in 1862.
From the Richmond Daily Dispatch July 10, 1862:
The Yankees Celebrating the Fourth of July in Norfolk.
–The Yankees had a celebration of the 4th of July in Norfolk. A paragraph from the Norfolk Union, of the 5th will show how a part of it was done:
Next came the Union car, most beautifully decorated, drawn by eight of the finest dark bay horses that were ever put in harness, and driven by a driver (we could not learn his name) whose tact and masterly manÃ…uvring through the streets, entitled him to the hand of the most beautiful young lady on the wheel car. On the car was seated a blooming young lady, representing the Goddess of Liberty, supported by two little sailor boys, and supported by 34 young ladies, representing all the States of the Union. Each had a flag in her hand, as did the miniature sailors, the Goddess wearing the cap of Liberty, and waving over the whole were two large and beautiful American flags. This was truly a gorgeous spectacle, and many were the impressions upon the street that it was the grandest display they ever saw. Next to the car was a carriage, drawn by a noble span of fine black horses, and in it were the clergyman and editor of the Union, and a noble old patriot, now counting nearly 80 years, who believed that to be the last celebration of independence Day that his eyes would ever be permitted to behold.
After the exhibition of the gay equipage to the “indulging thousands,” all hands repaired to the Opera House, where “Gov.-Gen.Viele” made a short speech. Several other individuals connected with the Hessian army made speeches, after which the assemblage desperate. [departed?]
Egbert Ludovicus Viele was military governor of Norfolk in 1862. He returned to civil engineering in 1863. He was buried at West Point after his death in 1902.