The dad-of-two from Owensboro, Indiana, was attending the town's Trail of Treats Halloween event and said he came up with the costume ideas at the last minute.
But he couldn't explain why he had a full Nazi uniform close to hand in his house.
He went as a Nazi soldier while his young son was dressed as Adolf Hitler, complete with a drawn-on moustache.
Speaking to local media, Goldbach admitted the outfits had been a mistake but claimed he had innocent intentions.
At first, Goldbach was defiant, and on Facebook slammed parents for "threatening" him and his son.
Goldbach said it was when people started to come up to the pair and say they were offended by their costumes that he realised what he had done was wrong.
"About the time the first person approached us — it was at that time that they clued me to, 'we shouldn't have done this'," he went on.
Wanda Anthony, a resident who had been disturbed by the costume told reporters Goldbach should have gone as a clown instead.
"A lot of people are afraid of clowns. He could have been anything," she said.
Despite apologising, Goldbach claims he isn't a Nazi, just a lover of history.
Source: Daily Star