Why didn't General Lee attack the U.S. Army in the city of Fredericksburg after they retreated?

He didn't feel strong enough to counter-attack. Despite the Union casualties, they still outnumbered Lee's army.