Ryan Blake netted in the 77th minute of the tie to give the visitors – currently 17th in the Southern League – a day to remember.
Elsewhere, National League South side Whitehawk won a 5-3 thriller against Lincoln City to reach the second round for the first time in their history.
FA Cup first round results, Sunday November 8 2015
Didcot Town 0-3 Exeter
Aldershot 0-0 Bradford
Brackley Town 2-2 Newport
Braintree 1-1 Oxford Utd
Bristol Rovers 0-1 Chesham United
Halifax 0-4 Wycombe
Gainsborough Trinity 0-1 Shrewsbury
Maidstone United 0-1 Yeovil
Whitehawk 5-3 Lincoln City