If a return is rejected, any scheduled direct debit withdrawal (payment) is canceled. If you chose to pay by credit card, the tax payment portion of that transaction is also canceled (payment of TaxAct fees will go through as scheduled).
The only exception to your tax payment being canceled would be if you chose to use your credit card and the payUSAtax website to make the payment.