If you made an IRA contribution in excess of the limit, you will be assessed an additional tax on that excess.
The excess amount will flow to Line 15 of Form 5329 Additional Taxes on Qualified Plans (Including IRAs) and Other Tax-Favored Accounts for the penalty to be calculated. There will be a Yellow Alert in the program listing the options available for the excess contribution and a Red Alert prompting for input of the value of Traditional IRAs as of the end of the year in order to calculate the additional tax on excess contributions. Once that is entered, the additional tax will calculate on Line 17 of Federal Form 5329.
No amount would flow to Form 8606 Nondeductible IRAs as it is not a nondeductible contribution, but an excess contribution.
Per IRS Publication 590-A Contributions to Individual Retirement Arrangements (IRAs), page 15:
Nondeductible Contributions
Although your deduction for IRA contributions may be reduced or eliminated, contributions can be made to your IRA of up to the general limit or, if it applies, the Kay Bailey Hutchison Spousal IRA limit. The difference between your total permitted contributions and your IRA deduction, if any, is your nondeductible contribution.
Page 35:
Excess Contributions
Generally, an excess contribution is the amount contributed to your traditional IRAs for the year that is more than the smaller of:
The taxable compensation limit applies whether your contributions are deductible or nondeductible.
An excess contribution could be the result of your contribution, your spouse's contribution, your employer's contribution, or an improper rollover contribution. If your employer makes contributions on your behalf to a SEP IRA,see chapter 2 of Pub. 560.
Tax on Excess Contributions
In general, if the excess contributions for a year aren’t withdrawn by the date your return for the year is due (including extensions), you are subject to a 6% tax. You must pay the 6% tax each year on excess amounts that remain in your traditional IRA at the end of your tax year. The tax can’t be more than 6% of the combined value of all your IRAs as of the end of your tax year.
The additional tax is figured on Form 5329. For information on filing Form 5329, see Reporting Additional Taxes, later.
Note that any link in the information above is updated each year automatically and will take you to the most recent version of the webpage or document at the time it is accessed.