If you modify a penalty rule, the new settings apply to all currently due invoices and any future invoices you create.
You cannot delete the penalty rule but can make the penalty amount as zero and it will be instantly applied.