perf(inference): skip padding-mask work the batch already proved unnecessary

perf(inference): skip padding-mask work the batch already proved unnecessary #3568