Personally, I would expect the opposite - the optimizer doing a lot of work on optimization passes that have minimal effect on one particular architecture.
Remember that the optimizer is working on a generic intermediate form - as long as that form is a reasonable analogue of the target architecture (e.g. you're not targeting a registerless stack machine), then any "optimization" applicable to the target architecture is also (in theory) capable of being applied to the intermediate representation.
This is, of course, assuming a well-implemented backend that efficiently transforms the IR to the target machine code, that can use optimal instructions even when there's no direct IR analogue to them.
Remember that the optimizer is working on a generic intermediate form - as long as that form is a reasonable analogue of the target architecture (e.g. you're not targeting a registerless stack machine), then any "optimization" applicable to the target architecture is also (in theory) capable of being applied to the intermediate representation.
This is, of course, assuming a well-implemented backend that efficiently transforms the IR to the target machine code, that can use optimal instructions even when there's no direct IR analogue to them.