A quick perusal of the innerwebs says:
Salicylic acid is the active substance that can visibly improve razor bumps. It is a dermatological-grade ingredient that exfoliates, moisturizes, clears pores and can help prevent infection. Use a post-shave product with salicylic acid so it remains on your skin the whole day.
Use only a non-acnegenic shaving cream specially formulated for sensitive skin, with lots of lubricating agents (foam-based shaving creams can dry and irritate your skin).
Do not use any product that has alcohol, it will seriously worsen ingrown hairs by drying the skin and closing the pores.
Exfoliate. Daily use of a gentle face scrub with glycolic and salicylic acid is particularly effective.
Use a soft-bristle face brush and liquid cleanser in a circular motion on your beard to dislodge the tips of ingrown hairs, eliminate dead skin cells and clear follicles to allow hairs to surface unimpeded.
Shaving too closely is one of the triggers for razor bumps. Hair stubs cut too closely will get trapped inside the hair follicle and dig inward or sideways. Don't worry, the disappearance of unsightly ingrown hairs will more than make up for the "five-o'clock shadow" appearance.
To avoid shaving too close, don't pull the skin when you shave; don't put too much pressure on the blades; shave with the grain and use a single-blade razor.
You will need to maintain this approach over time, as one extra-close shave will be enough to cause a recurrence of ingrown hairs that will take weeks to heal.