Irby was born in Palm Springs, California and grew up in Cabazon, California, the son of Ernie and Cindy Irby. He is of mixed Mexican-American and African-American heritage and has two brothers, Jason and Ernest III.
With a lifelong love for soccer, he played in Europe as part of Team USA before he was forced to give it up due to injury.
Following the end of his soccer career, he attended College of the Desert in Palm Desert, California and Orange Coast College in Costa Mesa, California. His drama professor encouraged him to pursue a career in theater at the American Academy of Dramatic Arts, New York City.