Your fish won’t have anywhere to hide, and you may end up feeling like it’s too sterile. Not really a display any more at that point.
However, If your enjoyment comes from seeing the coral grow, and you don’t care as much about the display aspect, you could make it work. Be careful not to loose too much bio media and throw off your tank’s balance.