In trying to set up a lighting test for the shader and seeing how fast it rendered, I tried to turn up the diffuse bounces to get a more realistic feel on the render. It didn't do anything. Apparently, the way I made the shader with the lambert making the diffuse calculations did not take into account bounces. Now, it does! I just plugged in the physically based diffuse node rather than the lambert.