@printguy111 I've dabbled a bit with multi-material 3D printing (E3D Toolchanger).
My theory on this is:
there is a small offset between the outline of every other layer, this is what you call "uneven" or "less smooth".
the "smaller" layer should be the first layer of this material after a toolchange. The extruder is not properly primed at this point, the material has been sitting for a while in the nozzle and some might have oozed out.
Edit: also check the temperatures. Cura uses standby temp, initial printing temp, printing temp, and final printing temp. Don't know why, it starts the layer with "initial printing temp" which at default might be 10-15 °C lower than printing temp. For consistent extrusion, I suppose all printing temperatures should be the same.
after finishing this layer, the printer directly jumps to the next layer from the same material. This time the extruder is perfectly primed since it has just finished the layer before.
to solve this problem, you could try the following:
print the infill before the walls. This should prime the extruder accordingly before it reaches the walls. Doesn't work however with small cross-sections.
add a prime tower or waste part that is printed first for each layer. The last bit is important and might not be quite easy to reach - IIRC, Cura for example seems to scan the build platform from left to right.
hope this helps,
Niklas