Could be a network or disk issue. For each TVE stream, it must first go through your network to your DVR server. Then it gets written to a temporary buffer on disk in your recording directory. Then it is read from the disk and sent to its final destination: either another file for a recording, or back across the network to the client.
Those steps are repeated for each TVE stream; so 2 streams means 4 concurrent network streams of that content and 2 concurrent writes on disk, and 2 concurrent writes if you have 2 live streams going.
To improve performance, make sure everything is wired gigabit ethernet, and check that your drives are healthy and not failing.