Commercial skipping when show not complete

Why do I have to wait 40 minutes after a show is complete for commercial skip to work?

Because the processing for commercial detection does not begin until after the recording has been completed.

You can start watching before commercial detection is finished, and once it completes the commercial markers will be available, so you can use commercial skipping for the second half of the recording.