Constrained time measure

[Balazs Dezso]

It is not simple to implement benchmarking tool in C++. One of the difficulties is that some algorithms can run for unacceptable long time on some inputs, so the benchmark running time can be impractical.

There are workarounds for this problem. For example, we can blacklist some test cases for some algorithms, but this can be a tedious work, and it requires updates on algorithm changes. Other solution is to use a controller script which starts the benchmarking tool with different parameters, but this requires the separation of the controller tool and the benchmark tool.

Instead of the previous solutions, I propose a time measure tool, which executes a function with a time constraint. Technically, this can be implemented with a forked subprocess in unix systems (WIN32 doesn't have proper tool for it). This solution is similar to using a controller script, but we don't need to split out the test cases to the script.

My implementation is [7a1a282efbb4].

[Peter Kovacs]

I like this proposal and would have been used this tool if it had been developed before. :)

Let's add it to the main branch!

Minor comments:

• The method name: it is a bit strange for me, but I don't have a better suggestion. Maybe using "limited" or "timeout" in the name?
• The possible exceptions thrown by the method in different cases should be mentioned in its API doc.
• What about combining this new method with the currently existing runningTimeTest()? That method accepts a min_time parameter and calls the measured function several times in order to achieve more reliable measurement. Using both min_time and max_time seems to be reasonable in several cases.
• In write_full(pipefd[1], &success, sizeof(bool)), I would use sizeof(success) for the sake of consistency.

[Alpar Juttner]

What about runningTimeTestWithTimeOut()? I understand it is stupidly long, but it is not something we have to write million times.

[Peter Kovacs]

Replying to Alpar Juttner:

What about runningTimeTestWithTimeOut()? I understand it is stupidly long, but it is not something we have to write million times.

I think it's better than the original name constrainedRunningTimeTest(), as it clearly expresses the meaning of the function.

However, I would prefer the spelling runningTimeTestWithTimeout(), ie. considering "timeout" instead of "time out". Both versions are used, but the single word seems to be more common in US English and computer science:
​https://en.wikipedia.org/wiki/Timeout_(computing)