Once the problem has been identified, the solution becomes obvious: make sure the work we push to thread pools describes the resources to acquire before running the code in a dedicated thread.
My favourite approach assigns one global thread pool (queue) to each function or processing step. The arguments to the functions will change, but the code is always the same, so the resource requirements are also well understood.
I think I get what he's saying. To avoid one stage in the multi-step computation acquiring too many resources and starving other stages:
- Cap the resources used by each stage.
- Create one threadpool per stage.
- Include sufficient information in pending jobs that the threadpool can calculate the resources needed to process each job.
- Only start a job running when it can be run without exceeding resource cap for the stage.
Conceptually, you can think of it as pre-allocating a bundle of resources to each processing stage: threads, memory, etc. Each processing stage then runs as many jobs concurrently as it can with the allocated bundle of resources. That's the idea, but in reality each job allocates its own resources from a common pool that is shared among all the processing stages, so the processing stage has to calculate how much memory, etc. each job will allocate before giving it a thread to run on.
Pretty much, plus if you're even running, you have high confidence that lock acquisition will succeed, because the entity that gave you a thread to run on has checked that all the resources you need are available.
These are not the same things. The major difference is that hardware for instructions can do only one thing so it can be thought of more like a physical process.
For a software pipeline, you don't need to hand data off to a different thread, you just need to make sure the data is run with the right execution. That could be on any thread, and the execution/function can be run on any number of threads at the same time as long as the data is separate.
The solution proposed seems to be some variant on dataflow programming: <https://en.wikipedia.org/wiki/Dataflow_programming> Am I misunderstanding?