If we have a set of things (e.g. language models) and some measures we care about (e.g. cost, speed, whether weights are open, scores for a few benchmarks, etc.), then some of those things will be "pareto optimal" (see below) and some won't. The "pareto front" is the subset that is pareto optimal.
Some thing is "pareto optimal" when there isn't another thing that's AT LEAST AS GOOD in ALL measures, and BETTER in at least one way. For example, if we say there are no ties (for simplicity), then the cheapest language model is pareto optimal; the fastest model is pareto optimal; those which score highest on each benchmark are pareto optimal; and so on.
Tradeoffs can also be pareto optimal: for example, if the cheapest model is also slow, then there will be more pareto optimal models which are "cheapest for their speed"; and so on for other tradeoffs (e.g. fastest that achieves a certain benchmark score; cheapest model with open weights; etc.).
If you're making a decision about which thing to choose, you only need to care about those in the pareto front (since, by definition, anything that's not pareto optimal is objectively worse on at least one measure).
Pareto optimality does not compare one measure against another: something that's 10000x slower can still be pareto optimal, if it's 1% cheaper than the alternatives. To pick a "best" thing, you could give a weight/importance to each measure, and combine them into an overall score: but that's subjective, and might vary between people and tasks. In contrast, focusing on the pareto front is a way to ignore those things that will never be the best, regardless of weighting.
I honestly think the wikipedia article is too complicated. My own image example here as an another attempt to explain: https://imgur.com/a/5ZQIJDb
Mapping the cost of something (like an algorithm), and the time it takes (so lower is better for both). 1, 3 and 5 are all optimal in their own sense. No one is strictly better than the other, just different tradeoffs you have to choose yourself. However, you would never choose 2, because for a lower cost you could get the same result choosing 3. Same with 4, 6 and 7, they all have something that's both faster and at the same time just as cheap you could choose.
A pareto front is a bit like the classical "fast, cheap, good, choose 2". There are always tradeoffs, but if something is both slow, expensive and not better than something that's faster and cheaper, it's a bad choice, and thus not on the "pareto front".
If you optimize one criterion, it's simple: lowest is best or highest is best.
If you optimize multiple criteria, all optimal trade offs between any of the selected criteria are "best" in some way.
When you have a tradeoff between two parameters, which points dominate the others in the sense that you can't choose another point without getting less of one of the parameters.
"what's the family of optimal choices when you have multiple dimensions to rank on?"
Say a race vehicle has acceleration, top speed as defining parameters. Some are slow but accelerate hard, others need a long time to reach very high top speeds. Others are in between, or just flat out bad at both.
The pareto frontier is the set of vehicles that are best: pick one from the frontier and you can be sure that for it's given top speed, none accelerate faster. And vice versa, pick one with a given acceletation and you are sure none have a better top speed
The Pareto points are where you sacrifice the least of anything to get the most of everything.
There's the saying about buying computers. Good, Cheap, Fast, pick any two. That's where you would prioritise.
If someone makes something that better, cheaper, and faster, or even pretty close to the best on two of those and clearly better on the other. It's a Pareto point.
Over time computers are getting better, cheaper and faster (software notwithstanding). The leading edge of that advance of all of the things is the Pareto front.
> a Pareto front represents the set of solutions where no solution outperforms any other solution in the set at every objective
I do not believe you are correct when you say
> something that better, cheaper, and faster, or even pretty close to the best on two of those and clearly better on the other. It's a Pareto point.
Since that would outperform on every objective
GP's point that it's prioritisation does not seem incorrect to me. Prioritisation involves considering trade-offs of various approaches and deciding which aspects & attributes to optimise for, at the expense of others.
Prioritising is when you choose something over another. A drag racer prioritises time to travel a quarter mile.
Going for the Pareto is when you elect not to prioritise. It is explicitly deciding to not choose one property over another ant to keep everything as much as you can.
I read it as finding the point of maximum effectiveness. The point at which the most is done for the most.
Getting to that point can be calculated (in some cases), but I suspect most folks get there by trial and error. Finding out what is effective, and what is not, and choosing what is effective, over what is not, until there's no longer a choice. That often becomes tribal knowledge, and is handed down. There's always someone trying to improve it, and when they figure it out, that gets added to the tribal knowledge. Basically, that's how nature does it, so there's some serious prior art. Natural Selection is brutal prioritization.
In Morocco, they used to announce the end of the Ramadan fast, by holding up a black thread and a white thread, and waiting until they could not tell the difference.
Then, they'd fire a cannon, and everybody would dig into some awesome soup. Sort of the same thing.
If we have a set of things (e.g. language models) and some measures we care about (e.g. cost, speed, whether weights are open, scores for a few benchmarks, etc.), then some of those things will be "pareto optimal" (see below) and some won't. The "pareto front" is the subset that is pareto optimal.
Some thing is "pareto optimal" when there isn't another thing that's AT LEAST AS GOOD in ALL measures, and BETTER in at least one way. For example, if we say there are no ties (for simplicity), then the cheapest language model is pareto optimal; the fastest model is pareto optimal; those which score highest on each benchmark are pareto optimal; and so on.
Tradeoffs can also be pareto optimal: for example, if the cheapest model is also slow, then there will be more pareto optimal models which are "cheapest for their speed"; and so on for other tradeoffs (e.g. fastest that achieves a certain benchmark score; cheapest model with open weights; etc.).
If you're making a decision about which thing to choose, you only need to care about those in the pareto front (since, by definition, anything that's not pareto optimal is objectively worse on at least one measure).
Pareto optimality does not compare one measure against another: something that's 10000x slower can still be pareto optimal, if it's 1% cheaper than the alternatives. To pick a "best" thing, you could give a weight/importance to each measure, and combine them into an overall score: but that's subjective, and might vary between people and tasks. In contrast, focusing on the pareto front is a way to ignore those things that will never be the best, regardless of weighting.
I honestly think the wikipedia article is too complicated. My own image example here as an another attempt to explain: https://imgur.com/a/5ZQIJDb
Mapping the cost of something (like an algorithm), and the time it takes (so lower is better for both). 1, 3 and 5 are all optimal in their own sense. No one is strictly better than the other, just different tradeoffs you have to choose yourself. However, you would never choose 2, because for a lower cost you could get the same result choosing 3. Same with 4, 6 and 7, they all have something that's both faster and at the same time just as cheap you could choose.
A pareto front is a bit like the classical "fast, cheap, good, choose 2". There are always tradeoffs, but if something is both slow, expensive and not better than something that's faster and cheaper, it's a bad choice, and thus not on the "pareto front".
If you optimize one criterion, it's simple: lowest is best or highest is best. If you optimize multiple criteria, all optimal trade offs between any of the selected criteria are "best" in some way.
When you have a tradeoff between two parameters, which points dominate the others in the sense that you can't choose another point without getting less of one of the parameters.
thanks, may you tell me where we can use them?
The current thing that comes up regularly is choosing an LLM setup.
"what's the family of optimal choices when you have multiple dimensions to rank on?"
Say a race vehicle has acceleration, top speed as defining parameters. Some are slow but accelerate hard, others need a long time to reach very high top speeds. Others are in between, or just flat out bad at both.
The pareto frontier is the set of vehicles that are best: pick one from the frontier and you can be sure that for it's given top speed, none accelerate faster. And vice versa, pick one with a given acceletation and you are sure none have a better top speed
Basically, prioritization.
It’s really that simple.
Eschew obfuscation.
Almost the complete opposite of prioritisation.
The Pareto points are where you sacrifice the least of anything to get the most of everything.
There's the saying about buying computers. Good, Cheap, Fast, pick any two. That's where you would prioritise.
If someone makes something that better, cheaper, and faster, or even pretty close to the best on two of those and clearly better on the other. It's a Pareto point.
Over time computers are getting better, cheaper and faster (software notwithstanding). The leading edge of that advance of all of the things is the Pareto front.
Given this in the TFA
> a Pareto front represents the set of solutions where no solution outperforms any other solution in the set at every objective
I do not believe you are correct when you say
> something that better, cheaper, and faster, or even pretty close to the best on two of those and clearly better on the other. It's a Pareto point.
Since that would outperform on every objective
GP's point that it's prioritisation does not seem incorrect to me. Prioritisation involves considering trade-offs of various approaches and deciding which aspects & attributes to optimise for, at the expense of others.
[dead]
I’m not sure how that’s the opposite of prioritizing, but if I’m wrong, then I’ll happily admit it.
We choose our items/workflows/technologies/whatever, so we get the best/most efficient/most effective/whatever, across the widest possible set.
Sounds like prioritizing, to me, but I’m just a dumb hick, so I suppose I can be wrong.
Prioritising is when you choose something over another. A drag racer prioritises time to travel a quarter mile.
Going for the Pareto is when you elect not to prioritise. It is explicitly deciding to not choose one property over another ant to keep everything as much as you can.
I read it as finding the point of maximum effectiveness. The point at which the most is done for the most.
Getting to that point can be calculated (in some cases), but I suspect most folks get there by trial and error. Finding out what is effective, and what is not, and choosing what is effective, over what is not, until there's no longer a choice. That often becomes tribal knowledge, and is handed down. There's always someone trying to improve it, and when they figure it out, that gets added to the tribal knowledge. Basically, that's how nature does it, so there's some serious prior art. Natural Selection is brutal prioritization.
In Morocco, they used to announce the end of the Ramadan fast, by holding up a black thread and a white thread, and waiting until they could not tell the difference.
Then, they'd fire a cannon, and everybody would dig into some awesome soup. Sort of the same thing.
No, it's more about maximizing the utility function and finding the "shield" where you'd start getting diminishing return beyond that
you have confused me