I can't tell how it should or shouldn't be done, but can share the experience from my organization.
We've adopted Holacracy 2 years ago and have 20 partners (to give context). For long, we interpreted it in the same way you do (duty applies to all, not just to fellow circle members).
Only later we found out that this restriction actually exists in the constitution. Since then we've tried to pay more attention to this and indeed it gave us some more clarity.
1) Partners were more likely to reject a project that they didn't feel is really helping them to achieve the purpose which led to evolution
2) It became much more clear which roles *really* belong to which circle and resolving the tensions thereof benefited the circles as well.