As a generalization of skew braces, the notion of skew trusses was introduced by T. Brzezinski. It was shown that every Rota-Baxter group has the structure of skew braces by V. G. Bardakov and V. Gubarev. To investigate an analogue of Rota-Baxter groups which has the structure of skew trusses, we define RotaBaxter systems. We study the relationship between Rota-Baxter systems and Rota-Baxter groups. Furthermore, we prove that a Rota-Baxter system can be decomposed as a direct sum of two semigroups. A factorization theorem is proved, generalizing the factorization theorem of Rota-Baxter groups. The notion of Rota-Baxter systems of Lie algebras was introduced, as a generalization of Rota-Baxter Lie algebras. The connection between Rota-Baxter systems of Lie algebras and Lie groups is studied. Finally, as a generalization of the modified Yang-Baxter equation, we define twisted modified Yang-Baxter equations. We give solutions of twisted modified Yang-Baxter equations by Rota-Baxter systems of Lie algebras.
