There might be a problem with the implementation of the solver. This part of FOMCON code belongs to the unmodified fotf toolbox code base. The solver *may* have some numerical issues that need to be addressed. I will review the code shortly. Meanwhile, kindly see the following references that document the implementation of the solver:
[1] Y. Q. Chen, I. Petráš, and D. Xue, “Fractional order control – A tutorial,” in Proc. ACC ’09. American Control Conference, 2009, pp. 1397–1411.
[2] D. Xue, Y. Chen, and D. P. Atherton, Linear Feedback Control: Analysis and Design with MATLAB (Advances in Design and Control), 1st ed.. Philadelphia, PA, USA: Society for Industrial and Applied Mathematics, 2008.
[3] C. A. Monje, Y. Chen, B. Vinagre, D. Xue, and V. Feliu, Fractional-order Systems and Controls: Fundamentals and Applications, ser. Advances in Industrial Control. Springer Verlag, 2010.
If you identify an explicit problem with the implementation, do not hesitate to file a bug.