Hybrid metal-organic compounds as relatively new and prosperous magnetoelectric multiferroics provide opportunities to improve the polarization, magnetization and magneto-electric coupling at the same time, which usually have some limitations in the common type-I and type-II multiferroics. In this work we investigate the crystal of guanidinium copper (II) formate [C(NH2)3]Cu(HCOO)3 and give novel insights concerning the structure, magnetic, electric and magneto-electric behaviour of this interesting material. Detailed analysis of crystal structure at 100 K is given. Magnetization points to the copper (II) formate spin-chain phase that becomes ordered below 4.6 K into the canted antiferromagnetic (AFM) state, as a result of super-exchange interaction over different formate bridges. The performed ab-initio colinear density functional theory (DFT) calculations confirm the AFM-like ground state as a first approximation and explain the coupling of spin-chains into the AFM ordered lattice. In versatile measurements of magnetization of a crystal, including transverse component besides the longitudinal one, very large anisotropy is found that might originate from canting of the coordination octahedra around copper (II) in cooperation with the canted AFM order. With cooling down in zero fields the generation of spontaneous polarization is observed step-wise below 270 K and 210 K and the effect of magnetic field on its value is observed also in the paramagnetic phase. Measured polarization is somewhat smaller than the DFT value in the c-direction, possibly due to twin domains present in the crystal. The considerable magneto-electric coupling below the magnetic transition temperature is measured with different orientations of the crystal in magnetic field, giving altogether the new light onto the magneto-electric effect in this material.