In this thesis we have set some terms of classical potential theory into context of Markov chains. As the basic example of Markov chain in this thesis, we use the simple symmetric random walk on the \(l\)-dimensional lattice \(\mathbb{Z}^l\), for \(l \geq 1\). In the study of properties and definitions of certain terms, it is important whether we talk about recurrent or transient Markov chains. We have shown that symmetric random walk is recurrent if \(l\) is 1 or 2, while it is transient for \(l \geq 3\). Harmonic functions play an important role in the theory of Markov chains and are necessary for defining other classical potential theory analogs. They are strongly connected with discrete martingale theory. Considering Markov chain \(X, f\) is harmonic if and only if \(f(X)\) is martingale. Superharmonic functions make broader class of functions than harmonic. Nonnegative superharmonic functions are called excessive functions. Riesz decomposition gives us existence and uniqueness of an excessive function decomposition into a sum of a potential and a nonnegative harmonic function. It turns out that probability of visiting some set \(B\) by a Markov chain, depending on initial state, is an excessive function. In particular, in case of an irreducible and transient Markov chain, harmonic function from Riesz decomposition of mentioned probability equals zero, so we actually talk about potential. Capacity is defined for transient Markov chains, since the potential is well-defined in this case. Regardless, all results for capacity in the case of transient Markov chains can be applied for transient as well as recurrent random walks. We only have to consider transient sets. In the end of this thesis, we give an important application of capacity in the theory of Markov chains. Recurrence criterion of a set for the simple symmetric random walk on \(\mathbb{Z}^3\) is given in terms of capacity and represents a discrete counterpart of the Wiener criterion.