Shanks' square forms factorization is a method for integer factorization devised by Daniel Shanks as an improvement on Fermat's factorization method.

The success of Fermat's method depends on finding integers ${\displaystyle x}$ and ${\displaystyle y}$ such that ${\displaystyle x^{2}-y^{2}=N}$, where ${\displaystyle N}$ is the integer to be factored. An improvement (noticed by Kraitchik) is to look for integers ${\displaystyle x}$ and ${\displaystyle y}$ such that ${\displaystyle x^{2}\equiv y^{2}{\pmod {N))}$. Finding a suitable pair ${\displaystyle (x,y)}$ does not guarantee a factorization of ${\displaystyle N}$, but it implies that ${\displaystyle N}$ is a factor of ${\displaystyle x^{2}-y^{2}=(x-y)(x+y)}$, and there is a good chance that the prime divisors of ${\displaystyle N}$ are distributed between these two factors, so that calculation of the greatest common divisor of ${\displaystyle N}$ and ${\displaystyle x-y}$ will give a non-trivial factor of ${\displaystyle N}$.

A practical algorithm for finding pairs ${\displaystyle (x,y)}$ which satisfy ${\displaystyle x^{2}\equiv y^{2}{\pmod {N))}$ was developed by Shanks, who named it Square Forms Factorization or SQUFOF. The algorithm can be expressed in terms of continued fractions or in terms of quadratic forms. Although there are now much more efficient factorization methods available, SQUFOF has the advantage that it is small enough to be implemented on a programmable calculator. Shanks programmed it on an HP-65, made in 1974, which has storage for only nine digit numbers and allows only 100 steps/keystrokes of programming. There are versions of the algorithm that use little memory and versions that store a list of values that run more quickly.

In 1858, the Czech mathematician Václav Šimerka used a method similar to SQUFOF to factor ${\displaystyle (10^{17}-1)/9}$ ${\displaystyle =}$ ${\displaystyle 11111111111111111}$ ${\displaystyle =}$ ${\displaystyle 2071723\cdot 5363222357}$.^[1]

Note This version of the algorithm works on some examples but often gets stuck in a loop.

This version does not use a list.

Input: ${\displaystyle N}$, the integer to be factored, which must be neither a prime number nor a perfect square, and a small positive integer, ${\displaystyle k}$.

Output: a non-trivial factor of ${\displaystyle N}$.

The algorithm:

Initialize ${\displaystyle i=0,P_{0}=\lfloor {\sqrt {kN))\rfloor ,Q_{-1}=1,Q_{0}=kN-P_{0}^{2}.}$

Repeat

${\displaystyle i=i+1,b_{i}=\left\lfloor {\frac {P_{0}+P_{i-1)){Q_{i-1))}\right\rfloor ,P_{i}=b_{i}Q_{i-1}-P_{i-1},Q_{i}=Q_{i-2}+b_{i}(P_{i-1}-P_{i})}$

until ${\displaystyle Q_{i))$ is a perfect square at some odd value of ${\displaystyle i}$.

Start the second phase (reverse cycle).

Initialize ${\displaystyle b_{0}=\left\lfloor {\frac {P_{0}-P_{i)){\sqrt {Q_{i))))\right\rfloor }$, ${\displaystyle Q_{-1}={\sqrt {Q_{i))))$, and ${\displaystyle P_{0}=b_{0}{\sqrt {Q_{i))}+P_{i))$, where ${\displaystyle P_{0},P_{i))$, and ${\displaystyle Q_{i))$ are from the previous phase. The ${\displaystyle b_{0))$ used in the calculation of ${\displaystyle P_{0))$ is the recently calculated value of ${\displaystyle b_{0))$.

Set ${\displaystyle i=0}$ and ${\displaystyle Q_{0}={\frac {kN-P_{0}^{2)){Q_{-1))))$, where ${\displaystyle P_{0))$ is the recently calculated value of ${\displaystyle P_{0))$.

Repeat

${\displaystyle i=i+1,b_{i}=\left\lfloor {\frac {P_{0}+P_{i-1)){Q_{i-1))}\right\rfloor ,P_{i}=b_{i}Q_{i-1}-P_{i-1},Q_{i}=Q_{i-2}+b_{i}(P_{i-1}-P_{i})}$

until ${\displaystyle P_{i}=P_{i-1}.}$^{[citation needed]}

Then if ${\displaystyle f=\gcd(N,P_{i})}$ is not equal to ${\displaystyle 1}$ and not equal to ${\displaystyle N}$, then ${\displaystyle f}$ is a non-trivial factor of ${\displaystyle N}$. Otherwise try another value of ${\displaystyle k}$.^{[citation needed]}

Shanks' method has time complexity ${\displaystyle O({\sqrt[{4}]{N)))}$.^[2]

Stephen S. McMath wrote a more detailed discussion of the mathematics of Shanks' method, together with a proof of its correctness.^[3]

Let ${\displaystyle N=11111}$

${\displaystyle Q_{-1}=1}$

Cycle forward
${\displaystyle i}$	${\displaystyle b_{i))$	${\displaystyle P_{i))$	${\displaystyle Q_{i))$
${\displaystyle 0}$	${\displaystyle }$	${\displaystyle 105}$	${\displaystyle 86}$
${\displaystyle 1}$	${\displaystyle 2}$	${\displaystyle 67}$	${\displaystyle 77}$
${\displaystyle 2}$	${\displaystyle 2}$	${\displaystyle 87}$	${\displaystyle 46}$
${\displaystyle 3}$	${\displaystyle 4}$	${\displaystyle 97}$	${\displaystyle 37}$
${\displaystyle 4}$	${\displaystyle 5}$	${\displaystyle 88}$	${\displaystyle 91}$
${\displaystyle 5}$	${\displaystyle 2}$	${\displaystyle 94}$	${\displaystyle 25}$

Here ${\displaystyle Q_{5}=25}$ is a perfect square, so the first phase ends.

For the second phase, set ${\displaystyle Q_{-1}={\sqrt {25))=5}$. Then:

Reverse cycle
${\displaystyle i}$	${\displaystyle b_{i))$	${\displaystyle P_{i))$	${\displaystyle Q_{i))$
${\displaystyle 0}$	${\displaystyle 2}$	${\displaystyle 104}$	${\displaystyle 59}$
${\displaystyle 1}$	${\displaystyle 3}$	${\displaystyle 73}$	${\displaystyle 98}$
${\displaystyle 2}$	${\displaystyle 1}$	${\displaystyle 25}$	${\displaystyle 107}$
${\displaystyle 3}$	${\displaystyle 1}$	${\displaystyle 82}$	${\displaystyle 41}$
${\displaystyle 4}$	${\displaystyle 4}$	${\displaystyle 82}$	${\displaystyle }$

Here ${\displaystyle P_{3}=P_{4}=82}$, so the second phase ends. Now calculate ${\displaystyle gcd(11111,82)=41}$, which is a factor of ${\displaystyle 11111}$.

Thus, ${\displaystyle N=11111=41\cdot 271}$.

Below is an example of C function for performing SQUFOF factorization on unsigned integer not larger than 64 bits, without overflow of the transient operations. ^{[citation needed]}

#include <inttypes.h>
#define nelems(x) (sizeof(x) / sizeof((x)[0]))

const int multiplier[] = {1, 3, 5, 7, 11, 3*5, 3*7, 3*11, 5*7, 5*11, 7*11, 3*5*7, 3*5*11, 3*7*11, 5*7*11, 3*5*7*11};

uint64_t SQUFOF( uint64_t N )
{
    uint64_t D, Po, P, Pprev, Q, Qprev, q, b, r, s;
    uint32_t L, B, i;
    s = (uint64_t)(sqrtl(N)+0.5);
    if (s*s == N) return s;
    for (int k = 0; k < nelems(multiplier) && N <= UINT64_MAX/multiplier[k]; k++) {
        D = multiplier[k]*N;
        Po = Pprev = P = sqrtl(D);
        Qprev = 1;
        Q = D - Po*Po;
        L = 2 * sqrtl( 2*s );
        B = 3 * L;
        for (i = 2 ; i < B ; i++) {
            b = (uint64_t)((Po + P)/Q);
            P = b*Q - P;
            q = Q;
            Q = Qprev + b*(Pprev - P);
            r = (uint64_t)(sqrtl(Q)+0.5);
            if (!(i & 1) && r*r == Q) break;
            Qprev = q;
            Pprev = P;
        };
        if (i >= B) continue;
        b = (uint64_t)((Po - P)/r);
        Pprev = P = b*r + P;
        Qprev = r;
        Q = (D - Pprev*Pprev)/Qprev;
        i = 0;
        do {
            b = (uint64_t)((Po + P)/Q);
            Pprev = P;
            P = b*Q - P;
            q = Q;
            Q = Qprev + b*(Pprev - P);
            Qprev = q;
            i++;
        } while (P != Pprev);
        r = gcd(N, Qprev);
        if (r != 1 && r != N) return r;
    }
    return 0;
}

• D. A. Buell (1989). Binary Quadratic Forms. Springer-Verlag. ISBN 0-387-97037-1.
• D. M. Bressoud (1989). Factorisation and Primality Testing. Springer-Verlag. ISBN 0-387-97040-1.
• Riesel, Hans (1994). Prime Numbers and Computer Methods for Factorization (2nd ed.). Birkhauser. ISBN 0-8176-3743-5.
• Samuel S. Wagstaff, Jr. (2013). The Joy of Factoring. Providence, RI: American Mathematical Society. pp. 163–168. ISBN 978-1-4704-1048-3.

• Daniel Shanks: Analysis and Improvement of the Continued Fraction Method of Factorization, (transcribed by S. McMath 2004)
• Daniel Shanks: SQUFOF Notes, (transcribed by S. McMath 2004)
• Stephen S. McMath: Parallel integer factorization using quadratic forms, 2005
• S. McMath, F. Crabbe, D. Joyner: Continued fractions and parallel SQUFOF, 2005
• Jason Gower, Samuel Wagstaff: Square Form Factorisation (Published)
• Shanks's SQUFOF Factoring Algorithm
• java-math-library