In 1998 Gerhard Frey firstly proposed using trace zero varieties for cryptographic purpose. These varieties are subgroups of the divisor class group on a low genus hyperelliptic curve defined over a finite field. These groups can be used to establish asymmetric cryptography using the discrete logarithm problem as cryptographic primitive.

Trace zero varieties feature a better scalar multiplication performance than elliptic curves. This allows fast arithmetic in these groups, which can speed up the calculations with a factor 3 compared with elliptic curves and hence speed up the cryptosystem.

Another advantage is that for groups of cryptographically relevant size, the order of the group can simply be calculated using the characteristic polynomial of the Frobenius endomorphism. This is not the case, for example, in elliptic curve cryptography when the group of points of an elliptic curve over a prime field is used for cryptographic purpose.

However to represent an element of the trace zero variety more bits are needed compared with elements of elliptic or hyperelliptic curves. Another disadvantage, is the fact, that it is possible to reduce the security of the TZV of ¹/₆^th of the bit length using cover attack.

A hyperelliptic curve C of genus g over a prime field ${\displaystyle \mathbb {F} _{q))$ where q = pⁿ (p prime) of odd characteristic is defined as

${\displaystyle C:~y^{2}+h(x)y=f(x),}$

where f monic, deg(f) = 2g + 1 and deg(h) ≤ g. The curve has at least one ${\displaystyle \mathbb {F} _{q))$-rational Weierstraßpoint.

The Jacobian variety ${\displaystyle J_{C}(\mathbb {F} _{q^{n)))}$ of C is for all finite extension ${\displaystyle \mathbb {F} _{q^{n))}$ isomorphic to the ideal class group ${\displaystyle \operatorname {Cl} (C/\mathbb {F} _{q^{n)))}$. With the Mumford's representation it is possible to represent the elements of ${\displaystyle J_{C}(\mathbb {F} _{q^{n)))}$ with a pair of polynomials [u, v], where u, v ∈ ${\displaystyle \mathbb {F} _{q^{n))[x]}$.

The Frobenius endomorphism σ is used on an element [u, v] of ${\displaystyle J_{C}(\mathbb {F} _{q^{n)))}$ to raise the power of each coefficient of that element to q: σ([u, v]) = [u^q(x), v^q(x)]. The characteristic polynomial of this endomorphism has the following form:

${\displaystyle \chi (T)=T^{2g}+a_{1}T^{2g-1}+\cdots +a_{g}T^{g}+\cdots +a_{1}q^{g-1}T+q^{g},}$

where a_i in ${\displaystyle \mathbb {Z} }$

With the Hasse–Weil theorem it is possible to receive the group order of any extension field ${\displaystyle \mathbb {F} _{q^{n))}$ by using the complex roots τ_i of χ(T):

${\displaystyle |J_{C}(\mathbb {F} _{q^{n)))|=\prod _{i=1}^{2g}(1-\tau _{i}^{n})}$

Let D be an element of the ${\displaystyle J_{C}(\mathbb {F} _{q^{n)))}$ of C, then it is possible to define an endomorphism of ${\displaystyle J_{C}(\mathbb {F} _{q^{n)))}$, the so-called trace of D:

${\displaystyle \operatorname {Tr} (D)=\sum _{i=0}^{n-1}\sigma ^{i}(D)=D+\sigma (D)+\cdots +\sigma ^{n-1}(D)}$

Based on this endomorphism one can reduce the Jacobian variety to a subgroup G with the property, that every element is of trace zero:

${\displaystyle G=\{D\in J_{C}(\mathbb {F} _{q^{n)))~|~{\text{Tr))(D)={\textbf {0))\},~~~({\textbf {0)){\text{ neutral element in ))J_{C}(\mathbb {F} _{q^{n)))}$

G is the kernel of the trace endomorphism and thus G is a group, the so-called trace zero (sub)variety (TZV) of ${\displaystyle J_{C}(\mathbb {F} _{q^{n)))}$.

The intersection of G and ${\displaystyle J_{C}(\mathbb {F} _{q})}$ is produced by the n-torsion elements of ${\displaystyle J_{C}(\mathbb {F} _{q})}$. If the greatest common divisor ${\displaystyle \gcd(n,|J_{C}(\mathbb {F} _{q})|)=1}$ the intersection is empty and one can compute the group order of G:

${\displaystyle |G|={\dfrac {|J_{C}(\mathbb {F} _{q^{n)))|}{|J_{C}(\mathbb {F} _{q})|))={\dfrac {\prod _{i=1}^{2g}(1-\tau _{i}^{n})}{\prod _{i=1}^{2g}(1-\tau _{i})))}$

The actual group used in cryptographic applications is a subgroup G₀ of G of a large prime order l. This group may be G itself.^[1][2]

There exist three different cases of cryptographical relevance for TZV:^[3]

• g = 1, n = 3
• g = 1, n = 5
• g = 2, n = 3

The arithmetic used in the TZV group G₀ based on the arithmetic for the whole group ${\displaystyle J_{C}(\mathbb {F} _{q^{n)))}$, But it is possible to use the Frobenius endomorphism σ to speed up the scalar multiplication. This can be archived if G₀ is generated by D of order l then σ(D) = sD, for some integers s. For the given cases of TZV s can be computed as follows, where a_i come from the characteristic polynomial of the Frobenius endomorphism :

• For g = 1, n = 3: ${\displaystyle s={\dfrac {q-1}{1-a_{1))}{\bmod {\ell ))}$
• For g = 1, n = 5: ${\displaystyle s={\dfrac {q^{2}-q-a_{1}^{2}q+a_{1}q+1}{q-2a_{1}q+a_{1}^{3}-a_{1}^{2}+a_{1}-1)){\bmod {\ell ))}$
• For g = 2, n = 3: ${\displaystyle s=-{\dfrac {q^{2}-a_{2}+a_{1)){a_{1}q-a_{2}+1)){\bmod {\ell ))}$

Knowing this, it is possible to replace any scalar multiplication mD (|m| ≤ l/2) with:

${\displaystyle m_{0}D+m_{1}\sigma (D)+\cdots +m_{n-1}\sigma ^{n-1}(D),~~~~{\text{where ))m_{i}=O(\ell ^{1/(n-1)})=O(q^{g})}$

With this trick the multiple scalar product can be reduced to about 1/(n − 1)^th of doublings necessary for calculating mD, if the implied constants are small enough.^[3][2]

The security of cryptographic systems based on trace zero subvarieties according to the results of the papers^[2][3] comparable to the security of hyper-elliptic curves of low genus g' over ${\displaystyle \mathbb {F} _{p'))$, where p' ~ (n − 1)(g/g' ) for |G| ~128 bits.

For the cases where n = 3, g = 2 and n = 5, g = 1 it is possible to reduce the security for at most 6 bits, where |G| ~ 2²⁵⁶, because one can not be sure that G is contained in a Jacobian of a curve of genus 6. The security of curves of genus 4 for similar fields are far less secure.

The attack published in^[4] shows, that the DLP in trace zero groups of genus 2 over finite fields of characteristic diverse than 2 or 3 and a field extension of degree 3 can be transformed into a DLP in a class group of degree 0 with genus of at most 6 over the base field. In this new class group the DLP can be attacked with the index calculus methods. This leads to a reduction of the bit length ¹/₆^th.

• Wienecke, Malte; Paar, Christof (17 February 2008). Cryptography on Trace-Zero Varieties (PDF). Ruhr University Bochum.
• Sutherland, Andrew V. (2007). "101 Useful Trace Zero Varieties". Massachusetts Institute of Technology.