Jump to content

Kronecker product

From Wikipedia, the free encyclopedia
(Redirected from Kronecker sum)

In mathematics, the Kronecker product, sometimes denoted by ⊗, is an operation on two matrices of arbitrary size resulting in a block matrix. It is a specialization of the tensor product (which is denoted by the same symbol) from vectors to matrices and gives the matrix of the tensor product linear map with respect to a standard choice of basis. The Kronecker product is to be distinguished from the usual matrix multiplication, which is an entirely different operation. The Kronecker product is also sometimes called matrix direct product.[1]

The Kronecker product is named after the German mathematician Leopold Kronecker (1823–1891), even though there is little evidence that he was the first to define and use it. The Kronecker product has also been called the Zehfuss matrix, and the Zehfuss product, after Johann Georg Zehfuss [de], who in 1858 described this matrix operation, but Kronecker product is currently the most widely used term.[2][3] The misattribution to Kronecker rather than Zehfuss was due to Kurt Hensel.[4]

Definition

[edit]

If A is an m × n matrix and B is a p × q matrix, then the Kronecker product AB is the pm × qn block matrix:

more explicitly:

Using and to denote truncating integer division and remainder, respectively, and numbering the matrix elements starting from 0, one obtains

For the usual numbering starting from 1, one obtains

If A and B represent linear transformations V1W1 and V2W2, respectively, then the tensor product of the two maps is represented by AB, which is the same as V1V2W1W2.

Examples

[edit]

Similarly:

Properties

[edit]

Relations to other matrix operations

[edit]
  1. Bilinearity and associativity:

    The Kronecker product is a special case of the tensor product, so it is bilinear and associative:

    where A, B and C are matrices, 0 is a zero matrix, and k is a scalar.
  2. Non-commutative:

    In general, AB and BA are different matrices. However, AB and BA are permutation equivalent, meaning that there exist permutation matrices P and Q such that[5]

    If A and B are square matrices, then AB and BA are even permutation similar, meaning that we can take P = QT.

    The matrices P and Q are perfect shuffle matrices.[6] The perfect shuffle matrix Sp,q can be constructed by taking slices of the Ir identity matrix, where .

    MATLAB colon notation is used here to indicate submatrices, and Ir is the r × r identity matrix. If and , then

  3. The mixed-product property:

    If A, B, C and D are matrices of such size that one can form the matrix products AC and BD, then[7]

    This is called the mixed-product property, because it mixes the ordinary matrix product and the Kronecker product.

    As an immediate consequence (again, taking and ),

    In particular, using the transpose property from below, this means that if

    and Q and U are orthogonal (or unitary), then A is also orthogonal (resp., unitary).

    The mixed Kronecker matrix-vector product can be written as:

    where is the vectorization operator applied on (formed by reshaping the matrix).
  4. Hadamard product (element-wise multiplication):

    The mixed-product property also works for the element-wise product. If A and C are matrices of the same size, B and D are matrices of the same size, then[7]

  5. The inverse of a Kronecker product:

    It follows that AB is invertible if and only if both A and B are invertible, in which case the inverse is given by

    The invertible product property holds for the Moore–Penrose pseudoinverse as well,[7][8] that is

    In the language of Category theory, the mixed-product property of the Kronecker product (and more general tensor product) shows that the category MatF of matrices over a field F, is in fact a monoidal category, with objects natural numbers n, morphisms nm are n×m matrices with entries in F, composition is given by matrix multiplication, identity arrows are simply n × n identity matrices In, and the tensor product is given by the Kronecker product.[9]

    MatF is a concrete skeleton category for the equivalent category FinVectF of finite dimensional vector spaces over F, whose objects are such finite dimensional vector spaces V, arrows are F-linear maps L : VW, and identity arrows are the identity maps of the spaces. The equivalence of categories amounts to simultaneously choosing a basis in every finite-dimensional vector space V over F; matrices' elements represent these mappings with respect to the chosen bases; and likewise the Kronecker product is the representation of the tensor product in the chosen bases.
  6. Transpose:

    Transposition and conjugate transposition are distributive over the Kronecker product:

    and
  7. Determinant:

    Let A be an n × n matrix and let B be an m × m matrix. Then

    The exponent in |A| is the order of B and the exponent in |B| is the order of A.
  8. Kronecker sum and exponentiation:

    If A is n × n, B is m × m and Ik denotes the k × k identity matrix then we can define what is sometimes called the Kronecker sum, ⊕, by

    This is different from the direct sum of two matrices. This operation is related to the tensor product on Lie algebras, as detailed below (#Abstract properties) in the point "Relation to the abstract tensor product".

    We have the following formula for the matrix exponential, which is useful in some numerical evaluations.[10]

    Kronecker sums appear naturally in physics when considering ensembles of non-interacting systems.[citation needed] Let Hk be the Hamiltonian of the kth such system. Then the total Hamiltonian of the ensemble is

  9. Vectorization of a Kronecker product:

    Let be an matrix and a matrix. When the order of the Kronecker product and vectorization is interchanged, the two operations can be linked linearly through a function that involves the commutation matrix. That is, and have the following relationship:

    Furthermore, the above relation can be rearranged in terms of either or as follows:

    where

  10. Outer Product:
    If and are arbitrary vectors, then the outer product between and is defined as . The Kronecker product is related to the outer product by: .

Abstract properties

[edit]
  1. Spectrum:

    Suppose that A and B are square matrices of size n and m respectively. Let λ1, ..., λn be the eigenvalues of A and μ1, ..., μm be those of B (listed according to multiplicity). Then the eigenvalues of AB are

    It follows that the trace and determinant of a Kronecker product are given by

  2. Singular values:

    If A and B are rectangular matrices, then one can consider their singular values. Suppose that A has rA nonzero singular values, namely

    Similarly, denote the nonzero singular values of B by

    Then the Kronecker product AB has rArB nonzero singular values, namely

    Since the rank of a matrix equals the number of nonzero singular values, we find that

  3. Relation to the abstract tensor product:

    The Kronecker product of matrices corresponds to the abstract tensor product of linear maps. Specifically, if the vector spaces V, W, X, and Y have bases {v1, ..., vm}, {w1, ..., wn}, {x1, ..., xd}, and {y1, ..., ye}, respectively, and if the matrices A and B represent the linear transformations S : VX and T : WY, respectively in the appropriate bases, then the matrix AB represents the tensor product of the two maps, ST : VWXY with respect to the basis {v1w1, v1w2, ..., v2w1, ..., vmwn} of VW and the similarly defined basis of XY with the property that AB(viwj) = (Avi) ⊗ (Bwj), where i and j are integers in the proper range.[11]

    When V and W are Lie algebras, and S : VV and T : WW are Lie algebra homomorphisms, the Kronecker sum of A and B represents the induced Lie algebra homomorphisms VWVW.[citation needed]
  4. Relation to products of graphs:
    The Kronecker product of the adjacency matrices of two graphs is the adjacency matrix of the tensor product graph. The Kronecker sum of the adjacency matrices of two graphs is the adjacency matrix of the Cartesian product graph.[12]

Matrix equations

[edit]

The Kronecker product can be used to get a convenient representation for some matrix equations. Consider for instance the equation AXB = C, where A, B and C are given matrices and the matrix X is the unknown. We can use the "vec trick" to rewrite this equation as

Here, vec(X) denotes the vectorization of the matrix X, formed by stacking the columns of X into a single column vector.

It now follows from the properties of the Kronecker product that the equation AXB = C has a unique solution, if and only if A and B are invertible (Horn & Johnson 1991, Lemma 4.3.1).

If X and C are row-ordered into the column vectors u and v, respectively, then (Jain 1989, 2.8 Block Matrices and Kronecker Products)

The reason is that

Applications

[edit]

For an example of the application of this formula, see the article on the Lyapunov equation. This formula also comes in handy in showing that the matrix normal distribution is a special case of the multivariate normal distribution. This formula is also useful for representing 2D image processing operations in matrix-vector form.

Another example is when a matrix can be factored as a Kronecker product, then matrix multiplication can be performed faster by using the above formula. This can be applied recursively, as done in the radix-2 FFT and the Fast Walsh–Hadamard transform. Splitting a known matrix into the Kronecker product of two smaller matrices is known as the "nearest Kronecker product" problem, and can be solved exactly[13] by using the SVD. To split a matrix into the Kronecker product of more than two matrices, in an optimal fashion, is a difficult problem and the subject of ongoing research; some authors cast it as a tensor decomposition problem.[14][15]

In conjunction with the least squares method, the Kronecker product can be used as an accurate solution to the hand–eye calibration problem.[16]

[edit]

Two related matrix operations are the Tracy–Singh and Khatri–Rao products, which operate on partitioned matrices. Let the m × n matrix A be partitioned into the mi × nj blocks Aij and p × q matrix B into the pk × q blocks Bkl, with of course Σi mi = m, Σj nj = n, Σk pk = p and Σ q = q.

Tracy–Singh product

[edit]

The Tracy–Singh product is defined as[17][18][19]

which means that the (ij)-th subblock of the mp × nq product A B is the mi p × nj q matrix Aij B, of which the (kℓ)-th subblock equals the mi pk × nj q matrix AijBkℓ. Essentially the Tracy–Singh product is the pairwise Kronecker product for each pair of partitions in the two matrices.

For example, if A and B both are 2 × 2 partitioned matrices e.g.:

we get:

Khatri–Rao product

[edit]
  • Block Kronecker product
  • Column-wise Khatri–Rao product

Face-splitting product

[edit]

Mixed-products properties[20]

where denotes the Face-splitting product.[21][22]

Similarly:[23]

where and are vectors,[24]

where and are vectors, and denotes the Hadamard product.

Similarly:

where is vector convolution and is the Fourier transform matrix (this result is an evolving of count sketch properties[25]),[21][22]

where denotes the column-wise Khatri–Rao product.

Similarly:

where and are vectors.

See also

[edit]

Notes

[edit]
  1. ^ Weisstein, Eric W. "Kronecker product". mathworld.wolfram.com. Retrieved 2020-09-06.
  2. ^ Zehfuss, G. (1858). "Ueber eine gewisse Determinante". Zeitschrift für Mathematik und Physik. 3: 298–301.
  3. ^ Henderson, Harold V.; Pukelsheim, Friedrich; Searle, Shayle R. (1983). "On the history of the kronecker product". Linear and Multilinear Algebra. 14 (2): 113–120. doi:10.1080/03081088308817548. hdl:1813/32834. ISSN 0308-1087.
  4. ^ Sayed, Ali H. (2022-12-22). Inference and Learning from Data: Foundations. Cambridge University Press. ISBN 978-1-009-21812-2.
  5. ^ Henderson, H.V.; Searle, S.R. (1980). "The vec-permutation matrix, the vec operator and Kronecker products: A review" (PDF). Linear and Multilinear Algebra. 9 (4): 271–288. doi:10.1080/03081088108817379. hdl:1813/32747.
  6. ^ Van Loan, Charles F. (2000). "The ubiquitous Kronecker product". Journal of Computational and Applied Mathematics. 123 (1–2): 85–100. Bibcode:2000JCoAM.123...85L. doi:10.1016/s0377-0427(00)00393-9.
  7. ^ a b c Liu, Shuangzhe; Trenkler, Götz; Kollo, Tõnu; von Rosen, Dietrich; Baksalary, Oskar Maria (2023). "Professor Heinz Neudecker and matrix differential calculus". Statistical Papers. doi:10.1007/s00362-023-01499-w.
  8. ^ Langville, Amy N.; Stewart, William J. (1 June 2004). "The Kronecker product and stochastic automata networks". Journal of Computational and Applied Mathematics. 167 (2): 429–447. Bibcode:2004JCoAM.167..429L. doi:10.1016/j.cam.2003.10.010.
  9. ^ Macedo, Hugo Daniel; Oliveira, José Nuno (2013). "Typing linear algebra: A biproduct-oriented approach". Science of Computer Programming. 78 (11): 2160–2191. arXiv:1312.4818. Bibcode:2013arXiv1312.4818M. CiteSeerX 10.1.1.747.2083. doi:10.1016/j.scico.2012.07.012. S2CID 9846072.
  10. ^ Brewer, J.W. (1969). "A note on Kronecker matrix products and matrix equation systems". SIAM Journal on Applied Mathematics. 17 (3): 603–606. doi:10.1137/0117057.
  11. ^ Dummit, David S.; Foote, Richard M. (1999). Abstract Algebra (2 ed.). New York: John Wiley and Sons. pp. 401–402. ISBN 978-0-471-36857-1.
  12. ^ See Knuth, D.E. "Pre-Fascicle 0a: Introduction to Combinatorial Algorithms" (zeroth printing, revision 2 ed.). answer to Exercise 96. Archived from the original on 2019-05-13. Retrieved 2007-10-24, to appear as part of Knuth, D.E. The Art of Computer Programming. Vol. 4A.
  13. ^ Van Loan, C.; Pitsianis, N. (1992). Approximation with Kronecker Products. Ithaca, NY: Cornell University Press.
  14. ^ King Keung Wu; Yam, Yeung; Meng, Helen; Mesbahi, Mehran (2016). "Kronecker product approximation with multiple factor matrices via the tensor product algorithm". 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC). pp. 004277–004282. doi:10.1109/SMC.2016.7844903. ISBN 978-1-5090-1897-0. S2CID 30695585.
  15. ^ Dantas, Cássio F.; Cohen, Jérémy E.; Gribonval, Rémi (2018). "Learning Fast Dictionaries for Sparse Representations Using Low-Rank Tensor Decompositions". Latent Variable Analysis and Signal Separation (PDF). Lecture Notes in Computer Science. Vol. 10891. pp. 456–466. doi:10.1007/978-3-319-93764-9_42. ISBN 978-3-319-93763-2. S2CID 46963798.
  16. ^ Li, Algo; et al. (4 September 2010). "Simultaneous robot-world and hand-eye calibration using dual-quaternions and Kronecker product" (PDF). International Journal of the Physical Sciences. 5 (10): 1530–1536. S2CID 7446157. Archived from the original (PDF) on 9 February 2020.
  17. ^ Tracy, D.S.; Singh, R.P. (1972). "A new matrix product and its applications in matrix differentiation". Statistica Neerlandica. 26 (4): 143–157. doi:10.1111/j.1467-9574.1972.tb00199.x.
  18. ^ Liu, Shuangzhe (1999). "Matrix Results on the Khatri–Rao and Tracy–Singh Products". Linear Algebra and Its Applications. 289 (1–3): 267–277. doi:10.1016/S0024-3795(98)10209-4.
  19. ^ Liu, Shuangzhe; Trenkler, Götz (2008). "Hadamard, Khatri-Rao, Kronecker and other matrix products". International Journal of Information and Systems Sciences. 4 (1): 160–177.
  20. ^ Slyusar, V.I. (1998) [27 December 1996]. "End products in matrices in radar applications" (PDF). Radioelectronics and Communications Systems. 41 (3): 50–53.
  21. ^ a b Slyusar, Vadym (1999). "New matrix operations for DSP" (self-published lecture). doi:10.13140/RG.2.2.31620.76164/1 – via ResearchGate. {{cite journal}}: Cite journal requires |journal= (help)
  22. ^ a b Slyusar, V.I. (March 13, 1998). "A Family of Face Products of Matrices and its Properties" (PDF). Cybernetics and Systems Analysis C/C of Kibernetika I Sistemnyi Analiz. 1999. 35 (3): 379–384. doi:10.1007/BF02733426. S2CID 119661450.
  23. ^ Slyusar, V.I. (1997-09-15). New operations of matrices product for applications of radars (PDF). Direct and Inverse Problems of Electromagnetic and Acoustic Wave Theory (DIPED-97), Lviv. pp. 73–74.
  24. ^ Ahle, Thomas Dybdahl; Knudsen, Jakob Bæk Tejs (2019-09-03). "Almost optimal tensor sketch". arXiv:1909.01821 [cs.DS].
  25. ^ Ninh, Pham; Pagh, Rasmus (2013). Fast and scalable polynomial kernels via explicit feature maps. SIGKDD international conference on Knowledge discovery and data mining. Association for Computing Machinery. CiteSeerX 10.1.1.718.2766. doi:10.1145/2487575.2487591.

References

[edit]
[edit]