Clear Filters
Clear Filters

Inverse huge matrix with tall array

7 views (last 30 days)
chen zhe
chen zhe on 20 Jun 2024
Commented: halleyhit on 23 Jun 2024
Hi,
I am trying to find x in my system Ax=b. A is a huge matrix, it is a size of 10 billion *9, and out of my memory, b is 10 billion *1 vector.
Therefore, I am using tall array, while I don know how to compute the inverse of A? (pseudo inverse)
A = tall(ds);
b_t = tall(ones(1,1));
SOL = gather(A\b);
This code works with SOL is 9*1 vector, however, the vector b_t has all elements equal to 1.
If I use my verctor b, it shows an error with incompatible tall array argument in tall/qrLeftSolve.
Any one can help this case?
Thanks in advance.

Answers (2)

halleyhit
halleyhit on 21 Jun 2024
Hi chen, I am writing some lines code and find they are working fine. You can directly solve equeation AX=B or get x_inv by tall array. Would you please share more information, like MATLAB version and full error message?
A=tall(rand(100,100));
B1=tall(rand(100,1));
X=A\B1; % solve X in equation A*X=B1
B2=tall(eye(100));
X_inv=A\B2; % solve inv(X)
X=gather(X)
Evaluating tall expression using the Local MATLAB Session: - Pass 1 of 1: 0% complete - Pass 1 of 1: Completed in 0.45 sec Evaluation completed in 0.79 sec
X = 100x1
0.3758 0.0945 1.6328 0.5906 -0.6462 0.2529 1.8940 1.8789 -0.1144 0.0478
<mw-icon class=""></mw-icon>
<mw-icon class=""></mw-icon>
X_inv=gather(X_inv)
Evaluating tall expression using the Local MATLAB Session: - Pass 1 of 1: 0% complete - Pass 1 of 1: Completed in 0.12 sec Evaluation completed in 0.28 sec
X_inv = 100x100
0.7743 -0.0693 0.8170 0.8577 -0.3118 0.3562 -0.9650 -0.7043 1.7218 -1.4824 -0.3475 -0.1169 0.8369 0.6821 0.7398 0.2457 -0.4201 0.1088 0.1201 -0.2784 -0.0559 -1.2019 -0.5790 -0.3116 0.0530 -0.0006 0.3367 0.9886 -1.0026 -0.1638 -0.0558 0.4031 0.0743 -0.3376 0.2502 -0.0059 -0.4564 0.2059 -0.6678 -0.0947 -0.0655 0.4893 -0.3262 -0.0874 -0.4319 -0.5594 0.1062 0.0800 -0.5376 0.1450 -0.1239 0.2719 0.1130 -0.3080 -0.2385 0.8645 -0.9948 -0.2167 0.1881 0.0734 -0.0784 0.1249 0.2214 -0.3601 0.2519 -0.1688 -0.2442 -0.2114 -0.6418 0.4962 0.1870 -0.0858 -0.1424 -0.2332 -0.6112 -0.1980 -0.3017 0.5306 -0.4729 -0.0702 -0.1993 -0.3099 -0.0528 -0.1944 -0.1935 0.5906 -0.3473 -0.4316 0.2758 0.0678 0.1676 -0.4320 0.4946 0.1798 -0.4843 -0.1374 0.0295 -0.1248 0.8595 -0.2044 -0.2073 -0.3594 0.5226 -0.0816 0.6570 0.5772 -0.1445 0.1138 0.3007 -0.0707 -0.0532 -0.2033 -0.3197 -0.0642 0.2278 -0.7270 0.5173 0.4822 -0.1869 -0.0182 -1.0476 0.2718 -1.6123 -0.8449 0.4432 -0.3391 2.0997 1.0481 -1.5932 1.9536 0.6653 0.2580 -0.8852 -0.7632 -1.1088 -0.2306 0.3132 -0.4577 -0.2821 0.2983 0.3981 1.8093 0.9012 1.0211 0.2953 -0.9724 -0.1735 -1.0113 0.8892 0.0465 -0.1037 0.5654 -0.1123 -0.3934 0.4696 0.1766 -0.2697 0.2524 -0.6914 0.1512 0.0448 0.3207 -0.4046 -0.0659 -0.6561 -0.4781 -0.1712 -0.1590 -0.2617 0.0846 0.1668 0.3875 0.2468 -0.0771 -0.4338 0.5762 -0.6102 -0.3239 0.3225 0.1269 -0.0470 0.2204 -0.9893 -0.5586 0.7420 -0.8195 1.3611 0.7751 -1.1705 1.5683 0.4142 0.4074 -0.6538 -0.4146 -0.7123 -0.2845 -0.1614 0.2445 -0.9458 0.3083 0.2387 1.3137 0.4728 0.6883 0.0753 0.0331 -0.7331 -0.5261 -0.0115 0.4966 0.6588 -0.6303 1.2606 0.5455 -0.4444 0.0983 -1.5520 -1.2083 1.4617 -1.1599 -0.4837 -0.3701 0.6419 0.6479 0.8795 0.4380 -0.6280 0.5090 0.2999 -0.4201 -0.3082 -1.9395 -0.8663 -0.8497 -0.1915 0.5109 0.5848 0.8014 -0.6520 -0.0784 -1.1458 0.2776 -0.8118 -0.8092 0.5263 -0.2181 0.5056 0.2499 -1.7848 1.4072 0.7064 0.0356 -0.8161 -0.5013 -1.4115 -0.3823 -0.0601 0.0124 -0.2746 0.0059 -0.2659 0.3024 0.4954 0.3484 -0.1463 0.3013 -0.2698 -1.1395 0.9805 -0.3267 -0.5385 0.7469 -0.4072 -0.7453 0.7073 0.1720 -0.2972 0.1257 -1.7257 0.7864 0.7099 0.4504 -0.7276 -0.0966 -1.5364 -0.8592 -0.3177 0.2224 -0.6054 -0.1317 -0.1317 0.0955 0.4519 -0.2018 -0.4185 0.9981 -0.6300 -0.9723 0.7656 -0.1510
<mw-icon class=""></mw-icon>
<mw-icon class=""></mw-icon>
  2 Comments
chen zhe
chen zhe on 21 Jun 2024
Hi halleyhit,
Thanks for your answer. It works well for me.
But I have another issue in pinv, if I need a Pseudo inverse.
X=pinv(A)*B1; % solve X in equation A*X=B1
X=gather(X);
The error shows incorrect number or type for function diag().
Do we have any good solution for this pinv()?
Thanks in advance.
halleyhit
halleyhit on 23 Jun 2024
as dos shows, pinv does not support tall array

Sign in to comment.


Ayush Modi
Ayush Modi on 21 Jun 2024
Edited: Ayush Modi on 21 Jun 2024
Hi Chen,
"If I use my verctor b, it shows an error with incompatible tall array argument in tall/qrLeftSolve."
Operator '\' requires both the inputs to be of same type. If either one is tall array and the other one is not, this error will be thrown. To resolve the error, you would need to convert b vector into a tall array.
Here is the sample code for your reference:
% Defining two random matrix
A = rand(100,100)
A = 100x100
0.9947 0.4626 0.6908 0.9572 0.3930 0.2770 0.4175 0.1024 0.9190 0.9325 0.8319 0.8676 0.5394 0.6506 0.8565 0.8859 0.9501 0.9520 0.2930 0.4328 0.5271 0.1691 0.2264 0.2029 0.5648 0.0718 0.9155 0.5690 0.0737 0.3573 0.0923 0.2488 0.7940 0.2614 0.8866 0.7586 0.4054 0.6688 0.5740 0.3765 0.7618 0.2080 0.1918 0.5318 0.7129 0.9550 0.4707 0.9276 0.0998 0.5540 0.7819 0.4010 0.8166 0.4177 0.1151 0.4744 0.3886 0.6050 0.6691 0.9335 0.0192 0.5947 0.6057 0.9769 0.7346 0.8659 0.5971 0.8688 0.4415 0.4081 0.3207 0.3461 0.0195 0.1080 0.9258 0.0734 0.2192 0.3056 0.8795 0.5699 0.1733 0.7057 0.6939 0.8609 0.3452 0.2789 0.9620 0.2709 0.1687 0.0543 0.0302 0.7422 0.7112 0.3936 0.1013 0.8690 0.9021 0.8979 0.7237 0.5793 0.3438 0.3597 0.6744 0.1376 0.2294 0.5965 0.6801 0.2358 0.0972 0.6335 0.5269 0.8279 0.6645 0.2513 0.0440 0.1707 0.9959 0.6991 0.8024 0.3426 0.5075 0.7772 0.1627 0.6762 0.6031 0.6545 0.7705 0.3404 0.4068 0.9728 0.8714 0.3318 0.0450 0.7205 0.8281 0.0297 0.5191 0.5438 0.6343 0.3165 0.5878 0.9132 0.9126 0.8596 0.8247 0.0339 0.4097 0.7741 0.0685 0.4046 0.0702 0.9067 0.2362 0.9370 0.2169 0.0860 0.8906 0.7138 0.4679 0.5370 0.5455 0.1653 0.3492 0.2454 0.8324 0.1050 0.2765 0.9222 0.0056 0.3227 0.7864 0.9552 0.3336 0.6613 0.0659 0.7436 0.2887 0.8740 0.7749 0.9455 0.5397 0.2529 0.7490 0.6075 0.7590 0.4994 0.8931 0.9234 0.2500 0.1606 0.8784 0.5149 0.1548 0.2430 0.5835 0.8846 0.0499 0.2962 0.8501 0.6615 0.2866 0.0072 0.9357 0.1078 0.1873 0.4558 0.5208 0.8972 0.9301 0.1669 0.5393 0.8417 0.4028 0.8963 0.9119 0.8722 0.5371 0.8855 0.2501 0.0766 0.1744 0.5684 0.1131 0.2407 0.8461 0.0656 0.9170 0.9014 0.1651 0.3962 0.1488 0.3718 0.7011 0.1383 0.8359 0.8441 0.6202 0.0033 0.6855 0.6126 0.1507 0.4941 0.0794 0.4621 0.2253 0.4462 0.3381 0.9387 0.4232 0.8856 0.9501 0.0953 0.7915 0.1792 0.6617 0.6468 0.9575 0.4384 0.1039 0.6810 0.3056 0.1640 0.5208 0.2375 0.3266 0.8771 0.6450 0.7747 0.9686 0.1680 0.7484 0.2843 0.4546 0.6945 0.6782 0.4360 0.6349 0.4398 0.5671 0.7586 0.3504 0.3579 0.2416 0.3331 0.8744 0.4183 0.3727 0.0479 0.9523 0.9842 0.7074 0.0672 0.3116 0.6295 0.6847 0.3250 0.4385 0.5122 0.3505 0.7533
<mw-icon class=""></mw-icon>
<mw-icon class=""></mw-icon>
b = rand(100,1)
b = 100x1
0.5972 0.5849 0.6938 0.5226 0.1276 0.1076 0.8850 0.6889 0.6440 0.6508
<mw-icon class=""></mw-icon>
<mw-icon class=""></mw-icon>
% Converting them to tall arrays
tallA = tall(A)
tallA = 100x100 tall double matrix Columns 1 through 18 0.9947 0.4626 0.6908 0.9572 0.3930 0.2770 0.4175 0.1024 0.9190 0.9325 0.8319 0.8676 0.5394 0.6506 0.8565 0.8859 0.9501 0.9520 0.0923 0.2488 0.7940 0.2614 0.8866 0.7586 0.4054 0.6688 0.5740 0.3765 0.7618 0.2080 0.1918 0.5318 0.7129 0.9550 0.4707 0.9276 0.0192 0.5947 0.6057 0.9769 0.7346 0.8659 0.5971 0.8688 0.4415 0.4081 0.3207 0.3461 0.0195 0.1080 0.9258 0.0734 0.2192 0.3056 0.0302 0.7422 0.7112 0.3936 0.1013 0.8690 0.9021 0.8979 0.7237 0.5793 0.3438 0.3597 0.6744 0.1376 0.2294 0.5965 0.6801 0.2358 0.5075 0.7772 0.1627 0.6762 0.6031 0.6545 0.7705 0.3404 0.4068 0.9728 0.8714 0.3318 0.0450 0.7205 0.8281 0.0297 0.5191 0.5438 0.0702 0.9067 0.2362 0.9370 0.2169 0.0860 0.8906 0.7138 0.4679 0.5370 0.5455 0.1653 0.3492 0.2454 0.8324 0.1050 0.2765 0.9222 0.5397 0.2529 0.7490 0.6075 0.7590 0.4994 0.8931 0.9234 0.2500 0.1606 0.8784 0.5149 0.1548 0.2430 0.5835 0.8846 0.0499 0.2962 0.5393 0.8417 0.4028 0.8963 0.9119 0.8722 0.5371 0.8855 0.2501 0.0766 0.1744 0.5684 0.1131 0.2407 0.8461 0.0656 0.9170 0.9014 : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : Columns 19 through 36 0.2930 0.4328 0.5271 0.1691 0.2264 0.2029 0.5648 0.0718 0.9155 0.5690 0.0737 0.3573 0.6437 0.8474 0.7992 0.8280 0.0741 0.1939 0.0998 0.5540 0.7819 0.4010 0.8166 0.4177 0.1151 0.4744 0.3886 0.6050 0.6691 0.9335 0.4819 0.5598 0.4953 0.4115 0.4782 0.3217 0.8795 0.5699 0.1733 0.7057 0.6939 0.8609 0.3452 0.2789 0.9620 0.2709 0.1687 0.0543 0.0242 0.3062 0.0516 0.6704 0.0306 0.7953 0.0972 0.6335 0.5269 0.8279 0.6645 0.2513 0.0440 0.1707 0.9959 0.6991 0.8024 0.3426 0.4167 0.6541 0.2446 0.0995 0.2396 0.4477 0.6343 0.3165 0.5878 0.9132 0.9126 0.8596 0.8247 0.0339 0.4097 0.7741 0.0685 0.4046 0.8334 0.2130 0.6875 0.6769 0.3877 0.4635 0.0056 0.3227 0.7864 0.9552 0.3336 0.6613 0.0659 0.7436 0.2887 0.8740 0.7749 0.9455 0.5906 0.3272 0.5108 0.3575 0.0480 0.4427 0.8501 0.6615 0.2866 0.0072 0.9357 0.1078 0.1873 0.4558 0.5208 0.8972 0.9301 0.1669 0.3334 0.5827 0.3716 0.8463 0.3661 0.2470 0.1651 0.3962 0.1488 0.3718 0.7011 0.1383 0.8359 0.8441 0.6202 0.0033 0.6855 0.6126 0.3333 0.6425 0.1980 0.4854 0.8663 0.5186 : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : Columns 37 through 54 0.5355 0.0628 0.5854 0.7285 0.3470 0.4697 0.0952 0.2652 0.5367 0.3317 0.5564 0.9495 0.7961 0.8552 0.7955 0.6598 0.0790 0.2758 0.1317 0.1247 0.5925 0.3688 0.3156 0.2509 0.2378 0.4523 0.1257 0.3662 0.2784 0.7689 0.7046 0.9002 0.3912 0.0351 0.6458 0.8515 0.0865 0.6577 0.8280 0.9986 0.2409 0.1537 0.6579 0.1930 0.0249 0.4879 0.7781 0.6314 0.9368 0.0240 0.0143 0.2097 0.7062 0.7341 0.9884 0.2811 0.9029 0.6874 0.0348 0.4320 0.6434 0.8410 0.5180 0.4068 0.9110 0.1273 0.4360 0.5249 0.4062 0.9801 0.6685 0.9761 0.0288 0.8181 0.5334 0.4100 0.8383 0.0385 0.5114 0.2457 0.0255 0.4325 0.5701 0.6968 0.2131 0.8715 0.9579 0.5394 0.1259 0.3427 0.6658 0.1941 0.3923 0.7473 0.8542 0.1991 0.5978 0.0734 0.1934 0.6157 0.8895 0.7805 0.9647 0.7126 0.5998 0.1779 0.3588 0.6053 0.4122 0.1348 0.8091 0.4853 0.9496 0.1586 0.8800 0.0572 0.1071 0.4716 0.7755 0.0318 0.7086 0.1122 0.3572 0.6548 0.3940 0.6840 0.7034 0.2817 0.6467 0.2609 0.2060 0.7199 0.8099 0.2952 0.0169 0.8177 0.3036 0.2856 0.1215 0.2466 0.9243 0.0432 0.3923 0.5641 : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : Columns 55 through 72 0.1232 0.8125 0.3342 0.3624 0.6900 0.7983 0.9980 0.8447 0.7832 0.7575 0.0067 0.1952 0.2662 0.5321 0.0664 0.9825 0.9733 0.8893 0.0374 0.8396 0.6496 0.5587 0.9625 0.6297 0.3435 0.6432 0.2192 0.5261 0.8477 0.8670 0.6865 0.4448 0.0893 0.0777 0.3255 0.2446 0.7908 0.6795 0.1366 0.4217 0.7251 0.2381 0.5144 0.5793 0.6458 0.7338 0.7580 0.5616 0.4369 0.6664 0.6267 0.3804 0.3380 0.2379 0.5076 0.2598 0.8806 0.6885 0.9630 0.9466 0.9165 0.0427 0.4639 0.2303 0.8713 0.6334 0.5789 0.5165 0.1985 0.1758 0.5908 0.5495 0.9266 0.6116 0.9526 0.6330 0.8698 0.4270 0.3496 0.9524 0.4305 0.7161 0.9226 0.2925 0.0856 0.7509 0.1776 0.9825 0.3673 0.4231 0.3078 0.9878 0.5069 0.3448 0.8062 0.8439 0.0511 0.2587 0.2708 0.8813 0.3075 0.0586 0.4917 0.8142 0.1890 0.0054 0.9303 0.4693 0.9174 0.2580 0.3145 0.4633 0.1351 0.6166 0.6503 0.9601 0.2325 0.5454 0.5390 0.4688 0.0713 0.5822 0.1256 0.0251 0.5389 0.3943 0.8867 0.6729 0.7401 0.9365 0.5946 0.3014 0.0888 0.9111 0.3976 0.8435 0.8404 0.6172 0.3288 0.5484 0.8531 0.1393 0.3151 0.2739 : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : Columns 73 through 90 0.1418 0.4728 0.5474 0.4756 0.7492 0.7575 0.9610 0.9560 0.3065 0.2035 0.2640 0.2454 0.1366 0.5136 0.2233 0.1296 0.6005 0.8043 0.1893 0.4862 0.4133 0.2853 0.9044 0.8362 0.7122 0.3697 0.4223 0.9724 0.4888 0.3134 0.2218 0.4102 0.4233 0.1391 0.1185 0.7081 0.0672 0.4929 0.4596 0.5012 0.8577 0.7600 0.2484 0.3413 0.3446 0.0896 0.4449 0.8577 0.8031 0.8912 0.0473 0.3659 0.5192 0.9020 0.8129 0.0201 0.7949 0.7919 0.3383 0.9086 0.1702 0.7679 0.5813 0.3368 0.8705 0.3670 0.9006 0.9876 0.4270 0.3445 0.5984 0.8060 0.5353 0.4185 0.3806 0.4322 0.0480 0.7821 0.2039 0.3469 0.5749 0.2550 0.5728 0.0580 0.8129 0.2231 0.5661 0.7490 0.9359 0.3340 0.4876 0.9417 0.9716 0.1828 0.2480 0.4886 0.1627 0.5514 0.1248 0.7397 0.3727 0.1695 0.9214 0.4645 0.9173 0.5430 0.6221 0.0176 0.5162 0.1587 0.7328 0.5016 0.3008 0.5404 0.9609 0.4512 0.3910 0.8881 0.9625 0.0213 0.6435 0.8747 0.2997 0.8007 0.4106 0.3149 0.8028 0.6989 0.2706 0.2003 0.8490 0.4253 0.8486 0.8889 0.2723 0.4114 0.3909 0.1060 0.2967 0.8794 0.5920 0.0155 0.5504 0.6084 : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : Columns 91 through 100 0.3253 0.8026 0.1349 0.1779 0.5445 0.7229 0.8003 0.1008 0.3925 0.6121 0.3294 0.3681 0.5884 0.7623 0.0899 0.5957 0.0224 0.3019 0.7777 0.6986 0.3077 0.1976 0.9783 0.5364 0.0945 0.2875 0.4918 0.1006 0.7761 0.0023 0.6713 0.1236 0.1948 0.5170 0.1872 0.9505 0.3202 0.6634 0.0846 0.9351 0.6131 0.7419 0.9973 0.6152 0.3016 0.2948 0.8884 0.6059 0.3350 0.1469 0.5383 0.3988 0.6944 0.4074 0.4004 0.8078 0.5827 0.2220 0.3726 0.4620 0.8897 0.4469 0.8567 0.9446 0.6511 0.4413 0.7349 0.8421 0.3107 0.8229 0.5451 0.0544 0.0301 0.8424 0.8300 0.1674 0.7776 0.5627 0.4716 0.4457 : : : : : : : : : : : : : : : : : : : :
tallb = tall(b)
tallb = 100x1 tall double column vector 0.5972 0.5849 0.6938 0.5226 0.1276 0.1076 0.8850 0.6889 : :
% We can do operation A\b
A\b
ans = 100x1
-2.1302 -0.6963 2.2138 -4.1181 3.3345 -0.1259 2.6496 -4.5379 4.1526 -0.1002
<mw-icon class=""></mw-icon>
<mw-icon class=""></mw-icon>
% We can also do tallA\tallb
gather(tallA\tallb)
Evaluating tall expression using the Local MATLAB Session: - Pass 1 of 1: 0% complete - Pass 1 of 1: Completed in 0.16 sec Evaluation completed in 0.3 sec
ans = 100x1
-2.1302 -0.6963 2.2138 -4.1181 3.3345 -0.1259 2.6496 -4.5379 4.1526 -0.1002
<mw-icon class=""></mw-icon>
<mw-icon class=""></mw-icon>
% ERROR - if they are different
gather(tallA\b) % Note b is matrix, not tall array
Error using tall/qrLeftSolve (line 16)
Incompatible non-scalar tall array arguments. Each of the tall arrays must be the same size in the first dimension, must be derived from a single tall array, and must not have been indexed differently in the first dimension (indexing operations include functions such as VERTCAT, SPLITAPPLY, SORT, CELL2MAT, SYNCHRONIZE, RETIME and so on).

Error in \ (line 29)
[~,Z] = qrLeftSolve(X,Y);
  1 Comment
chen zhe
chen zhe on 21 Jun 2024
This much clear. Thanks Ayush.
My another issue is pinv, if I need a Pseudo inverse.
X=pinv(A)*B1; % solve X in equation A*X=B1
X=gather(X);
The error shows incorrect number or type for function diag().
Do we have any good solution for this pinv()?
Thanks in advance.

Sign in to comment.

Products

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!