lmstat
I got the following question:
Hi Art, Will you please tell me a little about the lmstat function? How to execute the function? What the output shows? Thanks,
lmstat does a bunch of things related to managing software licenses.
The license manager software is called "flexlm". This is probably not 100% technically precise, but basically flexlm uses a "license server" (basically, a computer with the flexlm software running on it), license "daemons" (details provided by individual software vendors on which "features" need to be licensed), and then calls from within the client to "check out" features.
Best to look at some examples (on pegasus2):
Call lmstat with the -a option to see everything:
login5 868% /share/opt/MATLAB/R2013a/etc/lmstat -a
------------------------------------------------------------------
lmstat - Copyright (c) 1989-2012 Flexera Software LLC. All Rights Reserved.
Flexible License Manager status on Thu 9/18/2014 13:31
License server status: 28518@m2
License file(s) on m2: /share/opt/flexlm/server.lic:
m2: license server UP (MASTER) v11.11
Vendor daemon status (on m2):
INTEL: UP v11.11
Feature usage info:
Users of I0C64C577: (Total of 5 licenses issued; Total of 0 licenses in use)
Users of AdvXEl: (Total of 5 licenses issued; Total of 0 licenses in use)
.....Cutting out a bunch of lines here....
pgroupd: UP v11.11
Feature usage info:
Users of PGI2013-cdk-linux: (Total of 2 licenses issued; Total of 0 licenses in use)
Users of PGI71-cdk-linux: (Total of 2 licenses issued; Total of 0 licenses in use)
.....Cutting out a bunch of lines here....
MLM: UP v11.11
Feature usage info:
Users of MATLAB: (Total of 6901 licenses issued; Total of 6 licenses in use)
.....Cutting out a bunch of lines here....
------------------------------------------------------------------
With lmstat -a you get the output of all the daemons running on the license server. In this case there are three: "INTEL", "pgroupd" and "MLM". MLM is the matlab license manager (the relevant flexlm daemon for matlab). The other ones are for I don't know what. A set of intel compilers and something else.. not sure..
Under each daemon are a list of the features licensed by that daemon. For example, the INTEL daemon has "I0C64C577" "AdvXEl" and a bunch more features.
For each feature you see a summary of how many copies of that feature can be simultaneously used, and how many are already checked out (in use), e.g.:
Users of AdvXEl: (Total of 5 licenses issued; Total of 0 licenses in use)
So 5 clients can use "AdvXEl" at the same time. None are using it right now. The definition of what is exactly a "client" and what it means to "check out" a feature is left up to the individual vendors. More on that below....
Presumably you are mostly interested in the matlab licenses. You can restrict the lmstat output to only the matlab part with the -S flag:
login5 869% /share/opt/MATLAB/R2013a/etc/lmstat -S MLM
------------------------------------------------------------------
lmstat - Copyright (c) 1989-2012 Flexera Software LLC. All Rights Reserved.
Flexible License Manager status on Thu 9/18/2014 13:49
License server status: 28518@m2
License file(s) on m2: /share/opt/flexlm/server.lic:
m2: license server UP (MASTER) v11.11
Vendor daemon status (on m2):
MLM: UP v11.11
Feature usage info:
Users of MATLAB: (Total of 6901 licenses issued; Total of 7 licenses in use)
"MATLAB" v31, vendor: MLM
floating license
agleason login5 /dev/pts/4 (v29) (m2/28518 10208), start Thu 9/18 10:09
agleason n171 /dev/tty (v29) (m2/28518 8826), start Thu 9/18 12:31
gmarques visx2 /dev/pts/16 (v29) (m2/28518 11705), start Wed 9/17 17:55
mhaghighat n303 /dev/tty (v29) (m2/28518 2741), start Thu 9/18 12:20
mhaghighat n149 /dev/tty (v29) (m2/28518 11917), start Thu 9/18 12:20
mmehari apollo2.ccs.miami.edu /dev/pts/1 (v29) (m2/28518 3637), start Thu 9/18 12:23
wlzhao login3.pegasus.edu /dev/pts/41 (v29) (m2/28518 11833), start Thu 9/18 13:39
Users of SIMULINK: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of Aerospace_Blockset: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of Aerospace_Toolbox: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of Bioinformatics_Toolbox: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of Communication_Toolbox: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of Video_and_Image_Blockset: (Total of 6901 licenses issued; Total of 2 licenses in use)
"Video_and_Image_Blockset" v31, vendor: MLM
floating license
mhaghighat n149 /dev/tty (v29) (m2/28518 8633), start Thu 9/18 12:26
mhaghighat n303 /dev/tty (v29) (m2/28518 337), start Thu 9/18 12:29
Users of Control_Toolbox: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of Curve_Fitting_Toolbox: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of Signal_Blocks: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of Data_Acq_Toolbox: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of Database_Toolbox: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of Econometrics_Toolbox: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of RTW_Embedded_Coder: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of Fin_Instruments_Toolbox: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of Financial_Toolbox: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of Fixed_Point_Toolbox: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of Fuzzy_Toolbox: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of GADS_Toolbox: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of Image_Acquisition_Toolbox: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of Image_Toolbox: (Total of 6901 licenses issued; Total of 2 licenses in use)
"Image_Toolbox" v31, vendor: MLM
floating license
mhaghighat n303 /dev/tty (v29) (m2/28518 1804), start Thu 9/18 12:20
mhaghighat n149 /dev/tty (v29) (m2/28518 7627), start Thu 9/18 12:20
Users of Instr_Control_Toolbox: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of MATLAB_Builder_for_dot_Net: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of MATLAB_Coder: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of Compiler: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of MAP_Toolbox: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of MPC_Toolbox: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of MBC_Toolbox: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of Neural_Network_Toolbox: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of Optimization_Toolbox: (Total of 6901 licenses issued; Total of 1 license in use)
"Optimization_Toolbox" v31, vendor: MLM
floating license
agleason n171 /dev/tty (v29) (m2/28518 1337), start Thu 9/18 12:36
Users of Distrib_Computing_Toolbox: (Total of 6901 licenses issued; Total of 1 license in use)
"Distrib_Computing_Toolbox" v31, vendor: MLM
floating license
agleason login5 /dev/pts/4 (v29) (m2/28518 2440), start Thu 9/18 10:10
Users of PDE_Toolbox: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of RF_Toolbox: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of Robust_Toolbox: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of Signal_Toolbox: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of SimBiology: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of SimEvents: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of SimHydraulics: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of SimMechanics: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of Power_System_Blocks: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of Simscape: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of Real-Time_Workshop: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of Simulink_Control_Design: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of Simulink_Design_Optim: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of Excel_Link: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of Stateflow: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of Statistics_Toolbox: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of Symbolic_Toolbox: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of Identification_Toolbox: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of Wavelet_Toolbox: (Total of 6901 licenses issued; Total of 0 licenses in use)
Users of MATLAB_Distrib_Comp_Engine: (Total of 32 licenses issued; Total of 32 licenses in use)
"MATLAB_Distrib_Comp_Engine" v29, vendor: MLM
floating license
agleason n051 /dev/tty (v29) (m2/28518 10736), start Thu 9/18 12:13
agleason n051 /dev/tty (v29) (m2/28518 6856), start Thu 9/18 12:13
agleason n051 /dev/tty (v29) (m2/28518 7549), start Thu 9/18 12:13
agleason n051 /dev/tty (v29) (m2/28518 9715), start Thu 9/18 12:13
agleason n319 /dev/tty (v29) (m2/28518 7034), start Thu 9/18 12:13
agleason n319 /dev/tty (v29) (m2/28518 11514), start Thu 9/18 12:13
agleason n319 /dev/tty (v29) (m2/28518 1445), start Thu 9/18 12:13
agleason n254 /dev/tty (v29) (m2/28518 10115), start Thu 9/18 12:13
agleason n319 /dev/tty (v29) (m2/28518 6229), start Thu 9/18 12:13
agleason n254 /dev/tty (v29) (m2/28518 7132), start Thu 9/18 12:13
agleason n254 /dev/tty (v29) (m2/28518 6729), start Thu 9/18 12:13
agleason n254 /dev/tty (v29) (m2/28518 8438), start Thu 9/18 12:13
agleason n257 /dev/tty (v29) (m2/28518 9424), start Thu 9/18 12:13
agleason n257 /dev/tty (v29) (m2/28518 3334), start Thu 9/18 12:13
agleason n257 /dev/tty (v29) (m2/28518 1744), start Thu 9/18 12:13
agleason n257 /dev/tty (v29) (m2/28518 3544), start Thu 9/18 12:13
agleason n057 /dev/tty (v29) (m2/28518 11613), start Thu 9/18 12:13
agleason n057 /dev/tty (v29) (m2/28518 3946), start Thu 9/18 12:13
agleason n057 /dev/tty (v29) (m2/28518 6131), start Thu 9/18 12:13
agleason n057 /dev/tty (v29) (m2/28518 11215), start Thu 9/18 12:13
agleason n317 /dev/tty (v29) (m2/28518 11317), start Thu 9/18 12:13
agleason n317 /dev/tty (v29) (m2/28518 4293), start Thu 9/18 12:13
agleason n316 /dev/tty (v29) (m2/28518 10019), start Thu 9/18 12:13
agleason n113 /dev/tty (v29) (m2/28518 10513), start Thu 9/18 12:13
agleason n113 /dev/tty (v29) (m2/28518 9927), start Thu 9/18 12:13
agleason n316 /dev/tty (v29) (m2/28518 2173), start Thu 9/18 12:13
agleason n317 /dev/tty (v29) (m2/28518 4458), start Thu 9/18 12:13
agleason n317 /dev/tty (v29) (m2/28518 9815), start Thu 9/18 12:13
agleason n316 /dev/tty (v29) (m2/28518 3033), start Thu 9/18 12:13
agleason n113 /dev/tty (v29) (m2/28518 12212), start Thu 9/18 12:13
agleason n316 /dev/tty (v29) (m2/28518 12116), start Thu 9/18 12:13
agleason n113 /dev/tty (v29) (m2/28518 12058), start Thu 9/18 12:13
Lots of good information in here. Matlab organizes its features to correspond with the matlab "toolboxes" (if you look at the feature names they are the same as the toolbox names). When you run matlab, the matlab client "checks out" a license for each feature that your program uses. Let's look at some examples:
---------------------------------------------------------------------------- Users of MATLAB: (Total of 6901 licenses issued; Total of 7 licenses in use) "MATLAB" v31, vendor: MLM floating license agleason login5 /dev/pts/4 (v29) (m2/28518 10208), start Thu 9/18 10:09 agleason n171 /dev/tty (v29) (m2/28518 8826), start Thu 9/18 12:31 gmarques visx2 /dev/pts/16 (v29) (m2/28518 11705), start Wed 9/17 17:55 mhaghighat n303 /dev/tty (v29) (m2/28518 2741), start Thu 9/18 12:20 mhaghighat n149 /dev/tty (v29) (m2/28518 11917), start Thu 9/18 12:20 mmehari apollo2.ccs.miami.edu /dev/pts/1 (v29) (m2/28518 3637), start Thu 9/18 12:23 wlzhao login3.pegasus.edu /dev/pts/41 (v29) (m2/28518 11833), start Thu 9/18 13:39 ----------------------------------------------------------------------------
Every matlab client checks out a "MATLAB" feature. lmstat lists details about each one that is checked out (7 in use in this case). On pegasus2, the node names tell you a bit about what each is being used for:
agleason login5 /dev/pts/4 (v29) (m2/28518 10208), start Thu 9/18 10:09 gmarques visx2 /dev/pts/16 (v29) (m2/28518 11705), start Wed 9/17 17:55 mmehari apollo2.ccs.miami.edu /dev/pts/1 (v29) (m2/28518 3637), start Thu 9/18 12:23 wlzhao login3.pegasus.edu /dev/pts/41 (v29) (m2/28518 11833), start Thu 9/18 13:39
These are all interactive sessions, or at least running on interactive nodes.
agleason n171 /dev/tty (v29) (m2/28518 8826), start Thu 9/18 12:31 mhaghighat n303 /dev/tty (v29) (m2/28518 2741), start Thu 9/18 12:20 mhaghighat n149 /dev/tty (v29) (m2/28518 11917), start Thu 9/18 12:20
These ones (nXXX) are running in batch mode on the compute nodes.
If you scroll down you can see which toolboxes are being used by which session. For example, I have matlab and the parallel toolbox on my interactive session:
Users of MATLAB: (Total of 6901 licenses issued; Total of 7 licenses in use) "MATLAB" v31, vendor: MLM floating license agleason login5 /dev/pts/4 (v29) (m2/28518 10208), start Thu 9/18 10:09 Users of Distrib_Computing_Toolbox: (Total of 6901 licenses issued; Total of 1 license in use) "Distrib_Computing_Toolbox" v31, vendor: MLM floating license agleason login5 /dev/pts/4 (v29) (m2/28518 2440), start Thu 9/18 10:10
And, I use matlab and the optimization toolbox on my batch job:
Users of MATLAB: (Total of 6901 licenses issued; Total of 7 licenses in use) "MATLAB" v31, vendor: MLM floating license agleason n171 /dev/tty (v29) (m2/28518 8826), start Thu 9/18 12:31 Users of Optimization_Toolbox: (Total of 6901 licenses issued; Total of 1 license in use) "Optimization_Toolbox" v31, vendor: MLM floating license agleason n171 /dev/tty (v29) (m2/28518 1337), start Thu 9/18 12:36
MOST of the matlab toolbox licenses work like this: for each job that you run, one license is checked out for each toolbox used in that job. The exception to this is the distributed computing engine (DCE). The DCE checks out one DCE license for each CPU that you are using, but does not check out individual toolbox licenses (as long as you have a license "issued" - not necessarily available - for the toolbox you want to use).
Again, if you look at the output above, you can see I am running a job using DCE on nodes: 051,319,254,257,057,317,113,316 however, there are NO other features besides DCE checked out by me on those nodes. This behavior is unique to the DCE. Once you know this, you can tell from the above output that I have two batch jobs running. One using DCE and the other not. This is confirmed with bjobs:
login5 874% bjobs
JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME
1274484 agleaso RUN general login5.pega 4*n113 Job8 Sep 18 12:13
4*n316
4*n317
4*n319
4*n051
4*n254
4*n257
4*n057
1274584 agleaso RUN general login5.pega n171 13t3_gp1 Sep 18 12:31
Finally, some points about how does this impact running jobs on pegasus2. The good news is that because UM now has a site license for matlab, there are TONS of licenses available for the individual toolboxes. You can see most of the features have 6091 licenses available!!! Thus, for all practical purposes, if you run your jobs using bsub and matlab < script.m it is pretty much impossible to not have a license available. You can use this method, and the parallel computing toolbox, to run basically as many simultaneous jobs as you want on ONE physical compute node and up to 12 CPUs.
The bad news is that there are only 32 seats of the DCE, which can easily - one might argue by definition usually will - be consumed by a single job. It is possible for a job to use less than the maximum 32 seats, in which case more than one job can use the DCE at a time. But, if someone wants to use all 32 processors for a given job then all other jobs using the DCE will fail until the first job finishes. I have not been able to figure out a way to "queue" jobs to wait for free licenses, but I have to believe there must be a way to do that....
Hope this helps,