https://www.knightli.com/ja/tags/mellanox-infiniband/ Recent content in Mellanox Infiniband on KnightLiブログ Hugo -- gohugo.io ja Thu, 28 Apr 2022 00:00:00 +0000 https://www.knightli.com/ja/2022/04/28/mellanox-infiniband-%E5%88%87%E6%8D%A2-ib-ethernet-vpi/ Thu, 28 Apr 2022 00:00:00 +0000 https://www.knightli.com/ja/2022/04/28/mellanox-infiniband-%E5%88%87%E6%8D%A2-ib-ethernet-vpi/ <h2 id="ネットワークカードの現在の動作ステータスを確認します">ネットワークカードの現在の動作ステータスを確認します </h2><p>ibstat を実行すると、ネットワーク カードの動作モードが返されるリンク層情報が返されます。</p> <div class="highlight"><div class="chroma"> <table class="lntable"><tr><td class="lntd"> <pre tabindex="0" class="chroma"><code><span class="lnt"> 1 </span><span class="lnt"> 2 </span><span class="lnt"> 3 </span><span class="lnt"> 4 </span><span class="lnt"> 5 </span><span class="lnt"> 6 </span><span class="lnt"> 7 </span><span class="lnt"> 8 </span><span class="lnt"> 9 </span><span class="lnt">10 </span><span class="lnt">11 </span><span class="lnt">12 </span><span class="lnt">13 </span><span class="lnt">14 </span><span class="lnt">15 </span><span class="lnt">16 </span><span class="lnt">17 </span><span class="lnt">18 </span><span class="lnt">19 </span><span class="lnt">20 </span><span class="lnt">21 </span><span class="lnt">22 </span><span class="lnt">23 </span><span class="lnt">24 </span><span class="lnt">25 </span><span class="lnt">26 </span><span class="lnt">27 </span><span class="lnt">28 </span></code></pre></td> <td class="lntd"> <pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback"><span class="line"><span class="cl">knightli@epyc:~$ sudo ibstat </span></span><span class="line"><span class="cl">CA &#39;mlx4_0&#39; </span></span><span class="line"><span class="cl"> CA type: MT4103 </span></span><span class="line"><span class="cl"> Number of ports: 2 </span></span><span class="line"><span class="cl"> Firmware version: 2.42.5700 </span></span><span class="line"><span class="cl"> Hardware version: 0 </span></span><span class="line"><span class="cl"> Node GUID: 0x98f2b3ffffd5c3c0 </span></span><span class="line"><span class="cl"> System image GUID: 0x98f2b3ffffd5c3c3 </span></span><span class="line"><span class="cl"> Port 1: </span></span><span class="line"><span class="cl"> State: Down </span></span><span class="line"><span class="cl"> Physical state: Disabled </span></span><span class="line"><span class="cl"> Rate: 40 </span></span><span class="line"><span class="cl"> Base lid: 0 </span></span><span class="line"><span class="cl"> LMC: 0 </span></span><span class="line"><span class="cl"> SM lid: 0 </span></span><span class="line"><span class="cl"> Capability mask: 0x00010000 </span></span><span class="line"><span class="cl"> Port GUID: 0x9af2b3fffed5c3c1 </span></span><span class="line"><span class="cl"> Link layer: Ethernet </span></span><span class="line"><span class="cl"> Port 2: </span></span><span class="line"><span class="cl"> State: Down </span></span><span class="line"><span class="cl"> Physical state: Disabled </span></span><span class="line"><span class="cl"> Rate: 40 </span></span><span class="line"><span class="cl"> Base lid: 0 </span></span><span class="line"><span class="cl"> LMC: 0 </span></span><span class="line"><span class="cl"> SM lid: 0 </span></span><span class="line"><span class="cl"> Capability mask: 0x00010000 </span></span><span class="line"><span class="cl"> Port GUID: 0x9af2b3fffed5c3c2 </span></span><span class="line"><span class="cl"> Link layer: Ethernet </span></span></code></pre></td></tr></table> </div> </div><h2 id="作業モードを切り替える">作業モードを切り替える </h2><h3 id="mst-ソフトウェアをインストールする">MST ソフトウェアをインストールする </h3><div class="highlight"><div class="chroma"> <table class="lntable"><tr><td class="lntd"> <pre tabindex="0" class="chroma"><code><span class="lnt">1 </span></code></pre></td> <td class="lntd"> <pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback"><span class="line"><span class="cl">root@epyc:/home/knightli# systemctl start mst </span></span></code></pre></td></tr></table> </div> </div><h3 id="デバイス名を問い合わせる">デバイス名を問い合わせる </h3><p>MST デバイスの下には、返されたデバイス名が表示されます。 2 つのポートはデバイス名にも対応していることに注意してください。</p> <div class="highlight"><div class="chroma"> <table class="lntable"><tr><td class="lntd"> <pre tabindex="0" class="chroma"><code><span class="lnt"> 1 </span><span class="lnt"> 2 </span><span class="lnt"> 3 </span><span class="lnt"> 4 </span><span class="lnt"> 5 </span><span class="lnt"> 6 </span><span class="lnt"> 7 </span><span class="lnt"> 8 </span><span class="lnt"> 9 </span><span class="lnt">10 </span><span class="lnt">11 </span><span class="lnt">12 </span><span class="lnt">13 </span><span class="lnt">14 </span></code></pre></td> <td class="lntd"> <pre tabindex="0" class="chroma"><code class="language-gdscript3" data-lang="gdscript3"><span class="line"><span class="cl"><span class="n">root</span><span class="err">@</span><span class="n">epyc</span><span class="o">-</span><span class="mi">7551</span><span class="n">p</span><span class="p">:</span><span class="o">/</span><span class="n">home</span><span class="o">/</span><span class="n">knightli</span><span class="c1"># mst status</span> </span></span><span class="line"><span class="cl"><span class="n">MST</span> <span class="n">modules</span><span class="p">:</span> </span></span><span class="line"><span class="cl"><span class="o">------------</span> </span></span><span class="line"><span class="cl"> <span class="n">MST</span> <span class="n">PCI</span> <span class="n">module</span> <span class="n">loaded</span> </span></span><span class="line"><span class="cl"> <span class="n">MST</span> <span class="n">PCI</span> <span class="n">configuration</span> <span class="n">module</span> <span class="n">loaded</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="n">MST</span> <span class="n">devices</span><span class="p">:</span> </span></span><span class="line"><span class="cl"><span class="o">------------</span> </span></span><span class="line"><span class="cl"><span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">mst</span><span class="o">/</span><span class="n">mt4103_pciconf0</span> <span class="o">-</span> <span class="n">PCI</span> <span class="n">configuration</span> <span class="n">cycles</span> <span class="n">access</span><span class="o">.</span> </span></span><span class="line"><span class="cl"> <span class="n">domain</span><span class="p">:</span><span class="n">bus</span><span class="p">:</span><span class="n">dev</span><span class="o">.</span><span class="n">fn</span><span class="o">=</span><span class="mi">0000</span><span class="p">:</span><span class="mi">28</span><span class="p">:</span><span class="mf">00.0</span> <span class="n">addr</span><span class="o">.</span><span class="n">reg</span><span class="o">=</span><span class="mi">88</span> <span class="n">data</span><span class="o">.</span><span class="n">reg</span><span class="o">=</span><span class="mi">92</span> <span class="n">cr_bar</span><span class="o">.</span><span class="n">gw_offset</span><span class="o">=-</span><span class="mi">1</span> </span></span><span class="line"><span class="cl"> <span class="n">Chip</span> <span class="n">revision</span> <span class="n">is</span><span class="p">:</span> <span class="mi">00</span> </span></span><span class="line"><span class="cl"><span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">mst</span><span class="o">/</span><span class="n">mt4103_pci_cr0</span> <span class="o">-</span> <span class="n">PCI</span> <span class="n">direct</span> <span class="n">access</span><span class="o">.</span> </span></span><span class="line"><span class="cl"> <span class="n">domain</span><span class="p">:</span><span class="n">bus</span><span class="p">:</span><span class="n">dev</span><span class="o">.</span><span class="n">fn</span><span class="o">=</span><span class="mi">0000</span><span class="p">:</span><span class="mi">28</span><span class="p">:</span><span class="mf">00.0</span> <span class="n">bar</span><span class="o">=</span><span class="mh">0xeb700000</span> <span class="n">size</span><span class="o">=</span><span class="mh">0x100000</span> </span></span><span class="line"><span class="cl"> <span class="n">Chip</span> <span class="n">revision</span> <span class="n">is</span><span class="p">:</span> <span class="mi">00</span> </span></span></code></pre></td></tr></table> </div> </div><h3 id="作業モードを切り替える-1">作業モードを切り替える </h3><p>mlxconfig -d /dev/mst/mt4103_pciconf0 set LINK_TYPE_P1=1</p> <ul> <li>-d デバイス名</li> <li>LINK_TYPE<br> LINK_TYPE_P1: ポート 1<br> LINK_TYPE_P2: ポート 2<br> 1 : IB<br> 2 : Ethernet<br> 3 : VPI</li> </ul> <div class="highlight"><div class="chroma"> <table class="lntable"><tr><td class="lntd"> <pre tabindex="0" class="chroma"><code><span class="lnt"> 1 </span><span class="lnt"> 2 </span><span class="lnt"> 3 </span><span class="lnt"> 4 </span><span class="lnt"> 5 </span><span class="lnt"> 6 </span><span class="lnt"> 7 </span><span class="lnt"> 8 </span><span class="lnt"> 9 </span><span class="lnt">10 </span><span class="lnt">11 </span><span class="lnt">12 </span><span class="lnt">13 </span><span class="lnt">14 </span><span class="lnt">15 </span><span class="lnt">16 </span><span class="lnt">17 </span><span class="lnt">18 </span><span class="lnt">19 </span><span class="lnt">20 </span><span class="lnt">21 </span><span class="lnt">22 </span><span class="lnt">23 </span><span class="lnt">24 </span><span class="lnt">25 </span><span class="lnt">26 </span><span class="lnt">27 </span><span class="lnt">28 </span><span class="lnt">29 </span></code></pre></td> <td class="lntd"> <pre tabindex="0" class="chroma"><code class="language-gdscript3" data-lang="gdscript3"><span class="line"><span class="cl"><span class="n">root</span><span class="err">@</span><span class="n">epyc</span><span class="p">:</span><span class="o">/</span><span class="n">home</span><span class="o">/</span><span class="n">knightli</span><span class="c1"># mlxconfig -d /dev/mst/mt4103_pciconf0 set LINK_TYPE_P1=1</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="n">Device</span> <span class="c1">#1:</span> </span></span><span class="line"><span class="cl"><span class="o">----------</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="n">Device</span> <span class="n">type</span><span class="p">:</span> <span class="n">ConnectX3Pro</span> </span></span><span class="line"><span class="cl"><span class="n">Device</span><span class="p">:</span> <span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">mst</span><span class="o">/</span><span class="n">mt4103_pciconf0</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="n">Configurations</span><span class="p">:</span> <span class="n">Next</span> <span class="n">Boot</span> <span class="n">New</span> </span></span><span class="line"><span class="cl"> <span class="n">LINK_TYPE_P1</span> <span class="n">IB</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="n">IB</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"> <span class="n">Apply</span> <span class="n">new</span> <span class="n">Configuration</span><span class="err">?</span> <span class="p">(</span><span class="n">y</span><span class="o">/</span><span class="n">n</span><span class="p">)</span> <span class="p">[</span><span class="n">n</span><span class="p">]</span> <span class="p">:</span> <span class="n">y</span> </span></span><span class="line"><span class="cl"><span class="n">Applying</span><span class="o">...</span> <span class="n">Done</span><span class="o">!</span> </span></span><span class="line"><span class="cl"><span class="o">-</span><span class="n">I</span><span class="o">-</span> <span class="n">Please</span> <span class="n">reboot</span> <span class="n">machine</span> <span class="n">to</span> <span class="nb">load</span> <span class="n">new</span> <span class="n">configurations</span><span class="o">.</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="n">root</span><span class="err">@</span><span class="n">epyc</span><span class="p">:</span><span class="o">/</span><span class="n">home</span><span class="o">/</span><span class="n">knightli</span><span class="c1"># mlxconfig -d /dev/mst/mt4103_pciconf0 set LINK_TYPE_P2=1</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="n">Device</span> <span class="c1">#1:</span> </span></span><span class="line"><span class="cl"><span class="o">----------</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="n">Device</span> <span class="n">type</span><span class="p">:</span> <span class="n">ConnectX3Pro</span> </span></span><span class="line"><span class="cl"><span class="n">Device</span><span class="p">:</span> <span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">mst</span><span class="o">/</span><span class="n">mt4103_pciconf0</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="n">Configurations</span><span class="p">:</span> <span class="n">Next</span> <span class="n">Boot</span> <span class="n">New</span> </span></span><span class="line"><span class="cl"> <span class="n">LINK_TYPE_P2</span> <span class="n">IB</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="n">IB</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"> <span class="n">Apply</span> <span class="n">new</span> <span class="n">Configuration</span><span class="err">?</span> <span class="p">(</span><span class="n">y</span><span class="o">/</span><span class="n">n</span><span class="p">)</span> <span class="p">[</span><span class="n">n</span><span class="p">]</span> <span class="p">:</span> <span class="n">y</span> </span></span><span class="line"><span class="cl"><span class="n">Applying</span><span class="o">...</span> <span class="n">Done</span><span class="o">!</span> </span></span><span class="line"><span class="cl"><span class="o">-</span><span class="n">I</span><span class="o">-</span> <span class="n">Please</span> <span class="n">reboot</span> <span class="n">machine</span> <span class="n">to</span> <span class="nb">load</span> <span class="n">new</span> <span class="n">configurations</span><span class="o">.</span> </span></span></code></pre></td></tr></table> </div> </div><h3 id="ホストを再起動します">ホストを再起動します </h3><div class="highlight"><div class="chroma"> <table class="lntable"><tr><td class="lntd"> <pre tabindex="0" class="chroma"><code><span class="lnt">1 </span></code></pre></td> <td class="lntd"> <pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback"><span class="line"><span class="cl">root@epyc-7551p:/home/knightli# sudo reboot </span></span></code></pre></td></tr></table> </div> </div>