====== Tahoe ======

Tahoe is a research-oriented platform for the development of numerical methods, finite element models, and material models. It is divided into 2 projects, the public Tahoe project (Tahoe v 2.1) hosted by SourceForge and the private development project hosted by Codesion. The open Tahoe project includes methods and models that have been published and implementations that have been verified. The development project contains unpublished methods models and implementations that have not been verified and thus not yet ready for public release. Go to the tahoe project page for a user guide (incomplete), discussion forum, and developer's wiki [[http://tahoe.sourceforge.net/|http://tahoe.sourceforge.net/]]

==== TAHOE User Manual ====

An incomplete but useful user manual for Tahoe is available here: {{ ::tahoe.user.pdf }}


==== TAHOE on GitHub ====

Tahoe is now available on GitHub: [[https://github.com/NguyenLabJHU/Tahoe-FEM]]. Follow the instructions on GitHub to install Tahoe on macOS and Linux (Ubuntu). Installation instructions on GitHub are based on the following instructions and may need some tweaks. Please contact Bibek if you would like to install Tahoe.

==== Installation of TAHOE ====

Below are the directions to download and install both the open and development modules and associated libraries and to update both projects using CVS. Try it first. If you encounter errors, read the log files to figure out the error. If you can't, then email me the log files so that I can help you. Please do not just email me and say that you encountered an error. I need to know the specific error to help you debug.

Nguyen Lab: Please don't just do nothing or copy someone else's executable if you encounter an error. Past experience has shown that doing the latter will only create more problems and will severely hinder my ability to help you with programming, debugging, etc. Ask google, your lab mates, and me for help.  This is our primary computing tool and if you can't compile it you won't be getting any work done.

Old instructions for installing TAHOE on macOS and Linux are [[tahoe-old|archived here]]. You might find it useful in specific circumstances.

==== Installing on macOS: ====

''%%TAHOE%%'' was installed and tested on macOS Catalina 10.15 with ''%%open-mpi%%'' 4.1.3 and ''%%gcc%%'' 11.2.0. Sourceforge doesn’t have the current version ''%%tahoe-manager%%'' available yet but will be updated soon.

  - Install ''%%xcode%%'' command line tools to enable basic development libraries and features. On macOS terminal: <code>
$ xcode-select --install </code>
  - If you don’t have an X11 client on macOS, download and install XQuartz. Visit [[https://www.xquartz.org]]. If you are accessing macOS via the terminal and have no GUI, you can install XQuartz using ''%%homebrew%%'' later. \\ \\ 
  - Apple changed the default shell from ''%%bash%%'' to ''%%zsh%%''. In case you don’t have an existing ''%%zsh%%'' profile, open one. You can skip the step if you have an existing ''%%zsh%%'' profile. <code> $ open ~/.zshrc </code>
  - Save and close the ''%%zshrc%%'' file. We will use this file later to save the environment variables and path related to ''%%TAHOE%%'' installation.
  - Download and install ''%%homebrew%%'' package manager: <code>
$ /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" </code>
  - Change the read and write permission for ''%%homebrew%%'': <code>
$ sudo chown -R $(whoami) $(brew --prefix)/* </code>
  - Download and install XQuartz via ''%%homebrew%%'', if you haven’t done it before: <code>
$ brew install --cask xquartz </code>
  - Install the following packages using ''%%homebrew:%%'' <code>
$ brew install wget
$ brew install cmake
$ brew install automake
$ brew install open-mpi 
$ brew options subversion
$ brew install subversion </code>
  - The current stable version of ''%%open-mpi%%'' available via ''%%homebrew%%'' is 4.1.3 and depends on ''%%gcc%%'' 11.2.0. Please check the installed versions of ''%%gcc%%'', ''%%g++%%'', and ''%%gfortran%%'' from the following directory before proceeding to next step. <code>
$ cd /usr/local/bin </code>
  - Create a symlink for ''%%gcc%%'', ''%%g++%%'', and ''%%gfortran%%'' compilers using (Please make sure to use the correct version of the compilers): <code>
$ ln -s gcc-11 gcc
$ ln -s g++-11 g++
$ ln -s gfortran-11 gfortran </code>
  - Log out of ''%%zsh%%'' terminal to activate the changes and log back in. \\  \\ 
  - In your home directory, create a directory for downloading and installing Tahoe and its dependencies: <code>
$ mkdir tahoe && cd tahoe </code>
  - Clone the ACCESS library using git. We will use ''%%exodus%%'' format. Install the Third-Party Libraries (TPL), build and install SEACAS using the following: <code>
$ git clone https://github.com/gsjaardema/seacas.git
$ cd seacas
$ ./install-tpl.sh
$ mkdir build && cd build
$ ../cmake-exodus
$ make && make install </code>
  - Save the following environment variables to your ''%%.zshrc%%'' file. Go back to the Tahoe directory using ''%%   - cd ..%%'' commands. Please make sure to use the correct directory. <code>
$ TAHOE_MAIN=$PWD
$ echo export TAHOE_MAIN=$TAHOE_MAIN >> ~/.zshrc
$ echo export ACCESS=$TAHOE_MAIN/seacas >> ~/.zshrc
$ echo export LD_LIBRARY_PATH=$ACCESS/lib >> ~/.zshrc
$ echo export CVS_RSH=ssh >> ~/.zshrc
$ source ~/.zshrc </code>
  - Create symlinks for ''%%exodus%%'' libraries. <code>
$ cd $ACCESS
$ ln -s include/ inc
$ cd lib
$ ln -s libexodus.dylib libexoIIv2c.dylib
$ ln -s libexodus.a libexoIIv2c.a
</code>
  - Log out of ''%%zsh%%'' terminal to activate the changes and log back in. \\ \\ 
  - Download the latest version of ''%%TAHOE-Installer%%'' (''%%tahoe-manager%%'', installation macros, and scripts) to ''%%/Users/username/Downloads/TAHOE-Installer%%''. \\ \\ 
  - Create a backup directory for downloading all the default and optional modules for ''%%TAHOE%%''. <code>
$ cd $TAHOE_MAIN
$ mkdir tahoe-backup && cd tahoe-backup </code>
  - Set the following variables for ease of usage ("username" should be changed to your own username): <code>
$ TAHOE_DOWNLOAD=/Users/username/Downloads/TAHOE-Installer
$ TAHOE_BACKUP=$TAHOE_MAIN/tahoe-backup </code>
  - Copy ''%%tahoe-manager%%'' to the ''%%tahoe-backup%%'' directory. <code>
$ cp -i $TAHOE_DOWNLOAD/tahoe-manager $TAHOE_BACKUP </code>
  - If the ''%%tahoe-manager%%'' isn’t already an executable, convert it to an executable file. If ''%%tahoe-manager%%'' is already an executable, you can skip this step below (it should be under "$TAHOE_BACKUP" directory). <code>
$ sudo chmod 755 ./tahoe-manager </code>
  - Run ''%%tahoe-manager%%'' to download the macros, default modules, and optional modules: <code>
$ ./tahoe-manager </code>
  - Select connection type: %%svn%%, and enter SourceForge user name. It will start downloading the macros. Once the macros are downloaded, enter any architecture type to continue the download. \\ \\ 
  - For optional modules, select 0 (''%%CBLAS%%''), 6 (''%%ACCESS%%''), 8 (''%%benchmark_XML%%''), 10 (''%%contrib%%''), 11 (''%%development%%''), 12 (''%%development_benchmark_xml%%''), 13 (''%%f2c%%''), 14 (''%%metis%%''), 15 (''%%spooles%%''), 16 (''%%spoolesMPI%%''). \\ \\ 
  - Copy the new macro files, ''%%GNU-GCC-MPI-9.3.macros%%'' and ''%%GNU-GCC9.3.macros%%'', to the downloaded ''%%marcos%%'' directory. This is a temporary step until the newer macros are available on SourceForge. <code>
$ cp -i $TAHOE_DOWNLOAD/*.macros $TAHOE_BACKUP/macros </code>
  - Edit the .tahoe-config file to change the architecture to either GNU-GCC-MPI-9.3 or GNU-GCC9.3. \\ \\ 
  - Use the libf2c_.a file here to replace the one under ''%%tahoe-install/f2c/lib%%'' directory. \\ \\ 
  - Make a copy of the backup directory for installation and rename it. <code>
$ cd $TAHOE_MAIN
$ cp -r tahoe-backup tahoe-install </code>
  - Go to the installation directory and run %%tahoe-manager%% to update the installation settings with the correct architecture. <code>
$ cd tahoe-install
$ ./tahoe-manager update </code>
  - Select a connection type as before, enter a user name, and select an architecture. For the parallel version of ''%%TAHOE%%'', select ''%%GNU-GCC-MPI-9.3%%'' or for serial version of ''%%TAHOE%%'', select ''%%GNU-GCC-9.3%%''. Select the optional modules as before. If the serial version of ''%%TAHOE%%'' is being installed, skip optional module 16 (''%%spoolesMPI%%''). \\ \\ 
  - Build and compile ''%%TAHOE:%%'' <code>
$ ./tahoe-manager build </code>
  - Once ''%%TAHOE%%'' is compiled and built, add the following environment variables and path to your ''%%.zshrc%%'' file: <code>
$ echo export TAHOE_MOD=$TAHOE_MAIN/tahoe-install >> ~/.zshrc
$ echo export TAHOE_DIR=$TAHOE_MOD/tahoe >> ~/.zshrc
$ echo export PATH=$PATH:$TAHOE_MOD/bin >> ~/.zshrc
$ source ~/.zshrc </code>
  - If you are using macOS with Apple Silicon processors, then you may have to add the following to your %%.bashrc%% file. We have not found a workaround for it yet. Make sure to choose your username. <code>
install_name_tool -add_rpath “$ACCESS/lib” /Users/beijunshen/tahoe/tahoe-install/tahoe/tahoe </code>

==== Installing on Ubuntu: ====

The parallel version of ''%%TAHOE%%'' was installed and tested on Ubuntu 20.04 via Windows Subsystem for Linux (WSL). Download the latest version of ''%%tahoe-manager%%'', installation macros, and scripts from Nguyen Lab OneDrive. Sourceforge doesn’t have this version and will be updated soon.

  - Download the latest version of ''%%TAHOE-Installer%%'' (''%%tahoe-manager%%'', installation macros and scripts) from Nguyen Lab OneDrive to ''%%/mnt/c/Users/username/Downloads%%''. \\ \\ 
  - Go to your home directory, and create a directory for downloading and installing Tahoe modules and their dependencies: <code>
$ cd
$ mkdir tahoe && cd tahoe
$ mkdir tahoe-backup && cd tahoe-backup </code>
  - Set the following variables for ease of usage. Make sure to use the correct directory. <code>
$ TAHOE_DOWNLOAD=/mnt/c/Users/username/Downloads/TAHOE-Installer
$ TAHOE_MAIN=/home/username/tahoe
$ TAHOE_BACKUP=/home/username/tahoe/tahoe-backup </code>
  - Copy ''%%install_pre.sh%%'' and ''%%tahoe-manager%%'' to the ''%%tahoe-backup%%'' and ''%%tahoe%%'' directory: <code>
$ cp -i $TAHOE_DOWNLOAD/install_pre.sh $TAHOE_MAIN
$ cp -i $TAHOE_DOWNLOAD/tahoe-manager $TAHOE_BACKUP
</code>
  - Open ''%%install_pre.sh%%'' using a text editor (for example, VS Code) and change the installation directory variable, ''%%TAHOE_MAIN%%'', within the file. Save and close the file. <code>
$ code install_pre.sh </code>
  - Change the permission for ''%%install_pre.sh%%'' to make it an executable. Run the script to download and install pre-requisite packages for ''%%TAHOE%%''. <code>
$ chmod a+x install_pre.sh
$ ./install_pre.sh
</code>
  - Log out of the ''%%bash%%'' shell to activate the change and log back in.
  - Set the following variables for ease of usage: <code>
$ TAHOE_DOWNLOAD=/mnt/c/Users/username/Downloads/TAHOE-Installer
$ TAHOE_BACKUP=/home/bibek/TAHOE/tahoe-backup </code>
  - Go to ''%%tahoe-backup%%'' directory. Run %%tahoe-manager%% to download the macros, default modules, and optional modules: <code>
$ cd $TAHOE_BACKUP
$ ./tahoe-manager </code>
  - Select connection type: %%svn%%, and enter the SourceForge user name. It will start downloading the macros. Once the macros are downloaded, enter any architecture type to continue downloading. \\ \\ 
  - For optional modules, select 0 (''%%CBLAS%%''), 6 (''%%ACCESS%%''), 8 (''%%benchmark_XML%%''), 10 (''%%contrib%%''), 11 (''%%development%%''), 12 (''%%development_benchmark_xml%%''), 13 (''%%f2c%%''), 14 (''%%metis%%''), 15 (''%%spooles%%''), 16 (''%%spoolesMPI%%''). \\ \\ 
  - Copy the new macro files, ''%%GNU-GCC-MPI-9.3.macros%%'' and ''%%GNU-GCC-9.3.macros%%'', to the downloaded ''%%marcos%%'' directory. This is a temporary step until the newer macros are available on SourceForge. <code>
$ cp -i $TAHOE_DOWNLOAD/*.macros $TAHOE_BACKUP/macros </code>
  - Clean all the settings before making a copy of the backup directory for installation purposes. <code>
$ ./tahoe-manager clean </code>
  - Make a copy of the backup directory for installation and rename it. <code>
$ cd $TAHOE_MAIN
$ cp -r tahoe-backup tahoe-install </code>
  - Go to the installation directory and run %%tahoe-manager%% to update the installation settings with the correct architecture. <code>
$ cd tahoe-install
$ ./tahoe-manager update </code>
  - Select a connection type, enter a user name, and select an architecture. For the parallel version of ''%%TAHOE%%'', select ''%%GNU-GCC-MPI-9.3%%'' or for serial version of ''%%TAHOE%%'', select %%GNU-GCC-9.3%%. Select the optional modules as before. If the serial version of ''%%TAHOE%%'' is being installed, skip optional module 16 (''%%spoolesMPI%%''). \\ \\ 
  - Build and compile ''%%TAHOE:%%''<code>$ ./tahoe-manager build </code>
  - Once ''%%TAHOE%%'' is compiled and built, add the following environment variables and path to your ''%%.bashrc%%'' file: <code>
$ echo export TAHOE_MOD=$TAHOE_MAIN/Tahoe-Install >> ~/.bashrc
$ echo export TAHOE_DIR=$TAHOE_MOD/tahoe >> ~/.bashrc
$ echo export PATH="$TAHOE_MOD/bin:$PATH" >> ~/.bashrc
$ source ~/.bashrc </code>

==== Installing Tahoe for MARCC ====

**Assuming what is needed in Tahoe serials compiling has been established.**

1) load the module required for Tahoe parallel compile on MARCC (GCC, MARCC, CMake)

    module load MARCC

    module load gcc

    module load cmake

  * These can be added to the .bashrc (for bash shell) file

2) download the tahoe-install directory

    cvs -d username@tahoe.cvs.sourceforge.net:/cvsroot/tahoe checkout tahoe-install

3) configure tahoe for build:

    cd tahoe-install

   ./tahoe-manager config
   enter Sourceforge "username"
   enter Cloudforge "tahoe.username"
   enter your password for Sourceforge or Cloudforge when prompted (if you don't have ssh key pairs setup)
   default build architecture number: ?   choose 112 (for openmpi on linux)
                                                              * you chose : RedHat.OpenMPI
   do want this to be the default architecture for all modules (y/n) [y]: 
                                                              * default modules : macros expat toolbox tahoe
   do want to add optional modules (y/n) [n]: y
   Choice or modules to toggle (applied left to right): 0 6 7 10 11 13 14 15 16 (turn on CBLAS access aztec contrib development f2c metis spooles spoolesMPI)

4) Build Tahoe

./tahoe-manager build

After installation, If you get the following error when running Tahoe:  error while loading shared libraries: libnetcdf.so.11: cannot open shared object file: No such file or directory.  Then add "export LD_LIBRARY_PATH=/XXX/lib" (replace XXX with the directory of ACCESS, e.g. LD_LIBRARY_PATH=/home-2/hdong6@jhu.edu/seacas/lib) into the .bashrc file and restart the bash.

Tips: 1. When running parallel, make sure to use the command: $ mpirun -np number_of_cores tahoe -f. Using tahoe -f will run program in serial.

2. When running parallel, make sure that you choose SPOOLES under solver

3. MARCC changed compilers, if your Tahoe does not compile, connect Zheliang Wang (wzhelia1@jhu.edu) for an updated zip file of Tahoe.

4. For more information, see tahoe user manual.

Building Tahoe using Make

The Tahoe project is built using Make. Each module contains a makefile containing the build options and configurations for each module is.

1) To create a makefile for a module (e.g. tahoe), cd to the directory associated with the module (e.g. full_path_to_tahoe-install/tahoe) and cp makefile.template to makefile. Add the HOME_DIR for the module (e.g., full_path_to_tahoe-install/tahoe), and the build architecture (e.g., DARWIN-GCC4.X). Uncomment modules that you want to link to in the build options (e.g. access).

2) Type make help to get all targets. For example, the targets for the Tahoe module are:

  * init - (re-)initialize headers and file dependencies
  * build - (re-)build binaries
  * objects - (re-)compile object file but don't link
  * clean - remove object files and build temporaries
  * help - display this list
  * additional targets for configuration:

  * clean_config - remove config files in src/config
  * update_config - update config files in src/config
  * additional targets for (un-)installing optional modules:

  * touch_spooles - sparse, direct solver library
  * touch_SuperLU - sparse, direct solver library
  * touch_Aztec - sparse, iterative solver library
  * touch_f2c - f2c support
  * touch_SEACAS - Sandia tools
  * touch_MPI - support for parallel execution
  * touch_METIS - domain decomposition using METIS
  * touch_qhull - computational geometry using qhull
  * touch_NOX - NOX nonlinear solver library
  * touch_fossum - geomaterial development module
  * touch_expat - XML parsing
  * touch_SIMOD - Shared Interface MODels libraries

3) Each module can be built individually with make init build -s. Initialize and build the toolbox and libraries before the tahoe directory.

4)Resources/instructions for Make

[[http://www.gnu.org/software/make/manual/make.html#Top|GNU Make]]

[[http://www.cs.utah.edu/dept/old/texinfo/make/make.html|Free Software Foundation]]

==== Using SVN ====

SVN [[https://subversion.apache.org/|Subversion]] is an open-source version control system used to manage the collaborative development of Tahoe. Browse through the tahoe svn repositories to see previous versions and track the difference between versions.

The svn repository for the public tahoe project: [[https://sourceforge.net/p/tahoe/codesvn/HEAD/tree/]]

The svn repository for the development project:  [[https://sourceforge.net/p/tahoe/codesvn_dev/HEAD/tree/]]

The svn repository for the development project benchmark:  [[https://sourceforge.net/p/tahoe/codesvn_dev_bench/HEAD/tree/]]

A tutorial to svn: [[http://svnbook.red-bean.com/]]

key commands: commit, update, checkout

==== Creating Tahoe input files ====

1) Tahoe uses [[http://www.w3.org/XML/|XML]] files as input files. There are many XML editors available for purchase and free download from the web. A common one used in the lab is the no-frills GUI [[http://www.felixgolubov.com/XMLEditor/|XAmple]] editor.

==== Installing Xample XML editor on macOS ====

Xample is probably the best open-source XML editor.  It allows you to easily load custom XML schemas (.xsd files), informs you of all the possible inputs for lists and sublists, and allows for easy adding and removing of nodes. Unfortunately, Xample is no longer supported by its original creator, which means it will not run on the latest Java version.  The last Java version compatible that Xample ran on was version 8 (1.8.0_91).  If you do not have version 8, you must install it.  The easiest way to do this is to use homebrew.

  * $ brew install cask
  * $ brew cask install java8

check the paths of the versions installed:

  * $ /usr/libexec java_home -V

You should see something like this:

  * 9.0.1, x86_64:      "Java SE 9.0.1" /Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk/Contents/Home
  * 1.8.0_91, x86_64:   "Java SE 8"     /Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home

Alternatively, download the following disk image for v8 and try installing it yourself without brew

{{ ::jre-8u151-macosx-x64.dmg }}

Download the following distribution of Xample

{{ :xample.tar }}

untar: tar -xvf xample.tar

If you're using csh or tcsh, you will execute Xample using the run-xample.csh script, but first you will need to set a few variables. open the the run-xample.csh script and add the following after "#!/bin/csh -f "

1) setenv JAVA_HOME /Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home

2) set JAVA = /usr/bin/java

3) set xampledir = [Full path to Xample]/XAmple-distr You can run Xample by running the script in the X-window.

Next, create the xml schema by typing tahoe -xsd (to create file tahoe.xsd). 3) Open tahoe.xsd in Xample. 4) At this point, you can create an .xml input file from scratch or open and edit an existing .xml input file (for example using a benchmark).

==== Install jedit ====

Another good open-source xml editor is jedit. I allows you to load custom .xsd or .dtd xml schema and has a tree view of the inputs as well as a table of allowable inputs for each sublist.  It's still not as convenient as Xample, but it is an actively supported application that runs on the latest version of java.

To install jedit, download the installation package for your platform from: [[http://www.jedit.org/index.php?page=download]] Install according to the directions on the website.   For mac, you may have to run the application from the command line using the latest version of java.

<code>% java -jar [full path to jEdit]/jEdit.app/Contents/Java jedit.jar</code>

Next, install the XML plug-in.  You can do this in the application's pull-down menu Plugins -> Plugin Manager and check the boxes for XML, XercesPlugin, and SideKick.  To download and install the plug-ins manually, follow the instructions at [[http://plugins.jedit.org/]] .

When you open an existing tahoe xml file or create a new one, you have to associate either the tahoe.xsd or tahoe.dtd schema with it.  To do so got to Plugins -> Set Schema Type and choose between XSD or DTD.  Next, go to Plugins -> Set Schema.  The sidekick window will now show you a tree view of all of your lists and sublists of the input file. You can dock the sidekick window by clicking on the down arrow at the top of the window. To insert a new field,  scroll to the line that you want to insert something in, then go to Plugins -> XML Insert.  This will create another window of all the input fields allowed for that list item.  You can also dock this window in the main window by clicking on the down arrow at the top of the menu.

==== Benchmark examples ====

All benchmark examples reside in tahoe-install/benchmark_XML and are organized as follows:

  * level.0 - element formulations (e.g., elastodynamic, elastostatic, diffusion, adhesion, etc..)
  * level.1 - material formulations for solid and cohesive elements; stress smoothing algorithms
  * level.2 - boundary controllers and conditions, contact
  * level.3 - parallel execution

Running Tahoe

1) Run interactively by typing tahoe at the command line. You'll be prompted to enter the input file path or option.

2) Run tahoe with command line options (tahoe [-option]):

  * f [input_file.xml] executes tahoe with the specified input file.
  * xsd generates the xml template tahoe.xsd file.
  * dtd generates the xml template tahoe.dtd file
  * verbose executes tahoe with verbose message logs
  * decomp decomposes geometry file for parallel execution
  * join joins output files for parallel execution

3) Run tahoe batch processes and create a text file with @ in the first line. List the input files in subsequent lines. Download example:

{{ ::run.batch }}

Adding users and permissions Add user to sourceforge and cloudforge repositories. For sourceforge:

In the tahoe directory, checkout CVSROOT

  * cd CVSROOT
  * vi avail
  * Add the username to the avail file
  * cvs commit