CLEAR File Storage and Data Management

There are several ways to copy files to and from the CLEAR systems. The best method depends on your local operating system and where your files are currently stored.

Before transferring files, make sure you have the necessary SSH tools installed on your computer. See SSH Tools for additional information.

Cloud Storage to Your Local Computer to CLEAR

If your files are stored in a cloud storage service, you can first download them to your local computer and then transfer them to CLEAR.

For information about available cloud storage services, see Storage Options for Undergraduate and Graduate Students.

You can also use rclone on your local computer. See CLEAR Cloud Storage with Rclone Guide.

Windows

Windows users can transfer files to and from CLEAR using several tools, including:

• WinSCP
• MobaXterm
• PuTTY PSCP
• OpenSSH for Windows

For OpenSSH instructions, see CLEAR SSH Access Guide.

macOS and Linux

macOS and Linux users can use the scp command from a terminal. The general syntax is:

scp <source> <destination>

Copy a File from Your Local Computer to CLEAR

For example, if you have a file named test.txt on your local computer and want to copy it to your CLEAR home directory:

1. Open a terminal.
2. Change to the directory containing the file.
3. Verify that the file exists.
4. Use scp to transfer the file.

cd /<directory>
ls -la test.txt

# Copy the file to your CLEAR home directory
scp ./test.txt <NetID>@ssh.clear.rice.edu:~/

# Copy the file and give it a new name
scp ./test.txt <NetID>@ssh.clear.rice.edu:~/<new_name>

# Copy the file to an existing subdirectory
scp ./test.txt <NetID>@ssh.clear.rice.edu:~/<directory>/<new_name>

Replace <NetID> with your Rice NetID. Depending on your SSH configuration, you may be prompted to complete authentication before the transfer begins.

Copy a File from CLEAR to Your Local Computer

To retrieve a file from your CLEAR home directory, reverse the source and destination:

scp <NetID>@ssh.clear.rice.edu:~/<directory>/<file> ./

Cloud Storage Directly to CLEAR

To transfer files directly between supported cloud storage services and CLEAR, you can use rclone. See CLEAR Cloud Storage with Rclone Guide.

macOS users can also transfer files using SFTP.

Make sure your computer is connected to the appropriate Rice network or VPN if required. Open Terminal and enter:

sftp <NetID>@ssh.clear.rice.edu

Replace <NetID> with your Rice NetID and complete authentication when prompted.

After connecting, you will be placed in an SFTP shell. Useful commands include:

• help — Display available SFTP commands.
• ls — List files on the remote CLEAR system.
• get — Download a file from CLEAR to your local computer.
• put — Upload a file from your local computer to CLEAR.
• exit — Close the SFTP session.

This section provides information for faculty or course administrators who may still need to migrate legacy course materials from older Owlnet or RUF systems to CLEAR.

Course Accounts and Storage

CLEAR course materials are typically stored under:

/clear/courses

A CLEAR system administrator can create a course directory for the class you are teaching, for example:

/clear/courses/<course_name>

If teaching assistants or other users require access to the course space, include that information when submitting your request so the appropriate permissions can be configured.

Migrating Existing Course Data

If you have legacy course materials on an older Owlnet or RUF system, there are two general options:

1. Use a secure file transfer method such as scp to copy the files to your CLEAR course space, if the legacy system is still accessible.
2. Submit a help ticket requesting assistance with migrating existing course data to CLEAR.

If you need assistance creating course space or migrating legacy course data, submit a help ticket and include CLEAR and the course name in your request.

Snapshots of CLEAR home directories may be available for recovering recently deleted or modified files. File recovery from available snapshots can be performed directly by the user.

Log in to a CLEAR system using SSH and begin from your home directory:

cd
mkdir recover
cd ../.snapshot
ls

The ls command displays the available snapshots. Select the appropriate snapshot and navigate to your home directory within it:

cd <snapshot_name>/<NetID>

You can then copy a directory or individual file to the recovery directory you created:

# Recover an entire directory
cp -R <directory> ~/recover

# Recover an individual file
cp <file> ~/recover

# Return to your home directory
cd

After recovery, review the files in ~/recover and move them to the appropriate location.

NOTE: The syntax for the cp command is cp <source> <destination>. Verify the source and destination carefully before copying files to avoid unintentionally overwriting existing data.

Storage space in CLEAR home directories is limited. If you are approaching or exceeding your quota, use the following Linux tools to identify large files and directories before deciding what can safely be removed.

Important Notes Before Cleanup

• Perform disk-usage investigation and cleanup through an SSH session to a CLEAR system.
• If you are already over your storage limit, applications such as VS Code may fail to start correctly because they attempt to create or modify files in your home directory.
• Use standard Linux commands such as du, cd, sort, find, and rm when cleaning your home directory.
• Be cautious when deleting files. Deleted files may not be easily recoverable if they are not available in a snapshot.

Locate Large Files and Directories

Start in your home directory and display the size of top-level files and directories:

cd
du -skh .[0-9a-zA-Z]* * | sort -h

This displays disk usage from smallest to largest. If you identify a large directory, change into that directory and run the command again:

cd <directory>
du -skh .[0-9a-zA-Z]* * | sort -h

Repeat this process until you locate the files or directories consuming the most storage.

To display the largest items first:

cd
du -skh .[0-9a-zA-Z]* * | sort -hr | less

To display files and directories recursively with the largest entries first:

cd
du -ah | sort -rh | less

Using the tree Command

If the tree command is available, it can provide another view of directory usage.

cd
tree -dsh | less

To include hidden directories:

cd
tree -adsh | less

Using Manual Pages

Use the man command to learn more about Linux commands and available options:

man tree
man less
man du
man find
man rm

Removing Files

After identifying a file that you are certain is no longer needed, you can remove it interactively using:

rm -i <filename>

The -i option prompts you for confirmation before deleting the file.

Core Files

Programs may generate core files when they crash or when core dumps are required for debugging. These files can accumulate over time and consume storage.

To locate core files:

cd
find . -name "core.[0-9][0-9][0-9]*" -exec sh -c 'file "$1" | grep -q "LSB core file" && du -sh "$1"' _ {} \; | sort -h

Review the results carefully. If you are certain the identified core files are no longer needed, remove individual files using:

rm -i <core_file>

VS Code Remote Storage

Remote VS Code sessions can create hidden directories in your CLEAR home directory. Over time, these directories may consume a significant amount of storage.

Common locations include:

• ~/.vscode
• ~/.vscode-server
• ~/.cache/vscode-cpptools

To check how much space these directories are using:

cd
du -skh .vscode .vscode-server .cache/vscode-cpptools 2>/dev/null | sort -h

Cleaning VS Code Remote Files

Important: Before performing cleanup, close all VS Code sessions connected to CLEAR.

If .vscode-server or .cache/vscode-cpptools is consuming a significant amount of storage, review the contents before removing files.

If you determine that the remote VS Code data can safely be removed, you can clean the contents of the directories. Use extreme caution with rm -rf.

cd ~/.vscode-server && rm -rf .[0-9a-zA-Z]* *
cd ~/.cache/vscode-cpptools && rm -rf .[0-9a-zA-Z]* *

CAUTION: The rm -rf command removes files recursively without prompting for confirmation. Verify that you are in the correct directory before running the command. The use of && helps ensure that the removal command runs only if changing to the specified directory succeeds.

If VS Code Cleanup Fails Because Files Are in Use

If cleanup fails because files are reported as busy or in use, a remote VS Code process may still be running on a CLEAR system.

First, make sure all VS Code sessions on your local computer that connect to CLEAR are closed.

Log in to the CLEAR systems where you may have established remote VS Code sessions and check for processes using your .vscode-server directory:

lsof | grep "$HOME/.vscode-server"

If processes are returned, review them before terminating anything:

lsof | grep "$HOME/.vscode-server" | awk '{ print $2 }' | sort -u

To display detailed information about the identified processes:

for I in $(lsof | grep "$HOME/.vscode-server" | awk '{ print $2 }' | sort -u); do
    ps -lfjww --pid "$I"
    echo
done

Only after verifying that the processes belong to old VS Code sessions should you terminate them.

for I in $(lsof | grep "$HOME/.vscode-server" | awk '{ print $2 }' | sort -u); do
    kill "$I"
done

Verify that no processes are still using the directory:

lsof | grep "$HOME/.vscode-server"

If nothing is returned, retry the VS Code cleanup.

To use the CLEAR systems successfully, you need:

1. Access to the CLEAR systems.
2. A CLEAR home directory.

Accessing CLEAR

The standard SSH access point for CLEAR is:

ssh.clear.rice.edu

You can connect using an SSH client appropriate for your operating system, such as PuTTY, and OpenSSH, or MobaXterm.

• CLEAR SSH Access Guide
• CLEAR X11 Forwarding and Graphical Applications

CLEAR Home Directory

After logging in, verify that you are placed in your CLEAR home directory. If you receive an error indicating that the system cannot change to your assigned home directory, your CLEAR home directory may not yet exist or may require administrator assistance.

Need Help?

If you cannot log in to CLEAR or do not have a working CLEAR home directory, submit a ticket to the Rice Help Desk:

Contact the TS&S Service Desk

Include CLEAR in the subject or description of your request and provide any relevant error messages.