Sunday, June 1, 2008

A mini-tutorial on the Unix/Linux find command

Locating Files:

The find command is used to locate files on a Unix or Linux system.
find will search any set of directories you specify for files that
match the supplied search criteria. You can search for files by name,
owner, group, type, permissions, date, and other criteria. The search
is recursive in that it will search all subdirectories too. The
syntax looks like this:

find where-to-look criteria what-to-do

All arguments to find are optional, and there are defaults for all
parts. (This may depend on which version of find is used. Here we
discuss the freely available GNU version of find, which is the version
available on YborStudent.) For example where-to-look defaults to .
(that is, the current working directory), criteria defaults to none
(that is, show all files), and what-to-do (known as the find action)
defaults to -print (that is, display found files to standard output).

For example:

find

will display all files in the current directory and all
subdirectories. The commands

find . -print
find .

do the exact same thing. Here's an example find command using a
search criteria and the default action:

find / -name foo

will search the whole system for any files named foo and display them.
Here we are using the criteria -name with the argument foo to tell
find to perform a name search for the filename foo. The output might
look like this:

/home/wpollock/foo
/home/ua02/foo
/tmp/foo

If find doesn't locate any matching files, it produces no output.

The above example said to search the whole system, by specifying the
root directory (/) to search. If you don't run this command as root,
find will display a error message for each directory on which you
don't have read permission. This can be a lot of messages, and the
matching files that are found may scroll right off your screen. A
good way to deal with this problem is to redirect the error messages
so you don't have to see them at all:

find / -name foo 2>/dev/null

Other Features And Applications:

The -print action lists the files separated by a space when the output
is piped to another command. This can lead to a problem if any found
files contain spaces in their names, as the output doesn't use any
quoting. In such cases, when the output of find contains a file name
such as foo bar and is piped into another command, that command sees
two file names, not one file name containing a space.

In such cases you can specify the action -print0 instead, which lists
the found files separated not with a space, but with a null character
(which is not a legal character in Unix or Linux file names). Of
course the command that reads the output of find must be able to
handle such a list of file names. Many commands commonly used with
find (such as tar or cpio) have special options to read in file names
separated with nulls instead of spaces.

You can use shell-style wildcards in the -name search argument:

find . -name foo\*bar

This will search from the current directory down for foo*bar (that is,
any filename that begins with foo and ends with bar). Note that
wildcards in the name argument must be quoted so the shell doesn't
expand them before passing them to find. Also, unlike regular shell
wildcards, these will match leading periods in filenames. (For
example find -name \*.txt.)

You can search for other criteria beside the name. Also you can list
multiple search criteria. When you have multiple criteria any found
files must match all listed criteria. That is, there is an implied
Boolean AND operator between the listed search criteria. find also
allows OR and NOT Boolean operators, as well as grouping, to combine
search criteria in powerful ways (not shown here.)

Here's an example using two search criteria:

find / -type f -mtime -7 | xargs tar -rf weekly_incremental.tar
gzip weekly_incremental.tar

will find any regular files (i.e., not directories or other special
files) with the criteria -type f, and only those modified seven or
fewer days ago (-mtime -7). Note the use of xargs, a handy utility
that coverts a stream of input (in this case the output of find) into
command line arguments for the supplied command (in this case tar,
used to create a backup archive). 1

Another use of xargs is illustrated below. This command will
efficiently remove all files named core from your system (provided you
run the command as root of course):

find / -name core | xargs /bin/rm -f
find / -name core -exec /bin/rm -f '{}' \; # same thing
find / -name core -delete # same if using Gnu find

(The last two forms run the rm command once per file, and are not as
efficient as the first form.)

One of my favorite find criteria is to locate files modified less than
10 minutes ago. I use this right after using some system
administration tool, to learn which files got changed by that tool:

find / -mmin -10

(This search is also useful when I've downloaded some file but can't locate it.)

Another common use is to locate all files owned by a given user (-user
username). This is useful when deleting user accounts.

You can also find files with various permissions set. -perm
/permissions means to find files with any of the specified permissions
on, -perm -permissions means to find files with all of the specified
permissions on, and -perm permissions means to find files with exactly
permissions. Permisisons can be specified either symbolically
(preferred) or with an octal number. The following will locate files
that are writeable by others:

find . -perm +o=w

(Using -perm is more complex than this example shows. You should
check both the POSIX documentation for find (which explains how the
symbolic modes work) and the Gnu find man page (which describes the
Gnu extensions).

When using find to locate files for backups, it often pays to use the
-depth option, which forces the output to be depth-first—that is,
files first and then the directories containing them. This helps when
the directories have restrictive permissions, and restoring the
directory first could prevent the files from restoring at all (and
would change the time stamp on the directory in any case). Normally,
find returns the directory first, before any of the files in that
directory. This is useful when using the -prune action to prevent
find from examining any files you want to ignore:

find / -name /dev -prune | xargs tar ...

When specifying time with find options such as -mmin (minutes) or
-mtime (24 hour periods, starting from now), you can specify a number
n to mean exactly n, -n to mean less than n, and +n to mean more than
n. 2 For example:

find . -mtime 0 # find files modified within the past 24 hours
find . -mtime -1 # find files modified within the past 24 hours
find . -mtime 1 # find files modified between 24 and 48 hours ago
find . -mtime +1 # find files modified more than 48 hours ago
find . -mmin +5 -mmin -10 # find files modifed between 6 and 9 minutes ago

The following displays non-hidden (no leading dot) files in the
current directory only (no subdirectories), with an arbitrary output
format (see the man page for the dozens of possibilities with the
-printf action):

find . -maxdepth 1 -name '[!.]*' -printf 'Name: %16f Size: %6s\n'

As a system administrator you can use find to locate suspicious files
(e.g., world writable files, files with no valid owner and/or group,
SetUID files, files with unusual permissions, sizes, names, or dates).
Here's a final more complex example (which I save as a shell script):

find / -noleaf -wholename '/proc' -prune \
-o -wholename '/sys' -prune \
-o -wholename '/dev' -prune \
-o -wholename '/windows-C-Drive' -prune \
-o -perm -2 ! -type l ! -type s \
! \( -type d -perm -1000 \) -print

This says to seach the whole system, skipping the directories /proc,
/sys, /dev, and /windows-C-Drive (presumably a Windows partition on a
dual-booted computer). The -noleaf option tells find to not assume
all remaining mounted filesystems are Unix file systems (you might
have a mounted CD for instance). The -o is the Boolean OR operator,
and ! is the Boolean NOT operator (applies to the following criteria).
So this criteria says to locate files that are world writable (-perm
-2) and NOT symlinks (! -type l) and NOT sockets (! -type s) and NOT
directories with the sticky (or text) bit set (! \( -type d -perm
-1000 \)). (Symlinks, sockets and directories with the sticky bit set
are often world-writable and generally not suspicious.)

A common request is a way to find all the hard links to some file.
Using ls -li file will tell you how many hard links the file has, and
the inode number. You can locate all pathnames to this file with:

find mount-point -xdev -inum inode-number

Since hard links are restricted to a single filesystem, you need to
search that whole filesystem so you start the search at the
filesystem's mount point. (This is likely to be either /home or / for
files in your home directory.) The -xdev options tells find to not
search any other filesystems.

(While most Unix and all Linux systems have a find command that
supports the -inum criteria, this isn't POSIX standard. Older Unix
systems provided the ncheck command instead that could be used for
this.)
Using -exec Efficiently

The -exec option to find is great, but since it runs the command
listed for every found file, it isn't very efficient. On a large
system this makes a difference! One solution is to combine find with
xargs as discussed above:

find whatever... | xargs command

However this approach has two limitations. Firstly not all commands
accept the list of files at the end of the command. A good example is
cp:

find . -name \*.txt | xargs cp /tmp # This won't work!

(Note the Gnu version of cp has a non-POSIX option -t for this.)

Secondly filenames may contain spaces or newlines, which would confuse
the command used with xargs. (Again Gnu tools have options for that,
find ... -print0 |xargs -0 ....)

There are POSIX (but non-obvious) solutions to both problems. An
alternate form of -exec ends with a plus-sign, not a semi-colon. This
form collects the filenames into groups or sets, and runs the command
once per set. (This is exactly what xargs does, to prevent argument
lists from becoming too long for the system to handle.) In this form
the {} argument expands to the set of filenames. For example:

find / -name core -exec /bin/rm -f '{}' +

This form of -exec can be combined with a shell feature to solve the
other problem. The POSIX shell allows us to use:

sh -c 'command-line' [ command-name [ args... ] ]

(We don't usually care about the command-name, so X, dummy, or inline
cmd is used.) Here's an example of efficiently copying found files,
in a POSIX-compliant way 3:

find . -name '*.txt' -exec sh -c 'cp "$@" /tmp' dummy {} +

Or even better:

find . -name '*.txt' -type f \
-exec sh -c 'exec cp -f "$@" /tmp' find-copy {} +

The find command can be amazingly useful. See the man page to learn
all the criteria and options you can use.

Monday, May 12, 2008

Smartly load your properties


Strive for disk location-independent code nirvana

August 8, 2003

QWhat is the best strategy for loading property and configuration files in Java?

A When you think about how to load an external resource in Java, several options immediately come to mind: files, classpath resources, and URLs. Although all of them eventually get the job done, experience shows that classpath resources and URLs are by far the most flexible and user-friendly options.

In general, a configuration file can have an arbitrarily complex structure (e.g., an XML schema definition file). But for simplicity, I assume below that we're dealing with a flat list of name-value pairs (the familiar .properties format). There's no reason, however, why you can't apply the ideas shown below in other situations, as long as the resource in question is constructed from an InputStream.

Evil java.io.File

Using good old files (via FileInputStream, FileReader, and RandomAccessFile) is simple enough and certainly the obvious route to consider for anyone without a Java background. But it is the worst option in terms of ease of Java application deployment. Using absolute filenames in your code is not the way to write portable and disk position-independent code. Using relative filenames seems like a better alternative, but remember that they are resolved relative to the JVM's current directory. This directory setting depends on the details of the JVM's launch process, which can be obfuscated by startup shell scripts, etc. Determining the setting places an unfair amount of configuration burden on the eventual user (and in some cases, an unjustified amount of trust in the user's abilities). And in other contexts (such an Enterprise JavaBeans (EJB)/Web application server), neither you nor the user has much control over the JVM's current directory in the first place.

An ideal Java module is something you add to the classpath, and it's ready to go. Think EJB jars, Web applications packaged in .war files, and other similarly convenient deployment strategies. java.io.File is the least platform-independent area of Java. Unless you absolutely must use them, just say no to files.

Classpath resources

Having dispensed with the above diatribe, let's talk about a better option: loading resources through classloaders. This is much better because classloaders essentially act as a layer of abstraction between a resource name and its actual location on disk (or elsewhere).

Let's say you need to load a classpath resource that corresponds to a some/pkg/resource.properties file. I use classpath resource to mean something that's packaged in one of the application jars or added to the classpath before the application launches. You can add to the classpath via the -classpath JVM option each time the application starts or by placing the file in the <jre home>\classes directory once and for all. The key point is that deploying a classpath resource is similar to deploying a compiled Java class, and therein lies the convenience.

You can get at some/pkg/resource.properties programmatically from your Java code in several ways. First, try:

  ClassLoader.getResourceAsStream ("some/pkg/resource.properties");
Class.getResourceAsStream ("/some/pkg/resource.properties");
ResourceBundle.getBundle ("some.pkg.resource");


Additionally, if the code is in a class within a some.pkg Java package, then the following works as well:

  Class.getResourceAsStream ("resource.properties");


Note the subtle differences in parameter formatting for these methods. All getResourceAsStream() methods use slashes to separate package name segments, and the resource name includes the file extension. Compare that with resource bundles where the resource name looks more like a Java identifier, with dots separating package name segments (the .properties extension is implied here). Of course, that is because a resource bundle does not have to be backed by a .properties file: it can be a class, for a example.

To slightly complicate the picture, java.lang.Class's getResourceAsStream() instance method can perform package-relative resource searches (which can be handy as well, see "Got Resources?"). To distinguish between relative and absolute resource names, Class.getResourceAsStream() uses leading slashes for absolute names. In general, there's no need to use this method if you are not planning to use package-relative resource naming in code.

It is easy to get mixed up in these small behavioral differences for ClassLoader.getResourceAsStream(), Class.getResourceAsStream(), and ResourceBundle.getBundle(). The following table summarizes the salient points to help you remember:

Behavioral differences

Method Parameter format Lookup failure behavior Usage example
ClassLoader.
getResourceAsStream()
"/"-separated names; no leading "/" (all names are absolute) Silent (returns null) this.getClass().getClassLoader()
.getResourceAsStream
("some/pkg/resource.properties")
Class.
getResourceAsStream()
"/"-separated names; leading "/" indicates absolute names; all other names are relative to the class's package Silent (returns null) this.getClass()
.getResourceAsStream
("resource.properties")
ResourceBundle.
getBundle()
"."-separated names; all names are absolute; .properties suffix is implied Throws unchecked
java.util.MissingResourceException
ResourceBundle.getBundle
("some.pkg.resource")


From data streams to java.util.Properties

You might have noticed that some previously mentioned methods are half measures only: they return InputStreams and nothing resembling a list of name-value pairs. Fortunately, loading data into such a list (which can be an instance of java.util.Properties) is easy enough. Because you will find yourself doing this over and over again, it makes sense to create a couple of helper methods for this purpose.

The small behavioral difference among Java's built-in methods for classpath resource loading can also be a nuisance, especially if some resource names were hardcoded but you now want to switch to another load method. It makes sense to abstract away little things like whether slashes or dots are used as name separators, etc. Without further ado, here's my PropertyLoader API that you might find useful (available with this article's download):

public abstract class PropertyLoader
{
/**
* Looks up a resource named 'name' in the classpath. The resource must map
* to a file with .properties extention. The name is assumed to be absolute
* and can use either "/" or "." for package segment separation with an
* optional leading "/" and optional ".properties" suffix. Thus, the
* following names refer to the same resource:
* <pre>
* some.pkg.Resource
* some.pkg.Resource.properties
* some/pkg/Resource
* some/pkg/Resource.properties
* /some/pkg/Resource
* /some/pkg/Resource.properties
* </pre>
*
* @param name classpath resource name [may not be null]
* @param loader classloader through which to load the resource [null
* is equivalent to the application loader]
*
* @return resource converted to java.util.Properties [may be null if the
* resource was not found and THROW_ON_LOAD_FAILURE is false]
* @throws IllegalArgumentException if the resource was not found and
* THROW_ON_LOAD_FAILURE is true
*/
public static Properties loadProperties (String name, ClassLoader loader)
{
if (name == null)
throw new IllegalArgumentException ("null input: name");

if (name.startsWith ("/"))
name = name.substring (1);

if (name.endsWith (SUFFIX))
name = name.substring (0, name.length () - SUFFIX.length ());

Properties result = null;

InputStream in = null;
try
{
if (loader == null) loader = ClassLoader.getSystemClassLoader ();

if (LOAD_AS_RESOURCE_BUNDLE)
{
name = name.replace ('/', '.');
// Throws MissingResourceException on lookup failures:
final ResourceBundle rb = ResourceBundle.getBundle (name,
Locale.getDefault (), loader);

result = new Properties ();
for (Enumeration keys = rb.getKeys (); keys.hasMoreElements ();)
{
final String key = (String) keys.nextElement ();
final String value = rb.getString (key);

result.put (key, value);
}
}
else
{
name = name.replace ('.', '/');

if (! name.endsWith (SUFFIX))
name = name.concat (SUFFIX);

// Returns null on lookup failures:
in = loader.getResourceAsStream (name);
if (in != null)
{
result = new Properties ();
result.load (in); // Can throw IOException
}
}
}
catch (Exception e)
{
result = null;
}
finally
{
if (in != null) try { in.close (); } catch (Throwable ignore) {}
}

if (THROW_ON_LOAD_FAILURE && (result == null))
{
throw new IllegalArgumentException ("could not load [" + name + "]"+
" as " + (LOAD_AS_RESOURCE_BUNDLE
? "a resource bundle"
: "a classloader resource"));
}

return result;
}

/**
* A convenience overload of {@link #loadProperties(String, ClassLoader)}
* that uses the current thread's context classloader.
*/
public static Properties loadProperties (final String name)
{
return loadProperties (name,
Thread.currentThread ().getContextClassLoader ());
}

private static final boolean THROW_ON_LOAD_FAILURE = true;
private static final boolean LOAD_AS_RESOURCE_BUNDLE = false;
private static final String SUFFIX = ".properties";
} // End of class


The Javadoc comment for the loadProperties() method shows that the method's input requirements are quite relaxed: it accepts a resource name formatted according to any of the native method's schemes (except for package-relative names possible with Class.getResourceAsStream()) and normalizes it internally to do the right thing.

The shorter loadProperties() convenience method decides which classloader to use for loading the resource. The solution shown is reasonable but not perfect; you might consider using techniques described in "Find a Way Out of the ClassLoader Maze" instead.

Note that two conditional compilation constants control loadProperties() behavior, and you can tune them to suit your tastes:

  • THROW_ON_LOAD_FAILURE selects whether loadProperties() throws an exception or merely returns null when it can't find the resource
  • LOAD_AS_RESOURCE_BUNDLE selects whether the resource is searched as a resource bundle or as a generic classpath resource


Setting LOAD_AS_RESOURCE_BUNDLE to true isn't advantageous unless you want to benefit from localization support built into java.util.ResourceBundle. Also, Java internally caches resource bundles, so you can avoid repeated disk file reads for the same resource name.

More things to come

I intentionally omitted an interesting classpath resource loading method, ClassLoader.getResources(). Despite its infrequent use, ClassLoader.getResources() allows for some very intriguing options in designing highly customizable and easily configurable applications.

I didn't discuss ClassLoader.getResources() in this article because it's worthy of a dedicated article. As it happens, this method goes hand in hand with the remaining way to acquire resources: java.net.URLs. You can use these as even more general-purpose resource descriptors than classpath resource name strings. Look for more details in the next Java Q&A installment.

Google+