prev back next

How to create & add binary classes/libraries

Contents

Introduction

The most interresting feature distinguishing Smalltalk/X from other smalltalk implementations is the ability to create native object files and (possibly shared) libraries from smalltalk classes.

This document will give you information and a step-by-step guide on how to compile your classes to machine code, how to create & package class libraries for distribution and how to include such a binary class library in your system.

Notice, that beside speed advantage and the ability to include inline C code, the semantic of binary compiled code is equivalent to that of interpreted bytecode. All language features (such as context handling, block handling, stack unwinding and exception handling) are available and perform transparent whether methods are compiled or interpreted.

Configuration files

Stc can be used like any other (batch-) compiler by calling it directly from the unix command line. However, since there are many possible command line arguments and configuration settings that have to be managed, it is easier and suggested that you use Makefiles to compile classes and build systems.

The system as delivered includes shell scripts and make rules to handle different architectures and configurations. To avoid the need for rewriting all Makefiles for different architectures, all relevant Makefiles are created from prototypes; these are named "Make.proto"; one is found in every directory.

The configurator reads these prototypes and creates the actual Makefiles depending on the choosen configuration.
To do so, the configurator (called "CONFIG") uses so called config and rule files, which define all architecture and configuration dependent settings.

Rule files are found in the "rules" subdirectory; config files in subdirectories of "configurations".
You dont have to care for the rule files normally - they are independent of the configuration.
The config files are organized by machine architecture and configuration. They define things like C-compiler flags, include paths, classes to be included in the building process etc.
The generic paths of config files looks like:

    configurations/COMMON/defines
    configurations/architecture/COMMON/defines
    configurations/architecture/configuration/defines
For example, the linux definitions for linux versions above 1.0, with C-optimizer turned on and GL-simulator classes included are taken from:
    configurations/COMMON/defines
    configurations/linux/COMMON/defines
    configurations/linux/linux1.x-opt-vgl/defines
the definitions used for a configuration without GL classes is in:
    configurations/COMMON/defines
    configurations/linux/COMMON/defines
    configurations/linux/linux1.x-opt/defines
and so on. The name of the directory has no semantic meaning - its only for human readers (I could have called them "linux-1", "linux-2" etc).

When Makefiles are built, all defines are simply concatenated in the above order - this allows definitions from the COMMON config files to be redefined. "CONFIG" needs the name of the architecture and the name of the configuration to create the "Makefiles".

Once built, Makefiles know from which configuration they were created and know how to recreate "Makefiles".
You should therefore never copy Makefiles to other architectures, use the same physical directory for multiple platforms (i.e. dont NFS-mount to another platform and make there). Finally, you should not edit the Makefiles, since they may be recreated and your changes be lost. Only edit "Make.proto" files.

Summary:

Whats in a config file

Remember, that for all of the following definitions, useful defaults are setup in
configurations/COMMON/defines and configurations/arch/COMMON/defines.

In normal situations, no changes are required. However, if you need anything to be changed, do so in your configuration specific config file; never in a common one. Since later definitions overwrite previous ones, you can always change things by adding a define to the privat defines file.

The following is a short extract - there are many more things that can (but are not required to) be changed:

	CC=                     defines the c-compilers name

	O=.o                    extension for object files
				(usually .o on unix, .obj on MSDOS)


	#
	# for make install only:
	#
	DESTBINDIR=             where are binaries to be installed
	DESTLIBDIR=             where are libraries to be installed
	DESTINCLDIR=            where are include files to be installed
	DESTMANDIR=             where are man pages to be installed

	#
	# for addeditional directories and libs (see below)
	#
	OTHERLIBDIRS=           specifies other directories, where
				Make.protos are to be built and classes
				are to be compiled.

	PRIVATELIBS=            specifies names of other classlibraries to be
				included in the final link

	PRIVATEOBJ=             specifies corresponding filenames

	PRIVATE_SO=             specifies corresponding filenames for shared
				objects

	OTHERLIBS=              specifies names of other (c-)libraries to be
				included in the final link

	#
	# Xlib stuff
	#
	XINCLUDE=               path to X include files
	LIBX=                   whats the name of your X-lib
	LIBXEXT=                whats the name of your Xext-lib


	#
	# c compiler flags and (preprocessor) defines
	#
	DEFS=                   addidtional defines passed to the c-compiler

	OPT=                    addidtional optimizer flags passed to the c-compiler

	#
	# additional individual class objects
	#
	EXTRA_CLASSES=          extra individual compiled classes to be 
				included in the final link. These are classes
				that do not come from class libraries.

	EXTRA_OBJ=              the corresponding object file names

	#
	# class libraries to be included
	#
	LIBLIST=                names all class libraries to be included in the
				final link (already includes PRIVATELIBS)

	LIBOBJS=                the corresponding object file names
				(already includes PRIVATEOBJ)
To make the above more transparent, lets look at a concrete example.
The default linux configuration specifies:
	CC=gcc                          from configurations/linux/COMMON
	DESTBINDIR=/usr/local/bin       from configurations/COMMON
	DESTLIBDIR=/usr/local/lib       from configurations/COMMON

	#
	# no other directories to visit
	#
	OTHERLIBDIRS=                   from configurations/COMMON

	#
	# no other class libraries to include
	#
	PRIVATELIBS=                    from configurations/COMMON
	PRIVATEOBJ=                     from configurations/COMMON
	PRIVATE_SO=                     from configurations/COMMON

	#
	# no other c libraries to include
	#
	OTHERLIBS=                      from configurations/COMMON

	#
	# X stuff
	#
	XINCLUDE=/usr/include/X11       from configurations/COMMON
	LIBX=-lX                        from configurations/COMMON
	LIBXEXT=-lXext                  from configurations/COMMON

	#
	# #define SHAPE adds support for the X shape extension
	# 
	DEFS=-Di386 -DSHAPE             from configurations/linux/opt-vgl/defines
	#
	# O6 and omit-frame-pointer are c-compiler switches
	#
	OPT=-O6 -fomit-frame-pointer    from configurations/linux/opt-vgl/defines
	EXTRA_CLASSES=XWorkstation      from configurations/linux/opt-vgl/defines
	EXTRA_OBJ=$(LIBVIEWDIR)/XWorkstat.o
					from configurations/linux/opt-vgl/defines
	#
	# names of class libraries to be included
	# (if not redefined, this already includes PRIVATELIBS)
	#
	LIBLIST="libbasic libview libwidg .... libtool"
					by expansion of $(STANDARD_LIST),
					which is the default definition

	#
	# filenames of objects to be included
	# (if not redefined, this already includes PRIVATEOBJ)
	#
	LIBOBJS=$TOP/libbasic/libbasic.o .... $TOP/libtool/libtool.o
					by expansion of $(STANDARD_LIB),
					which is the default definition
the above may not be up to date, when you read this document.
Please read on, some things will be clear in a minute ...

Recompiling the system

Although recompilation of the complete system is easily done with:
	cd TOP
	make clobber            - to clean everything
	make target             - make a target system as set by CONFIG
it takes hours on some machines. For local changes, you will get a feeling of what needs to be recompiled. For example, after adding a method to a class in libbasic, you can also say:
	cd libbasic
	make
	cd ../projects/smalltalk
	make smalltalk
which is much faster, since not all directories are visited.

However, whenever instance variables are added to classes which are subclasses somewhere else, these subclasses have to be recompiled as well (I hope you have added dependencies to your "Make.proto"; otherwise that other make will not help).

When new source files (i.e. new classes) are added to the system, use:

	...
	cd theDirectoryWhereTheSourceWasAdded
	make 
	...
	cd ../projects/smalltalk
	make
the difference is that "make smalltalk" only relinks the smalltalk executable, while the later "make" also scans all directories for ".st" files and creates symbolic links in the "source" subdirectory (remember: these are required for the browser to find a methods sourcecode).

To create the sourcelinks only, use:

	...
	cd ../projects/smalltalk
	make sourceLink
(there are corresponding rules to link the bitmap, resource and style files.

Whats in a Make.proto file

Since all architecture and configuration specific things are handled by the config files, Make.protos need not (should not) include any defines which are not valid for all configurations.

Therefore, Make.proto files are rather short. They only define the name of the library to be created, the subdirectories (if any) that should be visited and the names of the object files, which make up the library.

The required defines are:

	TOP=                            defines the position relative to the TOP
					directory (see example below)

	SUBDIRS=                        names of subdirectories (if any)

	LIBNAME=                        name of the class library
					(empty, if this is not for a classlib)

	all::                           default rule; should include
					abbrev.stc, objs, genClassList
					and a target rule.
					Additional C file targets should go here

	objs::                          names all object files
optional are definitions which change compilation flags:
	STCOPT=                         stc options; defaulted in COMMON/defines

	STCLOCALOPT=                    stc options to be used in addition
					to standard settings;
					additional package or optimization 
					flags should be given here.
optional but highly recommended rules are:
	clean::                         any cleanup of temporary files;
					should leave the target and intermediate
					object files for fast remake

	clobber::                       should clean everything that can be
					reconstructed by make - even if it
					will take a while to do so.

	tar:                            create a tar file for source
					distribution

	dependencies                    add dependencies for all object files
For example, the "Make.proto" for the basic class library looks (somewhat) like:
	TOP=..                          - its directly under TOP
	SUBDIRS=                        - I have no subdirectories to make

	LIBNAME=libbasic                - thats the name of the classlibrary

	STCOPT=$(LIBBASIC_STCOPT)       - override default options
					  (LIBBASIC_STCOPT is +optinline)

	STCLOCALOPT=-Pbasic-classes \   - define the package of all classes
		    -warnGlobalAssign \   compiled here, turn off some warnings
		    +optinline2           and turn on more inlining


	all::   abbrev.stc \            - default rule: create abbrev file
		objs genClassList \       objects, classList and a prelinked
		$(OBJTARGET)              class library (OBJTARGET)

	objs::  Object.$(O) \           - names all object files which are
		Boolean.$(O) \            to be created from corresponding .st
		...                       files. Notice the $(O) instead of .o;
					  This allows the same Make.proto to be
					  be used on MSDOS and Mac systems.

Creating binary class libraries

The following gives step-by-step information on how to add a new class library. For all of your new classes, this is the recommended way of doing things. Adding classes to existing directories may lead to problems and/or added work whenever a new ST/X release is delivered and installed: you would have to reedit all of your changed Make.proto files.

Lets assume, that your new library is to be called "libfoo" and shall contain the classes Foo, Bar and Baz.

  1. Write the classes
    Enter, test and debug the classes in the existing environment.

  2. Create a home for the sources
    Create a directory - a good strategy is to name the directory according the class library that is to be created. In this example, a good name would be "libfoo".

  3. Create the source files
    Save the source using fileOut in the SystemBrowser.

    Notice, that by default, the browser creates the files in your startup directory (which is usually "projects/smalltalk"); so you have to manually move the files into "TOP/libfoo" in this case.
    However, specifying a director for the active project, the files will be saved there. Therefore, its better to open a project, activate it, and set its directoryname to "../../libfoo". Then switch back to your browser and fileOut your classes.

    You may put all of your classes into a common category - this allows easier file out. Since stc requires that each class is in a separate source file, the fileOut-category function cannot be used here (since it creates one huge file containing all classes).
    Use fileOut-each, which saves each class's in a separate sourcefile.

  4. Create a Make.proto file
    The best strategy is to copy an existing "Make.proto", and change it as required.

    Here is what it could look like after your changes:

        #
        # required: where are we relative to TOP
        #
        TOP=..
    
        #
        # not really required: the default is empty anyway
        #
        SUBDIRS=
    
        #
        # required: the name of the resulting class library
        #
        LIBNAME=libfoo
    
        #
        # not required: a useful default is set anyway
        # take same optimization flags as for libbasic
        #
        STCOPT=$(LIBBASIC_STCOPT)
    
        #
        # not required: the default is empty
        # but defining a package is highly recommended
        # (see the ProjectViews browse function ...)
        #
        STCLOCALOPT=-Pfoo-classes
    
        #
        # required: define whats to be done
        #
        # this says:
        #   - create an abbreviation file
        #   - then compile all objects
        #   - then create a classList file
        #   - then create a classlibrary (OBJTARGET)
        #
        all::  abbrev.stc objs genClassList $(OBJTARGET)
    
        #
        # required: what are my objects
        #
        objs::
    		Foo.$(O)        \
    		Bar.$(O)        \
    		Baz.$(O)
    
        #
        # not required (but highly recommended):
        # dependency rules
        # later versions of ST/X will create these automatically for you;
        # till then, we have to add them here.
        #
        # the following is correct for
        #    Foo subclassOf: Object
        #    Bar subclassOf: Foo
        #    Baz subclassOf: Object
        #
        Foo.$(O):   Foo.st $(INCLUDE)/Object.H
        Bar.$(O):   Bar.st $(INCLUDE)/Foo.H $(INCLUDE)/Object.H
        Baz.$(O):   Baz.st $(INCLUDE)/Object.H
    
    You find the above prototype in "rules/Make.proto.lib", which includes even more detailed comments on what the settings do. Take that as a base for your own "Make.protos".

  5. Create a Makefile from the proto
    For a quick test, use an existing Makefile which is located in the same hierarchy (i.e. the same number of subdirectories under TOP):
        cd libfoo
        make -f ../libbasic/Makefile Makefile
    

    However, the long term solution is to add the "libfoo" directory to the list of automatically visited directories. This is done by first defining:

        OTHERLIBDIRS=libfoo
    
    in your configuration file (in "configurations/arch/myconf/defines"), then changing to the TOP directory, and recreating all makefiles.
        ...
        add OTHERLIBDIRS definitions to your config file 
        ...
        cd TOP
        make Makefile
        make Makefiles
    
    Watch the output of the "make Makefiles" command: the added "libfoo" directory should be visited along with the other directories and a "Makefile" be generated there.

    After this, every make started in the TOP directory will always include your libfoo directory. You will never again have to do things manually there.

  6. Compile your library
    While testing, you probably do not want to start the (time consuming) global make in TOP, but instead only compile things local to libfoo.
    	cd libfoo
    	make
    
    this should (after a while) leave you with your new classLibrary libfoo in that directory.
    Depending on the architecture and/or configuration, the filename extension of the library varies; any of "libfoo.a", "libfoo.obj", "libfoo.o" or "libfoo.so" may be found there after the make. Dont care for this detail - the make rules create whatever is best for your architecture.
    (For example, on some systems archives (".a" extension) lead to very long link times - on these, classLibraries are prelinked, relocatable objects (".obj"). Some do not allow ".obj", therefore ".o" is used. Finally, some support shared libraries named ".so")
    If you plan to pass the compiled class library to others, all you have to distribute is the libfoo object just created.
    The others (those who include your binary) must change their config files too, but only have to perform the following (and final) step.

  7. Get the classes into the executable
    If you are on a system which supports dynamic loading (currently the SGI Indy and Unixware SYS5.4), and ST/X has been configured to support it, you can attach this library to your running system without leaving it (use fileIn in the FileBrowser).
    On all others, you have to leave any running smalltalk, relink the smalltalk executable and start anew (this time with those new classes being part of the built-in classes).

    Since saved snapshot images are (currently) unusable after a system rebuild, it is now time to save all your work in source form (i.e. fileOut all other classes and make certain that you can reconstruct your universe later from these and/or the changes file).

    Until now, our config file only defined which directories should be visited in addition to the others; it did not yet specify that libfoo is to be included in the executable. This is done by adding the lines:

    	PRIVATELIBS=libfoo
    	PRIVATEOBJ=$(TOP)/libfoo/libfoo$(OBJNAME)
    	PRIVATE_SO=$(TOP)/libfoo/libfoo$(SO_NAME)
    
    to your config file, and again recreate the Makefiles from TOP. (in this example, we did the config-file changes as two separate steps for didactic reasons - you will later do it in one step, and thus avoid recreating Makefiles twice ...)
    In the above, OBJNAME will expand to whatever the extension of class libraries is in that configuration; SO_NAME expands to the name of shared libraries. Even if your system does not support shared libraries now, define the PRIVATE_SO in case you switch to another architecture or a new ST/X release later.
    After that, change to the smalltalk directory and relink the executable:
    	cd projects/smalltalk
    	make
    
    (later, you will only use "make smalltalk"; but the first time, we need the source-links to be created).
Congratulations ! you should now find your foo-classes in the system when starting the new executable. (use "smalltalk -I" to have it ignore any existing snapshot file.)

For your tests, keep the old snapshot and smalltalk executable around as "st.img.sav" and "smalltalk.sav" for a while - just in case.

Adding more classes

Once you are through the above hard work, adding more classes is easy:
	...
	Save more classes into the libfoo directory
	...

	cd TOP/libfoo

	...
	add entries for the class-file names in Make.proto
	add dependency information
	...

	make Makefile
	make

	cd ../projects/smalltalk
	make

Adding more classlibraries

You may not want to add classes forever to that single libfoo library. At some point, it may be required to add another library. To do so, follow the above steps again, leading to entries in your config file which look like:
	OTHERLIBDIRS=libfoo libfoo2 ...
        
	PRIVATEOBJS=$(TOP)/libfoo/libfoo$(OBJNAME) \
		    $(TOP)/wherever/libfoo2$(OBJNAME) \
		    ...

	PRIVATE_SO=$(TOP)/libfoo/libfoo$(SO_NAME) \
		    $(TOP)/wherever/libfoo2$(SO_NAME) \
		    ...

	PRIVATELIBS=libfoo libfoo2 ...
If you are working in a group with others, a good strategy is to place classlibraries into a central directory, which can be used from anybode.
In this case, all users of the class library (i.e. those that do not care for how these are created) will only need the PRIVATE definitions in their config file, NOT the OTHERLIBDIRS line.

For example, if you have placed the classlibraries to "/usr/local/lib", these other config files should look like:

	PRIVATEOBJS=/usr/local/lib/libfoo$(OBJNAME) \
		    /usr/local/lib/libfoo2$(OBJNAME)

	PRIVATE_SO=/usr/local/lib/libfoo$(SO_NAME) \
		   /usr/local/lib/libfoo2$(SO_NAME)

	PRIVATELIBS=libfoo libfoo2
Remember again, that after every such change, your Makefile in projects/smalltalk has to be regenerated.

Interfacing C functions

To add C functions, you have various options of where these are located (in any case you will need a Smalltalk wrapper method which calls those functions - this is described in how to write primitives & inline C code): Except for the first case above (putting them into the primitiveFunctions section), you will have to change your config file to have the C library included. See the next chapter on how this is done.

Adding C libraries

Additional C libraries are included in the link of the smalltalk executable by specifying them as "OTHERLIBS" in the config file. For example, if you have a C library called "libUseful.a", placed under TOP/libfoo/cStuff, the line should look like:
	OTHERLIBS=$(TOP)/libfoo/cStuff/libUseful.a
Of course, if the library is located in some standard place (i.e. in /usr/lib or /usr/local/lib), you can also write:
	OTHERLIBS=-lUseful
Read the C-compilers and linkers man pages if you need more information on this.

All OTHERLIBS are linked in the order specified; thus, if you have dependencies between your extra libraries, these may be fixed by changing the order. For example, if you have two libraries "libUseful1.a" and "libUseful2.a", of which the second needs entries in the first, you will get a link error if the config define looks like:

	OTHERLIBS=$(TOP)/libfoo/cStuff/libUseful1.a \
		  $(TOP)/libfoo/cStuff/libUseful2.a
in this case, change the order as in:
	OTHERLIBS=$(TOP)/libfoo/cStuff/libUseful2.a \
		  $(TOP)/libfoo/cStuff/libUseful1.a

An example for C Interfacing

In the following, an interface to C functions which are compiled from separate sources is shown.
For the example, we assume that a directory "libfoo" exists, and the library is to be named "libfoo" as well.
For better structuring, we place the C sources into a subdirectory of "libfoo" called "cstuff" and create a C library called "cstuff.a" there.
The C sources are in "libfoo/cstuff/module1.c":
	cFuntion1() {
	    printf("here is your C function call ....\n");
	}
and "libfoo/cstuff/module2.c":
	cFuntion2(arg) 
	    int arg;
	{
	    printf("here is your 2nd C funtion; the args value is %d\n");
	}
The interface wrapping code for smalltalk is in "libfoo/CInterface.st":
	Object subclass:#Cinterface
	       instanceVariableNames:''
	       classVariableNames:''
	       poolDictionaries:''
	!

	!Cinterface class methodsFor:'C calling'!

	cFuntion1
	%{
	    cFuntion1();
	%}
	!

	cFuntion2:argument
	%{
	    if (__isSmallInteger(argument)) {
		cFuntion2( _intVal(argument) );
	    }
	%}
	! !
The "Make.proto" to compile all of these is:
	TOP=..
	LIBNAME=libfoo

	all::   CSTUFF abbrev.stc objs genClassList $(OBJTARGET)

	objs::  CInterface.$(O)

	CSTUFF::
		(cd cstuff ; make)
The C library is built by the following rules in "cstuff/Makefile":
	all:    module1.o module2.o
		ar rv cstuff.a module1.o module2.o
Thats it, after a "make" in libfoo, you will find "libfoo.obj" (or "libfoo.so") and "libfoo/cstuff.a ready for linkage.

To get those into the executable, your config file should have the definitions:

	OTHERLIBDIRS=$(TOP)/libfoo

	PRIVATEOBJS=$(TOP)/libfoo/libfoo$(OBJNAME)

	PRIVATE_SO=$(TOP)/libfoo/libfoo$(SO_NAME)

	PRIVATELIBS=libfoo

	OTHERLIBS=$(TOP)/libfoo/cstuff/cstuff.a
The files of the example can be found in "doc/coding/libfoo_example".

For details on the primitive wrapper code, read how to write primitives & inline C code.

Automatic building

Automatic creation and building via the project management is being prepared. However, at this time, this feature is not fully implemented and things should be done manually.

Once completed, you will be able to create per project directories, Make.protos and the sources by the click of a button.


Copyright © Claus Gittinger Development & Consulting, all rights reserved

(cg@ssw.de)