==============================================================
Linux Kernel State Tracer (LKST)
--------------------------------

Date:	Oct, 29, 2003

Version: 2.0.0

Authors: Kazuyoshi Serizawa <serizawa@sdl.hitachi.co.jp>
	 Tetsuhito Nakamura <tetsu@crl.hitachi.co.jp>
	 Keisuke Hatasaki   <keisukeh@crl.hitachi.co.jp>
	 Takahiro Tsurukubo <tsurukubo.t@jp.fujitsu.com>
	 Yasunori Goto	    <y-goto@jp.fujitsu.com>
	 Satoshi Sakamoto   <s-sakamoto@jp.fujitsu.com>
	 Takao Indoh	    <indou.takao@jp.fujitsu.com>
	 Masami Hiramatsu   <hiramatu@sdl.hitachi.co.jp>
	 Katsuhisa Ogasawara<ogasawar@sdl.hitachi.co.jp>

==============================================================
COPYRIGHT (C) HITACHI,LTD. 2001-2003 ALL RIGHTS RESERVED.
COPYRIGHT (C) FUJITSU,LTD. 2001-2003 ALL RIGHTS RESERVED.
==============================================================

INDEX
-----
1. Introduction
 1.1 Release Notes
2. Installation
3. How To Use
 3.1 How to insert new trace point (and new event)
 3.2 How to extract LKST trace data from LKCD dump file
 3.3 How to program an event handler
 3.4 How to port an event handler to LKST 2.0
 3.5 How to make new event handlers
4. Known Issues
5. Future plans
6. Reporting Problems and Discussing LKST
7. Licence
8. Change History


1. Introduction
---------------
    Linux Kernel State Tracer(LKST) records information as trace data
    about events in the Linux Kernel. It records various events like
    process context switch, send signal, exception, memory allocation,
    send packet, and so on.

    This program is a facility for analyzing trouble of the Linux Kernel.
    It becomes possible that developers investigate troubles by this facility
    without stopping important(used for Mission Critical purposes) server.

    Functions of present Linux could not offer enough information for
    developers to solve the trouble of the system.

       - Standard Kernel:When the kernel is panic, it displays only the
			 registers and stack information. If developers
			 want more information, it is necessary to insert
			 printk() in the kernel and recompile it.
       - LKCD		:LKCD store a dump when the system crash.
			 But, if developers use it only, they cannot
			 investigate the sequence of events till kernel panic.
	 (LKCD is a.k.a. Linux Kernel Crash Dumps)

    Event information of the kernel is always recorded as trace data
    on the server under running by using LKST. So, these problems can be
    solved.

    Moreover, LKST has some features.
      - It is possible to change dynamically which events are recorded.
	Developers can obtain information about the events which they concern
	only. And, It reduces the overhead of components which is not related
	with trouble.
      - It is possible to change each handlers related each events.
	A default handler of all events is just recording the events.
	But, if it is necessary, default handler can be changed.
	This function can be used as following,
	  +Notify user processes when the handler detects abnormal status.
	  +Change a maskset,which controls what type of events should be
	   recorded, dynamically by the handler.
	   ->  The system can usually run with a few events trace for the cause
	       of good performance. And, when the handler detects abnormal
	       status, it can change a maskset to get more detail information.


1.1 Release Notes
-----------------
   - port for linux-2.6.0-test9
   - support sysfs/procfs
   - separate LKCD feature
   - Fix some bugs.

2. Installation
---------------
   See file of "Install".


3. How To Use
-------------
   Please refer the online manual for details of each command.

  1) Display the present kernel trace data
      It is necessary to execute 2 commands.

      a. Get a log buffer from kernel
	 Enter the command as follows. Then LKST log buffer is saved to
	  <logfile> file.

	     % lkstbuf read -f logfile

      b. Display the trace data
	 Enter the command as follows. Trace data is written to stdout.
	 This command can be specified various options to filter the traced
	 record.

	     % lkstbuf print -f logfile

	<output example>
	-------------------------------------------------------------------
	event_type=context_switch
		cpu=02, pid=00001008
		time=Tue Mar 26 18:48:53.143274134 2002
		arg1=0xd68a4000 0x00000000 : pointer to task_struct(prev)
		arg2=0xf2dde000 0x00000000 : pointer to task_struct(next)
		arg3=0x00000001 0x0000000a : process state/process count

	event_type=mem_get_freepage
		cpu=02, pid=00001008
		time=Tue Mar 26 18:48:53.143270345 2002
		arg1=0xecc0f000 0x00000000 : pointer to page
		arg2=0x000001f0 0x00000000 : type of page(gfp_mask)
		arg3=0x00000000 0x00000000 : the number of page(order)
		arg4=0xc0168197 0x00000000 : call address

	event_type=mem_malloc
		cpu=02, pid=00001008
		time=Tue Mar 26 18:48:53.143268419 2002
		arg1=0x00000018 0x00000000 : size
		arg2=0x000001f0 0x00000000 : flags
		arg3=0xdfe3e5a0 0x00000000 : objp
		arg4=0xc0168578 0x00000000 : call address
	...
	...
	-------------------------------------------------------------------


  2) Change which events are recorded.
     The traced event types can be changed by doing as follows.

     a.Get a maskset file.
	 At first, please get a maskset file which control what event types
	 should be recorded by LKST.

	   % lkstm read -m id -d | grep 0x > maskset_file

	 "id" is the decimal number which is used as an identifier of the
	  maskset. Id 0, 1, and 2 are the reserved maskset; Id 0 does not
	  record any event types, id 1 records all of the event types, and
	  id 2 records the default event types.

     b.Edit the maskset file.
	 Next, edit the saved maskset_file.
	 In default, event of spinlock() is not recorded, so id of
	 event handler for event 0x080 is 0x0ff.
	 (Event type number of spin_lock() is 0x080.)

	    :		    :
	  0x07f		  0x01
	  0x080		  0xff		     <<<
	  0x081		  0xff
	    :		    :

	 To record the event, please change the value "0x0ff" to "0x001"
	 at right side of the line which contains its event number.

	    :		    :
	  0x07f		  0x01
	  0x080		  0x01		     <- change
	  0x081		  0xff
	    :		    :


	 Oppositely, If one event is not wanted to record,
	 please delete the line which is written it's event number.
	 ex) event type of system call entry ( number = 0x030).

	    :		    :
	  0x02f		  0x01
	  0x030		  0xff		     <- change
	  0x031		  0x01
	    :		    :

	 All the of Event type numbers are written in lkst-events-1.06.pdf
	 that is available from download page.

     c.Write the new maskset.
	 The new maskset is written when the next command is executed.

	   % lkstm write -m id -f maskset_file

	 "id" can be specified from 4 to 254.
	 At the same time, you can specify the name of the new maskset with
	 "-n" option. Instead of previous, do as below.

	   % lkstm write -m id -n new_maskset_name -f maskset_file

	 "new_maskset_name" must be different from the name of other masksets.
	 Please repeat the operation from a. to c., if you want to make more
	 masksets.

	 If you want to confirm the state of the maskset, maskset_read
	 have to be executed alone.

	   % lkstm read -m id

     d.Select maskset
	 New maskset becomes effective if maskset_set command is executed,
	 and the traced event types are changed.

	   % lkstm set -m id

     e.Confirm which maskset is currently selected.
	 If you want to know which maskset has been selected,
	 use lkst_status command.

	   % lkst status

	  <output example>
	-------------------------------------------------------------------
	<Current status>
	version of LKST		: 2.0.0
	number of cpus		: 4
	number of masksets	: 4
	number of event-handlers: 4
	current_maskset_id	: 2
	current_buffer_id (cpu: 000): 0
	current_buffer_id (cpu: 001): 0
	current_buffer_id (cpu: 002): 0
	current_buffer_id (cpu: 003): 0
	-------------------------------------------------------------------

	 ID of the selected maskset is shown to the right of
	 "current_maskset_id" in the forth line. In this example, it is 2.
	 And the number of right of "number of masksets" is total number of
	 the masksets. In this example, it is 4.
	 cf. Other information means as follows.
	    "number of event-handlers" :number of registered event handlers
	    "number of cpus"	       :number of system cpus
	    "current_buffer_id"	       :ID of currently selected buffers for
					all CPUs


  3) Invoke LKST logging daemon
     LKST logging daemon continues to output the containts of event
     buffer to log files.
     Please refer the online manual of lkstlogd(8) for details.
     Name of the files are /var/log/lkst/sebufCC.S, CC represents
     cpu id corresponds to the file and S represents a sequence
     number (described later).
     If directory /var/log/lkst does not exist, please make it as follows.

	   % mkdir /var/log/lkst

     This is an example of command to invoke the daemon,

	   % lkstlogd -a -b 2M -l 10M -n 2

     LKST logging daemon will stop and restart to write the file
     when the daemon receives a signal SIGUSR1.
     To send the signal, this is convenient,

	   % kill -USR1 `cat /var/run/lkstlogd.pid`

     The size of each file are limited as specified by -l option,
     so old records would be overwritten. To avoid overwitten,
     LKST logging daemon can change files. Said sequence number
     will be increased on this change and to discriminate new
     set of files and old files. To change file,

	   % kill -USR2 `cat /var/run/lkstlogd.pid`

     examples/lkst_watch_daemon is an example of this feature.
     This script watch unexpected termination of another daemon,
     then change the files.


  4) Dump logs in panic()
     Please do "3) Install addon tools" in "Install".
     And just do,

		 % insmod lkst_mod_panicdump.o maskset_id_base=2

     A parameter "maskset_id_base" specifies which maskset to
     be used as a template of the new maskset.
     For testing, this module can call panic(). See 6)-c below.


  5) How to display stack trace on each event
     a.Insmod the stack trace module.
	  Please do "3) Install addon tools" in "Install".
	  And do,

		 % insmod lkst_mod_stacktrace maskset_id_base=2

       A parameter "maskset_id_base" specifies a template.

     b.Execute special formatter
	  Execute lkst_mod_stacktrace_read.pl.

		 % lkstbuf read > logfile
		 % perl lkst_mod_stacktrace_read.pl \
		     -s /boot/System.map-`uname -r` -k /proc/ksyms -d logfile

	  All of the options can be omitted. The default value for -s and -k
	  are the same as those values of above command line. If option -d
	  is omitted, this formatter reads from current event buffers.

	  <output example>
	-------------------------------------------------------------------
	0x00057f24: 0000269005494693 [mc], \
	 type:	      INT_HARDWARE_ENTRY(0x0010), pid: 0000000000, cpu: 000 \
	  0x00000000 0x00000000 0x00000001 0x00000000 \
	  0x00000000 0x00000000 0x00000000 0x00000000 \
	    <LKST_ETYPE_INT_HARDWARE_ENTRY_HEADER+98> <default_idle+0> <defau
	lt_idle+0> <call_do_IRQ+5> <default_idle+0> <default_idle+0> <default
	_idle+44> <cpu_idle+82>

	0x00057f26: 0000269005511255 [mc], \
	 type:	     INT_TASKLETHI_ENTRY(0x0012), pid: 0000000000, cpu: 000 \
	  0xc01257d0 0x00000000 0x00000000 0x00000000 \
	  0x00000000 0x00000000 0x00000000 0x00000000 \
	    <LKST_ETYPE_INT_TASKLETHI_ENTRY_HEADER+60> <bh_action+0> <do_soft
	irq_Rsmp_f0a529b7+123> <LKST_ETYPE_INT_HARDWARE_ENTRY_HEADER+459> <de
	fault_idle+0> <default_idle+0> <call_do_IRQ+5> <default_idle+0>

	0x00057f28: 0000269005513862 [mc], \
	 type:		    INT_BH_ENTRY(0x0016), pid: 0000000000, cpu: 000 \
	  0x00000000 0x00000000 0xc012a390 0x00000000 \
	  0x00000000 0x00000000 0x00000000 0x00000000 \
	    <LKST_ETYPE_INT_BH_ENTRY_HEADER+71> <timer_bh+0> <LKST_ETYPE_INT_
	TASKLETHI_ENTRY_HEADER+74> <do_softirq_Rsmp_f0a529b7+123> <LKST_ETYPE
	_INT_HARDWARE_ENTRY_HEADER+459> <default_idle+0> <default_idle+0> <ca
	ll_do_IRQ+5>
	-------------------------------------------------------------------

	NOTE: Contents of stack trace might include some unrelated address.


  6) Change event handlers
     As described on the "Introduction", users can register additional
     event handler in order to add function more than recording the events.
     Please refer addons/lkst_mod_panicdump.c as an example.

     a.Register new event-handler.
     b.Change maskset.
	  When lkst_mod_panicdump.o is loaded, the initialiser in
	  the module will do them automatically.
	  The initialiser will register lkst_evhandler_panicdump()
	  as a new eventhandler that ID is 128, and create a
	  new maskset named "maskset_panicdump".

     c.Control event handler.
	  The control function of the event handler can invoke panic()
	  for testing.

		 % echo panic | lksteh ctrl -e 128 -f -

	  The control function compares the argument and "panic",
	  if correspond, the call panic("LKST PANIC TEST");

  7) Add user buffer
     If a new user buffer is added, data can be recorded in the added
     buffer.
     It becomes possible that a lot of data can be recorded without
     overflowing the standard buffer.

     a.First, Create a buffer (or buffers if you run on SMP system).

		% lkstbuf create -s <bytesize>

       <bytesize> is the size of the new buffer.

     b.Next, Select the new buffer to record.

		% lkstbuf jump -b <buffer_id>

     Please refer exapmles/lkst_test_run for more information.
     This is an example of switching the buffer where data is recorded
     when the specified command is executed.

	   % cd examples
	   % sh lkst_test_run 65536 ps aux

     The first argument is the size of new buffer, and the latter arguments
     are command to execute.

  8) How to use LKST as an event counter
     a.Insmod the counter module.
	  Make addons/lkst_mod_event_count.o, and insmod it.
     b.Execute command.
	  Execute addons/lkst_event_count, and then
	  the total of each event counter is displayed.


3.1 How to insert new trace point (and new event)
-------------------------------------------------
  1) Define new event type in linux/lkst_events.h as follows.

	 LKST_ETYPE_DEF(event-ID, hook-type, event-symbol, "event-name", \
			"1st argument description", \
			"2nd argument description", \
			"3rd argument description", \
			"4th argument description")

	   event-ID    --- Value of the event type(0x000..0xfff)

			      0x000-0x0ff: preset
			      0x100-0x1ff: user
			      0x200-0xeff: reserved
			      0xf00-0xfff: system

	   hook-type   --- Type of hook header for Kernel Hooks
			   Specify either the following according to the insertion
			   location of the HOOK macro.

			   - NORMAL: If you insert HOOK macro in the kernel,
				     use this type.

			   - MODULE: If you insert HOOK macro in the module,
				     use this type.

			   - INLINE: If you insert HOOK macro in the in-line
				     function of the kernel, use this type.
				     If you insert the same HOOK macro
				     in the two or more places, use this.

			   - UNDEF:  If you do not use this event by HOOK
				     macro, use this type.

			   NOTE: If you insert HOOK macro in the in-line
				 function of the module, use MODULE type.
						 ~~~~~~

	   event-symbol --- The mnemonic of the event type
	   event-name --- The description of the event type
	   1..4th arguments descriptions : The description for each arguments.

	  (example)
	      LKST_ETYPE_DEF(0x100, NORMAL, NEW_EVENT, "new event", \
			     "current task",
			     "target task",
			     NULL,
			     NULL)


  2) Insert following HOOK macro where you want to add the trace point.

       Add following sentence to the file to be added the trace point.
       #include <linux/lkst.h>

       - LKST_HOOK_INLINE(etype, argument1, argument2, argument3, argument4)

	     The case of inserting HOOK macro in the in-line function or the
	     macro function. If you insert the same HOOK macros in the two or more
	     places, use this.

       - LKST_HOOK(etype, argument1, argument2, argument3, argument4)

	     The case excepts the above-mentioned.

	  * etype   : The value of etype is equal to event-ID. Also,
		      You can use LKST_ET_SYM(event-symbol) or
		      LKST_ETYPE_event-symbol.
	  * argument1..4 : Arguments for HOOK.


   3) When the trace point is in the module, the macro for declaration of HOOK
     header is put in it. And the macro for the initialization and
     termination of HOOK is inserted respectively in the function of the
     module-initialization and module-termination.

	 Declaration macro for HOOK
	   LKST_ETYPE_DEF_MODULE(event-ID, hook-type, event-symbol, \
				 "event-name", \
				 "1st argument description", \
				 "2nd argument description", \
				 "3rd argument description", \
				 "4th argument description")

	!NOTE!: you can spacify NORMAL to hook-type if this event is ONLY
		caused in the same module.

	 Initialization macro
	   lkst_hook_etype_register(&LKST_ETYPE_INFO(event-symbol));

	 Termination macro
	   lkst_hook_etype_unregister(&LKST_ETYPE_INFO(event-symbol));

       See sample program "examples/lkst_mod_trc.c".


3.2 How to extract LKST trace data from LKCD dump file
------------------------------------------------------
!!!WARNING!!! THIS FEATURE IS NOT SUPPORTED BY CURRENTLY VERSION(2.0.0)!!
   a.Create dump file.
       See LKCD documents which can be downloaded at
       http://lkcd.sourceforge.net/.

       <example>
	  (i) Save system memory to dump device.
	      Enter the command as follows.

	    % lkcd config
	    % echo 1 > /proc/sys/kernel/sysrq

	    Hit Sysrq+C, then start saving system memory to dump device,
	    and rebooting automatically.

	    NOTE:
	    At default, system memory is saved at swap disk.
	    So if swapon runs in booting your system, system memory saved at
	    swap disk is collapsed.
	    Please save system memory at another disk except swap disk,
	    or comment out the line of swapon in the rc.sysinit.

	  (ii) Crate dump file.
	      Enter the command as follows. Dump file is created at
	      /var/log/dump/N (N is an integer of 0 or more.).

	    % lkcd save

     b.Run lcrash command
	  (i) Enter the command as follows. lcrash command starts and prompt is
	      shown.

	    % cd /var/log/dump/N
	    (N is an integer of 0 or more.)
	    % lcrash -n N
	    map = map.N, dump = dump.N, outfile = stdout, kerntypes = kerntypes
	    .N

	    Please wait...
		    Initializing vmdump access ... Done.
		    Loading system map ................................. Done.
		    Loading type info (Kerntypes) ... Done.
		    Initializing arch specific data ... Done.
		    Loading ksyms from dump ....... Done.
	    lkstdumplib being initialized
	    >>

	  (ii) Run lkstdump sub-command
	       Enter the command as follows. Trace data is written to file
	       <logfile>.

	    >> lkstdump logfile
	    static buffer write(2048 records)
	    cpu[0].buffer[0] write(1087 records)
	    cpu[1].buffer[0] write(1087 records)

	      <logfile>'s file format is the same as the file created by
	      "lkstbuf read", so you can see <logfile> by using "lsktbuf print".


3.3 How to program an event handler
-----------------------------------
   You can make and add new event-handler as a kernel module.
   But it requires high-level of programing ability and knowledge about linux
  kernel. Please do it at your own risk.

  1) Include following 4 files.

	#include<linux/version.h>
	#include<linux/module.h>
	#include<linux/init.h>
	#include<linux/lkst_private.h>

   First 3 files are basically required by kernel modules, and last one is
  reqired by LKST.

  2) Describe module information.

	MODULE_AUTHOR(/*write your name*/);
	MODULE_DESCRIPTION(/*write description of this module*/);
	MODULE_LICENSE("GPL");/*support GPL only*/

   Do *NOT* forget last description. Only GPL'ed modules can use kernel APIs
  of LKST.

  3) Write an event-handler function.
   You can use lkst_evhandlerprim_* functions in this function.

	static void lkst_evhandler_example(void *phookrec, int event_type,
					   lkst_arg_t arg1, lkst_arg_t arg2,
					   lkst_arg_t arg3, lkst_arg_t arg4)
	{
		/*TODO: do something*/
	}

  4) Write an event-handler control function.(optional)
   You can do something to control handler's behavior.

	static int lkst_evhandler_example_ctrl(void *buf, size_t bufsize)
	{
		/*TODO: do something*/
	}

  5) Write sysfs interface functions.(optional)
	static ssize_t example_ctrl_store(struct lkst_eh_device *ed,
					  const char *buf, size_t count)
	{
		/*TODO: do something*/
		return count; /* return 0 means sleep forever.*/
	}
	static ssize_t example_ctrl_show(struct lkst_eh_device *ed, char *buf)
	{
		/*TODO: do something*/
		return strlen(buf) + 1; /*you must return how many
					  charactors you wrote.*/
	}

  6) Declare module initializer and terminator.

	static int mod_init(void);
	static void mod_cleanup(void);
	module_init(mod_init);		/*define as initializer*/
	module_exit(mod_cleanup);	/*define as terminator*/

  7) Define a lkst_eh_device for registering to lkst class.

	static LKST_EH_DEV_DEF(/*the name of evhandler*/,
			       lkst_evhandler_example,
			       lkst_evhandler_example_ctrl,
			       example_ctrl_store,
			       example_ctrl_show);
     !! If the name of evhandler is HOGE, you just write HOGE in 1st argument,
	Do NOT write "HOGE" !!

  8) Program module initializer.
    In initializer, you have to register new event-handler to LKST.
    Following is sample code.

	static int mod_init(void)
	{
		int	retval;
		/* register an event handler */
		retval = lkst_eh_device_register(LKST_EH_DEV(/*the name of evhandler*/));
		if (retval < 0)
			goto init_err;

		return 0;

	init_err:
		mod_cleanup() /*If occurs an error, invokes terminator.*/
		return retval;
	}

  9) Program module terminator.
    In terminator, you have to deregister your event-handler from LKST.
    Following is sample code. (WARNING: *MUST* deregister it, or kernel might
    panic.)

	static void mod_cleanup(void)
	{
		if (LKST_EH_DEV(/*the name of evhandler*/).id != LKST_EVHANDLER_ID_VOID) {
			lkst_evhandler_unregister(LKST_EH_DEV(/*the name of evhandler*/));
		}
	}


3.4 How to port an event handler to LKST 2.0
--------------------------------------------
* You can use old interface yet. But it is not recommended. *

  1)  Define a lkst_eh_device data structure out of functions.

	static LKST_EH_DEV_DEF(example, example_handler, example_ctrl,
			       NULL, NULL);

    This code define a structure named 'lkst_eh_dev_example', initialize
   that name member to "example", evhandler to example_handler, evhandler_ctrl
   to example_ctrl, and so on. In other words, this macro is expanded as
   following:

	static struct lkst_eh_device lkst_eh_dev_example = {
		.name = "example",
		.id = LKST_EVHANDLER_ID_VOID,
		.evhandler = example_handler,
		.evhandler_ctrl = example_ctrl,
		.ctrl_store = NULL,
		.ctrl_show = NULL,
		.class_dev.class = &lkst_class,
		.class_dev.dev = NULL,
	};

    There are new members, ctrl_store, ctrl_show, and class_dev. class_dev is
   a sysfs related data structure, so you should not touch that without
   thorough understanding. Remaining two new members are discussed
   in step 4.

  2) Register the lkst_eh_device data structure in the module initializer.

    Instead of using lkst_evhandler_register(), you can use
   lkst_eh_device_register() to register an event handler.

	lkst_eh_device_register(&LKST_EH_DEV(example));

  LKST_EH_DEV() macro make the symbol (name) from the argument. for example,
  LKST_EH_DEV(example) is expanded to lkst_eh_dev_example.

    If succeed, this function returns zero. If fail, it returns an error value
   less than zero. This behavior different from lkst_evhandler_register().
    If you want to know what id number is assigned, please check
   LKST_EH_DEV(example).id .

  3) Deregister the lkst_eh_device data structure in the module terminator.

    Instead of using lkst_evhandler_unregister(), you can use
   lkst_eh_device_unregister() to deregister an event handler.

	lkst_eh_device_unregister(&LKST_EH_DEV(example));

   !NOTE!: This function returns nothing (void).

  4) Optionally (But recommended), you can define ctrl_store and ctrl_show
     functions to be able to access from user-mode more easily.

    These two functions are new features. The ctrl_store function stores
   control data from user-mode via sysfs interface. And the ctrl_show shows
   module internal data to user-mode.

	static ssize_t example_ctrl_store(struct lkst_eh_device *ed,
					  const char *buf, size_t count)
	{
		/*TODO: do something*/
		return count; /* return 0 means sleep forever.*/
	}
	static ssize_t example_ctrl_show(struct lkst_eh_device *ed, char *buf)
	{
		/*TODO: do something*/
		return strlen(buf) + 1; /*you must return how many
					  charactors you wrote.*/
	}

    Do NOT return zero from the ctrl_store() function. It should return the
   number of read bytes. Also, the ctrl_show() should return the number of
   written bytes.
    After defined these functions, rewrite the definition of the
   lkst_eh_device data structure as following:

	static LKST_EH_DEV_DEF(example, example_handler, example_ctrl,
			       example_ctrl_store, example_ctrl_show);


3.5 How to make new event handlers
----------------------------------
   Suppose that you have two event handlers(lkst_mod_example1.c,
  lkst_mod_example2.c) in a directory (~/example_module/).

  1) copy and rename addons/Makefile.skel to Makefile in the directrory.
    $ cp addons/Makefile.skel example_modules/Makefile

  2) edit the Makefile according to comments written in it.
    for example:

----
KMODSRC := lkst_mod_example1.c lkst_mod_example2.c
KMODOBJ := $(KMODSRC:.c=.o)
KMOD_KO := $(KMODOBJ:.o=.ko)

PROGSRC :=
PROGOBJ := $(PROGSRC:.c=.o)
PROGBIN := $(PROGOBJ:.o=)

ifneq ($(KERNELRELEASE),)

obj-m	:= $(KMODOBJ)

else

TOPDIR := ../addons
SUB_DIRS :=
ADDCMD-clean :=
ADDCMD-distclean :=

include $(TOPDIR)/Makefile.base

endif
----

  3) run make command.
    $ cd example_modules
    $ make


4. Known Issues
---------------
     - The time data in the event records written by
       IOCTL(LKST_IOC_BUFFER_READ) and lkst_buffer_read do not conform to the
       draft of POSIX standard 1003.25.


5. Future plans
---------------
     - Support LKCD again.
     - Usability improvement.
     - Performance improvement.
     - Adding more extensions for analyzing various faults.
     - Add connectivity to Event Logging (http://evlog.sourceforge.net/).
     - Cleanup sources.


6. Reporting Problems and Discussing LKST
-----------------------------------------
email:	lkst-develop@lists.sourceforge.net
	or lkst-develop@lists.sourceforge.jp
visit:	https://sourceforge.net/projects/lkst/
	or https://sourceforge.jp/projects/lkst/
	or http://oss.hitachi.co.jp/sdl/english/kstracer_main.html


7. Licence
----------
COPYRIGHT (C) HITACHI,LTD. 2001-2003 ALL RIGHTS RESERVED.
COPYRIGHT (C) FUJITSU,LTD. 2001-2003 ALL RIGHTS RESERVED.

This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation; either version 2
of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.


8. Change History
-----------------
2.0	30th of Octorber, 2003	Release v2.0.0
				  Update for Linux 2.6.0
				  Support sysfs.
				  Support procfs.
				  Support new kernelhooks.
				  Add new APIs and macros.
				  Cleanup and split sources.
				  Add some scripts.
				  Unsupport LKCD (currently).
				  Fix some bugs.
1.5	 5th of August, 2003	Release v1.5
				  Add watchdog tools.
				  Make makefile more useful.
				  Add overwritten detection event.
				  Add -S option to lkstbuf print.
				  Modify default maskset.
				  Fix some bugs.
1.5-rc2	 8th of May, 2003	Trial release 2
				  Fix some bugs.
1.5-rc1	 6th of May, 2003	Trial release 1
				  Merge patches for IPF(IA64).
				  Change bitwidth and dword order of arguments.
1.4	13rd of March, 2003	Release v1.4
				  Fix the display format of time.
				  Change the results of LKST_IOC_BUFFER_LIST to
				  enable to get the number of buffers.
1.4-rc2	 18th of February, 2003	 Trial release 2
				  Fix same bugs.
				  Add some configurable kernel options.
				  Add lkst_mod_pmc.c .
				  Add LKST_ETYPE_OOPS_PGFAULT event.
				  Add LKST_ETYPE_OOPS_NMIWDOG event.
				  Simplify format of the list of buffers.
				  Disable logging while kernel dumping.
				  Add lkstbuf print -V option.
1.4-rc1	 24rd of January, 2003	 Trial release
				  Rewrite most of codes associated with event-
				  buffers.
				  Separate resources per processor.
				  Awake reading process in schedule().
				  Allow to log lock event with lkstlogd.
				  Support inter-buffer jump feature.
				  Update commands.
				  Update addons.
				  Add light-weight handler as an addon module.
				  Modify kernel APIs and ioctl interfaces.
				  Add maskset configuration feature.
				  Add the version of LKST in lkst.h.
				  Fix some bugs.
1.3.1	25th of November, 2002	Release v1.3.1
				  Update for kernel 2.4.19.
				  Separate Kernel Hooks from LKST patch.
				  Add module that record only specified process.
				  Make sure lkstlogd release own resources.
				  lkstlogd report warning to syslog.
				  Add and modify APIs for event-handler and maskset.
				  Fix some bugs.
1.3	30th of August, 2002	Release v1.3
				  Performance improvement of Kernel Hooks.
				  Modify default maskset.
				  Support CSV format by lkstbuf command.
				  Add and modify APIs for event-handler.
				  Add config to delete hooks on inline func.
				  Add script to make /dev/lkst.
				  Fix some bugs.
1.2	28th of June, 2002	Release v1.2
				  Performance improvement of default handler.
1.1	14th of May, 2002	Release v1.1
				  Replace LKCD patch with the official one.
1.01	26th of April, 2002	Release v1.01
				  Update lkstlogd command.
				  Add trace data of exception events.
				  Fix some bugs.
1.0	12th of April, 2002	Release v1.0
				  Add kernel crash dump function by using LKCD.
				  Update some command.
				  Add log header to log file.
				  Fix some bugs.
0.9	31st of March, 2002	Release v0.9 for some enhancements.
				  Add LKST logging daemon,
				  Replace trace point hook to use Kernel Hooks,
				  Add some add-ons and exaples.
0.1a	18th of January, 2002	Fix bug of the racing between
				  etrc_evhandler_default() and
				  etrc_buffer_{delete,create}().
				  Add some minor modification,
				  clearfy some comments.
0.1	14th of December, 2001	First public drop

See changelog to know more details.
